Commit Graph

1552 Commits

Author SHA1 Message Date
Kazu Hirata 4ba9d9c84f Use StringRef::contains (NFC) 2021-10-23 20:41:46 -07:00
Noah Shutty e678c51177 [Support][ThinLTO] Move ThinLTO caching to LLVM Support library
We would like to move ThinLTO’s battle-tested file caching mechanism to
the LLVM Support library so that we can use it elsewhere in LLVM.

Patch By: noajshu

Differential Revision: https://reviews.llvm.org/D111371
2021-10-18 18:57:25 -07:00
Petr Hosek 8e46e34d24 Revert "[Support][ThinLTO] Move ThinLTO caching to LLVM Support library"
This reverts commit 92b8cc52bb since
it broke the gold plugin.
2021-10-18 12:24:05 -07:00
Noah Shutty 92b8cc52bb [Support][ThinLTO] Move ThinLTO caching to LLVM Support library
We would like to move ThinLTO’s battle-tested file caching mechanism to
the LLVM Support library so that we can use it elsewhere in LLVM.

Patch By: noajshu

Differential Revision: https://reviews.llvm.org/D111371
2021-10-18 12:08:49 -07:00
Kazu Hirata 8568ca789e Use llvm::erase_if (NFC) 2021-10-18 09:33:42 -07:00
Nico Weber f09dce564e [lld] fix typos to cycle bots 2021-10-12 17:03:39 -04:00
Nico Weber f3091831f4 [lld] Use checkError more
No behavior change.
2021-10-04 11:46:16 -04:00
Amy Huang 6f7483b1ec Reland "[LLD] Remove global state in lld/COFF" after fixing asan and msan test failures
Original commit description:

  [LLD] Remove global state in lld/COFF

  This patch removes globals from the lldCOFF library, by moving globals
  into a context class (COFFLinkingContext) and passing it around wherever
  it's needed.

  See https://lists.llvm.org/pipermail/llvm-dev/2021-June/151184.html for
  context about removing globals from LLD.

  I also haven't moved the `driver` or `config` variables yet.

  Differential Revision: https://reviews.llvm.org/D109634

This reverts commit a2fd05ada9.

Original commits were b4fa71eed3
and e03c7e367a.
2021-09-17 17:18:42 -07:00
Amy Huang a2fd05ada9 Temporarily revert "[LLD] Remove global state in lld/COFF" and "[lld] Add test to
check for timer output"

Seems to be causing a number of asan test failures.

This reverts commit b4fa71eed3
and e03c7e367a.
2021-09-16 11:58:11 -07:00
Amy Huang e03c7e367a [lld] Add test to check for timer output
This test checks that timers are working and printing as expected.

I also seem to have changed the order of the timers in my globals refactoring
patch, so I fixed it here.

Differential Revision: https://reviews.llvm.org/D109904
2021-09-16 11:36:46 -07:00
Amy Huang b4fa71eed3 [LLD] Remove global state in lld/COFF
This patch removes globals from the lldCOFF library, by moving globals
into a context class (COFFLinkingContext) and passing it around wherever
it's needed.

See https://lists.llvm.org/pipermail/llvm-dev/2021-June/151184.html for
context about removing globals from LLD.

I also haven't moved the `driver` or `config` variables yet.

Differential Revision: https://reviews.llvm.org/D109634
2021-09-16 11:00:23 -07:00
Alexandre Ganea 7f0664f193 [LLD][COFF] Clean paths in PDB even when /pdbsourcepath is omitted
Differential Revision: https://reviews.llvm.org/D109030
2021-08-31 19:05:10 -04:00
Nico Weber 34ac7a7ac1 [lld/COFF] Ignore /LTCG, /LTCG:, /LTCGOUT:, /ILK: flags
We currently complain "could not open /LTCG: no such file or directory",
which isn't very useful.  We could emit a warning when we see this flag, but
just ignoring it seems fine.

Final missing part of PR38799.

Differential Revision: https://reviews.llvm.org/D108799
2021-08-27 09:13:30 -04:00
Nico Weber 66dc44f703 [lld/COFF] Use P_priv more
P_priv does the same as the old QF further down. Standardize on P_priv.

No behavior change.

Differential Revision: https://reviews.llvm.org/D108798
2021-08-27 08:48:05 -04:00
Nico Weber 400a1de3ac [lld/COFF] Improve handling of the /manifestdependency: flag
If multiple /manifestdependency: flags are passed, they are
naively deduped, but after that each of them should have an
effect, instead of just the last one.

Also, /manifestdependency: flags are allowed in .drectve sections
(from `#pragma comment(linker, ...`). To make the interaction between
/manifestdependency: flags enabling manifest by default but
/manifest:no overriding this work, add an explict ManifestKind::Default
state to represent no explicit /manifest flag being passed.
To make /manifestdependency: flags from input file .drectve sections
work with /manifest:embed, delay embedded manifest emission until
after input files have been read.

Differential Revision: https://reviews.llvm.org/D108628
2021-08-25 14:36:32 -04:00
Yolanda Chen 8fa16cc628 [LTO][lld] Add lto-pgo-warn-mismatch option
When enable CSPGO for ThinLTO, there are profile cfg mismatch warnings that will cause lld-link errors (with /WX)
due to source changes (e.g. `#if` code runs for profile generation but not for profile use)
To disable it we have to use an internal "/mllvm:-no-pgo-warn-mismatch" option.
In contrast clang uses option ”-Wno-backend-plugin“ to avoid such warnings and gcc has an explicit "-Wno-coverage-mismatch" option.

Add "lto-pgo-warn-mismatch" option to lld COFF/ELF to help turn on/off the profile mismatch warnings explicitly when build with ThinLTO and CSPGO.

Differential Revision: https://reviews.llvm.org/D104431
2021-08-11 09:45:55 -07:00
Wang, Pengfei 6c4809825d Revert "[lld] Add lto-pgo-warn-mismatch option"
This reverts commit 0cfb00a1c9.
2021-08-11 16:25:42 +08:00
Yolanda Chen 0cfb00a1c9 [lld] Add lto-pgo-warn-mismatch option
When enable CSPGO for ThinLTO, there are profile cfg mismatch warnings that will cause lld-link errors (with /WX).
To disable it we have to use an internal "/mllvm:-no-pgo-warn-mismatch" option.
In contrast clang uses option ”-Wno-backend-plugin“ to avoid such warnings and gcc has an explicit "-Wno-coverage-mismatch" option.

Add this "lto-pgo-warn-mismatch" option to lld to help turn on/off the profile mismatch warnings explicitly when build with ThinLTO and CSPGO.

Reviewed By: tejohnson

Differential Revision: https://reviews.llvm.org/D104431
2021-08-11 14:43:26 +08:00
Martin Storsjö 9dbc4b09af [LLD] [COFF] Make -export-all-symbols work as intended for EXEs
If some symbols are marked with dllexport, we still want to export
all symbols if -export-all-symbols is specified. Previously, this
only worked as it should for DLL output, not for EXE.

This should fix downstream bug
https://github.com/msys2/MINGW-packages/issues/9163.

Differential Revision: https://reviews.llvm.org/D106245
2021-07-22 23:34:03 +03:00
Martin Storsjö e0e09481ee [LLD] [COFF] Add a couple "MinGW only" comments re linking against DLLs. NFC.
This was requested in the post-commit review of D104530.
2021-07-20 23:57:24 +03:00
Fangrui Song db5e078690 [LTO] Add SelectionKind to IRSymtab and use it in ld.lld/LLVMgold
In PGO, a C++ external linkage function `foo` has a private counter
`__profc_foo` and a private `__profd_foo` in a `comdat nodeduplicate`.

A `__attribute__((weak))` function `foo` has a weak hidden counter `__profc_foo`
and a private `__profd_foo` in a `comdat nodeduplicate`.

In `ld.lld a.o b.o`, say a.o defines an external linkage `foo` and b.o
defines a weak `foo`. Currently we treat `comdat nodeduplicate` as `comdat any`,
ld.lld will incorrectly consider `b.o:__profc_foo` non-prevailing.  In the worst
case when `b.o:__profd_foo` is retained and `b.o:__profc_foo` isn't, there will
be dangling reference causing an `undefined hidden symbol` error.

Add SelectionKind to `Comdat` in IRSymtab and let linkers ignore nodeduplicate comdat.

Differential Revision: https://reviews.llvm.org/D106228
2021-07-20 13:22:00 -07:00
David Blaikie 1def2579e1 PR51018: Remove explicit conversions from SmallString to StringRef to future-proof against C++23
C++23 will make these conversions ambiguous - so fix them to make the
codebase forward-compatible with C++23 (& a follow-up change I've made
will make this ambiguous/invalid even in <C++23 so we don't regress
this & it generally improves the code anyway)
2021-07-08 13:37:57 -07:00
Jeremy Drake 7a7da69fbe [LLD] [COFF] Avoid thread exhaustion on 32-bit Windows host
LLD on 32-bit Windows would frequently fail on large projects with
an exception "thread constructor failed: Exec format error".  The stack
trace pointed to this usage of std::async, and looking at the
implementation in libc++ it seems using std::async with
std::launch::async results in the immediate creation of a new thread
for every call.  This could result in a potentially unbounded number
of threads, depending on the number of input files.  This seems to
be hitting some limit in 32-bit Windows host.

I took the easy route, and only use threads on 64-bit Windows, not all
Windows as before.  I was thinking a more proper solution might
involve using a thread pool rather than blindly spawning any number
of new threads, but that may have other unforeseen consequences.

Reviewed By: rnk

Differential Revision: https://reviews.llvm.org/D105506
2021-07-07 22:00:18 +03:00
Martin Storsjö ce211c505b [LLD] [COFF] Fix up missing stdcall decorations in MinGW mode
If linking directly against a DLL without an import library, the
DLL export symbols might not contain stdcall decorations.

If we have an undefined symbol with decoration, and we happen to have
a matching undecorated symbol (which either is lazy and can be loaded,
or already defined), then alias it against that instead.

This matches what's done in reverse, when we have a def file
declaring to export a symbol without decoration, but we only have
a defined decorated symbol. In that case we do a fuzzy match
(SymbolTable::findMangle). This case is more straightforward; if we
have a decorated undefined symbol, just strip the decoration and look
for the corresponding undecorated symbol name.

Add warnings and options for either silencing the warning or disabling
the whole feature, corresponding to how ld.bfd does it.

(This feature works for any symbol decoration mismatch, not only when
linking against a DLL directly; ld.bfd also tolerates it anywhere,
and also fixes up mismatches in the other direction, like
SymbolTable::findMangle, for any symbol, not only exports. But in
practice, at least for lld, it would primarily end up used for linking
against DLLs.)

Differential Revision: https://reviews.llvm.org/D104532
2021-07-02 09:49:14 +03:00
Martin Storsjö a9ff1ce1b9 [LLD] [COFF] Support linking directly against DLLs in MinGW mode
GNU ld.bfd supports linking directly against DLLs without using an
import library, and some projects have picked up on this habit.
(There's no one single unsurmountable issue with using import
libraries, but this is a regularly surfacing missing feature.)

As long as one is linking by name (instead of by ordinal), the DLL
export table contains most of the information needed. (One can
inspect what section a symbol points at, to see if it's a function
or data symbol. The practical implementation of this loops over all
sections for each symbol, but as long as they're not very many, that
should hopefully be tolerable performance wise.)

One exception where the information in the DLL isn't entirely enough
is on i386 with stdcall functions; depending on how they're done,
the exported function name can be a plain undecorated name, while
the import library would contain the full decorated symbol name. This
issue is addressed separately in a different patch.

This is implemented mimicing the structure of a regular import library,
with one InputFile corresponding to the static archive that just adds
lazy symbols, which then are fetched when they are needed. When such
a symbol is fetched, we synthesize a coff_import_header structure
in memory and create a regular ImportFile out of it.

The implementation could be even smaller by just creating ImportFiles
for every symbol available immediately, but that would have the
drawback of actually ending up importing all symbols unless running
with GC enabled (and mingw mode defaults to having it disabled for
historical reasons).

Differential Revision: https://reviews.llvm.org/D104530
2021-07-02 09:49:13 +03:00
Martin Storsjö d07f43641f [LLD] [COFF] Fix handling of LTO comdats with nontrivial selection types after 728cc0075e
Commit 728cc0075e made comdat symbols
from LTO objects be treated as any regular comdat symbol. This works
great for symbols that actually are IMAGE_COMDAT_SELECT_ANY, but
if the symbols have a less trivial selection type that require comparing
either the section chunk size or contents, we can't check that before
actually doing the LTO compilation.

Therefore bring back one aspect of handling from before; that comdat
resolution with a leader from an LTO symbol is essentially skipped,
like it was before 728cc0075e.

Differential Revision: https://reviews.llvm.org/D104605
2021-06-25 09:39:56 +03:00
Fangrui Song f1e2d5851b [OptTable] Rename PrintHelp to printHelp
To be consistent with other member functions and match the coding standard.
2021-06-24 14:47:03 -07:00
Martin Storsjö 3c6f8ca7c9 [lld] Rename StringRef _lower() method calls to _insensitive() 2021-06-25 00:22:01 +03:00
Reid Kleckner 8d84751ac4 Revert "[LLD] [COFF] Avoid doing repeated fuzzy symbol lookup for each iteration. NFC."
This reverts commit e1adf90826.

This appears to affect the way that C++ mangled symbols appear in the
import library when using a .def file that names a C++ free function
with no name decoration. I will follow up with a reduced test case
shortly.
2021-06-22 11:35:14 -07:00
Fangrui Song 50225112b5 [lld-link] Fix -Wunused-but-set-variable in -DLLVM_ENABLE_ASSERTIONS=off build. NFC 2021-06-20 11:35:02 -07:00
Martin Storsjö e1adf90826 [LLD] [COFF] Avoid doing repeated fuzzy symbol lookup for each iteration. NFC.
This is run every time around in the main linker loop. Once a match
has been found, stop trying to rematch such a symbol.

Not sure if this has any actual measurable performance impact though
(SymbolTable::findMangle() iterates over the whole symbol table for
each call and does fuzzy matching on top of that) but this makes the
code more reassuring to read at least. (This is in practice run for def
files listing undecorated stdcall functions to be exported.)

Differential Revision: https://reviews.llvm.org/D104529
2021-06-19 22:32:37 +03:00
Martin Storsjö ceee35e3e4 [LLD] [COFF] Remove a stray duplicate comment. NFC.
The following class isn't part of the export table; there's a
second correctly placed comment about the things that actually
belong to the export table.
2021-06-17 13:02:35 +03:00
Martin Storsjö 728cc0075e [LLD] [COFF] Fix autoexport from LTO objects with comdat symbols
Make sure that comdat symbols also have a non-null dummy
SectionChunk associated.

This requires moving around an existing FIXME regarding comdats in
LTO.

Differential Revision: https://reviews.llvm.org/D103012
2021-06-03 15:14:49 +03:00
Alexandre Ganea 2b9b9652ce [LLD][COFF] Reduce the maximum size of the GHASH table
Before this patch, the maximum size of the GHASH table was 2^31 buckets. However we were storing the bucket index into a TypeIndex which has an input limit of (2^31)-4095 indices, see this link. Any value above that limit will improperly set the TypeIndex's high bit, which is interpreted as DecoratedItemIdMask. This used to cause bad indices on extraction when calling TypeIndex::toArrayIndex().

Differential Revision: https://reviews.llvm.org/D103297
2021-05-28 09:46:17 -04:00
Reid Kleckner 99f023656b [PDB] Fix ubsan complaint about memcpy from null pointer 2021-05-27 19:49:09 -07:00
Reid Kleckner 109aac9212 [PDB] Enable parallel ghash type merging by default
Ghashing is probably going to be faster in most cases, even without
precomputed ghashes in object files.

Here is my table of results linking clang.pdb:

-------------------------------
| threads | GHASH   | NOGHASH |
-------------------------------
|  j1     | 51.031s | 25.141s |
|  j2     | 31.079s | 22.109s |
|  j4     | 18.609s | 23.156s |
|  j8     | 11.938s | 21.984s |
| j28     |  8.375s | 18.391s |
-------------------------------

This shows that ghashing is faster if at least four cores are available.
This may make the linker slower if most cores are busy in the middle of
a build, but in that case, the linker probably isn't on the critical
path of the build. Incremental build performance is arguably more
important than highly contended batch build link performance.

The -time output indicates that ghash computation is the dominant
factor:

    Input File Reading:             924 ms (  1.8%)
    GC:                             689 ms (  1.3%)
    ICF:                            527 ms (  1.0%)
    Code Layout:                    414 ms (  0.8%)
    Commit Output File:              24 ms (  0.0%)
    PDB Emission (Cumulative):    49938 ms ( 94.8%)
      Add Objects:                46783 ms ( 88.8%)
        Global Type Hashing:      38983 ms ( 74.0%)
        GHash Type Merging:        5640 ms ( 10.7%)
        Symbol Merging:            2154 ms (  4.1%)
      Publics Stream Layout:        188 ms (  0.4%)
      TPI Stream Layout:             18 ms (  0.0%)
      Commit to Disk:              2818 ms (  5.4%)
  --------------------------------------------------
  Total Link Time:                52669 ms (100.0%)

We can speed that up with a faster content hash (not SHA1).

Differential Revision: https://reviews.llvm.org/D102888
2021-05-27 14:19:36 -07:00
David Blaikie e5b66a3734 lld-coff: Simplify a few lambda uses after 7975dd033c 2021-05-24 17:26:46 -07:00
Axel Y. Rivera 4fb131b497 [LLD][COFF] PR49068: Include the IMAGE_REL_BASED_HIGHLOW relocation base type when the machine is 64 bits and the relocation type is ADDR32
The COFF driver produces an ABSOLUTE relocation base for an ADDR32
relocation type and the system is 64 bits (machine=AMD64). The
relocation information won't be added in the output and could
produce an incorrect address access during run-time. This change
set checks if the relocation type is IMAGE_REL_AMD64_ADDR32 and
if so, adds the relocated symbol as IMAGE_REL_BASED_HIGHLOW base.

Differential Revision: https://reviews.llvm.org/D96619
2021-05-21 23:45:55 +03:00
Reid Kleckner e73203a561 [PDB] Check the type server guid when ghashing
Previously we simply didn't check this. Prereq to make the test suite
pass with ghash enabled by default.

Differential Revision: https://reviews.llvm.org/D102885
2021-05-20 16:36:12 -07:00
Martin Storsjö 33b71ec9c6 [LLD] [COFF] Fix automatic export of symbols from LTO objects
Differential Revision: https://reviews.llvm.org/D101569
2021-05-21 00:36:58 +03:00
Reid Kleckner 12dd8df38b [PDB] Do not record PGO or coverage public symbols
These symbols are long, and they tend to cause the PDB file size to
overflow. They are generally not necessary when debugging problems in
user code.

This change reduces the size of chrome.dll.pdb with coverage from
6,937,108,480 bytes to 4,690,210,816 bytes.

Differential Revision: https://reviews.llvm.org/D102719
2021-05-19 12:41:31 -07:00
Reid Kleckner ac2226b0f5 [PDB] Improve error handling when writes fail
Handle PDB writing errors like any other error in LLD: emit an error and
continue. This allows the linker to print timing data and summary data
after linking, which can be helpful for finding PDB size problems. Also
report how large the file would have been.

Example output:

lld-link: error: Output data is larger than 4 GiB. File size would have been 6,937,108,480
lld-link: error: failed to write PDB file ./chrome.dll.pdb
                                    Summary
--------------------------------------------------------------------------------
          33282 Input OBJ files (expanded from all cmd-line inputs)
              4 PDB type server dependencies
              0 Precomp OBJ dependencies
       33396931 Input type records
... snip ...
  Input File Reading:           59756 ms ( 45.5%)
  GC:                            7500 ms (  5.7%)
  ICF:                           3336 ms (  2.5%)
  Code Layout:                   6329 ms (  4.8%)
  PDB Emission (Cumulative):    46192 ms ( 35.2%)
    Add Objects:                27609 ms ( 21.0%)
      Type Merging:             16740 ms ( 12.8%)
      Symbol Merging:           10761 ms (  8.2%)
    Publics Stream Layout:       9383 ms (  7.1%)
    TPI Stream Layout:           1678 ms (  1.3%)
    Commit to Disk:              3461 ms (  2.6%)
--------------------------------------------------
Total Link Time:               131244 ms (100.0%)

Differential Revision: https://reviews.llvm.org/D102713
2021-05-18 13:17:17 -07:00
Reid Kleckner ee23f8b36f [COFF] Remove a truncation assertion from setRVA
LLD already produces a nice error message when sections exceed 4GB, and
this setRVA assertion causes LLD to crash instead of diagnosing the
error properly.

No test because we don't want slow tests that create 4GB files.
2021-05-13 19:37:14 -07:00
Martin Storsjö 7e0768329c [LLD] [COFF] Fix including the personality function for DWARF EH when linking with --gc-sections
Since c579a5b1d9 we don't traverse
.eh_frame when doing GC. But the exception handling personality
function needs to be included, and is only referenced from within
.eh_frame.

Differential Revision: https://reviews.llvm.org/D102138
2021-05-12 22:23:01 +03:00
Martin Storsjö 518b7f9135 [LLD] [COFF] Add an assert regarding the RVA of exported symbols. NFC.
As this isn't handled as a regular relocation, the normal handling of
maybeReportRelocationToDiscarded in Chunks.cpp doesn't apply here.

This would have caught the issue fixed by
82de4e0753.

Differential Revision: https://reviews.llvm.org/D102115
2021-05-11 13:04:01 +03:00
Alex Reinking 7ac3fcc526 Allow /STACK in #pragma comment(linker, ...)
The Halide project uses `#pragma comment(linker, "/STACK:...")` to set
the stack size high enough for our embedded compiler to run in end-user
programs on Windows.

Unfortunately, lld-link.exe breaks on this when embedded in a COFF
object, despite supporting the flag on the command line. MSVC's link.exe
supports this fine. This patch extends support for this to lld-link.exe
for better compatibility with MSVC projects.

Differential Revision: https://reviews.llvm.org/D99680
2021-05-05 16:00:33 -07:00
Martin Storsjö 82de4e0753 [LLD] [COFF] Actually include the exported comdat symbols
This is a followup to 2b01a417d7ccb001ccc1185ef5fdc967c9fac8d7;
previously the RVAs of the exported symbols from comdats were left
zero.

Thanks to Kleis Auke Wolthuizen for the fix suggestion and pointing
out the omission.

Differential Revision: https://reviews.llvm.org/D101615
2021-05-04 22:13:08 +03:00
Zequan Wu 6b30240288 Reland "[lld-link] Enable addrsig table in COFF lto"
This reverts commit a78fa73bcf.

The commit cab48e2f0e fixes the issue on eabd55b1b2.
2021-04-29 15:54:12 -07:00
Martin Storsjö 2b01a417d7 [LLD] [COFF] Fix the mingw --export-all-symbols behaviour with comdat symbols
When looking for the "all" symbols that are supposed to be exported,
we can't look at the live flag - the symbols we mark as to be
exported will become GC roots even if they aren't yet marked as live.

With this in place, building an LLVM library with BUILD_SHARED_LIBS
produces the same set of symbols exported regardless of whether the
--gc-sections flag is specified, both with and without being built
with -ffunction-sections.

Differential Revision: https://reviews.llvm.org/D101522
2021-04-29 23:35:10 +03:00
Zequan Wu aa80955f63 [lld-link] Warn on exported deleting dtor
MSVC linker has this [[ https://docs.microsoft.com/en-us/cpp/error-messages/tool-errors/linker-tools-warning-lnk4102?view=msvc-160 | warning]], so lld-link should also warn on this.

Differential Revision: https://reviews.llvm.org/D100606
2021-04-20 14:06:31 -07:00
Reid Kleckner 18a9b18087 [COFF] Simplify ICF associated comdat handling
This is a different approach from D98993 that should achieve most of the
same benefit. The two changes are:
1. Sort the list of associated child sections by section name
2. Do not consider associated sections to have children themselves

This fixes the main issue, which was that we sometimes considered an
.xdata section to have a child .pdata section. That lead to slow links
and larger binaries (less xdata folding).

Otherwise, this should be NFC: we go back to ignoring .debug/.gljmp and
other metadata sections rather than only looking at pdata/xdata. We
discovered that we do care about other associated sections, like ASan
global registration metadata.
2021-04-14 10:40:16 -07:00
Bogdan Graur 7975dd033c
[NFC] Fix unused variable warning.
Differential Revision: https://reviews.llvm.org/D100451
2021-04-14 09:36:12 +02:00
Pengfei Wang 184377da5c [LLD] Implement /guard:[no]ehcont
Reviewed By: rnk

Differential Revision: https://reviews.llvm.org/D99078
2021-04-14 15:06:49 +08:00
Abhina Sreeskantharajan 82b3e28e83 [SystemZ][z/OS][Windows] Add new OF_TextWithCRLF flag and use this flag instead of OF_Text
Problem:
On SystemZ we need to open text files in text mode. On Windows, files opened in text mode adds a CRLF '\r\n' which may not be desirable.

Solution:
This patch adds two new flags

  - OF_CRLF which indicates that CRLF translation is used.
  - OF_TextWithCRLF = OF_Text | OF_CRLF indicates that the file is text and uses CRLF translation.

Developers should now use either the OF_Text or OF_TextWithCRLF for text files and OF_None for binary files. If the developer doesn't want carriage returns on Windows, they should use OF_Text, if they do want carriage returns on Windows, they should use OF_TextWithCRLF.

So this is the behaviour per platform with my patch:

z/OS:
OF_None: open in binary mode
OF_Text : open in text mode
OF_TextWithCRLF: open in text mode

Windows:
OF_None: open file with no carriage return
OF_Text: open file with no carriage return
OF_TextWithCRLF: open file with carriage return

The Major change is in llvm/lib/Support/Windows/Path.inc to only set text mode if the OF_CRLF is set.
```
  if (Flags & OF_CRLF)
    CrtOpenFlags |= _O_TEXT;
```

These following files are the ones that still use OF_Text which I left unchanged. I modified all these except raw_ostream.cpp in recent patches so I know these were previously in Binary mode on Windows.
./llvm/lib/Support/raw_ostream.cpp
./llvm/lib/TableGen/Main.cpp
./llvm/tools/dsymutil/DwarfLinkerForBinary.cpp
./llvm/unittests/Support/Path.cpp
./clang/lib/StaticAnalyzer/Core/HTMLDiagnostics.cpp
./clang/lib/Frontend/CompilerInstance.cpp
./clang/lib/Driver/Driver.cpp
./clang/lib/Driver/ToolChains/Clang.cpp

Reviewed By: MaskRay

Differential Revision: https://reviews.llvm.org/D99426
2021-04-06 07:23:31 -04:00
Nico Weber a78fa73bcf Revert "[lld-link] Enable addrsig table in COFF lto"
This reverts commit eabd55b1b2.
Speculative, for crbug.com/1195545
2021-04-03 10:56:33 -04:00
Zequan Wu eabd55b1b2 [lld-link] Enable addrsig table in COFF lto
This allow safe-icf mode to work when linking with LTO.

Differential Revision: https://reviews.llvm.org/D99613
2021-03-30 15:47:53 -07:00
Amy Huang 5127da0291 Revert "[COFF] Only consider associated EH sections during ICF"
This change causes an asan error for ODR violation.

This reverts commit 7ce9a3e9a9.
2021-03-29 19:15:35 -07:00
Abhina Sreeskantharajan c83cd8feef [NFC] Reordering parameters in getFile and getFileOrSTDIN
In future patches I will be setting the IsText parameter frequently so I will refactor the args to be in the following order. I have removed the FileSize parameter because it is never used.

```
  static ErrorOr<std::unique_ptr<MemoryBuffer>>
  getFile(const Twine &Filename, bool IsText = false,
          bool RequiresNullTerminator = true, bool IsVolatile = false);

  static ErrorOr<std::unique_ptr<MemoryBuffer>>
  getFileOrSTDIN(const Twine &Filename, bool IsText = false,
                 bool RequiresNullTerminator = true);

 static ErrorOr<std::unique_ptr<MB>>
 getFileAux(const Twine &Filename, uint64_t MapSize, uint64_t Offset,
            bool IsText, bool RequiresNullTerminator, bool IsVolatile);

  static ErrorOr<std::unique_ptr<WritableMemoryBuffer>>
  getFile(const Twine &Filename, bool IsVolatile = false);
```

Reviewed By: jhenderson

Differential Revision: https://reviews.llvm.org/D99182
2021-03-25 09:47:49 -04:00
Yolanda Chen 4f9c61ef72 [lld] add context-sensitive PGO options for COFF.
Add lld CSPGO (Contex-Sensitive PGO) options for COFF target.

Reference the ELF options from https://reviews.llvm.org/D56675

Reviewed By: tejohnson

Differential Revision: https://reviews.llvm.org/D98763
2021-03-24 23:40:09 -07:00
Reid Kleckner 7ce9a3e9a9 [COFF] Only consider associated EH sections during ICF
The only known reason why ICF should not merge otherwise identical
sections with differing associated sections has to do with exception
handling tables. It's not clear what ICF should do when there are other
kinds of associated sections. In every other case when this has come up,
debug info and CF guard metadata, we have opted to make ICF ignore the
associated sections.

For comparison, ELF doesn't do anything for comdat groups. Instead,
.eh_frame is parsed to figure out if a section has an LSDA, and if so,
ICF is disabled.

Another issue is that the order of associated sections is not defined.
We have had issues in the past (crbug.com/1144476) where changing the
order of the .xdata/.pdata sections in the object file lead to large ICF
slowdowns.

To address these issues, I decided it would be best to explicitly
consider only .pdata and .xdata sections during ICF. This makes it easy
to ignore the object file order, and I think it makes the intention of
the code clearer.

I've also made the children() accessor return an empty list for
associated sections. This mostly only affects ICF and GC. This was the
behavior before I made this a linked list, so the behavior change should
be good. This had positive effects on chrome.dll: more .xdata sections
were merged that previously could not be merged because they were
associated with distinct .pdata sections.

Reviewed By: mstorsjo

Differential Revision: https://reviews.llvm.org/D98993
2021-03-22 15:36:26 -07:00
Reid Kleckner b552adf8b3 [PDB] Improve warning for corrupt debug info
The S_[GL]PROC32_ID symbol records are supposed to point to function ID
records. If they don't, they are corrupt. The warning message here was
very technical, but a user has encountered it in the wild. Add some more
information and some more testing.
2021-03-11 14:28:09 -08:00
Reid Kleckner b69db4a7ab Re-land "[PDB] Defer relocating .debug$S until commit time and parallelize it"
This reverts commit bacf9cf2c5 and
reinstates commit 1a9bd5b813.

Reverting this commit did not appear to make the problem go away, so we
can go ahead and reland it.
2021-03-10 15:14:09 -08:00
Zequan Wu 5bdc5e7efd [lld-link] Add safe icf mode to lld-link, which does safe icf for all sections.
Differential Revision: https://reviews.llvm.org/D97436
2021-03-03 14:52:33 -08:00
Martin Storsjö 075539ddf6 [LLD] [COFF] Allow invoking lib.exe mode via -lib in addition to /lib
Remove a stray -lib argument in guardcf-lto.ll; llvm-lib doesn't
support generating import libs from a def file unlike lib.exe.
Previously this worked because the -lib argument was ignored
(printing only a warning).

Differential Revision: https://reviews.llvm.org/D96699
2021-02-24 11:16:12 +02:00
Nico Weber e6d1f261a5 [lld-link] Add /reproduce: support for several flags
/reproduce: now works correctly with:
- /call-graph-ordering-file:
- /def:
- /natvis:
- /order:
- /pdbstream:

I went through all instances of MemoryBuffer::getFile() and made sure
everything that didn't already do so called takeBuffer().

For natvis, that wasn't possible since DebugInfo/PDB wants to take
owernship of the natvis buffer. For that case, I'm manually adding the
tar file entry.

/natvis: and /pdbstream: is slightly awkward, since createResponseFile()
always adds these flags to the response file but createPDB() (which
ultimately adds the files referenced by the flags) is only called if
/debug is also passed. So when using /natvis: without /debug with
/reproduce:, lld won't warn, but when linking using the response
file from the archive, it won't find the natvis file since it's not
in the tar. This isn't a new issue though, and after this patch things
at least work with using /natvis: _with_ debug with /reproduce:.
(Same for /pdbstream:)

Differential Revison: https://reviews.llvm.org/D97212
2021-02-22 16:52:49 -05:00
Reid Kleckner bacf9cf2c5 Revert "[PDB] Defer relocating .debug$S until commit time and parallelize it"
This reverts commit 1a9bd5b813.

I suspect that this patch may have caused https://crbug.com/1171438.
2021-01-28 13:17:27 -08:00
Sam Clegg 299b0e5ee9 [lld] Consistent help text for `--save-temps`
I noticed that this option was not appearing at all in the `--help`
messages for `wasm-ld` or `ld.lld`.

Add help text and make it consistent across all ports.

Differential Revision: https://reviews.llvm.org/D94925
2021-01-25 10:27:18 -08:00
Reid Kleckner 1a9bd5b813 Reland "[PDB] Defer relocating .debug$S until commit time and parallelize it"
This reverts commit 5b7aef6eb4 and relands
6529d7c5a4.

The ASan error was debugged and determined to be the fault of an invalid
object file input in our test suite, which was fixed by my last change.
LLD's project policy is that it assumes input objects are valid, so I
have added a comment about this assumption to the relocation bounds
check.
2021-01-20 11:53:43 -08:00
Reid Kleckner 69e0bc77a5 [COFF] Use range for on relocations, NFC 2021-01-20 11:01:33 -08:00
Mitch Phillips 5b7aef6eb4 Revert "[PDB] Defer relocating .debug$S until commit time and parallelize it"
This reverts commit 6529d7c5a4.

Reason: Broke the ASan buildbots.
http://lab.llvm.org:8011/#/builders/99/builds/1567
2021-01-19 11:45:48 -08:00
Alexandre Ganea e7a371f9fd [LLD][COFF] Avoid std::vector resizes during type merging
Consistently saves approx. 0.6 sec (out of 18 sec) on a large output (400 MB EXE, 2 GB PDB).

Differential Revision: https://reviews.llvm.org/D94555
2021-01-13 14:35:03 -05:00
Reid Kleckner 6529d7c5a4 [PDB] Defer relocating .debug$S until commit time and parallelize it
This is a pretty classic optimization. Instead of processing symbol
records and copying them to temporary storage, do a first pass to
measure how large the module symbol stream will be, and then copy the
data into place in the PDB file. This requires defering relocation until
much later, which accounts for most of the complexity in this patch.

This patch avoids copying the contents of all live .debug$S sections
into heap memory, which is worth about 20% of private memory usage when
making PDBs. However, this is not an unmitigated performance win,
because it can be faster to read dense, temporary, heap data than it is
to iterate symbol records in object file backed memory a second time.

Results on release chrome.dll:
peak mem: 5164.89MB -> 4072.19MB (-1,092.7MB, -21.2%)
wall-j1:  0m30.844s -> 0m32.094s (slightly slower)
wall-j3:  0m20.968s -> 0m20.312s (slightly faster)
wall-j8:  0m19.062s -> 0m17.672s (meaningfully faster)

I gathered similar numbers for a debug, component build of content.dll
in Chrome, and the performance impact of this change was in the noise.
The memory usage reduction was visible and similar.

Because of the new parallelism in the PDB commit phase, more cores makes
the new approach faster. I'm assuming that most C++ developer machines
these days are at least quad core, so I think this is a win.

Differential Revision: https://reviews.llvm.org/D94267
2021-01-12 17:46:29 -08:00
Alexandre Ganea b14ad90b13 [LLD][COFF] Simplify function. NFC. 2021-01-07 22:37:11 -05:00
Alexandre Ganea eaadb41db6 [LLD][COFF] When using PCH.OBJ, ensure func_id records indices are remapped under /DEBUG:GHASH
Before this patch, when using LLD with /DEBUG:GHASH and MSVC precomp.OBJ files, we had a bunch of:

lld-link: warning: S_[GL]PROC32ID record in blabla.obj refers to PDB item index 0x206ED1 which is not a LF[M]FUNC_ID record

This was caused by LF_FUNC_ID and LF_MFUNC_ID which didn't have correct mapping to the corresponding TPI records. The root issue was that the indexMapStorage was improperly re-assembled in UsePrecompSource::remapTpiWithGHashes.

After this patch, /DEBUG and /DEBUG:GHASH produce exactly the same debug infos in the PDB.

Differential Revision: https://reviews.llvm.org/D93732
2021-01-07 17:27:13 -05:00
Nico Weber 77fb45e59e [lld/mac] Add --version flag
It's an extension to ld64, but all the other ports have it, and
someone asked for it in PR43721.

While here, change the COFF help text to match the other ports.

Differential Revision: https://reviews.llvm.org/D93491
2020-12-22 22:06:39 -05:00
Reshabh Sharma fdd6ed8e93 [LLD] Rename lld port driver entry function to a consistent name
Libraries linked to the lld elf library exposes a function named main.
When debugging code linked to such libraries and intending to set a
breakpoint at main, the debugger also sets breakpoint at the main
function at lld elf driver. The possible choice was to rename it to
link but that would again clash with lld::*::link. This patch tries
to consistently rename them to linkerMain.

Differential Revision: https://reviews.llvm.org/D91418
2020-12-18 12:18:37 +05:30
Nico Weber f710bb7063 lld: Replace some lld::outs()s with message()
No behavior change.
2020-12-17 16:19:09 -05:00
Nico Weber 9c6a884f67 fix typo to cycle bots 2020-12-12 20:16:14 -05:00
Reid Kleckner c8466a5731 Avoid a possible one-byte OOB read off of .drectve sections
Pointed out by Ryan Prichard
2020-12-09 13:32:28 -08:00
Arthur Eubanks fed7565ee2 [COFF][LTO][NPM] Use NPM for LTO with ENABLE_EXPERIMENTAL_NEW_PASS_MANAGER
Reviewed By: hans

Differential Revision: https://reviews.llvm.org/D92866
2020-12-09 08:53:50 -08:00
Fangrui Song 4701cb41ed [lld] Delete unused declarations
Notes:

* runMSVCLinker: remnant of r338615
* wasm markSymbol: remnant of r374275
* wasm addDataAddressGlobal: accidentally added by r372779
* MachO Writer::createSymtabContents: accidentally added by D76839
2020-12-06 15:26:37 -08:00
Nico Weber c8974af164 fix typos to cycle bots 2020-12-04 10:18:44 -05:00
Nico Weber a0994cbe27 lld-link: Let LLD_REPRODUCE control /reproduce:, like in ld.lld
Also sync help texts for the option between elf and coff ports.

Decisions:
- Do this even if /lldignoreenv is passed. /reproduce: does not affect
  the main output, and this makes the env var more convenient to use.
  (On the other hand, it's now possible to set this env var and forget
  about it, and all future builds in the same shell will be much slower.
  That's true for ld.lld, but posix shells have an easy way to set an
  env var for a single command; in cmd.exe this is not possible without
  contortions. Then again, lld-link runs in posix shells too.)

Original patch rebased across D68378 and D68381.

Differential Revision: https://reviews.llvm.org/D67707
2020-11-27 13:33:55 -05:00
Nico Weber da0aaedcd0 [gn build] (manually) port b534beabee 2020-11-25 20:19:46 -05:00
Amy Huang 1363dfaf31 [CodeView] Avoid emitting empty debug globals subsection.
In https://reviews.llvm.org/D89072 I added static const data members
to the debug subsection for globals. It skipped emitting an S_CONSTANT if it
didn't have a value, which meant the subsection could be empty.

This patch fixes the empty subsection issue.

Differential Revision: https://reviews.llvm.org/D92049
2020-11-25 16:13:32 -08:00
Nico Weber 11b7625833 [lld/mac] Implement basic typo correction for flags
Also use "unknown flag 'flag'" instead of "unknown flag: flag" for
consistency with the other ports.

Differential Revision: https://reviews.llvm.org/D91970
2020-11-24 11:33:39 -05:00
Martin Storsjö 0b2d84fba8 [LLD] [COFF] Allow wrapping dllimported functions
GNU ld doesn't seem to do this though, but it looks like a reasonable
use case, is easy to implement, and was requested in
https://bugs.llvm.org/show_bug.cgi?id=47384.

Differential Revision: https://reviews.llvm.org/D91689
2020-11-24 10:15:20 +02:00
Andrew Paverd 0139c8af8d [CFGuard] Add address-taken IAT tables and delay-load support
This patch adds support for creating Guard Address-Taken IAT Entry Tables (.giats$y sections) in object files, matching the behavior of MSVC. These contain lists of address-taken imported functions, which are used by the linker to create the final GIATS table.
Additionally, if any DLLs are delay-loaded, the linker must look through the .giats tables and add the respective load thunks of address-taken imports to the GFIDS table, as these are also valid call targets.

Reviewed By: rnk

Differential Revision: https://reviews.llvm.org/D87544
2020-11-17 18:24:45 -08:00
Nico Weber baa2aa28f5 lld: Add --color-diagnostic to MachO port, harmonize others
This adds `--[no-]color-diagnostics[=auto,never,always]` to
the MachO port and harmonizes the flag in the other ports:
- Consistently use MetaVarName
- Consistently document the non-eq version as alias of the eq version
- Use B<> in the ports that have it (no-op, shorter)
- Fix oversight in COFF port that made the --no flag have the wrong
  prefix

Differential Revision: https://reviews.llvm.org/D91640
2020-11-17 12:58:30 -05:00
Hans Wennborg 418f18c6cd Revert "Reland [CFGuard] Add address-taken IAT tables and delay-load support"
This broke both Firefox and Chromium (PR47905) due to what seems like dllimport
function not being handled correctly.

> This patch adds support for creating Guard Address-Taken IAT Entry Tables (.giats$y sections) in object files, matching the behavior of MSVC. These contain lists of address-taken imported functions, which are used by the linker to create the final GIATS table.
> Additionally, if any DLLs are delay-loaded, the linker must look through the .giats tables and add the respective load thunks of address-taken imports to the GFIDS table, as these are also valid call targets.
>
> Reviewed By: rnk
>
> Differential Revision: https://reviews.llvm.org/D87544

This reverts commit cfd8481da1.
2020-11-11 16:03:33 +01:00
rojamd b79e990f40 [lld][COFF] Add command line options for LTO with new pass manager
This is more or less a port of rL329598 (D45275) to the COFF linker.
Since there were already LTO-related settings under -opt:, I added
them there instead of new flags.

Differential Revision: https://reviews.llvm.org/D90624
2020-11-05 14:41:35 -05:00
Peter Penzin e59726220f [LLD] [COFF] Align all debug directories
Match MSVC linker output - align all debug directories on four bytes,
while removing debug directory alignment. This would have the same
effect on CETCOMPAT support as D89919.

Chromium bug: https://crbug.com/1136664

Differential Revision: https://reviews.llvm.org/D89921
2020-11-02 10:47:51 -08:00
Reid Kleckner 32cc962ef3 [COFF] Move ghash timers under the "add objects" timer
I had envisioned the ghash step as a big up front step, but as currently
written, the timers are nested, and we are notionally adding types from
objects, so we might as well arrange the timers this way.
2020-10-31 11:08:59 -07:00
Martin Storsjö 3785a413fe Reapply [LLD] [COFF] Implement a GNU/ELF like -wrap option
Add a simple forwarding option in the MinGW frontend, and implement
the private -wrap option in the COFF linker.

The feature in lld-link isn't gated by the -lldmingw option, but
the option is left as a private, undocumented option primarily
used by the MinGW driver.

The implementation is significantly based on the support for --wrap
in the ELF linker, but many small nuance details are different
between the ELF and COFF linkers, ending up with more than a few
implementation differences.

This fixes https://bugs.llvm.org/show_bug.cgi?id=47384.

Differential Revision: https://reviews.llvm.org/D89004

Reapplied with the bitfield member canInline fixed so it doesn't break
builds targeting windows.
2020-10-15 22:14:02 +03:00
Arthur Eubanks 3d338f6813 Revert "[LLD] [COFF] Implement a GNU/ELF like -wrap option"
This reverts commit a012c704b5.

Breaks Windows builds.

C:\src\llvm-mint\lld\COFF\Symbols.cpp(26,1): error: static_assert failed due to requirement 'sizeof(lld::coff::SymbolUnion) <= 48' "symbols should be optimized for memory usage"
static_assert(sizeof(SymbolUnion) <= 48,
2020-10-15 10:27:25 -07:00
Martin Storsjö a012c704b5 [LLD] [COFF] Implement a GNU/ELF like -wrap option
Add a simple forwarding option in the MinGW frontend, and implement
the private -wrap option in the COFF linker.

The feature in lld-link isn't gated by the -lldmingw option, but
the option is left as a private, undocumented option primarily
used by the MinGW driver.

The implementation is significantly based on the support for --wrap
in the ELF linker, but many small nuance details are different
between the ELF and COFF linkers, ending up with more than a few
implementation differences.

This fixes https://bugs.llvm.org/show_bug.cgi?id=47384.

Differential Revision: https://reviews.llvm.org/D89004
2020-10-15 18:34:02 +03:00
Martin Storsjö 9803cf57d6 [LLD] [COFF] Fix a condition that was missed in 7f0e6c31c2. NFC.
This should fix cases when e.g. auto import is enabled without
mingw mode in total being enabled.

Differential Revision: https://reviews.llvm.org/D89006
2020-10-15 18:34:01 +03:00
Luqman Aden 6a73d6564a [LLD] Set alignment as part of Characteristics in TLS table.
Fixes https://bugs.llvm.org/show_bug.cgi?id=46473

LLD wasn't previously specifying any specific alignment in the TLS table's Characteristics field so the loader would just assume the default value (16 bytes). This works most of the time except if you have thread locals that want specific higher alignments (e.g. 32 as in the bug) *even* if they specify an alignment on the thread local. This change updates LLD to take the max alignment from tls section.

Reviewed By: rnk

Differential Revision: https://reviews.llvm.org/D88637
2020-10-15 00:22:40 -07:00
Luqman Aden f87c98def8 Revert "[LLD] Set alignment as part of Characteristics in TLS table."
Revert individual wip commits and will instead follow up with a
single commit with all the changes. Makes cherry-picking easier
and will contain all the right tags.

This reverts commit 32a4ad3b6c.
This reverts commit 7fe13af676.
This reverts commit 51fbc1bef6.
This reverts commit f80950a8bb.
This reverts commit 0778cad9f3.
This reverts commit 8b70d527d7.
2020-10-15 00:21:36 -07:00
Luqman Aden 32a4ad3b6c [LLD] Set alignment as part of Characteristics in TLS table.
Fixes https://bugs.llvm.org/show_bug.cgi?id=46473

LLD wasn't previously specifying any specific alignment in the TLS table's Characteristics field so the loader would just assume the default value (16 bytes). This works most of the time except if you have thread locals that want specific higher alignments (e.g. 32 as in the bug) *even* if they specify an alignment on the thread local. This change updates LLD to take the max alignment from tls section.

Reviewed By: rnk

Differential Revision: https://reviews.llvm.org/D88637
2020-10-14 19:41:03 -07:00