Commit Graph

15258 Commits

Author SHA1 Message Date
Nico Weber cd52b35ee4 fix comment typos to cycle bots 2022-04-04 08:56:18 -04:00
Fangrui Song 388584d382 [ELF][test] Fix RUN lines in lto/sample-profile.ll
Reported at https://github.com/llvm/llvm-project/issues/54679#issuecomment-1086862116
2022-04-03 23:57:31 -07:00
Tobias Hieta 98bc304e9f [lld][COFF] Fix TypeServerSource lookup on GUID collisions
Microsoft shipped a bunch of PDB files with broken/invalid GUIDs
which lead lld to use 0xFF as the key for these files in an internal
cache. When multiple files have this key it will lead to collisions
and confused symbol lookup.

Several approaches to fix this was considered. Including making the key
the path to the PDB file, but this requires some filesystem operations
in order to normalize the file path.

Since this only happens with malformatted PDB files and we haven't
seen this before they malformatted files where shipped with visual
studio we probably shouldn't optimize for this use-case.

Instead we now just don't insert files with Guid == 0xFF into the
cache map and warn if we get collisions so similar problems can be
found in the future instead of being silent.

Discussion about the root issue and the approach to this fix can be found on Github: https://github.com/llvm/llvm-project/issues/54487

Reviewed By: aganea

Differential Revision: https://reviews.llvm.org/D122372
2022-04-02 10:09:07 +02:00
Nico Weber 663a7fa712 [lld/mac] Tweak a few comments
Addresses review feedback I had missed on https://reviews.llvm.org/D122624

No behavior change.

Differential Revision: https://reviews.llvm.org/D122904
2022-04-01 19:32:07 -04:00
Arthur Eubanks 79a9fe6c8a [test] Mark uuid.s as unsupported on Windows
For systems using gnuwin32, awk does not exist.
2022-04-01 15:32:51 -07:00
Leonard Grey a9e325116c Add output filename to UUID hash
Differential Revision: https://reviews.llvm.org/D122843
2022-03-31 18:50:05 -04:00
Roger Kim 34b9729561 [lld-macho][NFC] Encapsulate symbol priority implementation.
Just some code clean up.

Reviewed By: #lld-macho, int3

Differential Revision: https://reviews.llvm.org/D122752
2022-03-31 13:47:38 -04:00
Nico Weber 10cda6e36c [lld/mac] Give range extension thunks for local symbols local visibility
When two local symbols (think: file-scope static functions, or functions in
unnamed namespaces) with the same name in two different translation units
both needed thunks, ld64.lld previously created external thunks for both
of them. These thunks ended up with the same name, leading to a duplicate
symbol error for the thunk symbols.

Instead, give thunks for local symbols local visibility.

(Hitting this requires a jump to a local symbol from over 128 MiB away.
It's unlikely that a single .o file is 128 MiB large, but with ICF
you can end up with a situation where the local symbol is ICF'd with
a symbol in a separate translation unit. And that can introduce a
large enough jump to require a thunk.)

Fixes PR54599.

Differential Revision: https://reviews.llvm.org/D122624
2022-03-30 16:45:05 -04:00
Fangrui Song c0065f1182 [ELF] Default to --no-fortran-common
D86142 introduced --fortran-common and defaulted it to true (matching GNU ld
but deviates from gold/macOS ld64). The default state was motivated by transparently
supporting some FORTRAN 77 programs (Fortran 90 deprecated common blocks).
Now I think it again. I believe we made a mistake to change the default:

* this is a weird and legacy rule, though the breakage is very small
* --fortran-common introduced complexity to parallel symbol resolution and will slow down it
* --fortran-common more likely causes issues when users mix COMMON and
  STB_GLOBAL definitions (see https://github.com/llvm/llvm-project/issues/48570 and
  https://maskray.me/blog/2022-02-06-all-about-common-symbols).
  I have seen several issues in our internal projects and Android.
  On the other hand, --no-fortran-common is safer since
  COMMON/STB_GLOBAL have the same semantics related to archive member extraction.

Therefore I think we should switch back, not punishing the common uage.
A platform wanting --fortran-common can implement ld.lld as a shell script
wrapper around `lld -flavor gnu --fortran-common "$@"`.

Reviewed By: ikudrin, sfertile

Differential Revision: https://reviews.llvm.org/D122450
2022-03-30 09:12:09 -07:00
Fangrui Song 4645311933 [ELF] --emit-relocs: adjust offsets of .rel[a].eh_frame relocations
Two code paths may reach the EHFrame case in SectionBase::getOffset:

* .eh_frame reference
* relocation copy for --emit-relocs

The first may be used by clang_rt.crtbegin.o and GCC crtbeginT.o to get the
start address of the output .eh_frame. The relocation has an offset of 0 or
(x86-64 PC-relative leaq for clang_rt.crtbegin.o) -4. The current code just
returns `offset`, which handles this case well.

The second is related to InputSection::copyRelocations on .eh_frame (used by
--emit-relocs). .eh_frame pieces may be dropped due to GC/ICF, so we should
convert the input offset to the output offset. Use the same way as
MergeInputSection with a special case handling outSecOff==-1 for an invalid
piece (see eh-frame-marker.s).

This exposes an issue in mips64-eh-abs-reloc.s that we don't reliably
handle anyway. Just add --no-check-dynamic-relocations to paper over it.

Differential Revision: https://reviews.llvm.org/D122459
2022-03-29 09:51:41 -07:00
Fangrui Song 7370a489b1 [ELF] --emit-relocs: fix missing STT_SECTION when the first input section is synthetic
addSectionSymbols suppresses the STT_SECTION symbol if the first input section
is non-SHF_MERGE synthetic. This is incorrect when the first input section is synthetic
while a non-synthetic input section exists:

* `.bss : { *(COMMON) *(.bss) }`
  (abc388ed3c regressed the case because
  COMMON symbols precede .bss in the absence of a linker script)
* Place a synthetic section in another section: `.data : { *(.got) *(.data) }`

For `%t/a1` in the new test emit-relocs-synthetic.s, ld.lld produces incorrect
relocations with symbol index 0.
```
0000000000000000 <_start>:
       0: 8b 05 33 00 00 00             movl    51(%rip), %eax          # 0x39 <bss>
                0000000000000002:  R_X86_64_PC32        *ABS*+0xd
       6: 8b 05 1c 00 00 00             movl    28(%rip), %eax          # 0x28 <common>
                0000000000000008:  R_X86_64_PC32        common-0x4
       c: 8b 05 06 00 00 00             movl    6(%rip), %eax           # 0x18
                000000000000000e:  R_X86_64_GOTPCRELX   *ABS*+0x4
```

Fix the issue by checking every input section.

Reviewed By: ikudrin

Differential Revision: https://reviews.llvm.org/D122463
2022-03-29 08:56:21 -07:00
Fangrui Song 48e251b1d6 Revert D122459 "[ELF] --emit-relocs: adjust offsets of .rel[a].eh_frame relocations"
This reverts commit 6faba31e0d.

It may cause "offset is outside the section".
2022-03-28 20:26:21 -07:00
Fangrui Song 6faba31e0d [ELF] --emit-relocs: adjust offsets of .rel[a].eh_frame relocations
.eh_frame pieces may be dropped due to GC/ICF. When --emit-relocs adds
relocations against .eh_frame, the offsets need to be adjusted. Use the same
way as MergeInputSection with a special case handling outSecOff==-1 for an
invalid piece (see eh-frame-marker.s).

This exposes an issue in mips64-eh-abs-reloc.s that we don't reliably
handle anyway. Just add --no-check-dynamic-relocations to paper over it.

Original patch by Ayrton Muñoz

Differential Revision: https://reviews.llvm.org/D122459
2022-03-28 16:23:13 -07:00
Fangrui Song 27ef7494b1 [ELF][test] Refactor some .eh_frame tests
* Improve eh-frame-merge.s
* Delete invalid .eh_frame+5 test in ehframe-relocation.s
2022-03-28 15:55:46 -07:00
Fangrui Song 1db59dc8e2 [ELF] Fix llvm_unreachable failure when COMMON is placed in SHT_PROGBITS output section
Fix a regression in aa27bab5a1a17e9c4168a741a6298ecaa92c1ecb: COMMON in an
SHT_PROGBITS output section caused llvm_unreachable failure.
2022-03-28 11:05:52 -07:00
Fangrui Song 8565a87fd4 [ELF] Simplify MergeInputSection::getParentOffset. NFC
and remove overly verbose comments.
2022-03-28 10:02:35 -07:00
Fangrui Song c37accf0a2 [Option] Avoid using the default argument for the 3-argument hasFlag. NFC
The default argument true is error-prone: I think many would think the
default is false.
2022-03-26 00:57:06 -07:00
Sam McCall 57ee624d79 [cmake] Provide CURRENT_TOOLS_DIR centrally, replacing CLANG_TOOLS_DIR
CLANG_TOOLS_DIR holds the the current bin/ directory, maybe with a %(build_mode)
placeholder. It is used to add the just-built binaries to $PATH for lit tests.
In most cases it equals LLVM_TOOLS_DIR, which is used for the same purpose.
But for a standalone build of clang, CLANG_TOOLS_DIR points at the build tree
and LLVM_TOOLS_DIR points at the provided LLVM binaries.

Currently CLANG_TOOLS_DIR is set in clang/test/, clang-tools-extra/test/, and
other things always built with clang. This is a few cryptic lines of CMake in
each place. Meanwhile LLVM_TOOLS_DIR is provided by configure_site_lit_cfg().

This patch moves CLANG_TOOLS_DIR to configure_site_lit_cfg() and renames it:
 - there's nothing clang-specific about the value
 - it will also replace LLD_TOOLS_DIR, LLDB_TOOLS_DIR etc (not in this patch)

It also defines CURRENT_LIBS_DIR. While I removed the last usage of
CLANG_LIBS_DIR in e4cab4e24d, there are LLD_LIBS_DIR usages etc that
may be live, and I'd like to mechanically update them in a followup patch.

Differential Revision: https://reviews.llvm.org/D121763
2022-03-25 20:22:01 +01:00
Fangrui Song 940bd4c771 [ELF] addSectionSymbols: simplify isec->getOutputSection(). NFC 2022-03-24 21:54:20 -07:00
Fangrui Song d3e5b6f753 [ELF] Implement --build-id={md5,sha1} with truncated BLAKE3
--build-id was introduced as "approximation of true uniqueness across all
binaries that might be used by overlapping sets of people". It does not require
the some resistance mentioned below. In practice, people just use --build-id=md5
for 16-byte build ID and --build-id=sha1 for 20-byte build ID.

BLAKE3 has 256-bit key length, which provides 128-bit security against
(second-)preimage, collision, and differentiability attacks. Its portable
implementation is fast. It additionally provides Arm Neon/AVX2/AVX-512. Just
implement --build-id={md5,sha1} with truncated BLAKE3.

Linking clang 14 RelWithDebInfo with --threads=8 on a Skylake CPU:

* 1.13x as fast with --build-id=md5
* 1.15x as fast with --build-id=sha1

--threads=4 on Apple m1:

* 1.25x as fast with --build-id=md5
* 1.17x as fast with --build-id=sha1

Reviewed By: ikudrin

Differential Revision: https://reviews.llvm.org/D121531
2022-03-24 11:31:39 -07:00
Jakob Koschel 0c86198b27 Reland "[ELF] Enable new passmanager plugin support for LTO"
This is the orignal patch + a check that LLVM_BUILD_EXAMPLES is enabled before
adding a dependency on the 'Bye' example pass.

Original summary:

Add cli options for new passmanager plugin support to lld.

Currently it is not possible to load dynamic NewPM plugins with lld. This is an
incremental update to D76866. While that patch only added cli options for
llvm-lto2, this adds them for lld as well. This is especially useful for running
dynamic plugins on the linux kernel with LTO.

Reviewed By: MaskRay

Differential Revision: https://reviews.llvm.org/D120490
2022-03-24 16:29:18 +01:00
Raphael Isemann 1104d79261 Revert "[ELF] Enable new passmanager plugin support for LTO"
This reverts commit 32012eb11b.

Broke CMake configuration.
2022-03-24 09:57:15 +01:00
Jakob Koschel 32012eb11b [ELF] Enable new passmanager plugin support for LTO
Add cli options for new passmanager plugin support to lld.

Currently it is not possible to load dynamic NewPM plugins with lld. This is an
incremental update to D76866. While that patch only added cli options for
llvm-lto2, this adds them for lld as well. This is especially useful for running
dynamic plugins on the linux kernel with LTO.

Reviewed By: MaskRay

Differential Revision: https://reviews.llvm.org/D120490
2022-03-24 08:08:54 +01:00
Roger Kim f858fba631 [lld][Macho][NFC] Encapsulate priorities map in a priority class
`config->priorities` has been used to hold the intermediate state during the construction of the order in which sections should be laid out. This is not a good place to hold this state since the intermediate state is not a "configuration" for LLD. It should be encapsulated in a class for building a mapping from section to priority (which I created in this diff as the `PriorityBuilder` class).

The same thing is being done for `config->callGraphProfile`.

Reviewed By: #lld-macho, int3

Differential Revision: https://reviews.llvm.org/D122156
2022-03-23 13:57:26 -04:00
Jacob Lambert 71b162c4bd [AMDGPU][LLD] Adding support for ABI version 5 option
Code object version 5 will use the same EFlags as version 4, so we only need to add an additional case

Differential Revision: https://reviews.llvm.org/D122190
2022-03-23 01:22:37 -07:00
Jez Ng c9c2363048 [lld-macho][nfc] Don't mix file sizes with addresses
Update DataInCode's calculation of `endAddr` to use `getSize()` instead
of `getFileSize()` -- while in practice they're the same for
non-zerofill sections (which code sections are), we still should treat
address sizes / offsets as distinct from file sizes / offsets.
2022-03-22 17:52:53 -04:00
Jez Ng a993d607de [lld-macho][nfc] Add comment explaining why a cast<> is safe 2022-03-21 07:23:09 -04:00
Jez Ng 1c0234dfcc [lld-macho][nfc] Have findContainingSubsection take a Section
... instead of an instance of `Subsections`.

This simplifies the code slightly since all its callsites have a Section
instance anyway.
2022-03-21 07:23:09 -04:00
Sam Clegg a04a507714 [lld][WebAssembly] Fix crash accessing non-live __tls_base symbol
In programs that don't otherwise depend on `__tls_base` it won't
be marked as live.  However this symbol is used internally in
a couple of places do we need to mark it as live explictily in
those places.

Fixes: #54386

Differential Revision: https://reviews.llvm.org/D121931
2022-03-17 13:59:45 -07:00
henry wong 948d05324a [LTO][ELF] Require asserts for --stats-file= tests.
https://reviews.llvm.org/D121809 causes the build bot failure, add the `REQUIRES: asserts` to fix it.

Reviewed By: tejohnson

Differential Revision: https://reviews.llvm.org/D121888
2022-03-17 23:57:13 +08:00
wangliushuai 1c04b52b25 [LTO][ELF] Add --stats-file= option.
This patch adds a StatsFile option supported by gold to lld, related patch https://reviews.llvm.org/D45531.

Reviewed By: tejohnson, MaskRay

Differential Revision: https://reviews.llvm.org/D121809
2022-03-17 12:01:39 +08:00
Jez Ng f5ddcf25d6 [lld-macho] Extend lto-internalize-unnamed-addr.ll
* Test the case where a symbol is sometimes linkonce_odr and sometimes weak_odr
* Test the visibility of the symbols at the IR level, after the internalize
  stage of LTO is done. (Previously we only checked the visibility of
  symbols in the final output binary.)

Reviewed By: modimo

Differential Revision: https://reviews.llvm.org/D121428
2022-03-16 17:30:31 -04:00
Sam McCall 75acad41bc Use lit_config.substitute instead of foo % lit_config.params everywhere
This mechanically applies the same changes from D121427 everywhere.

Differential Revision: https://reviews.llvm.org/D121746
2022-03-16 09:57:41 +01:00
serge-sans-paille 989f1c72e0 Cleanup codegen includes
This is a (fixed) recommit of https://reviews.llvm.org/D121169

after:  1061034926
before: 1063332844

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D121681
2022-03-16 08:43:00 +01:00
Fangrui Song c9dbf407af [ELF] Move invalid binding diagnostic from initializeSymbols to postParse
It is excessive to have a diagnostic for STB_LOCAL. Just reuse the invalid
binding diagnostic for STB_LOCAL.
2022-03-16 00:31:29 -07:00
Fangrui Song bdb98bd979 [ELF] Use endianness-aware read32 to avoid dispatch. NFC 2022-03-15 23:51:11 -07:00
Fangrui Song 385573e07b [ELF] Inline ARMExidxSyntheticSection::classof. NFC
To optimize the only call site `dyn_cast<ARMExidxSyntheticSection>(first)` and
decrease code size.
2022-03-15 23:41:30 -07:00
Fangrui Song 1a590232f4 [ELF] Optimize "Strip sections"
If SHT_LLVM_SYMPART is unused, don't iterate over inputSections.
If neither --strip-debug/--strip-all, don't iterate over inputSections.
2022-03-15 23:15:43 -07:00
Fangrui Song 7c7702b318 [ELF] Move section assignment from initializeSymbols to postParse
https://discourse.llvm.org/t/parallel-input-file-parsing/60164

initializeSymbols currently sets Defined::section and handles non-prevailing
COMDAT groups. Move the code to the parallel postParse to reduce work from the
single-threading code path and make parallel section initialization infeasible.

Postpone reporting duplicate symbol errors so that the messages have the
section information. (`Defined::section` is assigned in postParse and another
thread may not have the information).

* duplicated-synthetic-sym.s: BinaryFile duplicate definition (very rare) now
  has no section information
* comdat-binding: `%t/w.o %t/g.o` leads to an undesired undefined symbol. This
  is not ideal but we report a diagnostic to inform that this is unsupported.
  (See release note)
* comdat-discarded-lazy.s: %tdef.o is unextracted. The new behavior (discarded
  section error) makes more sense
* i386-comdat.s: switched to a better approach working around
  .gnu.linkonce.t.__x86.get_pc_thunk.bx in glibc<2.32 for x86-32.
  Drop the ancient no-longer-relevant workaround for __i686.get_pc_thunk.bx

Depends on D120640

Differential Revision: https://reviews.llvm.org/D120626
2022-03-15 19:24:41 -07:00
Fangrui Song 9b61fff0eb Revert D120626 "[ELF] Move section assignment from initializeSymbols to postParse"
This reverts commit c30e6447c0.
It exposed brittle support for __x86.get_pc_thunk.bx.
Need to think a bit how to support __x86.get_pc_thunk.bx.
2022-03-15 19:00:54 -07:00
Fangrui Song 48a02152ab [ELF][test] Improve i386-linkonce.s
Make it behave like the glibc<2.32 .gnu.linkonce usage that we want to work around.
2022-03-15 18:47:52 -07:00
Sam Clegg 4690bf2ed3 [lld][WebAssembly] Take advantage of extended const expressions when available
In particular we use these in two places:

1. When building PIC code we no longer need to combine output segments
   into a single segment that can be initialized at `__memory_base`.
   Instead each segment can encode its offset from `__memory_base` in
   its initializer.  e.g.

```
(i32.add (global.get __memory_base) (i32.const offset)
```

2. When building PIC code we no longer need to relocation internalized
   global addresses.  We can just initialize them with their correct
   offsets.

Differential Revision: https://reviews.llvm.org/D121420
2022-03-15 17:50:05 -07:00
Jez Ng 8ce3750ff6 [lld-macho] Set FinalDefinitionInLinkageUnit on most LTO externs
Since Mach-O has a two-level namespace (unlike ELF), we can usually set
this property to true.

(I believe this setting is only available in the new LTO backend, so I
can't really use ld64 / libLTO's behavior as a reference here... I'm
just doing what I think is correct.)

See {D119294} for the work done to calculate the `interposable` used in
this diff.

Reviewed By: MaskRay

Differential Revision: https://reviews.llvm.org/D119506
2022-03-15 20:25:06 -04:00
Fangrui Song c1d4c67718 [ELF] Suppress duplicate symbol error for __x86.get_pc_thunk.bx 2022-03-15 17:20:29 -07:00
Sam Clegg 86c90f9bfd [lld][WebAssembly] Add --unresolved-symbols=import-dynamic
This is a new mode for handling unresolved symbols that allows all
symbols to be imported in the same that they would be in the case of
`-fpie` or `-shared`, but generting an otherwise fixed/non-relocatable
binary.

Code linked in this way should still be compiled with `-fPIC` so that
data symbols can be resolved via imports.

This essentially allows the building of static binaries that have
dynamic imports.  See:
https://github.com/emscripten-core/emscripten/issues/12682

As with other uses of the experimental dynamic linking ABI, this
behaviour will produce a warning unless run with `--experimental-pic`.

Differential Revision: https://reviews.llvm.org/D91577
2022-03-15 15:10:21 -07:00
Fangrui Song 6be457c14d [ELF] Work around not-fully-supported .gnu.linkonce.t.__x86.get_pc_thunk.bx 2022-03-15 14:48:29 -07:00
Jez Ng ceff23c6e3 [lld-macho] -flat_namespace for dylibs should make all externs interposable
All references to interposable symbols can be redirected at runtime to
point to a different symbol definition (with the same name). For
example, if both dylib A and B define symbol _foo, and we load A before
B at runtime, then all references to _foo within dylib B will point to
the definition in dylib A.

ld64 makes all extern symbols interposable when linking with
`-flat_namespace`.

TODO 1: Support `-interposable` and `-interposable_list`, which should
just be a matter of parsing those CLI flags and setting the
`Defined::interposable` bit.

TODO 2: Set Reloc::FinalDefinitionInLinkageUnit correctly with this info
(we are currently not setting it at all, so we're erring on the
conservative side, but we should help the LTO backend generate more
optimal code.)

Reviewed By: modimo, MaskRay

Differential Revision: https://reviews.llvm.org/D119294
2022-03-14 22:18:32 -04:00
Jez Ng 7f3ddf8443 [lld-macho][nfc] Allow Defined symbols to be placed in binding sections
Previously, we only allowed this for DylibSymbols. However, in order to
properly support `-flat_namespace` as well as `-interposable`, we need
to allow this for Defined symbols too. Therefore we hoist the
`lazyBindOffset` and the `stubsHelperIndex` into the parent Symbol
class.

The actual change to support interposition under `-flat_namespace` is in
{D119294}; the NFC changes here have been split out for easier review.

Perf regression isn't stat sig on my 3.2 GHz 16-Core Intel Xeon W linking
chromium_framework:

             base           diff           difference (95% CI)
  sys_time   1.227 ± 0.021  1.234 ± 0.031  [  -0.3% ..   +1.5%]
  user_time  3.665 ± 0.036  3.674 ± 0.035  [  -0.2% ..   +0.7%]
  wall_time  4.596 ± 0.055  4.609 ± 0.064  [  -0.3% ..   +0.9%]
  samples    34             47

Max RSS regression is barely stat sig:

           base                           diff                           difference (95% CI)
  time     1003664356.324 ± 15404053.912  1010380403.613 ± 10578309.455  [  +0.0% ..   +1.3%]
  samples  37                             31

Reviewed By: modimo

Differential Revision: https://reviews.llvm.org/D121351
2022-03-14 22:18:32 -04:00
Vy Nguyen 0d5e27623a Reland "[lld-macho] Avoid using bump-alloc in TrieBuider""
This reverts commit ee7a286cd3.
2022-03-14 19:33:13 -04:00
Sterling Augustine ee7a286cd3 Revert "[lld-macho] Avoid using bump-alloc in TrieBuider"
This reverts commit e049a87f04.

That commit breaks the build with errors of the form:

/usr/local/google/home/saugustine/llvm/llvm-project/lld/MachO/ExportTrie.cpp:148:11: error: definition of implicitly declared destructor
TrieNode::~TrieNode() {
2022-03-14 15:23:04 -07:00
Vy Nguyen e049a87f04 [lld-macho] Avoid using bump-alloc in TrieBuider
The code can be used in multi-threads and the allocator is not thread safe.

fixes PR/54378

Reviewed By: int3, #lld-macho

Differential Revision: https://reviews.llvm.org/D121638
2022-03-14 17:22:53 -04:00
Fangrui Song c30e6447c0 [ELF] Move section assignment from initializeSymbols to postParse
https://discourse.llvm.org/t/parallel-input-file-parsing/60164

initializeSymbols currently sets Defined::section and handles non-prevailing
COMDAT groups. Move the code to the parallel postParse to reduce work from the
single-threading code path and make parallel section initialization infeasible.

Postpone reporting duplicate symbol errors so that the messages have the
section information. (`Defined::section` is assigned in postParse and another
thread may not have the information).

* duplicated-synthetic-sym.s: BinaryFile duplicate definition (very rare) now
  has no section information
* comdat-binding: `%t/w.o %t/g.o` leads to an undesired undefined symbol. This
  is not ideal but we report a diagnostic to inform that this is unsupported.
  (See release note)
* comdat-discarded-lazy.s: %tdef.o is unextracted. The new behavior (discarded
  section error) makes more sense

Depends on D120640

Reviewed By: peter.smith

Differential Revision: https://reviews.llvm.org/D120626
2022-03-14 14:13:41 -07:00
Fangrui Song c7cf960d85 [ELF] Set the priority of STB_GNU_UNIQUE the same as STB_WEAK
In GCC -fgnu-unique output, STB_GNU_UNIQUE symbols are always defined relative
to a section in a COMDAT group. Currently `other` cannot be STB_GNU_UNIQUE for
valid input, so this patch is NFC.

If we switch to the model that ignores COMDAT resolution when performing symbol
resolution (D120626), this will fix bogus `relocation refers to a symbol in a
discarded section` errors when mixing -fno-gnu-unique objects with -fgnu-unique
objects.

Differential Revision: https://reviews.llvm.org/D120640
2022-03-14 12:00:15 -07:00
Sam Clegg 9504ab32b7 [WebAssembly] Second phase of implemented extended const proposal
This change continues to lay the ground work for supporting extended
const expressions in the linker.

The included test covers object file reading and writing and the YAML
representation.

Differential Revision: https://reviews.llvm.org/D121349
2022-03-14 08:55:47 -07:00
Nico Weber 17414150cf [lld-link] Tweak winsysroottest.test to have passing links on happy path
Previously, the test checked for a "undefined symbol" error
(instead of the "could not open std*.lib" which would happen without
the flag).

Instead, use /entry: so that the link succeeds.

No behavior change, but maybe makes the test a bit easier to understand.

Differential Revision: https://reviews.llvm.org/D121553
2022-03-14 10:44:26 -04:00
Fangrui Song 7b8fbb796c [ELF] Simplify addCopyRelSymbol with invokeELFT. NFC 2022-03-12 14:08:10 -08:00
Petr Hosek 0c0f6cfb7b [CMake] Rename TARGET_TRIPLE to LLVM_TARGET_TRIPLE
This clarifies that this is an LLVM specific variable and avoids
potential conflicts with other projects.

Differential Revision: https://reviews.llvm.org/D119918
2022-03-11 15:43:01 -08:00
Jez Ng 9b7b21d2f7 [lld-macho] Don't allocate memory in parallelForEach
... since BumpPtrAllocator isn't thread-safe.

Reviewed By: #lld-macho, Roger

Differential Revision: https://reviews.llvm.org/D121458
2022-03-11 13:32:24 -05:00
Fangrui Song 4a8de2832a [ELF] Add -z pack-relative-relocs
GNU ld 2.38 added -z pack-relative-relocs which is similar to
--pack-dyn-relocs=relr but synthesizes the `GLIBC_ABI_DT_RELR` version
dependency if a shared object named `libc.so.*` has a `GLIBC_2.*` version
dependency.

This is used to implement the (as some glibc folks call) version lockout
mechanism. Add this option, because glibc does not want to support
--pack-dyn-relocs=relr which does not add `GLIBC_ABI_DT_RELR`.
See https://maskray.me/blog/2021-10-31-relative-relocations-and-relr for
detail.

Close https://github.com/llvm/llvm-project/issues/53775

Reviewed By: peter.smith

Differential Revision: https://reviews.llvm.org/D120701
2022-03-10 19:54:21 -08:00
Jez Ng fc968bcba4
[lld-macho][nfc] Fix formatting in ld64-vs-lld.rst 2022-03-10 18:33:18 -05:00
Jez Ng 4308f031cd [lld-macho] Align cstrings less conservatively
Previously, we aligned every cstring to 16 bytes as a temporary hack to
deal with https://github.com/llvm/llvm-project/issues/50135. However, it
was highly wasteful in terms of binary size.

To recap, in contrast to ELF, which puts strings that need different
alignments into different sections, `clang`'s Mach-O backend puts them
all in one section.  Strings that need to be aligned have the .p2align
directive emitted before them, which simply translates into zero padding
in the object file. In other words, we have to infer the alignment of
the cstrings from their addresses.

We differ slightly from ld64 in how we've chosen to align these
cstrings. Both LLD and ld64 preserve the number of trailing zeros in
each cstring's address in the input object files. When deduplicating
identical cstrings, both linkers pick the cstring whose address has more
trailing zeros, and preserve the alignment of that address in the final
binary. However, ld64 goes a step further and also preserves the offset
of the cstring from the last section-aligned address.  I.e. if a cstring
is at offset 18 in the input, with a section alignment of 16, then both
LLD and ld64 will ensure the final address is 2-byte aligned (since
`18 == 16 + 2`). But ld64 will also ensure that the final address is of
the form 16 * k + 2 for some k (which implies 2-byte alignment).

Note that ld64's heuristic means that a dedup'ed cstring's final address is
dependent on the order of the input object files. E.g. if in addition to the
cstring at offset 18 above, we have a duplicate one in another file with a
`.cstring` section alignment of 2 and an offset of zero, then ld64 will pick
the cstring from the object file earlier on the command line (since both have
the same number of trailing zeros in their address). So the final cstring may
either be at some address `16 * k + 2` or at some address `2 * k`.

I've opted not to follow this behavior primarily for implementation
simplicity, and secondarily to save a few more bytes. It's not clear to me
that preserving the section alignment + offset is ever necessary, and there
are many cases that are clearly redundant. In particular, if an x86_64 object
file contains some strings that are accessed via SIMD instructions, then the
.cstring section in the object file will be 16-byte-aligned (since SIMD
requires its operand addresses to be 16-byte aligned). However, there will
typically also be other cstrings in the same file that aren't used via SIMD
and don't need this alignment. They will be emitted at some arbitrary address
`A`, but ld64 will treat them as being 16-byte aligned with an offset of
`16 % A`.

I have verified that the two repros in https://github.com/llvm/llvm-project/issues/50135
work well with the new alignment behavior.

Fixes https://github.com/llvm/llvm-project/issues/54036.

Reviewed By: #lld-macho, oontvoo

Differential Revision: https://reviews.llvm.org/D121342
2022-03-10 15:18:15 -05:00
serge-sans-paille f06d487dd6 Cleanup includes: WindowsDriver & WindowsManifest
Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D121330
2022-03-10 17:19:06 +01:00
Nico Weber a278250b0f Revert "Cleanup codegen includes"
This reverts commit 7f230feeea.
Breaks CodeGenCUDA/link-device-bitcode.cu in check-clang,
and many LLVM tests, see comments on https://reviews.llvm.org/D121169
2022-03-10 07:59:22 -05:00
serge-sans-paille 7f230feeea Cleanup codegen includes
after:  1061034926
before: 1063332844

Differential Revision: https://reviews.llvm.org/D121169
2022-03-10 10:00:30 +01:00
Fangrui Song 72bedf46c7 [ELF] Inline InputSection::getParent. NFC
Combined with the previous change, lld executable is ~2K smaller and some code
paths using InputSection::getParent are more efficient.

The fragmented headers lead to a design limitation that OutputSection has to be
incomplete, so we cannot use static_cast.
2022-03-08 11:26:12 -08:00
Fangrui Song 6c814931bc [ELF] Don't use multiple inheritance for OutputSection. NFC
Add an OutputDesc class inheriting from SectionCommand. An OutputDesc wraps an
OutputSection. This change allows InputSection::getParent to be inlined.

Differential Revision: https://reviews.llvm.org/D120650
2022-03-08 11:23:42 -08:00
Jez Ng ce2ae38124 [lld-macho] Deduplicate the `__objc_classrefs` section contents
ld64 breaks down `__objc_classrefs` on a per-word level and deduplicates
them. This greatly reduces the number of bind entries emitted (and
therefore the amount of work `dyld` has to do at runtime). For
chromium_framework, this change to LLD cuts the number of (non-lazy)
binds from 912 to 190, getting us to parity with ld64 in this aspect.

Reviewed By: #lld-macho, thakis

Differential Revision: https://reviews.llvm.org/D121053
2022-03-08 08:34:04 -05:00
Jez Ng 8ec1033933 [lld-macho] Deduplicate CFStrings during ICF
`__cfstring` has embedded addends that foil ICF's hashing / equality
checks. (We can ignore embedded addends when doing ICF because the same
information gets recorded in our Reloc structs.) Therefore, in order to
properly dedup CFStrings, we create a mutable copy of the CFString and
zero out the embedded addends before performing any hashing / equality
checks.

(We did in fact have a partial implementation of CFString deduplication
already. However, it only worked when the cstrings they point to are at
identical offsets in their object files.)

I anticipate this approach can be extended to other similar
statically-allocated struct sections in the future.

In addition, we previously treated all references with differing addends
as unequal. This is not true when the references are to literals:
different addends may point to the same literal in the output binary. In
particular, `__cfstring` has such references to `__cstring`. I've
adjusted ICF's `equalsConstant` logic accordingly, and I've added a few
more tests to make sure the addend-comparison code path is adequately
covered.

Fixes https://github.com/llvm/llvm-project/issues/51281.

Reviewed By: #lld-macho, Roger

Differential Revision: https://reviews.llvm.org/D120137
2022-03-08 08:34:03 -05:00
Jez Ng 0405920c5f Re-land [lld-macho][nfc] Don't use `stubsHelperIndex` in ICF hash
Previous attempt was commit 112135e774 and
reverted in d86d431814.
2022-03-07 16:58:00 -05:00
Nico Weber d86d431814 Revert "[lld-macho][nfc] Don't use `stubsHelperIndex` in ICF hash"
This reverts commit 112135e774.
Breaks lld/test/MachO/{icf.s,cfstring-dedup.s,invalid/cfstring.s}
2022-03-07 13:50:38 -05:00
Jez Ng ad1c32e9b3 [lld-macho][nfc] Reduce size of icfEqClass hash
... from a `uint64_t` to a `uint32_t`. (LLD-ELF uses a `uint32_t` too.)

About a 1.7% reduction in peak RSS when linking chromium_framework on my
3.2 GHz 16-Core Intel Xeon W Mac Pro, and no stat sig change in wall
time.

           </Users/jezng/test2.sh ["before"]>  </Users/jezng/test2.sh ["after"]>  difference (95% CI)
  RSS      1003036672.000 ± 9891065.259        985539505.231 ± 10272748.749       [  -2.3% ..   -1.2%]
  samples  27                                  26

             base           diff           difference (95% CI)
  sys_time   1.277 ± 0.023  1.277 ± 0.024  [  -0.9% ..   +0.9%]
  user_time  6.682 ± 0.046  6.598 ± 0.043  [  -1.6% ..   -0.9%]
  wall_time  5.904 ± 0.062  5.895 ± 0.063  [  -0.7% ..   +0.4%]
  samples    46             28

No appreciable change (~0.01%) in number of `equals` comparisons either:

Before:

  ld64.lld: ICF needed 8 iterations
  ld64.lld: equalsConstant() called 701643 times
  ld64.lld: equalsVariable() called 3438526 times

After:

  ld64.lld: ICF needed 8 iterations
  ld64.lld: equalsConstant() called 701729 times
  ld64.lld: equalsVariable() called 3438526 times

Reviewed By: #lld-macho, MaskRay, thakis

Differential Revision: https://reviews.llvm.org/D121052
2022-03-07 12:36:28 -05:00
Jez Ng 112135e774 [lld-macho][nfc] Don't use `stubsHelperIndex` in ICF hash
The existing hashing of stubsHelperIndex has mostly been a no-op* for
some time now (ever since we made ICF run before dylib symbols get their
stubs indices assigned). I guess we could consider hashing the name +
filename of the DylibSymbol instead, but I'm not sure the overhead's
worth it... moreover, LLD/ELF only hashes their Defined symbols as well.

*: Technically it does change the hash value since stubsHelperIndex is
initialized to `UINT32_MAX` by default. But since all stubsHelperIndex
values are the same at when ICF runs, they don't add any useful
information to the hash.
2022-03-07 12:36:28 -05:00
Jez Ng 7028799ca3 [lld-macho][nfc] Rename isec -> referentIsec to avoid shadowing
I found the shadowing a bit confusing
2022-03-07 12:36:28 -05:00
Jez Ng 64cc719766 [lld-macho][nfc] Track # of ICF calls to `equals*` methods
This is debug code that is disabled by default. It'll provide a easy way
to figure out the impact (if any) of tweaking ICF's hashing algorithm
(since a poor quality hash will result in many more `equals*` calls).

Reviewed By: #lld-macho, oontvoo

Differential Revision: https://reviews.llvm.org/D121051
2022-03-07 12:36:27 -05:00
Jez Ng 53e7eef43f [lld-macho][nfc] Use llvm::function_ref instead of std::function 2022-03-07 12:36:27 -05:00
Jez Ng c416f3fafd [lld-macho][nfc] Remove file statics from ICF.cpp
This gets us closer to the [LLD-as-a-library goal][1].

[1]: https://lists.llvm.org/pipermail/llvm-dev/2021-June/151184.html

Reviewed By: #lld-macho, thakis

Differential Revision: https://reviews.llvm.org/D121050
2022-03-07 12:36:26 -05:00
Fangrui Song a815424cc5 Reland D119909 [ELF] Parallelize initializeLocalSymbols
ObjFile::parse combines symbol initialization and resolution. Many tasks
unrelated to symbol resolution can be postponed and parallelized. This patch
extracts local symbol initialization and parallelizes it.

Technically the new function initializeLocalSymbols can be merged into
ObjFile::postParse, but functions like getSrcMsg may access the
uninitialized (all nullptr) local part of InputFile::symbols.

Linking chrome: 1.02x as fast with glibc malloc, 1.04x as fast with mimalloc

Depends on f456c3ae3f and D119908

Reviewed By: ikudrin

Differential Revision: https://reviews.llvm.org/D119909
2022-03-04 19:00:10 -08:00
Fangrui Song f456c3ae3f [ELF] Move addWrappedSymbols before postParseObjectFile
addWrappedSymbols may trigger archive extraction: split stack implementation
uses --wrap=pthread_create, which extracts libgcc.a(generic-morestack-thread.o).

This fixes the regression caused by 09602d3b47 by
making the invariant satisfied: no more non-compileBitcodeFiles object file is
produced at postParseObjectFile.
2022-03-04 18:56:37 -08:00
Jorge Gorbe Moya 449b649fec Revert "[ELF] Parallelize initializeLocalSymbols"
This reverts commit 09602d3b47.
2022-03-04 15:01:17 -08:00
Jez Ng 72c5b26f3d [lld-macho][nfc] Use %X in mapfile test
LLD (and ld64) emits uppercase hex addresses in the mapfile. The
map-file.s test passes right now because the addresses we emit happen
not to include any alphabets, but that can easily change.

I noticed this while dealing with
https://github.com/llvm/llvm-project/issues/54184.

Reviewed By: #lld-macho, thakis

Differential Revision: https://reviews.llvm.org/D120941
2022-03-04 14:21:17 -05:00
Jez Ng 984197612c [lld-macho][nfc] Rename some tests for consistency
Now all the tests that cover symbol resolution / precedence have
"resolution" in their filename.

I also added a couple of extra comments.

Reviewed By: #lld-macho, thakis

Differential Revision: https://reviews.llvm.org/D120938
2022-03-04 14:21:16 -05:00
Jez Ng 070af48d13 [lld-macho][nfc] Decouple tapi-link.s test from libSystem
If we fix https://github.com/llvm/llvm-project/issues/54184, we will end
up including libSystem in every %lld invocation, which would break
tapi-link.s as it assumes that libSystem isn't directly linked (instead
it goes through libReexportSystem).

Let's remove this unnecessary coupling, as well as use `split-file`
instead of having a separate file under `Inputs`.

Reviewed By: #lld-macho, oontvoo

Differential Revision: https://reviews.llvm.org/D120939
2022-03-03 19:48:59 -05:00
Jez Ng dd29597e10 [LTO] Initialize canAutoHide() using canBeOmittedFromSymbolTable()
Per discussion on
https://reviews.llvm.org/D59709#inline-1148734, this seems like the
right course of action. `canBeOmittedFromSymbolTable()` subsumes and
generalizes the previous logic. In addition to handling `linkonce_odr`
`unnamed_addr` globals, we now also internalize `linkonce_odr` +
`local_unnamed_addr` constants.

Reviewed By: tejohnson

Differential Revision: https://reviews.llvm.org/D120173
2022-03-03 19:04:11 -05:00
Jez Ng 5c268743da [lld-macho][nfc] Use %lld-watchos substitution in bind-opcodes.s
Previously, we were using a syslibroot that pointed to macos while
linking against arch arm64_32, which didn't really make sense. It isn't
currently an issue, but will be if we add the `-lSystem` as part of
dealing with https://github.com/llvm/llvm-project/issues/54184.
2022-03-03 19:00:28 -05:00
Jez Ng f7547558c9 [lld-macho][nfc] Avoid using absolute addresses in cgprofile-icf.s
If we fix https://github.com/llvm/llvm-project/issues/54184, the
`dyld_stub_binder` symbol will get included in every output dylib. This
would cause the addresses of the other symbols to shift, breaking the
test as it currently stands. Let's make the test more flexible.

Reviewed By: lgrey

Differential Revision: https://reviews.llvm.org/D120940
2022-03-03 19:00:28 -05:00
Martin Storsjö 4c3b74b7f5 [LLD] [COFF] Order .debug_* sections at the end, to avoid leaving gaps if stripped
So far, we sort all discardable sections at the end, with only some
extra logic to make sure that the .reloc section is at the start
of that group of sections. But if there are other discardable
sections, other than .reloc, they must also be ordered before
.debug_* sections, to avoid leaving gaps if the executable is
stripped.

(Stripping executables doesn't remove all discardable sections,
only the ones named .debug_*).

Rust binaries seem to include a .rmeta section, which is marked
discardable. This fixes stripping such binaries if built with
dwarf debug info included.

This fixes issues observed in MSYS2 in
https://github.com/msys2/MINGW-packages/pull/10555.

Differential Revision: https://reviews.llvm.org/D120805
2022-03-03 10:08:51 +02:00
Douglas Yung e81e5d788c Add "REQUIRES: x86" to test as it calls llc with an x86_64 triple. 2022-03-02 11:12:41 -08:00
Sam Clegg 1cf6ebc0e9 [lld][WebAssembly] Improve error reporting for bad ar archive members
Show the name of of the archive in the error message as well as the name
of the object within it.

Differential Revision: https://reviews.llvm.org/D120689
2022-03-01 15:21:53 -08:00
Zequan Wu 5c9e20d7d0 [PDB] Add char8_t type
Differential Revision: https://reviews.llvm.org/D120690
2022-03-01 13:39:51 -08:00
Martin Storsjö 9ffeaaa0ea [LLD] [COFF] Use StringTableBuilder to optimize the string table
This does tail merging (and deduplication) of the strings.

On a statically linked clang.exe, this shrinks the ~17 MB string
table by around 0.5 MB. This adds ~160 ms to the linking time
which originally was around 950 ms.

For cases where `-debug:symtab` or `-debug:dwarf` isn't set, the
string table is only used for long section names, where this
shouldn't make any difference at all.

Differential Revision: https://reviews.llvm.org/D120677
2022-03-01 18:44:03 +02:00
Martin Storsjö 9dd2d50984 [LLD] [COFF] Use the new encodeSectionName() helper for long section names
The previous code used an unbounded sprintf, which in theory can
overflow, writing either the null terminator or the last digits
into the next struct member.

In practice, in LLD, all long section names are written sequentially
first at the start of the string table, followed by all the long
symbol names. Due to this, even if the total string table would
end up large, the long section names have fairly short offsets,
which is why this hasn't been an issue in practice.

I don't think it's worth trying to write a test that produces an
executable with enough long section names to make the section names
themselves exceed 10^6 bytes, which is currently necessary to trigger
faults with the previous form.

Differential Revision: https://reviews.llvm.org/D120676
2022-03-01 11:33:02 +02:00
Fangrui Song 87034ad2a4 [ELF] isKnownZFlag: move known literal flags to an array. NFC
The chain of == comparisons is a bit unwieldy to update.

While here, sort the entries alphabetically.
2022-02-28 23:23:33 -08:00
Jez Ng a552fb2a86 [lld-macho] Have relocation address included in range-check error message
This makes it easier to debug those errors. See e.g. https://github.com/llvm/llvm-project/issues/52767#issuecomment-1028713943

We take the approach of 'reverse-engineering' the InputSection from the
output buffer offset. This provides for a cleaner Target API, and is
similar to LLD-ELF's implementation of getErrorPlace().

Reviewed By: #lld-macho, Roger

Differential Revision: https://reviews.llvm.org/D118903
2022-02-28 21:56:38 -05:00
Fangrui Song 9e9c86fd67 [ELF] Change some non-null pointer parameters to references. NFC
To decrease difference for D120650. Also, rename some `OutputSection *sec` (and
`cmd`) to the more common `osec`.
2022-02-28 11:19:00 -08:00
Fangrui Song b07ef4d566 [ELF] Rename Symbol::compare to shouldReplace. NFC
The return value is not a boolean instead of a tri-state.
Suggested by Peter Smith in D120640.
2022-02-28 18:25:21 +00:00
Fangrui Song 8d01ac75e7 [ELF] Replace an unneeded dyn_cast_or_null with dyn_cast. NFC 2022-02-28 00:50:06 -08:00
Fangrui Song fee78961f5 [ELF] Optimize SectionBase::Kind values to make isa<InputSection> more efficient. NFC
Surprisingly my lld executable is 1.5KiB smaller.
2022-02-28 00:24:25 -08:00
Fangrui Song bb3eeac773 [ELF] Make InputSection::classof inline. NFC 2022-02-28 00:16:45 -08:00
Fangrui Song 4976d1fe58 [ELF] Move SyntheticSection check from InputSection::writeTo to OutputSection::writeTo. NFC
Simplify code and make the heavyweight operation to the call site so that it is
clearer how to improve the inefficient scheduling in the future.
2022-02-27 23:28:52 -08:00
Fangrui Song d07ff99591 [ELF] Enforce double-dash form --error-limit
It's ld.lld specific and by convention we enforce the double-dash form to avoid
collision with the short option -e (--entry).
2022-02-27 20:49:36 +00:00