Commit Graph

554 Commits

Author SHA1 Message Date
Vitaly Buka b01bfdfda6 [lld][MachO] Fix UB after D103006
ubsan detected:
lld/MachO/SyntheticSections.cpp:636:15: runtime error: null pointer
passed as argument 2, which is declared to never be null
2021-06-14 21:15:54 -07:00
Alexander Shaposhnikov 928394d109 [lld][MachO] Add support for LC_DATA_IN_CODE
Add first bits for emitting LC_DATA_IN_CODE.

Test plan: make check-lld-macho

Differential revision: https://reviews.llvm.org/D103006
2021-06-14 19:21:59 -07:00
Jez Ng cc17bfe489 [lld-macho] Fix "shift exponent too large" UBSAN error
UBSAN seems to have added this check somewhere along the way...

This might also fix the PPC buildbot, which is failing on the same test
2021-06-14 13:47:25 -04:00
Jez Ng e06b9ba485 [lld-macho] Reword comment for clarity 2021-06-14 13:47:25 -04:00
Jez Ng 9c5d43fb55 [lld-macho] Try to fix MSAN "uninitialized memory" error
I *think* this is the fix, with the regression being introduced by
D104199. Not 100% sure since MSAN isn't supported on my Mac machine, and
it'll take some time to spin up a Linux box... will look at the
buildbots for answers
2021-06-13 23:47:09 -04:00
Jez Ng da24e6d43e [lld-macho][nfc] Add `final` to classes where possible
I wanted to see if we would get any perf wins out of this, but
it doesn't seem to be the case. But it still seems worth committing.

Reviewed By: MaskRay

Differential Revision: https://reviews.llvm.org/D104200
2021-06-13 19:52:03 -04:00
Jez Ng c5c05ffa45 [lld-macho][nfc] Represent the image loader cache with a ConcatInputSection
We don't need to define any special behavior for this section,
so creating a subclass for it is redundant.

Reviewed By: #lld-macho, thakis

Differential Revision: https://reviews.llvm.org/D104199
2021-06-13 19:51:31 -04:00
Jez Ng b2a0739012 [lld-macho][nfc] Remove InputSection::outSecFileOff
`outSecFileOff` and the associated `getFileOffset()` accessors were
unnecessary.

For all the cases we care about, `outSecFileOff` is the same as
`outSecOff`. The only time they deviate is if there are zerofill
sections within a given segment. But since zerofill sections are always
at the end of a segment, the only sections where the two values deviate
are zerofill sections themselves. And we never actually query the
outSecFileOff of zerofill sections.

As for `getFileOffset()`, the only place it was being used was to
calculate the offset of the entry symbol. However, we can compute that
value by just taking the difference between the address of the entry
symbol and the address of the Mach-O header. In fact, this appears to be
what ld64 itself does. This difference is the same as the file offset as
long as there are no intervening zerofill sections, but since `__text`
is the first section in `__TEXT`, this never happens, so our previous
use of `getFileOffset()` was not wrong -- just inefficient.

Reviewed By: #lld-macho, thakis

Differential Revision: https://reviews.llvm.org/D104177
2021-06-13 19:51:30 -04:00
Nico Weber 7d4c8a2b8f [lld/mac] clarify comment
This is a "we should do X in the future" fixme, not an "X might go wrong"
fixme.
2021-06-13 13:30:07 -04:00
Nico Weber 5f9bc580d8 fix comment typos to cycle bots 2021-06-13 10:18:51 -04:00
Alexander Shaposhnikov b9095f5e1a [lld][MachO] Fix function starts section
Sort the addresses stored in FunctionStarts section.
Previously we were encoding potentially large numbers (due to unsigned overflow).

Test plan: make check-all

Differential revision: https://reviews.llvm.org/D103662
2021-06-11 17:47:28 -07:00
Jez Ng 5de7467e98 [lld-macho] Fix debug build
D103977 broke a bunch of stuff as I had only tested the release build
which eliminated asserts.

I've retained the asserts where possible, but I also removed a bunch
instead of adding a whole lot of verbose ConcatInputSection casts.
2021-06-11 20:21:27 -04:00
Jez Ng 464d3dc3d1 [lld-macho] Have dead-stripping work with literal sections
Literal sections are not atomically live or dead. Rather,
liveness is tracked for each individual literal they contain. CStrings
have their liveness tracked via a `live` bit in StringPiece, and
fixed-width literals have theirs tracked via a BitVector.

The live-marking code now needs to track the offset within each section
that is to be marked live, in order to identify the literal at that
particular offset.

Numbers for linking chromium_framework on my 3.2 GHz 16-Core Intel Xeon W
with both `-dead_strip` and `--deduplicate-literals`, with and without this diff
applied:

```
    N           Min           Max        Median           Avg        Stddev
x  20          4.32          4.44         4.375         4.372    0.03105174
+  20           4.3          4.39          4.36        4.3595   0.023277502
No difference proven at 95.0% confidence
```
This gives us size savings of about 0.4%.

Reviewed By: #lld-macho, thakis

Differential Revision: https://reviews.llvm.org/D103979
2021-06-11 19:50:09 -04:00
Jez Ng 681cfeb591 [lld-macho][nfc] Have InputSection ctors take some parameters
This is motivated by an upcoming diff in which the
WordLiteralInputSection ctor sets itself up based on the value of its
section flags. As such, it needs to be passed the `flags` value as part
of its ctor parameters, instead of having them assigned after the fact
in `parseSection()`. While refactoring code to make that possible, I
figured it would make sense for the other InputSections to also take
their initial values as ctor parameters.

Reviewed By: #lld-macho, thakis

Differential Revision: https://reviews.llvm.org/D103978
2021-06-11 19:50:09 -04:00
Jez Ng 7f2ba39b16 [lld-macho][nfc] Move liveness-tracking fields into ConcatInputSection
These fields currently live in the parent InputSection class,
but they should be specific to ConcatInputSection, since the other
InputSection classes (that contain literals) aren't atomically live or
dead -- rather their component string/int literals should have
individual liveness states. (An upcoming diff will add liveness bits for
StringPieces and fixed-sized literals.)

I also factored out some asserts for isCoalescedWeak() in MarkLive.cpp.
We now avoid putting coalesced sections in the `inputSections` vector,
so we don't have to check/assert against it everywhere.

Reviewed By: #lld-macho, thakis

Differential Revision: https://reviews.llvm.org/D103977
2021-06-11 19:50:08 -04:00
Jez Ng 5d88f2dd94 [lld-macho] Deduplicate fixed-width literals
Conceptually, the implementation is pretty straightforward: we put each
literal value into a hashtable, and then write out the keys of that
hashtable at the end.

In contrast with ELF, the Mach-O format does not support variable-length
literals that aren't strings. Its literals are either 4, 8, or 16 bytes
in length. LLD-ELF dedups its literals via sorting + uniq'ing, but since
we don't need to worry about overly-long values, we should be able to do
a faster job by just hashing.

That said, the implementation right now is far from optimal, because we
add to those hashtables serially. To parallelize this, we'll need a
basic concurrent hashtable (only needs to support concurrent writes w/o
interleave reads), which shouldn't be to hard to implement, but I'd like
to punt on it for now.

Numbers for linking chromium_framework on my 3.2 GHz 16-Core Intel Xeon W:

      N           Min           Max        Median           Avg        Stddev
  x  20          4.27          4.39         4.315        4.3225   0.033225703
  +  20          4.36          4.82          4.44        4.4845    0.13152846
  Difference at 95.0% confidence
          0.162 +/- 0.0613971
          3.74783% +/- 1.42041%
          (Student's t, pooled s = 0.0959262)

This corresponds to binary size savings of 2MB out of 335MB, or 0.6%.
It's not a great tradeoff as-is, but as mentioned our implementation can
be signficantly optimized, and literal dedup will unlock more
opportunities for ICF to identify identical structures that reference
the same literals.

Reviewed By: #lld-macho, gkm

Differential Revision: https://reviews.llvm.org/D103113
2021-06-11 19:50:08 -04:00
Nico Weber f2b1a1e10c [lld/mac] Use sectionType() more
Not sure sectionType() carries its weight, but while we have it
we should use it consistently.

No behavior change.

Differential Revision: https://reviews.llvm.org/D104027
2021-06-11 11:15:47 -04:00
Nico Weber 54418c5a35 [lld/mac] Make binaries written by lld strippable
Be less clever when writing the indirect symbols in LC_DYSYMTAB:
lld used to make point __stubs and __la_symbol_ptr point at the
same bytes in the indirect symbol table in the __LINKEDIT segment.
That confused strip, so write the same bytes twice and make
__stubs and __la_symbol_ptr point at one copy each, so that they
don't share data. This unconfuses strip, and seems to be what ld64
does too, so hopefully tools are generally more used to this.

This makes the output binaries a bit larger, but not much: 4 bytes
for roughly each called function from a dylib and each weak function.
Chromium Framewoork grows by 6536 bytes, clang-format by a few hundred.

With this, `strip -x Chromium\ Framework` works (244 MB before stripping
to 171 MB after stripping, compared to 236 MB=>164 MB with ld64). Running
strip without `-x` produces the same error message now for lld-linked
Chromium Framework as for when using ld64 as a linker.

`strip clang-format` also works now but didn't previously.

Fixes PR50657.

Differential Revision: https://reviews.llvm.org/D104081
2021-06-11 00:18:03 -04:00
Jez Ng 4b5c6c5c4b [lld-macho][nfc] Fix uninitialized members warning from Coverity
We were always assigning to this member before using it, but just to be
safe...

See https://lists.llvm.org/pipermail/llvm-dev/2021-June/151029.html
2021-06-10 15:09:07 -04:00
Nico Weber e87c095af3 [lld/mac] Print dylib search details with --print-dylib-search or RC_TRACE_DYLIB_SEARCHING
For debugging dylib loading, it's useful to have some insight into what
the linker is doing.

ld64 has the undocumented RC_TRACE_DYLIB_SEARCHING env var
for this printing dylib search candidates.

This adds a flag --print-dylib-search to make lld print the seame information.
It's useful for users, but also for writing tests. The output is formatted
slightly differently than ld64, but we still support RC_TRACE_DYLIB_SEARCHING
to offer at least a compatible way to trigger this.

ld64 has both `-print_statistics` and `-trace_symbol_output` to enable
diagnostics output. I went with "print" since that seems like a more
straightforward name.

Differential Revision: https://reviews.llvm.org/D103985
2021-06-09 22:08:20 -04:00
Nico Weber 0e399eb527 [lld/mac] When handling @loader_path, use realpath() of symlinks
This is important for Frameworks, which are usually symlinks.

ld64 gets this right for @rpath that's replaced with @loader_path, but not for
bare @loader_path -- ld64's code calls realpath() in that case too, but ignores
the result.

ld64 somehow manages to find libbar1.dylib in the test without the
explicit `-rpath` in Foo1. I don't understand why or how. But this
change is a step forward and fixes an immediate problem I'm having,
so let's start with this :)

Differential Revision: https://reviews.llvm.org/D103990
2021-06-09 20:36:07 -04:00
Jez Ng 447dfbe005 [lld-macho] Implement -force_load_swift_libs
It causes libraries whose names start with "swift" to be force-loaded.
Note that unlike the more general `-force_load`, this flag only applies
to libraries specified via LC_LINKER_OPTIONS, and not those passed on
the command-line. This is what ld64 does.

Reviewed By: #lld-macho, thakis

Differential Revision: https://reviews.llvm.org/D103709
2021-06-07 23:48:35 -04:00
Jez Ng 04259cde15 [lld-macho] Implement cstring deduplication
Our implementation draws heavily from LLD-ELF's, which in turn delegates
its string deduplication to llvm-mc's StringTableBuilder. The messiness of
this diff is largely due to the fact that we've previously assumed that
all InputSections get concatenated together to form the output. This is
no longer true with CStringInputSections, which split their contents into
StringPieces. StringPieces are much more lightweight than InputSections,
which is important as we create a lot of them. They may also overlap in
the output, which makes it possible for strings to be tail-merged. In
fact, the initial version of this diff implemented tail merging, but
I've dropped it for reasons I'll explain later.

**Alignment Issues**

Mergeable cstring literals are found under the `__TEXT,__cstring`
section. In contrast to ELF, which puts strings that need different
alignments into different sections, clang's Mach-O backend puts them all
in one section. Strings that need to be aligned have the `.p2align`
directive emitted before them, which simply translates into zero padding
in the object file.

I *think* ld64 extracts the desired per-string alignment from this data
by preserving each string's offset from the last section-aligned
address. I'm not entirely certain since it doesn't seem consistent about
doing this; but perhaps this can be chalked up to cases where ld64 has
to deduplicate strings with different offset/alignment combos -- it
seems to pick one of their alignments to preserve. This doesn't seem
correct in general; we can in fact can induce ld64 to produce a crashing
binary just by linking in an additional object file that only contains
cstrings and no code. See PR50563 for details.

Moreover, this scheme seems rather inefficient: since unaligned and
aligned strings are all put in the same section, which has a single
alignment value, it doesn't seem possible to tell whether a given string
doesn't have any alignment requirements. Preserving offset+alignments
for strings that don't need it is wasteful.

In practice, the crashes seen so far seem to stem from x86_64 SIMD
operations on cstrings. X86_64 requires SIMD accesses to be
16-byte-aligned. So for now, I'm thinking of just aligning all strings
to 16 bytes on x86_64. This is indeed wasteful, but implementation-wise
it's simpler than preserving per-string alignment+offsets. It also
avoids the aforementioned crash after deduplication of
differently-aligned strings. Finally, the overhead is not huge: using
16-byte alignment (vs no alignment) is only a 0.5% size overhead when
linking chromium_framework.

With these alignment requirements, it doesn't make sense to attempt tail
merging -- most strings will not be eligible since their overlaps aren't
likely to start at a 16-byte boundary. Tail-merging (with alignment) for
chromium_framework only improves size by 0.3%.

It's worth noting that LLD-ELF only does tail merging at `-O2`. By
default (at `-O1`), it just deduplicates w/o tail merging. @thakis has
also mentioned that they saw it regress compressed size in some cases
and therefore turned it off. `ld64` does not seem to do tail merging at
all.

**Performance Numbers**

CString deduplication reduces chromium_framework from 250MB to 242MB, or
about a 3.2% reduction.

Numbers for linking chromium_framework on my 3.2 GHz 16-Core Intel Xeon W:

      N           Min           Max        Median           Avg        Stddev
  x  20          3.91          4.03         3.935          3.95   0.034641016
  +  20          3.99          4.14         4.015        4.0365     0.0492336
  Difference at 95.0% confidence
          0.0865 +/- 0.027245
          2.18987% +/- 0.689746%
          (Student's t, pooled s = 0.0425673)

As expected, cstring merging incurs some non-trivial overhead.

When passing `--no-literal-merge`, it seems that performance is the
same, i.e. the refactoring in this diff didn't cost us.

      N           Min           Max        Median           Avg        Stddev
  x  20          3.91          4.03         3.935          3.95   0.034641016
  +  20          3.89          4.02         3.935        3.9435   0.043197831
  No difference proven at 95.0% confidence

Reviewed By: #lld-macho, gkm

Differential Revision: https://reviews.llvm.org/D102964
2021-06-07 23:48:35 -04:00
Nico Weber 17c43c4045 [lld/mac] Add reexports after reexporter to inputFiles
When a library "host"'s reexports change their installName with
`$ld$os10.11$install_name$host`, we used to write a load command for "host" but
write the version numbers of the reexport instead of "host". This fixes that.

I first thought that the rule is to take the version numbers from the library
that originally had that install name (implemented in D103819), but that's not
what ld64 seems to be doing: It takes the version number from the first dylib
with that install name it loads, and it loads the reexporting library before
the reexports. We already did most of that, we just added reexports before the
reexporter. After this change, we add the reexporter before the reexports.

Addresses https://bugs.llvm.org/show_bug.cgi?id=49800#c11 part 1.

(ld64 seems to add reexports after processing _all_ files on the command line,
while we add them right after the reexporter. For the common case of reexport +
$ld$ symbol changing back to the exporter name, this doesn't make a difference,
but you can construct a case where it does. I expect this to not make a
difference in practice though.)

Differential Revision: https://reviews.llvm.org/D103821
2021-06-07 17:04:03 -04:00
Nico Weber c5ffe97988 [lld/mac] Implement support for searching dylibs with @rpath/ in install name
Also adjust a few comments, and move the DylibFile comment talking about
umbrella next to the parameter again.

Differential Revision: https://reviews.llvm.org/D103783
2021-06-07 06:22:52 -04:00
Nico Weber 52489021cf [lld/mac] Implement support for searching dylibs with @loader_path/ in install name
Differential Revision: https://reviews.llvm.org/D103779
2021-06-06 20:19:50 -04:00
Nico Weber a48bd587f7 [lld/mac] Implement support for searching dylibs with @executable_path/ in install name
Differential Revision: https://reviews.llvm.org/D103775
2021-06-06 20:01:50 -04:00
Nico Weber 7def700667 [lld/mac] Rename DylibFile::dylibName to DylibFile::installName
The flag to set it is called `-install_name`, and it's called `installName` in tbd files.

No behavior change.

Differential Revision: https://reviews.llvm.org/D103776
2021-06-06 20:00:35 -04:00
Nico Weber e910437443 [lld/mac] Use fewer magic numbers in magic $ld$ handling code
Also simply a conditional and de-alias a variable.
Minor cleanups, no behavior change.

Differential Revision: https://reviews.llvm.org/D103774
2021-06-06 18:13:16 -04:00
Alexander Shaposhnikov 5e49ee8794 [lld][MachO] Add support for $ld$install_name symbols
This diff adds support for $ld$install_name symbols.

Test plan: make check-lld-macho

Differential revision: https://reviews.llvm.org/D103746
2021-06-05 12:58:59 -07:00
Alexander Shaposhnikov 1309c181a8 [lld][MachO] Add first bits to support special symbols
This diff adds first bits to support special symbols $ld$previous* in LLD.
$ld$* symbols modify properties/behavior of the library
(e.g. its install name, compatibility version or hide/add symbols)
for specific target versions.

Test plan: make check-lld-macho

Differential revision: https://reviews.llvm.org/D103505
2021-06-04 23:32:26 -07:00
Alex Richardson 90344499ae [lld-macho] Fix BUILD_SHARED_LIBS build
ca6751043d added a dependency on XAR (at
least for the shared libs build), so without this change we get the
following linker error:

Undefined symbols for architecture x86_64:
  "_xar_close", referenced from:
      lld::macho::BitcodeBundleSection::finalize() in SyntheticSections.cpp.o

Reviewed By: #lld-macho, int3, thakis

Differential Revision: https://reviews.llvm.org/D100999
2021-06-03 19:58:43 +01:00
Nikita Popov d93b678abb [lld] Add missing includes (NFC)
Fix lld build after 983565a6fe.
2021-06-03 18:55:18 +02:00
Jez Ng 6881f29a36 [lld-macho] Parse re-exports of nested TAPI documents
D103423 neglected to call `parseReexports()` for nested TBD
documents, leading to symbol resolution failures when trying to look up
a symbol nested more than one level deep in a TBD file. This fixes the
regression and adds a test.

It also appears that `umbrella` wasn't being set properly when calling
`parseLoadCommands` -- it's supposed to resolve to `this` if `nullptr`
is passed. I didn't write a failing test case for this but I've made
`umbrella` a member so the previous behavior should be preserved.

Reviewed By: #lld-macho, thakis

Differential Revision: https://reviews.llvm.org/D103586
2021-06-03 12:02:30 -04:00
Nico Weber a5645513db [lld/mac] Implement -dead_strip
Also adds support for live_support sections, no_dead_strip sections,
.no_dead_strip symbols.

Chromium Framework 345MB unstripped -> 250MB stripped
(vs 290MB unstripped -> 236M stripped with ld64).

Doing dead stripping is a bit faster than not, because so much less
data needs to be processed:

    % ministat lld_*
    x lld_nostrip.txt
    + lld_strip.txt
        N           Min           Max        Median           Avg        Stddev
    x  10      3.929414       4.07692     4.0269079     4.0089678   0.044214794
    +  10     3.8129408     3.9025559     3.8670411     3.8642573   0.024779651
    Difference at 95.0% confidence
            -0.144711 +/- 0.0336749
            -3.60967% +/- 0.839989%
            (Student's t, pooled s = 0.0358398)

This interacts with many parts of the linker. I tried to add test coverage
for all added `isLive()` checks, so that some test will fail if any of them
is removed. I checked that the test expectations for the most part match
ld64's behavior (except for live-support-iterations.s, see the comment
in the test). Interacts with:
- debug info
- export tries
- import opcodes
- flags like -exported_symbol(s_list)
- -U / dynamic_lookup
- mod_init_funcs, mod_term_funcs
- weak symbol handling
- unwind info
- stubs
- map files
- -sectcreate
- undefined, dylib, common, defined (both absolute and normal) symbols

It's possible it interacts with more features I didn't think of,
of course.

I also did some manual testing:
- check-llvm check-clang check-lld work with lld with this patch
  as host linker and -dead_strip enabled
- Chromium still starts
- Chromium's base_unittests still pass, including unwind tests

Implemenation-wise, this is InputSection-based, so it'll work for
object files with .subsections_via_symbols (which includes all
object files generated by clang). I first based this on the COFF
implementation, but later realized that things are more similar to ELF.
I think it'd be good to refactor MarkLive.cpp to look more like the ELF
part at some point, but I'd like to get a working state checked in first.

Mechanical parts:
- Rename canOmitFromOutput to wasCoalesced (no behavior change)
  since it really is for weak coalesced symbols
- Add noDeadStrip to Defined, corresponding to N_NO_DEAD_STRIP
  (`.no_dead_strip` in asm)

Fixes PR49276.

Differential Revision: https://reviews.llvm.org/D103324
2021-06-02 11:09:26 -04:00
Nico Weber 66a1ecd2cf [lld/mac] Implement -needed_framework, -needed_library, -needed-l
These allow overriding dead_strip_dylibs.

Differential Revision: https://reviews.llvm.org/D103499
2021-06-02 11:06:42 -04:00
Nico Weber e14fd7d879 [lld/mac] Don't strip explicit dylib also mentioned in LC_LINKER_OPTION
Noticed by Jez in D103499.

Differential Revision: https://reviews.llvm.org/D103521
2021-06-02 10:59:56 -04:00
Nico Weber 476e4d65d4 [lld/mac] Address review feedback and improve a comment
I forgot to move the message() call around as requested in D103428
before committing that change. Move it now.

Also, improve the ordinal uniq'ing comment. I hadn't realized that the
distinct-but-identical files happen with --reproduce and not in general.

No behavior change.

Differential Revision: https://reviews.llvm.org/D103522
2021-06-02 10:54:53 -04:00
Nico Weber 78ce89bb1e [lld/mac] Implement -reexport_framework, -reexport_library, -reexport-l
These are slightly easier-to-use versions of -sub_library and -sub_umbrella.

Differential Revision: https://reviews.llvm.org/D103497
2021-06-02 06:37:34 -04:00
Nico Weber 222a88a243 [lld/mac] Make -t work correctly with -flat_namespace
We used to not print dylibs referenced by other dylibs in `-t` mode. This
affected reexports, and with `-flat_namespace` also just dylibs loaded by
dylibs. Now we print them.

Fixes PR49514.

Differential Revision: https://reviews.llvm.org/D103428
2021-06-01 19:23:39 -04:00
Vy Nguyen 8f89c054af [lld-macho][nfc] Remove unnecessary use of Optional<T*>
In all of these cases, the functions could simply return a nullptr instead of {}.
There is no case where Optional<nullptr> has a special meaning.

Differential Revision: https://reviews.llvm.org/D103489
2021-06-01 18:35:31 -04:00
Nico Weber aeae3e0ba9 [lld/mac] Emit only one LC_LOAD_DYLIB per dylib
In some cases, we end up with several distinct DylibFiles that
have the same install name. Only emit a single LC_LOAD_DYLIB in
those cases.

This happens in 3 cases I know of:

1. Some tbd files are symlinks. libpthread.tbd is a symlink against
   libSystem.tbd for example, so `-lSystem -lpthread` loads
   libSystem.tbd twice. We could (and maybe should) cache loaded
   dylibs by realpath() to catch this.

2. Some tbd files are copies of each other. For example,
   CFNetwork.framework/CFNetwork.tbd and
   CFNetwork.framework/Versions/A/CFNetwork.tbd are two distinct
   copies of the same file. The former is found by
   `-framework CFNetwork` and the latter by the reexport in
   CoreServices.tbd. We could conceivably catch this by
   making `-framework` search look in `Versions/Current` instead
   of in the root, and/or by using a content hash to cache
   tbd files, but that's starting to sound complicated.

3. Magic $ld$ symbol processing can change the install name of
   a dylib based on the target platform_version. Here, two
   truly distinct dylibs can have the same install name.

So we need this code to deal with (3) anyways. Might as well use
it for 1 and 2, at least for now :)

With this (and D103430), clang-format links in the same dylibs
when linked with lld and ld64.

Differential Revision: https://reviews.llvm.org/D103488
2021-06-01 18:15:35 -04:00
Nico Weber 8d80139ccc [lld/mac] fix test failure after 24979e111
If there is an error reading the dylib, we shouldn't try
to load its reexports.

Caught e.g. by https://lab.llvm.org/buildbot/#/builders/36/builds/8946
2021-06-01 16:35:25 -04:00
Nico Weber 2c1903412b [lld/mac] Implement removal of unused dylibs
This omits load commands for unreferenced dylibs if:
- the dylib was loaded implicitly,
- it is marked MH_DEAD_STRIPPABLE_DYLIB
- or -dead_strip_dylibs is passed

This matches ld64.

Currently, the "is dylib referenced" state is computed before dead code
stripping and is not updated after dead code stripping. This too matches ld64.
We should do better here.

With this, clang-format linked with lld (like with ld64) no longer has
libobjc.A.dylib in `otool -L` output. (It was implicitly loaded as a reexport
of CoreFoundation.framework, but it's not needed.)

Differential Revision: https://reviews.llvm.org/D103430
2021-06-01 16:06:30 -04:00
Nico Weber 24979e1113 [lld/mac] Don't load DylibFiles from the DylibFile constructor
loadDylib() keeps a name->DylibFile cache, but it only writes
to the cache once the DylibFile constructor has completed.
So dylib loads done recursively from the DylibFile constructor
wouldn't use the cache.

Now, we load additional dylibs after writing to the cache,
which means the cache now gets used for dylibs loaded because
they're referenced from other dylibs.

Related to PR49514 and PR50101, but no dramatic behavior change in itself.
(Technically we no longer crash when a tbd file reexports itself,
but that doesn't happen in practice. We now accept it silently instead
of crashing; ld64 has a diag for the reexport cycle.)

Differential Revision: https://reviews.llvm.org/D103423
2021-06-01 15:31:02 -04:00
Nico Weber 0b39f055d8 [lld/mac] Don't write mtimes to N_OSO entries if ZERO_AR_DATE is set.
This is important for build determinism. This matches ld64.

Differential Revision: https://reviews.llvm.org/D103446
2021-06-01 15:29:38 -04:00
Nico Weber c4053cd14e [lld/mac] Don't crash on -order_file with assembly inputs on arm64
.s files with `-g` generate __debug_aranges on darwin/arm64 for some
reason, and those lead to `nullptr` symbols. Don't crash on that.

Fixes PR50517.

Differential Revision: https://reviews.llvm.org/D103350
2021-05-28 21:00:46 -04:00
Jez Ng 7599e98ab7 [lld-macho][nfc] Remove unnecessary parameterization of section sort
As @alexshap pointed out [here](https://reviews.llvm.org/D102972#inline-975208),
it's a bit confusing to have the option to sort OutputSections with any
comparator when in practice we only use one.

Reviewed By: #lld-macho, alexshap, thakis

Differential Revision: https://reviews.llvm.org/D102974
2021-05-25 14:58:30 -04:00
Jez Ng fcab06bd85 [lld-macho][nfc] Sort OutputSections based on explicit order of command-line inputs
This diff paves the way for {D102964} which adds a new kind of
InputSection.

We previously maintained section ordering implicitly: we created
InputSections as we parsed each file in command-line order, and passed
on this ordering when we created OutputSections and OutputSegments by
iterating over these InputSections. The implicitness of the ordering
made it difficult to refactor the code to e.g. handle a new type of
InputSection. As such, I've codified the ordering explicitly via
`inputOrder` fields. This also allows us to use `sort` instead of
`stable_sort`.

Benchmarking chromium_framework on my 3.2 GHz 16-Core Intel Xeon W:

      N           Min           Max        Median           Avg        Stddev
  x  20          4.23          4.35          4.27         4.274   0.030157481
  +  20          4.24          4.38          4.27        4.2815   0.033759989
  No difference proven at 95.0% confidence

Reviewed By: #lld-macho, alexshap

Differential Revision: https://reviews.llvm.org/D102972
2021-05-25 14:58:29 -04:00
Jez Ng 33706191d8 [lld-macho][nfc] Rename MergedOutputSection to ConcatOutputSection
The ELF format has the concept of merge sections (marked by SHF_MERGE),
which contain data that can be safely deduplicated. The Mach-O
equivalents are called literal sections (marked by S_CSTRING_LITERALS or
S_{4,8,16}BYTE_LITERALS). While the Mach-O format doesn't use the word
'merge', to avoid confusion, I've renamed our MergedOutputSection to
ConcatOutputSection. I believe it's a more descriptive name too.

This renaming sets the stage for {D102964}.

Reviewed By: #lld-macho, alexshap

Differential Revision: https://reviews.llvm.org/D102971
2021-05-25 14:58:29 -04:00
Jez Ng 9cc0d893f7 [lld-macho][nfc] clang-format everything 2021-05-25 14:58:29 -04:00
Jez Ng 8535834ef7 [lld-macho][nfc] Misc code cleanup
* Move `static_asserts` into cpp instead of header file. I noticed they
  had been separated from the main class definition in the header, so I
  set about to clean that up, then figured it made more sense as part of
  the cpp file so as not to incur unnecessary compile-time overhead.

* Remove unnecessary `virtual`s

* Remove unnecessary comment / reword another comment
2021-05-25 14:58:29 -04:00
Alexander Shaposhnikov 57501e512e [lld][MachO] Fix code formatting
Apply clang-format -style=llvm to InputFile.cpp. NFC.

Test plan: make check-all
2021-05-23 20:35:55 -07:00
Nico Weber fd09a764eb [lld/mac] Remove dead declaration 2021-05-19 14:18:03 -04:00
Mariusz Ceier 9383e9c1e6 Fix lld macho standalone build by including llvm/Config/llvm-config.h instead of llvm/Config/config.h
lld/MachO/Driver.cpp and lld/MachO/SyntheticSections.cpp include
llvm/Config/config.h which doesn't exist when building standalone lld.

This patch replaces llvm/Config/config.h include with llvm/Config/llvm-config.h
just like it is in lld/ELF/Driver.cpp and HAVE_LIBXAR with LLVM_HAVE_LIXAR and
moves LLVM_HAVE_LIBXAR from config.h to llvm-config.h

Also it adds LLVM_HAVE_LIBXAR to LLVMConfig.cmake and links liblldMachO2.so
with XAR_LIB if LLVM_HAVE_LIBXAR is set.

Differential Revision: https://reviews.llvm.org/D102084
2021-05-19 11:15:07 -04:00
Nico Weber b4ead2c37b [lld/mac] Correctly set nextdefsym
In LC_DYSYMTAB, private externs were still emitted as exported symbols instead
of as locals.

Fixes PR50373. See bug for details.

Differential Revision: https://reviews.llvm.org/D102662
2021-05-18 13:53:55 -04:00
Nico Weber 095c520fb4 [lld/mac] Propagate -(un)exported_symbol(s_list) to privateExtern in Driver
That way, it's done only once instead of every time shouldExportSymbol() is
called.

Possibly a bit faster:

    % ministat at_main at_symtodo
    x at_main
    + at_symtodo
        N           Min           Max        Median           Avg        Stddev
    x  30     3.9732189      4.114846      4.024621     4.0304692   0.037022865
    +  30       3.93766     4.0510042     3.9973931      3.991469   0.028472565
    Difference at 95.0% confidence
            -0.0390002 +/- 0.0170714
            -0.967635% +/- 0.423559%
            (Student's t, pooled s = 0.0330256)

In other runs with n=30 it makes no perf difference, so maybe it's just noise.
But being able to quickly and conveniently answer "is this symbol exported?"
is useful for fixing PR50373 and for implementing -dead_strip, so this seems
like a good change regardless.

No behavior change.

Differential Revision: https://reviews.llvm.org/D102661
2021-05-18 07:42:58 -04:00
Alexander Shaposhnikov dc2c6cf274 [lld][MachO] Adjust isCodeSection signature
This diff changes the type of the argument of isCodeSection to const InputSection *.
NFC.

Test plan: make check-lld-macho

Differential revision: https://reviews.llvm.org/D102664
2021-05-17 22:09:47 -07:00
Nico Weber bc588f9961 [lld/mac] Inline a check
`match()` can only return for non-empty vectors, but at least in
non-LTO builds that isn't clear to the compiler. Help it out.
This is a minor but measurable speedup on my machine (but less
than what we might've lost in https://reviews.llvm.org/D100818#2764272 --
bot note higher N on this measurement here, so higher confidence here):

    % ministat at_main at_branch
    x at_main
    + at_branch
        N           Min           Max        Median           Avg        Stddev
    x  30     3.9243979     4.0395119      3.987375     3.9826236   0.027567796
    +  30     3.8495831     4.0009291      3.931325     3.9347135   0.037832878
    Difference at 95.0% confidence
            -0.0479101 +/- 0.0171102
            -1.20298% +/- 0.429622%
            (Student's t, pooled s = 0.0331007)

No behavior change.

Eventually we should apply these lists at symbol parse time instead of
every time shouldExportSymbol() though :)

Differential Revision: https://reviews.llvm.org/D102655
2021-05-17 20:04:45 -04:00
Nico Weber 4a12248ee2 [lld/mac] Honor REFERENCED_DYAMICALLY, set it on __mh_execute_header
Has the effect that `__mh_execute_header` stays in the symbol table of
outputs even after running `strip` on the output. I don't know if that's
important for anything -- my motivation for the patch is just is to make
the output more similar to ld64.

(Corresponds to symbolTableInAndNeverStrip in ld64.)

Differential Revision: https://reviews.llvm.org/D102619
2021-05-17 14:22:12 -04:00
Greg McGary 93c8559baf [lld-macho] Implement branch-range-extension thunks
Extend the range of calls beyond an architecture's limited branch range by first calling a thunk, which loads the far address into a scratch register (x16 on ARM64) and branches through it.

Other ports (COFF, ELF) use multiple passes with successively-refined guesses regarding the expansion of text-space imposed by thunk-space overhead. This MachO algorithm places thunks during MergedOutputSection::finalize() in a single pass using exact thunk-space overheads. Thunks are kept in a separate vector to avoid the overhead of inserting into the `inputs` vector of `MergedOutputSection`.

FIXME:
* arm64-stubs.s test is broken
* add thunk tests
* Handle thunks to DylibSymbol in MergedOutputSection::finalize()

Differential Revision: https://reviews.llvm.org/D100818
2021-05-12 09:44:58 -07:00
Nico Weber 9ab49ae55d [lld/mac] Implement -sectalign
clang sometimes passes this flag along (see D68351), so we should implement it.

Differential Revision: https://reviews.llvm.org/D102247
2021-05-11 13:31:32 -04:00
Jez Ng b1c3c2e4fc [lld-macho] Fix order file arch filtering
We had a hardcoded check and a stale TODO, written back when we only had
support for one architecture.

Reviewed By: #lld-macho, thakis

Differential Revision: https://reviews.llvm.org/D102154
2021-05-10 15:45:54 -04:00
Jez Ng 2516b0b526 [lld-macho] Treat undefined symbols uniformly
In particular, we should apply the `-undefined` behavior to all
such symbols, include those that are specified via the command line
(i.e.  `-e`, `-u`, and `-exported_symbol`). ld64 supports this too.

Reviewed By: #lld-macho, thakis

Differential Revision: https://reviews.llvm.org/D102143
2021-05-10 15:45:54 -04:00
Nico Weber 7f673fcaa9 [lld/mac] Fix alignment on subsections
On a section with alignment of 16, subsections aligned to 16-byte
boundaries should keep their 16-byte alignment.

Fixes PR50274. (The same bug could have happened with -order_file
previously.)

Differential Revision: https://reviews.llvm.org/D102139
2021-05-09 21:00:56 -04:00
Jez Ng 0f8854f7f5 [lld-macho] Don't reference entry symbol for non-executables
This would cause us to pull in symbols (and code) that should
be unused.

Reviewed By: #lld-macho, thakis

Differential Revision: https://reviews.llvm.org/D102137
2021-05-09 20:30:26 -04:00
Greg McGary 5be8502271 [lld-macho] Explicitly undefine literal exported symbols
Symbols explicitly exported via command-line options `--exported_symbol SYM` and `--exported_symbols_list FILE` must be defined. Before this fix, lazy symbols defined in archives would be left to languish. We now force them to be included in the linked output.

Differential Revision: https://reviews.llvm.org/D102100
2021-05-08 11:37:00 -07:00
Nico Weber 7b6dd265ce [lld/mac] Copy some of the commit message of d5a70db193 into a comment 2021-05-08 13:03:17 -04:00
Nico Weber d5a70db193 [lld/mac] Write every weak symbol only once in the output
Before this, if an inline function was defined in several input files,
lld would write each copy of the inline function the output. With this
patch, it only writes one copy.

Reduces the size of Chromium Framework from 378MB to 345MB (compared
to 290MB linked with ld64, which also does dead-stripping, which we
don't do yet), and makes linking it faster:

        N           Min           Max        Median           Avg        Stddev
    x  10     3.9957051     4.3496981     4.1411121      4.156837    0.10092097
    +  10      3.908154      4.169318     3.9712729     3.9846753   0.075773012
    Difference at 95.0% confidence
            -0.172162 +/- 0.083847
            -4.14165% +/- 2.01709%
            (Student's t, pooled s = 0.0892373)

Implementation-wise, when merging two weak symbols, this sets a
"canOmitFromOutput" on the InputSection belonging to the weak symbol not put in
the symbol table. We then don't write InputSections that have this set, as long
as they are not referenced from other symbols. (This happens e.g. for object
files that don't set .subsections_via_symbols or that use .alt_entry.)

Some restrictions:
- not yet done for bitcode inputs
- no "comdat" handling (`kindNoneGroupSubordinate*` in ld64) --
  Frame Descriptor Entries (FDEs), Language Specific Data Areas (LSDAs)
  (that is, catch block unwind information) and Personality Routines
  associated with weak functions still not stripped. This is wasteful,
  but harmless.
- However, this does strip weaks from __unwind_info (which is needed for
  correctness and not just for size)
- This nopes out on InputSections that are referenced form more than
  one symbol (eg from .alt_entry) for now

Things that work based on symbols Just Work:
- map files (change in MapFile.cpp is no-op and not needed; I just
  found it a bit more explicit)
- exports

Things that work with inputSections need to explicitly check if
an inputSection is written (e.g. unwind info).

This patch is useful in itself, but it's also likely also a useful foundation
for dead_strip.

I used to have a "canoncialRepresentative" pointer on InputSection instead of
just the bool, which would be handy for ICF too. But I ended up not needing it
for this patch, so I removed that again for now.

Differential Revision: https://reviews.llvm.org/D102076
2021-05-07 17:11:40 -04:00
Jez Ng 9260760235 [lld-macho] Support loading of zippered dylibs
ld64 can emit dylibs that support more than one platform (typically macOS and
macCatalyst). This diff allows LLD to read in those dylibs. Note that this is a
super bare-bones implementation -- in particular, I haven't added support for
LLD to emit those multi-platform dylibs, nor have I added a variety of
validation checks that ld64 does. Until we have a use-case for emitting zippered
dylibs, I think this is good enough.

Fixes PR49597.

Reviewed By: #lld-macho, oontvoo

Differential Revision: https://reviews.llvm.org/D101954
2021-05-06 11:19:40 -04:00
Vy Nguyen 23233ad139 [lld-macho] Check simulator platforms to avoid issuing false positive errors.
Currently the linker causes unnecessary errors when either the target or the config's platform is a simulator.

Differential Revision: https://reviews.llvm.org/D101855
2021-05-05 18:07:58 -04:00
Jez Ng 75ba351300 [lld-macho] Try to unbreak build
Looks like the PointerUnion casting cares about const-ness...
2021-05-05 15:47:14 -04:00
Jez Ng 8806df4778 [lld-macho] Preliminary support for ARM_RELOC_BR24
ARM_RELOC_BR24 is used for BL/BLX instructions from within ARM (i.e. not
Thumb) code. This diff just handles the basic case: branches from ARM to
ARM, or from ARM to Thumb where no shimming is required. (See comments
in ARM.cpp for why shims are required.)

Note: I will likely be deprioritizing ARM work for the near future to
focus on other parts of LLD. Apologies for the half-done state of this;
I'm just trying to wrap up what I've already worked on.

Reviewed By: #lld-macho, alexshap

Differential Revision: https://reviews.llvm.org/D101814
2021-05-05 14:41:01 -04:00
Jez Ng 20f51ffe67 [lld-macho] Have --reproduce account for path rerooting
We need to account for path rerooting when generating the response
file. We could either reroot the paths before generating the file, or pass
through the original filenames and change just the syslibroot. I've opted for
the latter, in order that the reproduction run more closely mirrors the
original.

We must also be careful *not* to make an absolute path relative if it is
shadowed by a rerooted path. See repro6.tar in reroot-path.s for
details.

I've moved the call to `createResponseFile()` after the initialization of
`config->systemLibraryRoots`, since it now needs to know what those roots are.

Reviewed By: #lld-macho, oontvoo

Differential Revision: https://reviews.llvm.org/D101224
2021-05-05 14:41:01 -04:00
Greg McGary 27b426b0c8 [lld-macho] Implement builtin section renaming
ld64 automatically renames many sections depending on output type and assorted flags. Here, we implement the most common configs. We can add more obscure flags and behaviors as needed.

Depends on D101393

Differential Revision: https://reviews.llvm.org/D101395
2021-05-03 21:26:51 -07:00
Jez Ng 001ba65375 [lld-macho] De-templatize mach_header operations
@thakis pointed out that `mach_header` and `mach_header_64`
actually have the same set of (used) fields, with the 64-bit version
having extra padding. So we can access the fields we need using the
single `mach_header` type instead of using templates to switch between
the two.

I also spotted a potential issue where hasObjCSection tries to parse a
file w/o checking if it does indeed match the target arch... As such,
I've added a quick magic number check to ensure we don't access invalid
memory during `findCommand()`.

Addresses PR50180.

Reviewed By: #lld-macho, thakis

Differential Revision: https://reviews.llvm.org/D101724
2021-05-03 18:31:23 -04:00
Jez Ng 05c5363b39 [lld-macho] Parse & emit the N_ARM_THUMB_DEF symbol flag
Eventually we'll use this flag to properly handle bl/blx
opcodes.

Reviewed By: #lld-macho, gkm

Differential Revision: https://reviews.llvm.org/D101558
2021-04-30 16:17:26 -04:00
Jez Ng 2d28100bf2 [lld-macho] Initial scaffolding for ARM32 support
This just parses the `-arch armv7` and emits the right header flags.
The rest will be slowly fleshed out in upcoming diffs.

Reviewed By: #lld-macho, gkm

Differential Revision: https://reviews.llvm.org/D101557
2021-04-30 16:17:25 -04:00
Nico Weber 4b456038e4 [lld/mac] Tweak two comments and fix style on one variable name
Cosmetic, no behavior change.
2021-04-30 09:30:51 -04:00
Jez Ng 7e115da5df [lld-macho] Make everything PIE by default
Modern versions of macOS (>= 10.7) and in general all modern Mach-O
target archs want PIEs by default. ld64 defaults to PIE for iOS >= 4.3,
as well as for all versions of watchOS and simulators. Basically all the
platforms LLD is likely to target want PIE. So instead of cluttering LLD's
code with legacy version checks, I think it's simpler to just default to
PIE for everything.

Note that `-no_pie` still works, so users can still opt out of it.

Reviewed By: #lld-macho, thakis, MaskRay

Differential Revision: https://reviews.llvm.org/D101513
2021-04-29 15:11:23 -04:00
Greg McGary 465204d63a [lld-macho][NFC] define more strings in section_names:: and segment_names::
As preparation for a subsequent diff that implements builtin section renaming, define more `constexpr` strings in namespaces `lld::macho::segment_names` and `lld::macho::section_names`, and use them to replace string literals.

Differential Revision: https://reviews.llvm.org/D101393
2021-04-27 17:48:45 -07:00
Jez Ng 7ca133c360 [lld-macho] std::sort -> llvm::sort 2021-04-27 18:02:59 -04:00
Greg McGary c2419aae76 [lld-macho] Add option --error-limit=N
Add option to limit (or remove limits) on the number of errors printed before exiting. This option exists in the other lld ports: COFF & ELF.

Differential Revision: https://reviews.llvm.org/D101274
2021-04-26 07:10:12 -07:00
Nico Weber de266ce4f9 [lld/mac] Don't assert when using -exported_symbol with private symbol
When I added this assert in D93609, it asserted that a symbol that
is privateExtern is also isExternal().

In D98381 the privateExtern check moved into shouldExportSymbol()
but the assert didn't -- now it checked that _every_ non-exported
symbol is isExternal(), which isn't true. Move the assert into the
privateExtern check where it used to be.

Fixes PR50098.

Differential Revision: https://reviews.llvm.org/D101223
2021-04-24 10:21:51 -04:00
Jez Ng 3fe5c3b018 [lld-macho] Fix use-after-free in loadDylib()
We were taking a reference to a value in `loadedDylibs`, which in turn
called `make<DylibFile>()`, which could then recursively call
`loadDylibs`, which would then potentially resize `loadedDylibs` and
invalidate that reference.

Fixes PR50101.

Reviewed By: #lld-macho, oontvoo

Differential Revision: https://reviews.llvm.org/D101175
2021-04-23 18:05:49 -04:00
Jez Ng 035eb6d154 [lld-macho]][nfc] Fix some typos + rephrase a comment
I was a bit confused by the comment because I thought that "Tests
that..." was describing the tests contained within the same file.

Reviewed By: #lld-macho, thakis

Differential Revision: https://reviews.llvm.org/D101160
2021-04-23 18:05:48 -04:00
Nico Weber a61891d491 [lld/mac] Support more flags for --reproduce
I went through the callers of `readFile()` and `addFile()` in Driver.cpp
and checked that the options that use them all get rewritten in the
--reproduce response file. -(un)exported_symbols_list and -bundle_loader
weren't, so add them.

Also spruce up the test for reproduce a bit and actually try linking
with the exptracted repro archive.

Motivated by the response file in PR50098 complaining abou the
-exported_symbols_list path being wrong :)

Differential Revision: https://reviews.llvm.org/D101182
2021-04-23 14:40:24 -04:00
Nico Weber a38ebed258 [lld/mac] Implement support for .weak_def_can_be_hidden
I first had a more invasive patch for this (D101069), but while trying
to get that polished for review I realized that lld's current symbol
merging semantics mean that only a very small code change is needed.
So this goes with the smaller patch for now.

This has no effect on projects that build with -fvisibility=hidden
(e.g.  chromium), since these see .private_extern symbols instead.

It does have an effect on projects that build with -fvisibility-inlines-hidden
(e.g. llvm) in -O2 builds, where LLVM's GlobalOpt pass will promote most inline
functions from .weak_definition to .weak_def_can_be_hidden.

Before this patch:

    % ls -l out/gn/bin/clang out/gn/lib/libclang.dylib
    -rwxr-xr-x  1 thakis  staff  113059936 Apr 22 11:51 out/gn/bin/clang
    -rwxr-xr-x  1 thakis  staff   86370064 Apr 22 11:51 out/gn/lib/libclang.dylib
    % out/gn/bin/llvm-objdump --macho --weak-bind out/gn/bin/clang | wc -l
        8291
    % out/gn/bin/llvm-objdump --macho --weak-bind out/gn/lib/libclang.dylib | wc -l
        5698

With this patch:

    % ls -l out/gn/bin/clang out/gn/lib/libclang.dylib
    -rwxr-xr-x  1 thakis  staff  111721096 Apr 22 11:55 out/gn/bin/clang
    -rwxr-xr-x  1 thakis  staff   85291208 Apr 22 11:55 out/gn/lib/libclang.dylib
    thakis@MBP llvm-project % out/gn/bin/llvm-objdump --macho --weak-bind out/gn/bin/clang | wc -l
         725
    thakis@MBP llvm-project % out/gn/bin/llvm-objdump --macho --weak-bind out/gn/lib/libclang.dylib | wc -l
         542

Linking clang becomes a tiny bit faster with this patch:

    x 100    0.67263818    0.77847815    0.69430709    0.69877208   0.017715892
    + 100    0.67209601    0.73323393    0.68600798    0.68917346   0.012824377
    Difference at 95.0% confidence
            -0.00959861 +/- 0.00428661
            -1.37364% +/- 0.613449%
            (Student's t, pooled s = 0.0154648)

This only happens if lld with the patch and lld without the patch are both
linked with an lld with the patch or both linked with an lld without the patch
(...or with ld64). I accidentally linked the lld with the patch with an lld
without the patch and the other way round at first. In that setup, no
difference is found. That makese sense, since having fewer weak imports will
make the linked output a bit faster too. So not only does this make linking
binaries such as clang a bit faster (since fewer exports need to be written to
the export trie by lld), the linked output binary binary is also a bit faster
(since dyld needs to process fewer dynamic imports).

This also happens to fix the one `check-clang` failure when using lld as host
linker, but mostly for silly reasons: See crbug.com/1183336, mostly comment 26.
The real bug here is that c-index-test links all of LLVM both statically and
dynamically, which is an ODR violation. Things just happen to work with this
patch.

So after this patch, check-clang, check-lld, check-llvm all pass with lld as
host linker :)

Differential Revision: https://reviews.llvm.org/D101080
2021-04-22 22:51:34 -04:00
Jez Ng 8c17a87515 [re-land][lld-macho] Fix min version check
We had got it backwards... the minimum version of the target
should be higher than the min version of the object files, presumably
since new platforms are backwards-compatible with older formats.

Fixes PR50078.

The original commit (aa05439c9c) broke many tests that had inputs too
new for our target platform (10.0). This commit changes the inputs to
target 10.0, which was the simpler thing to do, but we should really
just have our lit.local.cfg default to targeting 10.15... we're not
likely to ever have proper support for the older versions anyway. I will
follow up with a change to that effect.

Differential Revision: https://reviews.llvm.org/D101114
2021-04-22 19:35:32 -04:00
Jez Ng 75ecb804b1 Revert "[lld-macho] Fix min version check"
This reverts commit aa05439c9c.
2021-04-22 19:07:41 -04:00
Jez Ng aa05439c9c [lld-macho] Fix min version check
We had got it backwards... the minimum version of the target
should be higher than the min version of the object files, presumably
since new platforms are backwards-compatible with older formats.

Fixes PR50078.

Reviewed By: #lld-macho, thakis

Differential Revision: https://reviews.llvm.org/D101114
2021-04-22 18:25:44 -04:00
Jez Ng 2618eaf614 [lld-macho][nfc] Clean up some constructor declarations
Remove unnecessary default ctor + add `explicit` to others
2021-04-22 18:25:44 -04:00
Nico Weber c1b2a7bfbf [lld/mac] make a few "named parameter comments" more consistent
Most of LLVM and almost all of lld/MachO uses `/*foo=*/bar` style.
No behavior change.
2021-04-22 10:48:03 -04:00
Jez Ng ed4a4e3312 [lld-macho][nfc] Add accessors for commonly-used PlatformInfo fields
As discussed here: https://reviews.llvm.org/D100523#inline-951543

Reviewed By: #lld-macho, thakis, alexshap

Differential Revision: https://reviews.llvm.org/D100978
2021-04-21 15:43:56 -04:00
Jez Ng ab9c21bbab [lld-macho] Support LC_ENCRYPTION_INFO
This load command records a range spanning from the end of the load
commands to the end of the `__TEXT` segment. Presumably the kernel will encrypt
all this data.

Reviewed By: #lld-macho, thakis

Differential Revision: https://reviews.llvm.org/D100973
2021-04-21 13:39:56 -04:00
Alexander Shaposhnikov b5720354ef [lld][MachO] Refactor findCommand
Refactor findCommand to allow passing multiple types. NFC.

Test plan: make check-lld-macho

Differential revision: https://reviews.llvm.org/D100954
2021-04-21 08:38:17 -07:00
Alexander Shaposhnikov 5c835e1ae5 [lld][MachO] Add support for LC_VERSION_MIN_* load commands
This diff adds initial support for the legacy LC_VERSION_MIN_* load commands.

Test plan: make check-lld-macho

Differential revision: https://reviews.llvm.org/D100523
2021-04-21 05:41:14 -07:00
Jez Ng 7208bd4b32 [lld-macho] Skip platform checks for a few libSystem re-exports
XCode 12 ships with mismatched platforms for these libraries,
so this hack is necessary...

Fixes PR49799.

Reviewed By: #lld-macho, gkm, smeenai

Differential Revision: https://reviews.llvm.org/D100913
2021-04-20 19:54:53 -04:00
Jez Ng bb62ef9943 [lld-macho] Ensure segments are laid out contiguously
codesign/libstuff checks that the `__LLVM` segment is directly
before `__LINKEDIT` by checking that `fileOff + fileSize == next segment
fileOff`. Previously, there would be gaps between the segments due to
the fact that their fileOffs are page-aligned but their fileSizes
aren't. In order to satisfy codesign, we page-align fileOff *before*
calculating fileSize. (I don't think codesign checks for the relative
ordering of other segments, so in theory we could do this just for
`__LLVM`, but ld64 seems to do it for all segments.)

Note that we *don't* round up the fileSize of the `__LINKEDIT` segment.
Since it's the last segment, it doesn't need to worry about contiguity;
in addition, codesign checks that the last (hidden) section in
`__LINKEDIT` covers the last byte of the segment, so if we rounded up
`__LINKEDIT`'s size we would have to do the same for its last section,
which is a bother.

While at it, I also addressed a FIXME in the linkedit-contiguity.s test
to cover more `__LINKEDIT` sections.

Reviewed By: #lld-macho, thakis, alexshap

Differential Revision: https://reviews.llvm.org/D100848
2021-04-20 16:58:57 -04:00
Jez Ng 1aa29dffce [lld-macho] Support subtractor relocations that reference sections
The minuend (but not the subtrahend) can reference a section.

Note that we do not yet properly validate that the subtrahend isn't
referencing a section; I've filed PR50034 to track that.

I've also extended the reloc-subtractor.s test to reorder symbols, to
make sure that the addends are being associated with the minuend (and not
the subtrahend) relocation.

Fixes PR49999.

Reviewed By: #lld-macho, thakis

Differential Revision: https://reviews.llvm.org/D100804
2021-04-20 16:58:57 -04:00
Jez Ng bb0e1ae7c4 [lld-macho] Add separator to error message 2021-04-16 17:01:14 -04:00
Jez Ng ca6751043d [lld-macho] Initial groundwork for -bitcode_bundle
This diff creates an empty XAR file and copies it into
`__LLVM,__bundle`. Follow-up work will actually populate the contents of
that XAR.

Reviewed By: #lld-macho, gkm

Differential Revision: https://reviews.llvm.org/D100650
2021-04-16 16:47:14 -04:00
Jez Ng 4938b090cf [lld-macho] Don't use arrays as template parameters
MSVC from VSCode 2017 appears unhappy with it (causes an
internal compiler error.)

This also means that we need to avoid doing `sizeof(stubCode)` as
`sizeof(int[N])` on function array parameters decays into `sizeof(int *)`.

Reviewed By: #lld-macho, gkm

Differential Revision: https://reviews.llvm.org/D100605
2021-04-15 21:16:34 -04:00
Jez Ng 1acda12d00 [lld-macho] Make load relaxation work for arm64_32
arm64_32 uses 32-bit GOT loads, so we should accept those
instructions in `ARM64Common::relaxGotLoad()` too.

Reviewed By: #lld-macho, gkm

Differential Revision: https://reviews.llvm.org/D100229
2021-04-15 21:16:34 -04:00
Jez Ng 1460942c15 [lld-macho] Add 32-bit compact unwind support
This could probably have been part of D99633, but I split it up to make
things a bit more reviewable. I also fixed some bugs in the implementation that
were masked through integer underflows when operating in 64-bit mode.

Reviewed By: #lld-macho, gkm

Differential Revision: https://reviews.llvm.org/D99823
2021-04-15 21:16:33 -04:00
Jez Ng 3bc88eb392 [lld-macho] Add support for arm64_32
From what I can tell, it's pretty similar to arm64. The two main differences
are:

1. No 64-bit relocations
2. Stub code writes to 32-bit registers instead of 64-bit

Plus of course the various on-disk structures like `segment_command` are using
the 32-bit instead of the 64-bit variants.

Reviewed By: #lld-macho, gkm

Differential Revision: https://reviews.llvm.org/D99822
2021-04-15 21:16:33 -04:00
Jez Ng db7a413e51 [lld-macho] Re-root absolute input file paths if -syslibroot is specified
Reviewed By: #lld-macho, gkm

Differential Revision: https://reviews.llvm.org/D100147
2021-04-15 21:16:33 -04:00
Jez Ng eb5b7d4497 [lld-macho] LTO: Unset VisibleToRegularObj where possible
This allows LLVM's LTO to internalize symbols that are not referenced
directly by regular objects. Naturally, this means we need to track
which symbols are referenced by regular objects. The approach taken here
is similar to LLD-COFF's: like the COFF port, we extend
`SymbolTable::insert()` to set the isVisibleToRegularObj bit. (LLD-ELF
relies on the Symbol constructor and `Symbol::mergeProperties()`, but
the Mach-O port does not have a `mergeProperties()` equivalent.)

From what I can tell, ld64 (which uses libLTO) doesn't do this
optimization at all. I'm not even sure libLTO provides a way to do this.
Not having ld64's behavior as a reference implementation is unfortunate;
instead, I am relying on LLD-ELF/COFF's behavior as references while
erring on the conservative side. In particular, LLD-MachO will only do
this optimization for executables right now.

We also don't attempt it when `-flat_namespace` is used -- otherwise
we'd need scan the symbol table to find matches for every un-namespaced
symbol reference, which is expensive.

internalize.ll is based off the LLD-ELF tests `internalize-basic.ll` and
`internalize-undef.ll`. Looks like @davide added some of LLD-ELF's internalize
tests, so adding him as a reviewer...

Reviewed By: #lld-macho, gkm

Differential Revision: https://reviews.llvm.org/D99105
2021-04-15 21:16:33 -04:00
Jez Ng 8ca366935b Revert "[lld-macho] Add support for arm64_32" and other stacked diffs
This reverts commits:
* 8914902b01
* 35a745d814
* 682d1dfe09
2021-04-13 12:40:58 -04:00
Jez Ng 84c52f3a19 [lld-macho] arm64_32 executables are always PIE
This should fix the assert that's currently breaking the build.
2021-04-13 11:55:45 -04:00
Jez Ng 682d1dfe09 [lld-macho] Make load relaxation work for arm64_32
arm64_32 uses 32-bit GOT loads, so we should accept those
instructions in `ARM64Common::relaxGotLoad()` too.

Reviewed By: #lld-macho, gkm

Differential Revision: https://reviews.llvm.org/D100229
2021-04-13 10:43:28 -04:00
Jez Ng 3142fc3b5b [lld-macho] Have toString() emit full path to archive files
It doesn't make sense to take just the base filename for archives when we emit
the full path for object files. (LLD-ELF emits the full path too.)

This will also make it easier to write a proper test for {D100147}.

Reviewed By: #lld-macho, oontvoo

Differential Revision: https://reviews.llvm.org/D100357
2021-04-13 10:43:28 -04:00
Jez Ng 35a745d814 [lld-macho] Add 32-bit compact unwind support
This could probably have been part of D99633, but I split it up to make
things a bit more reviewable. I also fixed some bugs in the implementation that
were masked through integer underflows when operating in 64-bit mode.

Reviewed By: #lld-macho, gkm

Differential Revision: https://reviews.llvm.org/D99823
2021-04-13 10:43:28 -04:00
Jez Ng 8914902b01 [lld-macho] Add support for arm64_32
From what I can tell, it's pretty similar to arm64. The two main differences
are:

1. No 64-bit relocations
2. Stub code writes to 32-bit registers instead of 64-bit

Plus of course the various on-disk structures like `segment_command` are using
the 32-bit instead of the 64-bit variants.

Reviewed By: #lld-macho, gkm

Differential Revision: https://reviews.llvm.org/D99822
2021-04-13 10:43:28 -04:00
Jez Ng 88cb786ec2 [lld-macho][nfc] Remove DYSYM8 reloc attribute
It's likely redundant, per discussion with @gkm. The BYTE8
attribute covers the bit width requirement already.

Reviewed By: #lld-macho, gkm

Differential Revision: https://reviews.llvm.org/D100133
2021-04-09 19:48:08 -04:00
Jez Ng c2e76a9a6d [lld-macho][nfc] Use varargs form of hasArg() 2021-04-08 14:12:55 -04:00
Jez Ng c23b92acd0 [lld-macho] Support -add_ast_path
Swift builds seem to use it. All it requires is emitting the
corresponding paths as STABS.

Fixes llvm.org/PR49385.

Reviewed By: clayborg

Differential Revision: https://reviews.llvm.org/D100076
2021-04-08 14:12:55 -04:00
Jez Ng 050a7a27ca [lld-macho] Support --thinlto-jobs
The test is loosely based off LLD-ELF's `thinlto.ll`. However, I
found that test questionable because the the -save_temps behavior it
checks for is identical regardless of whether we are running in single-
or multi-threaded mode. I tried writing a test based on `--time-trace`
but couldn't get it to run deterministically... so I've opted to just
skip checking that behavior for now.

Reviewed By: #lld-macho, gkm

Differential Revision: https://reviews.llvm.org/D99356
2021-04-08 12:21:01 -04:00
Jez Ng d9065fe8ea [lld-macho] Parallelize __LINKEDIT generation
Benchmarking chromium_framework on a 3.2 GHz 16-Core Intel Xeon W Mac Pro:

      N           Min           Max        Median           Avg        Stddev
  x  20          4.33          4.42          4.37          4.37   0.021026299
  +  20          4.12          4.23          4.18         4.175   0.035318103
  Difference at 95.0% confidence
    -0.195 +/- 0.0186025
    -4.46224% +/- 0.425686%
    (Student's t, pooled s = 0.0290644)

Reviewed By: #lld-macho, gkm

Differential Revision: https://reviews.llvm.org/D99998
2021-04-07 19:55:52 -04:00
Vy Nguyen db851dfb49 [lld-macho] Make time-trace* options more permissive.
If either `time-trace-granularity` or `time-trace-file` is specified, then don't make users specify `-time-trace`.
It seems silly that I have to type all three options, eg, `-time-trace -time-trace-file=- -time-trace-granularity=...`.

Differential Revision: https://reviews.llvm.org/D100011
2021-04-07 16:00:20 -04:00
Vy Nguyen ffc65824f0 [lld-macho][nfc] Minor refactoring + clang-tidy fixes
- use "empty()" instead of "size()"
- refactor the re-export code so it doesn't create a new vector every time.

Differential Revision: https://reviews.llvm.org/D100019
2021-04-07 13:55:52 -04:00
Jez Ng 982e3c0510 [lld-macho] Sibling N_SO symbols must have the empty string
We had been giving them a string index of zero, which actually corresponds to a
string with a single space due to {D89639}.

This was far from obvious in the old test because llvm-nm doesn't quote the
symbol names, making the empty string look identical to a string of a single
space. `dsymutil -s` quotes its strings, so I've changed the test accordingly.

Fixes llvm.org/PR48714. Thanks @clayborg for the tips!

Reviewed By: clayborg

Differential Revision: https://reviews.llvm.org/D100003
2021-04-07 12:08:14 -04:00
Jez Ng 2461804b48 [lld-macho] Symbol::value should always be uint64_t
D98837 migrated a bunch of `value`s to uint64_t, but missed these.
2021-04-06 17:54:11 -04:00
Jez Ng ceec610754 [lld-macho] Fix & refactor symbol size calculations
I noticed two problems with the previous implementation:

* N_ALT_ENTRY symbols weren't being handled correctly -- they should
  determine the size of the previous symbol, even though they don't
  cause a new section to be created
* The last symbol in a section had its size calculated wrongly;
  the first subsection's size was used instead of the last one

I decided to take the opportunity to refactor things as well, mainly to
realize my observation
[here](https://reviews.llvm.org/D98837#inline-931511) that we could
avoid doing a binary search to match symbols with subsections. I think
the resulting code is a bit simpler too.

      N           Min           Max        Median           Avg        Stddev
  x  20          4.31          4.43          4.37        4.3775   0.034162922
  +  20          4.32          4.43          4.38        4.3755    0.02799906
  No difference proven at 95.0% confidence

Reviewed By: #lld-macho, alexshap

Differential Revision: https://reviews.llvm.org/D99972
2021-04-06 15:10:01 -04:00
Jez Ng 94f75202ac [lld-macho][nfc] Remove HelpHidden from aliases to implemented flags
This is a no-op. Just cleaning up Options.td...

Reviewed By: #lld-macho, oontvoo

Differential Revision: https://reviews.llvm.org/D99874
2021-04-06 15:10:00 -04:00
Jez Ng 174deb0539 [lld-macho] clang-format cleanup
find . -type f -name "*.cpp" -o -name "*.h" | xargs clang-format -i
2021-04-06 14:26:13 -04:00
Jez Ng e0df2b540a [lld-macho] Rename SubsectionMapping to SubsectionMap
We bikeshedded about it here: https://reviews.llvm.org/D98837#inline-931557

I initially suggested SubsectionMapping, but I thought the discussion
landed on doing `std::vector<SubsectionEntry>`. @alexshap went and did
both, but on hindsight I regret adding 3 more characters to an already
long name, and I think SubsectionEntry is descriptive enough...

This diff also renames `subsectionMap` to `subsecMap` for consistency
with other variable names in the codebase.
2021-04-06 14:26:13 -04:00
Cyndy Ishida 0116d04d04 [TextAPI] move source code files out of subdirectory, NFC
TextAPI/ELF has moved out into InterfaceStubs, so theres no longer a
need to seperate out TextAPI between formats.

Reviewed By: ributzka, int3, #lld-macho

Differential Revision: https://reviews.llvm.org/D99811
2021-04-05 10:24:42 -07:00
Jez Ng bd115d0991 [lld-macho] Another attempt at fixing 32-bit builds 2021-04-03 11:58:23 -04:00
Jez Ng c04e1c8b66 [lld-macho] Fix build on 32-bit systems
Summary: Follow-up to D99633.
2021-04-03 11:12:11 -04:00
Fangrui Song c318746345 [lld-macho] Fix -Wsuggest-override after D99633. NFC 2021-04-02 17:04:11 -07:00
Jez Ng 817d98d841 [lld-macho][nfc] Refactor in preparation for 32-bit support
The main challenge was handling the different on-disk structures (e.g.
`mach_header` vs `mach_header_64`). I tried to strike a balance between
sprinkling `target->wordSize == 8` checks everywhere (branchy = slow, and ugly)
and templatizing everything (causes code bloat, also ugly). I think I struck a
decent balance by judicious use of type erasure.

Note that LLD-ELF has a similar architecture, though it seems to use more templating.

Linking chromium_framework takes about the same time before and after this
change:

      N           Min           Max        Median           Avg        Stddev
  x  20          4.52          4.67         4.595        4.5945   0.044423204
  +  20           4.5          4.71         4.575         4.582   0.056344803
  No difference proven at 95.0% confidence

Reviewed By: #lld-macho, oontvoo

Differential Revision: https://reviews.llvm.org/D99633
2021-04-02 18:46:39 -04:00
Greg McGary 3f8c6f493b [lld-macho][NFC] Remove redundant member from class Defined
`class Symbol` defines a data member `InputFile *file;`
`class Defined` inherits from `Symbol` and also defines a data member `InputFile *file;` for no apparent purpose.

Differential Revision: https://reviews.llvm.org/D99783
2021-04-02 09:19:15 -07:00
Yang Fan d441dee5c2
[lld][MachO] Fix -Wsign-compare warning (NFC)
GCC warning:
```
/llvm-project/lld/MachO/InputFiles.cpp:484:24: warning: comparison of integer expressions of different signedness: ‘int64_t’ {aka ‘long int’} and ‘uint64_t’ {aka ‘long unsigned int’} [-Wsign-compare]
484 |           return value < subsectionEntry.offset;
    |                  ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~
```
2021-04-02 11:33:56 +08:00
Alexander Shaposhnikov f6ad045366 [lld][MachO] Make emitEndFunStab independent from .subsections_via_symbols
This diff addresses FIXME in SyntheticSections.cpp and removes
the dependency of emitEndFunStab on .subsections_via_symbols.

Test plan: make check-lld-macho

Differential revision: https://reviews.llvm.org/D99054
2021-04-01 17:48:09 -07:00
Alexander Shaposhnikov f1e4e2fb20 [lld][MachO] Refactor handling of subsections
This diff is a preparation for fixing FunStabs (incorrect size calculation).
std::map<uint32_t, InputSection*> (SubsectionMap) is replaced with
a sorted vector + binary search. If .subsections_via_symbols is set
this vector will contain the list of subsections, otherwise,
the offsets will be used for calculating the symbols sizes.

Test plan: make check-all

Differential revision: https://reviews.llvm.org/D98837
2021-03-31 16:52:53 -07:00
Jez Ng 9b6dde8af8 [lld-macho] Parallelize UUID hash computation
This reuses the approach (and some code) from LLD-ELF.

It's a decent win when linking chromium_framework on a Mac Pro (3.2 GHz 16-Core Intel Xeon W):

      N           Min           Max        Median           Avg        Stddev
  x  20          4.58          4.83          4.66        4.6685   0.066591844
  +  20          4.42          4.61           4.5         4.505    0.04751731
  Difference at 95.0% confidence
          -0.1635 +/- 0.0370242
          -3.5022% +/- 0.793064%
          (Student's t, pooled s = 0.0578462)

The output binary is 381MB.

Reviewed By: #lld-macho, oontvoo

Differential Revision: https://reviews.llvm.org/D99279
2021-03-31 15:48:36 -04:00
Greg McGary 427d359721 [lld-macho][NFC] Drop unnecessary macho:: namespace prefix on unambiguous references to Symbol
Within `lld/macho/`, only `InputFiles.cpp` and `Symbols.h` require the `macho::` namespace qualifier to disambiguate references to `class Symbol`.

Add braces to outer `for` of a 5-level single-line `if`/`for` nest.

Differential Revision: https://reviews.llvm.org/D99555
2021-03-30 14:58:35 -07:00
Nico Weber 742f663705 fix comment typo to cycle bots 2021-03-29 14:35:57 -04:00
Jez Ng a43f588e01 [lld-macho] Implement -segprot
Addresses llvm.org/PR49405.

Reviewed By: #lld-macho, oontvoo

Differential Revision: https://reviews.llvm.org/D99389
2021-03-29 14:08:12 -04:00
Jez Ng ae7aa9ed15 [lld-macho] Add time tracing for LTO
The test is similar to the one used for LLD-ELF.

Differential Revision: https://reviews.llvm.org/D99318
2021-03-26 18:14:10 -04:00
Jez Ng 94e369400e [lld-macho] Fix parsing of --time-trace-{granularity,file}
Summary: We needed to use `Joined` instead of `Flag`. This wasn't caught
because the relevant test that was copied from LLD-ELF was still
invoking LLD-ELF instead of LLD-MachO...

Differential Revision: https://reviews.llvm.org/D99313
2021-03-26 18:14:10 -04:00
Jez Ng 45cdceb40c [lld-macho] Support -no_function_starts
Pretty simple code-wise. Also threw in some refactoring:

* Put the functionStartSection under Writer instead of InStruct, since
  it doesn't need to be accessed outside of Writer
* Adjusted the test to put all files under the temp dir instead of at
  the top-level
* Added some CHECK-LABELs to make it clearer where the function starts
  data is

Differential Revision: https://reviews.llvm.org/D99112
2021-03-26 18:14:10 -04:00
Vy Nguyen dee5787d3e Reland [lld-macho][nfc] minor clean up, follow up to D98559
This reverts commit 77b4230ed9.

New change: Fixed tests on windows

     Differential Revision: https://reviews.llvm.org/D99210
2021-03-25 16:46:37 -04:00
Vy Nguyen e2f34cc330 [lld-macho][nfc] Removed unnecessary static_cast
Differential Revision: https://reviews.llvm.org/D99365
2021-03-25 15:07:46 -04:00
Jez Ng 0113cf00b6 [lld-macho] Add support for --threads
Code and test are largely identical to the LLD-ELF equivalents.

Reviewed By: MaskRay

Differential Revision: https://reviews.llvm.org/D99312
2021-03-25 14:51:31 -04:00
Jez Ng 4bcaafeb0e [lld-macho] Add more TimeTraceScopes
I added just enough to allow us to see a top-level breakdown of time taken. This
is the result of loading the time-trace output into `chrome:://tracing`:

ef5e8234f3/tracing.png

Reviewed By: oontvoo

Differential Revision: https://reviews.llvm.org/D99311
2021-03-25 14:51:31 -04:00
Jez Ng 53fd1ada76 [lld-macho] Fix typo in diagnostic message 2021-03-25 14:51:31 -04:00
Vy Nguyen 77b4230ed9 Revert "[lld-macho][nfc] minor clean up, follow up to D98559"
This reverts commit 1bc33eb6a3.
tests failed on windows
2021-03-23 17:15:36 -04:00
Vy Nguyen 1bc33eb6a3 [lld-macho][nfc] minor clean up, follow up to D98559
Differential Revision: https://reviews.llvm.org/D99210
2021-03-23 16:13:09 -04:00