https://discourse.llvm.org/t/parallel-input-file-parsing/60164
initializeSymbols currently sets Defined::section and handles non-prevailing
COMDAT groups. Move the code to the parallel postParse to reduce work from the
single-threading code path and make parallel section initialization infeasible.
Postpone reporting duplicate symbol errors so that the messages have the
section information. (`Defined::section` is assigned in postParse and another
thread may not have the information).
* duplicated-synthetic-sym.s: BinaryFile duplicate definition (very rare) now
has no section information
* comdat-binding: `%t/w.o %t/g.o` leads to an undesired undefined symbol. This
is not ideal but we report a diagnostic to inform that this is unsupported.
(See release note)
* comdat-discarded-lazy.s: %tdef.o is unextracted. The new behavior (discarded
section error) makes more sense
* i386-comdat.s: switched to a better approach working around
.gnu.linkonce.t.__x86.get_pc_thunk.bx in glibc<2.32 for x86-32.
Drop the ancient no-longer-relevant workaround for __i686.get_pc_thunk.bx
Depends on D120640
Differential Revision: https://reviews.llvm.org/D120626
https://discourse.llvm.org/t/parallel-input-file-parsing/60164
initializeSymbols currently sets Defined::section and handles non-prevailing
COMDAT groups. Move the code to the parallel postParse to reduce work from the
single-threading code path and make parallel section initialization infeasible.
Postpone reporting duplicate symbol errors so that the messages have the
section information. (`Defined::section` is assigned in postParse and another
thread may not have the information).
* duplicated-synthetic-sym.s: BinaryFile duplicate definition (very rare) now
has no section information
* comdat-binding: `%t/w.o %t/g.o` leads to an undesired undefined symbol. This
is not ideal but we report a diagnostic to inform that this is unsupported.
(See release note)
* comdat-discarded-lazy.s: %tdef.o is unextracted. The new behavior (discarded
section error) makes more sense
Depends on D120640
Reviewed By: peter.smith
Differential Revision: https://reviews.llvm.org/D120626
ObjFile::parse combines symbol initialization and resolution. Many tasks
unrelated to symbol resolution can be postponed and parallelized. This patch
extracts local symbol initialization and parallelizes it.
Technically the new function initializeLocalSymbols can be merged into
ObjFile::postParse, but functions like getSrcMsg may access the
uninitialized (all nullptr) local part of InputFile::symbols.
Linking chrome: 1.02x as fast with glibc malloc, 1.04x as fast with mimalloc
Depends on f456c3ae3f and D119908
Reviewed By: ikudrin
Differential Revision: https://reviews.llvm.org/D119909
ObjFile::parse combines symbol initialization and resolution. Many tasks
unrelated to symbol resolution can be postponed and parallelized. This patch
extracts local symbol initialization and parallelizes it.
Technically the new function initializeLocalSymbols can be merged into
ObjFile::postParse, but functions like getSrcMsg may access the
uninitialized (all nullptr) local part of InputFile::symbols.
Linking chrome: 1.02x as fast with glibc malloc, 1.04x as fast with mimalloc
Reviewed By: ikudrin
Differential Revision: https://reviews.llvm.org/D119909
* detect `def_tls.o undef_nontls.o` violation
* place error checking code (checking duplicate symbol) together
* allow `--defsym tls1=tls2 def_tls.o`
As a degraded error checking, `--defsym tls1=42` violation will not be detected.
https://discourse.llvm.org/t/parallel-input-file-parsing/60164
To decouple symbol initialization and section initialization, `Defined::section`
assignment should be postponed after input file parsing. To avoid spurious
duplicate definition error due to two definitions in COMDAT groups of the same
signature, we should postpone the duplicate symbol check.
The function is called postScan instead of a more specific name like
checkDuplicateSymbols, because we may merge Symbol::mergeProperties into
postScan. It is placed after compileBitcodeFiles to apply to ET_REL files
produced by LTO. This causes minor diagnostic regression
for skipLinkedOutput configurations: ld.lld --thinlto-index-only a.bc b.o
(bitcode definition prevails) won't detect duplicate symbol error. I think this
is an acceptable compromise. The important cases where (a) both files are
bitcode or (b) --thinlto-index-only is unused are still detected.
Reviewed By: ikudrin
Differential Revision: https://reviews.llvm.org/D119908
In many call sites we know uncompression cannot happen (non-SHF_ALLOC, or the
data (even if compressed) must have been uncompressed by a previous pass).
Prefer rawData in these cases. data() increases code size and prevents
optimization on rawData.
https://maskray.me/blog/2022-01-16-archives-and-start-lib
For every definition in an extracted archive member, we intern the symbol twice,
once for the archive index entry, once for the .o symbol table after extraction.
This is inefficient.
Symbols in a --start-lib ObjFile/BitcodeFile are only interned once because the
result is cached in symbols[i].
Just handle an archive using the --start-lib code path. We can therefore remove
ArchiveFile and LazyArchive. For many projects, archive member extraction ratio
is high and it is a net performance win. Linking a Release build of clang is
1.01x as fast.
Note: --start-lib scans symbols in the same order that llvm-ar adds them to the
index, so in the common case the semantics should be identical. If the archive
symbol table was created in a different order, or is incomplete, this strategy
may have different semantics. Such cases are considered user error.
The `is neither ET_REL nor LLVM bitcode` error is changed to a warning.
Previously an archive may have such members without a diagnostic. Using a
warning prevents breakage.
* For some tests, the diagnostics get improved where we did not consider
the archive member name: `b.a:` => `b.a(b.o):`.
* `no-obj.s`: the link is now allowed, matching GNU ld
* `archive-no-index.s`: the `is neither ET_REL nor LLVM bitcode` diagnostic is
demoted to a warning.
* `incompatible.s`: even when an archive is unextracted, we may report an
"incompatible with" error.
---
I recently decreased sizeof(SymbolUnion) by 8 and decreased memory usage quite a
bit, so retaining `symbols` for un-extracted archive members should not cause a
memory usage problem.
Reviewed By: peter.smith
Differential Revision: https://reviews.llvm.org/D119074
SharedSymbol::SharedSymbol initializes verdefIndex and Symbol::replace
copies verdefIndex.
By move verdefIndex assignment outside of ctor, Symbol::replace can be changed
to not copy verdefIndex. This can be used to decrease work for for
ObjKind/BitcodeKind.
Currently `this->getName() == newSym.getName()`.
By keeping the old nameData/nameSize, newSym's nameData/nameSize will be
ignored. The call sites can avoid calling getName().
printTraceSymbol needs to take the symbol name since `other`'s name is empty.
When processing dependent libraries, if there's a directory of the same
name as the library being searched for, either in the current directory
or earlier in the search order, LLD will try to open it and report an
error. This is because LLD uses file existence check. To address this
issue we reverse the order, searching the library by basename first
and only considering search paths later, and current directory last.
Differential Revision: https://reviews.llvm.org/D118498
StringRefZ does not improve performance. Non-local symbols always have eagerly
computed nameSize. Most local symbols's lengths will be updated in either:
* shouldKeepInSymtab
* SymbolTableBaseSection::addSymbol
Its benefit is offsetted by strlen in every call site (sums up to 5KiB code in a
release x86-64 build), so using StringRefZ may be slower.
In a -s link (uncommon) there is minor speedup, like ~0.3% for clang and chrome.
Reviewed By: alexander-shaposhnikov
Differential Revision: https://reviews.llvm.org/D117644
This simplifies the code a bit. While here,
* change the `multiple relocation sections` diagnostic from `fatal` to `error` and include the relocated section name.
* drop less useful name from `getRelocTarget`. Without -r/--emit-relocs we don't need to get SHT_REL/SHT_RELA names.
Move all variables at file-scope or function-static-scope into a hosting structure (lld::CommonLinkerContext) that lives at lldMain()-scope. Drivers will inherit from this structure and add their own global state, in the same way as for the existing COFFLinkerContext.
See discussion in https://lists.llvm.org/pipermail/llvm-dev/2021-June/151184.html
Differential Revision: https://reviews.llvm.org/D108850
Similar to D62188: a BitcodeFile's symbol table may be iterated twice, once in
--start-lib (lazy) state, and once in the non-lazy state. This patch
makes `parseLazy` save `symbols[i]` so that the non-lazy state does not need to
re-insert to the global symbol table. Avoiding a redundant `saver.save` may save
memory.
`Maximum resident set size (kbytes)` for a large --thinlto-index-only link:
* without the patch: 10164000
* with the patch: 10095716 (0.6% decrease)
Note: we can remove `saver.save` if `BitcodeCompiler::add` does not transfer the ownership
of `f.obj` in `checkError(ltoObj->add(std::move(f.obj), resols));`.
Reviewed By: tejohnson
Differential Revision: https://reviews.llvm.org/D116390
@tejohnson noticed that freeing MemoryBuffer instances right before
`lto->compile` can save RSS, likely because the memory can be reused by
LTO indexing (e.g. ThinLTO import/export lists).).
For ELFFileBase instances, symbol and section names are backed by MemoryBuffer,
so destroying MemoryBuffer would make some infrequent passes (parseSymbolVersion,
reportBackrefs) crash and make debugging difficult.
For a BitcodeFile, its content is completely unused, but destroying its
MemoryBuffer makes the buffer identifier inaccessible and may introduce
constraints for future changes.
This patch leverages madvise(MADV_DONTNEED) which achieves the major gain
without the latent issues.
`Maximum resident set size (kbytes): ` for a large --thinlto-index-only link:
* current behavior: 10146104KiB
* destroy MemoryBuffer instances: 8555240KiB
* madvise(MADV_DONTNEED) just bitcodeFiles and lazyBitcodeFiles: 8737372KiB
* madvise(MADV_DONTNEED) all MemoryBuffers: 8739796KiB (16% decrease)
Depends on D116366
Reviewed By: tejohnson
Differential Revision: https://reviews.llvm.org/D116367
and remove associated make<XXX> calls.
gnuHash and sysvHash are unchanged, otherwise LinkerScript::discard would
destroy the objects which may be referenced by input section descriptions.
My x86-64 lld executable is 121+KiB smaller.
The new `lazy` state is the inverse of the previous `LazyObjFile::extracted`.
There are many advantages:
* previously when a LazyObjFile was extracted, a new ObjFile/BitcodeFile was created; now the file is reused, just with `lazy` cleared
* avoid the confusing transfer of `symbols` from LazyObjFile to the new file
* the `incompatible file:` diagnostic is unified with `is incompatible with`
* simpler code, smaller executable (6200+ bytes smaller on x86-64)
* make eager parsing feasible (for parallel section/symbol table initialization)
Calling `Allocate` with 0 size (when .symtab is absent, e.g.
`invalid/mips-invalid-options-descriptor.test`) may return a nullptr, which will
crash with -fsanitize=null (the underlying `Allocate` function is
LLVM_ATTRIBUTE_RETURNS_NONNULL).
* Avoid the name truncation quirk in SymbolTable::insert: the truncated name will be replaced by @@ again.
* Allow foo and foo@@v1 in different files to be diagnosed as duplicate definition error (GNU ld behavior)
* Avoid potential redundant strlen on symbol name due to StringRefZ in ObjFile<ELFT>::initializeSymbols
Fangrui Song