Essentially takes the lld/Common/Threads.h wrappers and moves them to
the llvm/Support/Paralle.h algorithm header.
The changes are:
- Remove policy parameter, since all clients use `par`.
- Rename the methods to `parallelSort` etc to match LLVM style, since
they are no longer C++17 pstl compatible.
- Move algorithms from llvm::parallel:: to llvm::, since they have
"parallel" in the name and are no longer overloads of the regular
algorithms.
- Add range overloads
- Use the sequential algorithm directly when 1 thread is requested
(skips task grouping)
- Fix the index type of parallelForEachN to size_t. Nobody in LLVM was
using any other parameter, and it made overload resolution hard for
for_each_n(par, 0, foo.size(), ...) because 0 is int, not size_t.
Remove Threads.h and update LLD for that.
This is a prerequisite for parallel public symbol processing in the PDB
library, which is in LLVM.
Reviewed By: MaskRay, aganea
Differential Revision: https://reviews.llvm.org/D79390
Otherwise an ArgumentParser is constructed for every directive section,
and that involves copying the entire table of options into a vector.
There is no need for this, just have one option table.
This speeds up linking chrome.dll with PGO instrumentation by 13%
(154271ms -> 134033ms).
LLVM's Option library is very slow. In particular, it allocates at least
one large-ish heap object (Arg) for every argument. When PGO
instrumentation is enabled, all the __profd_* symbols are added to the
@llvm.used list, which compiles down to these /INCLUDE: directives. This
means we have O(#symbols) directives to parse in the section, so we end
up allocating an Arg for every function symbol in the object file. This
is unnecessary.
To address the issue and speed up the link, extend the fast path that we
already have for /EXPORT:, which has similar scaling issues.
I promise that I took a hard look at optimizing the Option library, but
its data structures are very general and would need a lot of cleanup. We
have accumulated lots of optional features (option groups, aliases,
multiple values) over the years, and these are now properties of every
parsed argument, when the vast majority of arguments do not use these
features.
Reviewed By: thakis
Differential Revision: https://reviews.llvm.org/D78845
Summary:
/PDBSTREAM:<name>=<file> adds the contents of <file> to stream <name> in the resulting PDB.
This allows native uses with workflows that (for example) add srcsrv streams to PDB files to provide a location for the build's source files.
Results should be equivalent to linking with lld-link, then running Microsoft's pdbstr tool with the command line:
pdbstr.exe -w -p:<PDB LOCATION> -s:<name> -i:<file>
except in cases where the named stream overlaps with a default named stream, such as "/names". In those cases, the added stream will be overridden, making the /pdbstream option a no-op.
Reviewers: thakis, rnk
Reviewed By: thakis
Differential Revision: https://reviews.llvm.org/D77310
--no-threads is a name copied from gold.
gold has --no-thread, --thread-count and several other --thread-count-*.
There are needs to customize the number of threads (running several lld
processes concurrently or customizing the number of LTO threads).
Having a single --threads=N is a straightforward replacement of gold's
--no-threads + --thread-count.
--no-threads is used rarely. So just delete --no-threads instead of
keeping it for compatibility for a while.
If --threads= is specified (ELF,wasm; COFF /threads: is similar),
--thinlto-jobs= defaults to --threads=,
otherwise all available hardware threads are used.
There is currently no way to override a --threads={1,2,...}. It is still
a debate whether we should use --threads=all.
Reviewed By: rnk, aganea
Differential Revision: https://reviews.llvm.org/D76885
Before this patch, it wasn't possible to extend the ThinLTO threads to all SMT/CMT threads in the system. Only one thread per core was allowed, instructed by usage of llvm::heavyweight_hardware_concurrency() in the ThinLTO code. Any number passed to the LLD flag /opt:lldltojobs=..., or any other ThinLTO-specific flag, was previously interpreted in the context of llvm::heavyweight_hardware_concurrency(), which means SMT disabled.
One can now say in LLD:
/opt:lldltojobs=0 -- Use one std::thread / hardware core in the system (no SMT). Default value if flag not specified.
/opt:lldltojobs=N -- Limit usage to N threads, regardless of usage of heavyweight_hardware_concurrency().
/opt:lldltojobs=all -- Use all hardware threads in the system. Equivalent to /opt:lldltojobs=$(nproc) on Linux and /opt:lldltojobs=%NUMBER_OF_PROCESSORS% on Windows. When an affinity mask is set for the process, threads will be created only for the cores selected by the mask.
When N > number-of-hardware-threads-in-the-system, the threads in the thread pool will be dispatched equally on all CPU sockets (tested only on Windows).
When N <= number-of-hardware-threads-on-a-CPU-socket, the threads will remain on the CPU socket where the process started (only on Windows).
Differential Revision: https://reviews.llvm.org/D75153
Added support for /map and /map:[filepath].
The output was derived from Microsoft's Link.exe output when using that same option.
Note that /MAPINFO support was not added.
The previous implementation of MapFile.cpp/.h was meant for /lldmap, and was renamed to LLDMapFile.cpp/.h
MapFile.cpp/.h is now for /MAP
However, a small fix was added to lldmap, replacing a std::sort with std::stable_sort to enforce reproducibility.
Differential Revision: https://reviews.llvm.org/D70557
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.
This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.
This doesn't actually modify StringRef yet, I'll do that in a follow-up.
Remove the lld::enableColors function, as it just obscures which
stream it's affecting, and replace with explicit calls to the stream's
enable_colors.
Also, assign the stderrOS and stdoutOS globals first in link function,
just to ensure nothing might use them.
(Either change individually fixes the issue of using the old
stream, but both together seems best.)
Follow-up to b11386f9be.
Differential Revision: https://reviews.llvm.org/D70492
This change is for those who use lld as a library. Context:
https://reviews.llvm.org/D70287
This patch adds a new parmeter to lld::*::link() so that we can pass
an raw_ostream object representing stdout. Previously, lld::*::link()
took only an stderr object.
Justification for making stdoutOS and stderrOS mandatory: I wanted to
make link() functions to take stdout and stderr in that order.
However, if we change the function signature from
bool link(ArrayRef<const char *> args, bool canExitEarly,
raw_ostream &stderrOS = llvm::errs());
to
bool link(ArrayRef<const char *> args, bool canExitEarly,
raw_ostream &stdoutOS = llvm::outs(),
raw_ostream &stderrOS = llvm::errs());
, then the meaning of existing code that passes stderrOS silently
changes (stderrOS would be interpreted as stdoutOS). So, I chose to
make existing code not to compile, so that developers can fix their
code.
Differential Revision: https://reviews.llvm.org/D70292
/align is not supposed to be used without /driver, so it makes sense
to warn if only /align is passed. MSVC link.exe warns on this too.
Differential Revision: https://reviews.llvm.org/D70163
Recent versions of Microsoft's dumpbin tool cannot handle such PE files.
LLVM tools and GNU tools can, and use this to encode long section names
like ".debug_info", which is commonly used for DWARF. Don't do this in
mingw mode or when -debug:dwarf is passed, since the user probably wants
long section names for DWARF sections.
PR43754
Reviewers: ruiu, mstorsjo
Differential Revision: https://reviews.llvm.org/D69594
Also mention "basename" and "dirname" in Path.h since I tried
to find these functions by looking for these strings. It might
help others find them faster if the comments contain these strings.
No behavior change.
Differential Revision: https://reviews.llvm.org/D69458
This patch adds /reproduce:<path> option to lld/COFF. This is an
lld-specific option, so we can name it freely. I chose /reproduce
over other names (e.g. /lldlinkrepro) for consistency with other lld
ports.
Differential Revision: https://reviews.llvm.org/D68381
llvm-svn: 373704
This reverts commit r371729 because /linkrepro option also exists
in Microsoft link.exe and their linker takes not a filename but a
directory name as an argument for /linkrepro.
Differential Revision: https://reviews.llvm.org/D68378
llvm-svn: 373703
This is useful for enforcing that builds are independent of the
environment; it can be used when all system library paths are added
via /libpath: already. It's similar ot cl.exe's /X flag.
Since it should also affect %LINK% (the other caller of
`Process::GetEnv` in lld/COFF), the early-option-parsing needs
to move around a bit. The options are:
- Add a manual loop over the argv ArrayRef and look for "/lldignoreenv".
This repeats the name of the flag in both Options.td and in
DriverUtils.cpp.
- Add yet another table.ParseArgs() call just for /lldignoreenv before
adding %LINK%.
- Use the existing early ParseArgs() that's there for --rsp-quoting and use
it for /lldignoreenv for %LINK% as well. This means --rsp-quoting
and /lldignoreenv can't be passed via %LINK%.
I went with the third approach.
Differential Revision: https://reviews.llvm.org/D67456
llvm-svn: 371852
from thin archives.
Currently lld adds the archive name to MemoryBufferRef identifiers in order to
ensure they are unique. For thin archives, since the file name is already unique and we
want to keep the original path to the file, don't add the archive name.
Differential Revision: https://reviews.llvm.org/D67295
llvm-svn: 371778
This makes lld-link behave like ld.lld. I don't see a reason for
the two drivers to have different behavior here.
While here, also make lld-link add a version.txt to the tar, like
ld.lld does.
Differential Revision: https://reviews.llvm.org/D67461
llvm-svn: 371729
Patch by Markus Böck.
PR42951: When linking an archive with members that have the same name linking
fails when using the -wholearchive option. This patch passes the index
of the member in the archive to the offset parameter to disambiguate the
member.
Differential Revision: https://reviews.llvm.org/D66239
llvm-svn: 371509
Summary:
This is a re-land of r370487 with a fix for the use-after-free bug
that rev contained.
This implements -start-lib and -end-lib flags for lld-link, analogous
to the similarly named options in ld.lld. Object files after
-start-lib are included in the link only when needed to resolve
undefined symbols. The -end-lib flag goes back to the normal behavior
of always including object files in the link. This mimics the
semantics of static libraries, but without needing to actually create
the archive file.
Reviewers: ruiu, smeenai, MaskRay
Reviewed By: ruiu, MaskRay
Subscribers: akhuang, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D66848
llvm-svn: 370816
Summary:
This implements -start-lib and -end-lib flags for lld-link, analogous
to the similarly named options in ld.lld. Object files after
-start-lib are included in the link only when needed to resolve
undefined symbols. The -end-lib flag goes back to the normal behavior
of always including object files in the link. This mimics the
semantics of static libraries, but without needing to actually create
the archive file.
Reviewers: ruiu, smeenai, MaskRay
Reviewed By: ruiu, MaskRay
Subscribers: akhuang, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D66848
llvm-svn: 370487
Extend WindowsResourceParser to support using a ResourceSectionRef for
loading resources from an object file.
Only allow merging resource object files in mingw mode; keep the
existing error on multiple resource objects in link mode.
If there only is one resource object file and no .res resources,
don't parse and recreate the .rsrc section, but just link it in without
inspecting it. This allows users to produce any .rsrc section (outside
of what the parser supports), just like before. (I don't have a specific
need for this, but it reduces the risk of this new feature.)
Separate out the .rsrc section chunks in InputFiles.cpp, and only include
them in the list of section chunks to link if we've determined that there
only was one single resource object. (We need to keep other chunks from
those object files, as they can legitimately contain other sections as
well, in addition to .rsrc section chunks.)
Differential Revision: https://reviews.llvm.org/D66824
llvm-svn: 370436
Summary:
This adds the -lto-obj-path option to lld-link. This can be
used to specify a path at which to write a native object file for
the full LTO part when using LTO unit splitting.
Reviewers: ruiu, tejohnson, pcc, rnk
Reviewed By: ruiu, rnk
Subscribers: mehdi_amini, steven_wu, dexonsmith, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D65964
llvm-svn: 369559
This is used by Wine for manually crafting export tables.
If the input object contains .edata sections, GNU ld references them
in the export directory instead of synthesizing an export table using
either export directives or the normal auto export mechanism. (AFAIK,
historically, way way back, GNU ld didn't support synthesizing the
export table - one was supposed to generate it using dlltool and link
it in instead.)
If faced with --out-implib and --output-def, GNU ld still populates
those output files with the same export info as it would have generated
otherwise, disregarding the input .edata. As this isn't an intended
usage combination, I'm not adding checks for that in tests.
Differential Revision: https://reviews.llvm.org/D65903
llvm-svn: 369358
1. raw_ostream supports ANSI colors so that you can write messages to
the termina with colors. Previously, in order to change and reset
color, you had to call `changeColor` and `resetColor` functions,
respectively.
So, if you print out "error: " in red, for example, you had to do
something like this:
OS.changeColor(raw_ostream::RED);
OS << "error: ";
OS.resetColor();
With this patch, you can write the same code as follows:
OS << raw_ostream::RED << "error: " << raw_ostream::RESET;
2. Add a boolean flag to raw_ostream so that you can disable colored
output. If you disable colors, changeColor, operator<<(Color),
resetColor and other color-related functions have no effect.
Most LLVM tools automatically prints out messages using colors, and
you can disable it by passing a flag such as `--disable-colors`.
This new flag makes it easy to write code that works that way.
Differential Revision: https://reviews.llvm.org/D65564
llvm-svn: 367649
The Archive object created when loading an archive specified with
wholearchive got cleaned up immediately, when the owning std::unique_ptr
went out of scope, even if persisted StringRefs pointed to memory that
belonged to the archive, which no longer was mapped in memory.
This hasn't been an issue with regular (as opposed to thin) archives,
as references to the member objects has kept the mapping for the whole
archive file alive - but with thin archives, all such references point
to other files.
Add the std::unique_ptr to the arena allocator, to retain it as long
as necessary.
This fixes (the last issue raised in) PR42388.
Differential Revision: https://reviews.llvm.org/D65565
llvm-svn: 367599
Summary:
This allows reporting undefined symbols before LTO codegen is
run. Since LTO codegen can take a long time, this improves user
experience by avoiding that time spend if the link is going to
fail with undefined symbols anyway.
Fixes PR32400.
Reviewers: ruiu
Reviewed By: ruiu
Subscribers: mehdi_amini, steven_wu, dexonsmith, mstorsjo, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D62434
llvm-svn: 367136
This ports r366573 from COFF to ELF.
There are now to toString(Archive::Symbol), one doing MSVC demangling
in COFF and one doing Itanium demangling in ELF, so rename these two
to toCOFFString() and to toELFString() to not get a duplicate symbol.
Nothing ever passes a raw Archive::Symbol to CHECK(), so these not
being part of the normal toString() machinery seems ok.
There are two code paths in the ELF linker that emits this type of
diagnostic:
1. The "normal" one in InputFiles.cpp. This is covered by the tweaked test.
2. An additional one that's only used for libcalls if there's at least
one bitcode in the link, and if the libcall symbol is lazy, and
lazily loaded from an archive (i.e. not from a lazy .o file).
(This code path was added in r339301.) Since all libcall names so far
are C symbols and never mangled, the change there is not observable
and hence not covered by tests.
Differential Revision: https://reviews.llvm.org/D65095
llvm-svn: 366836
Also add test coverage for thin archives (which are the only way I could
come up with to test at least some of the diagnostic changes).
Differential Revision: https://reviews.llvm.org/D64927
llvm-svn: 366573
Summary:
Fixes PR41828. Before this, LLD always emitted SafeSEH chunks and
defined __safe_se_handler_table & size. Now, /safeseh:no leaves those
undefined.
Additionally, we were checking for the safeseh @feat.00 flag in two
places: once to emit errors, and once during safeseh table construction.
The error was set up to be off by default, but safeseh is supposed to be
on by default. I combined the two checks, so now LLD emits an error if
an input object lacks @feat.00 and safeseh is enabled. This caused the
majority of 32-bit LLD tests to fail, since many test input object files
lack @feat.00 symbols. I explicitly added -safeseh:no to those tests to
preserve behavior.
Finally, LLD no longer sets IMAGE_DLL_CHARACTERISTICS_NO_SEH if any
input file wasn't compiled for safeseh.
Reviewers: mstorsjo, ruiu, thakis
Reviewed By: ruiu, thakis
Subscribers: llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D63570
llvm-svn: 366238
Summary:
Adds the following two options to lld-link:
-thinlto-prefix-replace: allows replacing a prefix in paths generated
for ThinLTO. This can be used to ensure index files and native object
files are stored in unique directories, allowing multiple distributed
ThinLTO links to proceed concurrently.
-thinlto-object-suffix-replace: allows replacing a suffix in object
file paths involved in ThinLTO. This allows minimized index files to
be used for the thin link while storing the paths to the full bitcode
files for subsequent steps (code generation and final linking).
Reviewers: ruiu, tejohnson, pcc, rnk
Subscribers: mehdi_amini, steven_wu, dexonsmith, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D64542
llvm-svn: 365807
Summary:
This implements -thinlto-index-only, -thinlto-index-only:,
and -thinlto-emit-imports-files options in lld-link. They are
analogous to their counterparts in ld.lld: -thinlto-index-only
causes us to perform ThinLTO's thin link and write index files,
but not perform code generation. -thinlto-index-only: does the
same, but also writes a text file listing the native object
files expected to be generated. -thinlto-emit-imports-files
creates a text file next to each index file, listing the files
to import from.
Reviewers: ruiu, tejohnson, pcc, rnk
Subscribers: mehdi_amini, steven_wu, dexonsmith, arphaman, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D64461
llvm-svn: 365800
Reid Kleckner