Commit Graph

39 Commits

Author SHA1 Message Date
Alexandre Ganea 09158252f7 [ThinLTO] Allow usage of all hardware threads in the system
Before this patch, it wasn't possible to extend the ThinLTO threads to all SMT/CMT threads in the system. Only one thread per core was allowed, instructed by usage of llvm::heavyweight_hardware_concurrency() in the ThinLTO code. Any number passed to the LLD flag /opt:lldltojobs=..., or any other ThinLTO-specific flag, was previously interpreted in the context of llvm::heavyweight_hardware_concurrency(), which means SMT disabled.

One can now say in LLD:
/opt:lldltojobs=0 -- Use one std::thread / hardware core in the system (no SMT). Default value if flag not specified.
/opt:lldltojobs=N -- Limit usage to N threads, regardless of usage of heavyweight_hardware_concurrency().
/opt:lldltojobs=all -- Use all hardware threads in the system. Equivalent to /opt:lldltojobs=$(nproc) on Linux and /opt:lldltojobs=%NUMBER_OF_PROCESSORS% on Windows. When an affinity mask is set for the process, threads will be created only for the cores selected by the mask.

When N > number-of-hardware-threads-in-the-system, the threads in the thread pool will be dispatched equally on all CPU sockets (tested only on Windows).
When N <= number-of-hardware-threads-on-a-CPU-socket, the threads will remain on the CPU socket where the process started (only on Windows).

Differential Revision: https://reviews.llvm.org/D75153
2020-03-27 10:20:58 -04:00
Sam Clegg b062fe1816 [lld][WebAssembly] Fail if bitcode objects are pulled in after LTO
This can happen if lto::LTO::getRuntimeLibcallSymbols doesn't return
an complete/accurate list of libcalls.  In this case new bitcode
object can be linked in after LTO.

For example the WebAssembly backend currently calls:
  setLibcallName(RTLIB::FPROUND_F32_F16, "__truncsfhf2");

But `__truncsfhf2` is not part of `getRuntimeLibcallSymbols` so if
this symbol is generated during LTO the link will currently fail.

Without this change the linker crashes because the bitcode symbol
makes it all the way to the output phase.

See: https://bugs.llvm.org/show_bug.cgi?id=44353

Differential Revision: https://reviews.llvm.org/D71632
2020-02-11 17:36:15 -08:00
Sam Clegg 9cd985815a [lld][WebAssembly] Add libcall symbols to the link when LTO is being used.
This code is copied almost verbatim from the equivalent change to the
ELF linker:

- https://reviews.llvm.org/D50017
- https://reviews.llvm.org/D50475

The upshot is that libraries containing libcall (such as compiler-rt
and libc) can be compiled with LTO.

Fixes PR41384

Differential Revision: https://reviews.llvm.org/D71738
2020-01-10 11:01:05 -08:00
Sam Clegg 67b055841f [lld][WebAssebmly] Preserve custom import attributes with LTO
Undefined symbols in WebAssembly can come with custom `import-module`
and `import-field` attributes.  However when reading symbols from
bitcode object files during LTO those curtom attributes are not
available.

Once we compile the LTO object and read in the symbol table from the
object file we have access to these custom attributes.  In this case,
when undefined symbols are added and a symbol already exists in the
SymbolTable we can't simple return it, we may need to update the
symbol's attributes.

Fixes: PR43211

Differential Revision: https://reviews.llvm.org/D68959

llvm-svn: 375081
2019-10-17 05:16:54 +00:00
Amy Huang 7b1d793713 Reland "Change the X86 datalayout to add three address spaces
for 32 bit signed, 32 bit unsigned, and 64 bit pointers."
This reverts 57076d3199.

Original review at https://reviews.llvm.org/D64931.
Review for added fix at https://reviews.llvm.org/D66843.

llvm-svn: 371568
2019-09-10 23:15:38 +00:00
Vlad Tsyrklevich 57076d3199 Revert "Change the X86 datalayout to add three address spaces for 32 bit signed,"
This reverts commit r370083 because it caused check-lld failures on
sanitizer-x86_64-linux-fast.

llvm-svn: 370142
2019-08-28 01:08:54 +00:00
Amy Huang 1299945b81 Change the X86 datalayout to add three address spaces for 32 bit signed,
32 bit unsigned, and 64 bit pointers.

llvm-svn: 370083
2019-08-27 17:46:53 +00:00
Sam Clegg 7c663cde14 [WebAssembly] Improve lto/comdat.ll test. NFC.
We were not previously testing the comdat exclusion in bitcode objects
because we were linking two copies of the .bc file and the
`linkonce_odr` linkage type was removing the duplicate `_start` at
the LTO stage.

Now we link an bitcode and non-bitcode version both of which contains a
copy of _start.  We link them in both orders, which means this test will
fail if comdat exclusion is not working correctly in bitcode parsing.

Differential Revision: https://reviews.llvm.org/D62923

llvm-svn: 362650
2019-06-05 21:08:30 +00:00
Sam Clegg 7d4ec5af6c [WebAssembly] Don't export __data_end and __heap_base by default.
These can still be exported via --export if needed.

Differential Revision: https://reviews.llvm.org/D62744

llvm-svn: 362276
2019-05-31 22:51:59 +00:00
Sam Clegg d506b0a484 [WebAssembly] Fix signatures of undefined function in LTO object which are not called directly.
We recently added special handling for function that are not called
directly but failed to add testing for the LTO case.

See https://reviews.llvm.org/D62153

Differential Revision: https://reviews.llvm.org/D62561

llvm-svn: 361975
2019-05-29 15:36:42 +00:00
Sam Clegg 697f2149f1 [WebAssembly] LTO: Honor comdat groups when loading bitcode files
But don't apply comdat groups when loading the LTO object files.
This is basically the same logic used by the ELF linker.

Differential Revision: https://reviews.llvm.org/D61924

llvm-svn: 360782
2019-05-15 16:03:28 +00:00
Fangrui Song b159906a9a [test] Change llvm-readobj -long-option to --long-option or well-known short options. NFC
Also change some options that have different semantics (cause confusion) in llvm-readelf mode:

-s => -S
-t => --symbols
-sd => --section-data

llvm-svn: 359651
2019-05-01 05:49:01 +00:00
Sam Clegg 815a05ca6b [WebAssembly] LTO: Don't include bitcode-only symbols in the symtab
Fixes https://bugs.llvm.org/show_bug.cgi?id=40654

Differential Revision: https://reviews.llvm.org/D59012

llvm-svn: 355577
2019-03-07 02:43:19 +00:00
Sam Clegg 0e6b42f5eb [WebAssebmly] Allow __wasm_call_ctors to be GC'ed
Differential Revision: https://reviews.llvm.org/D58806

llvm-svn: 355240
2019-03-01 22:35:47 +00:00
Thomas Lively f3b4f99007 [WebAssembly] Remove uses of ThreadModel
Summary:
In the clang UI, replaces -mthread-model posix with -matomics as the
source of truth on threading. In the backend, replaces
-thread-model=posix with the atomics target feature, which is now
collected on the WebAssemblyTargetMachine along with all other used
features. These collected features will also be used to emit the
target features section in the future.

The default configuration for the backend is thread-model=posix and no
atomics, which was previously an invalid configuration. This change
makes the default valid because the thread model is ignored.

A side effect of this change is that objects are never emitted with
passive segments. It will instead be up to the linker to decide
whether sections should be active or passive based on whether atomics
are used in the final link.

Reviewers: aheejin, sbc100, dschuff

Subscribers: mehdi_amini, jgravelle-google, hiraditya, sunfish, steven_wu, dexonsmith, rupprecht, jfb, jdoerfert, cfe-commits, llvm-commits

Tags: #clang, #llvm

Differential Revision: https://reviews.llvm.org/D58742

llvm-svn: 355112
2019-02-28 18:39:08 +00:00
Sam Clegg 6540e57002 [WebAssembly] Don't generate invalid modules when function signatures mismatch
Previously we could emit a warning and generate a potentially invalid
wasm module (due to call sites and functions having conflicting
signatures). Now, rather than create invalid binaries we handle such
cases by creating stub functions containing unreachable, effectively
turning these into runtime errors rather than validation failures.

Differential Revision: https://reviews.llvm.org/D57909

llvm-svn: 354528
2019-02-20 23:19:31 +00:00
Thomas Lively 2e1504091e [WebAssembly] Update MC for bulk memory
Summary:
Rename MemoryIndex to InitFlags and implement logic for determining
data segment layout in ObjectYAML and MC. Also adds a "passive" flag
for the .section assembler directive although this cannot be assembled
yet because the assembler does not support data sections.

Reviewers: sbc100, aardappel, aheejin, dschuff

Subscribers: jgravelle-google, hiraditya, sunfish, rupprecht, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D57938

llvm-svn: 354397
2019-02-19 22:56:19 +00:00
Sam Clegg 230dc11d24 [WebAssembly] Refactor handling of weak undefined functions. NFC.
Also add to the docs.

This is refactor in preparation for https://reviews.llvm.org/D57909

Differential Revision: https://reviews.llvm.org/D57920

llvm-svn: 353478
2019-02-07 22:42:16 +00:00
Sam Clegg 34583e5f55 [WebAssembly] Update test expectations to match llvm-readobj change. NFC.
Differential Revision: https://reviews.llvm.org/D57868

llvm-svn: 353358
2019-02-07 01:21:58 +00:00
Sam Clegg 4d0cc83be7 [WebAssembly] Use wasm-ld rather than lld with -flavor in tests. NFC.
Differential Revision: https://reviews.llvm.org/D57758

llvm-svn: 353187
2019-02-05 16:53:33 +00:00
Sam Clegg b30623ff16 [WebAssembly] Update test to match new readobj output. NFC.
Differential Revision: https://reviews.llvm.org/D57715

llvm-svn: 353105
2019-02-04 22:34:07 +00:00
Sam Clegg 89e4dcb4be [WebAssembly] Fix crash with LTO + relocatable + undefined symbols
Change the way we create the symbol table to be closer to how its done
on ELF.  Now the output symbol table matches the internal symtab order
and includes local and undefined symbols.

Fixes PR40204

Differential Revision: https://reviews.llvm.org/D56947

llvm-svn: 352645
2019-01-30 18:55:15 +00:00
Sam Clegg e320cea5b9 [WebAssembly] Fix undefined weak function symbols in LTO builds
Summary: Fixes PR40219

Subscribers: dschuff, mehdi_amini, inglorion, jgravelle-google, aheejin, sunfish, steven_wu, dexonsmith, llvm-commits

Differential Revision: https://reviews.llvm.org/D57420

llvm-svn: 352575
2019-01-30 00:25:52 +00:00
Michal Gorny 92dc7dce4a [test] Mark atime-based tests unsupported on NetBSD
Mark tests requiring 'touch' to set atime unsupported on NetBSD
due to kernel limitation preventing it from working with noatime.

llvm-svn: 348607
2018-12-07 16:21:08 +00:00
Fangrui Song 7bafaf8ffe Reland "Change the timestamp of llvmcache-foo file to meet the thinLTO prune policy"
Summary:
It is difficult to touch a file with a relative mtime across different OSes as POSIX touch -d is rigid. While we may construct relative timestamps with `date`, POSIX date is inadequate to do so as various OSes' date do not agree on a common format (OpenBSD uses `date -r seconds`, FreeBSD uses `date -v-2M` while GNU accepts `-d '-2 min'`)

Just use python os.utime()

Original description:

    The case may randomly fail if we test it with command "
     while llvm-lit tools/lld/test/ELF/lto/cache.ll; do true; done". It is because the llvmcache-foo file is younger than llvmcache-349F039B8EB076D412007D82778442BED3148C4E and llvmcache-A8107945C65C2B2BBEE8E61AA604C311D60D58D6. But due to timestamp precision reason their timestamp is the same. Given the same timestamp, the file prune policy is to remove bigger size file first, so mostly foo file is removed for its bigger size. And the files size is under threshold after deleting foo file. That's what test case expect.

    However sometimes, the precision is enough to measure that timestamp of llvmcache-349F039B8EB076D412007D82778442BED3148C4E and llvmcache-A8107945C65C2B2BBEE8E61AA604C311D60D58D6 are smaller than foo, so llvmcache-349F039B8EB076D412007D82778442BED3148C4E and llvmcache-A8107945C65C2B2BBEE8E61AA604C311D60D58D6 are deleted first. Since the files size is still above the file size threshold after deleting the 2 files, the foo file is also deleted. And then the test case fails, because it expect only one file should be deleted instead of 3.

    The fix is to change the timestamp of llvmcache-foo file to meet the thinLTO prune policy.
    The same fix is applied to llvm code at https://reviews.llvm.org/D52452.

    Patch by Luo Yuanke.

Reviewers: ruiu, craig.topper, smaslov, Jianping, espindola, LuoYuanke, tejohnson

Reviewed By: tejohnson

Subscribers: sbc100, krytarowski, aheejin, llvm-commits, dexonsmith, steven_wu, arichardson, inglorion, emaste, bjope, rupprecht

Differential Revision: https://reviews.llvm.org/D54039

llvm-svn: 346006
2018-11-02 17:44:30 +00:00
Teresa Johnson 366d7285a1 Revert "Change the timestamp of llvmcache-foo file to meet the thinLTO prune policy"
This reverts commit r345977.

A few bots failing because the invocation of "touch" is not accepted on
a couple other OSes. Specifically the -d argument is not accepted or requires
a different format.

http://lab.llvm.org:8011/builders/lld-x86_64-darwin13/builds/27103/steps/test_lld/logs/stdio
http://lab.llvm.org:8011/builders/lld-x86_64-freebsd/builds/24974/steps/test_lld/logs/stdio

llvm-svn: 345980
2018-11-02 15:22:19 +00:00
Teresa Johnson c8325b4b59 Change the timestamp of llvmcache-foo file to meet the thinLTO prune policy
Summary:

The case may randomly fail if we test it with command "
 while llvm-lit tools/lld/test/ELF/lto/cache.ll; do true; done". It is because the llvmcache-foo file is younger than llvmcache-349F039B8EB076D412007D82778442BED3148C4E and llvmcache-A8107945C65C2B2BBEE8E61AA604C311D60D58D6. But due to timestamp precision reason their timestamp is the same. Given the same timestamp, the file prune policy is to remove bigger size file first, so mostly foo file is removed for its bigger size. And the files size is under threshold after deleting foo file. That's what test case expect.

However sometimes, the precision is enough to measure that timestamp of llvmcache-349F039B8EB076D412007D82778442BED3148C4E and llvmcache-A8107945C65C2B2BBEE8E61AA604C311D60D58D6 are smaller than foo, so llvmcache-349F039B8EB076D412007D82778442BED3148C4E and llvmcache-A8107945C65C2B2BBEE8E61AA604C311D60D58D6 are deleted first. Since the files size is still above the file size threshold after deleting the 2 files, the foo file is also deleted. And then the test case fails, because it expect only one file should be deleted instead of 3.

The fix is to change the timestamp of llvmcache-foo file to meet the thinLTO prune policy.
The same fix is applied to llvm code at https://reviews.llvm.org/D52452.

Patch by Luo Yuanke.

Reviewers: ruiu, craig.topper, smaslov, Jianping, espindola, LuoYuanke

Subscribers: rupprecht, bjope, emaste, inglorion, arichardson, steven_wu, dexonsmith, llvm-commits

Differential Revision: https://reviews.llvm.org/D53123

llvm-svn: 345977
2018-11-02 14:52:10 +00:00
Sam Clegg c1a3b9d51a [WebAssembly] Remove duplicate function. NFC.
Differential Revision: https://reviews.llvm.org/D53963

llvm-svn: 345806
2018-11-01 01:08:37 +00:00
Sam Clegg 8b0b48f343 [WebAssembly] Preserve function signatures during LTO
With LTO when and undefined function (with a known signature)
in replaced by a defined bitcode function we were loosing the
signature information (since bitcode functions don't have
signatures).

With this change we preserve the original signature from the
undefined function and verify that the post LTO compiled
function has the correct signature.

This change improves the error handling in the case where
there is a signature mismatch with a function defined in
a bitcode file.

Differential Revision: https://reviews.llvm.org/D50721

llvm-svn: 343340
2018-09-28 16:50:14 +00:00
Sam Clegg 305b0343ce [WebAssembly] Add --[no]-export-dynamic to replace --export-default
In a very recent change I introduced a --no-export-default flag
but after conferring with others it seems that this feature already
exists in gnu GNU ld and lld in the form the --export-dynamic flag
which is off by default.

This change replaces export-default with export-dynamic and also
changes the default to match the traditional linker behaviour.

Now, by default, only the entry point is exported.  If other symbols
are required by the embedder then --export-dynamic or --export can
be used to export all visibility hidden symbols or individual
symbols respectively.

This change touches a lot of tests that were relying on symbols
being exported by default.  I imagine it will also effect many
users but do think the change is worth it match of the traditional
behaviour and flag names.

Differential Revision: https://reviews.llvm.org/D52587

llvm-svn: 343265
2018-09-27 21:06:25 +00:00
Stella Stamenova 38eee643bd [lit, python] Always add quotes around the python path in lit
Summary:
The issue with the python path is that the path to python on Windows can contain spaces. To make the tests always work, the path to python needs to be surrounded by quotes.

This is a companion change to: https://reviews.llvm.org/D50206

Reviewers: asmith, zturner, espindola

Subscribers: emaste, sbc100, arichardson, aheejin, steven_wu, dexonsmith, llvm-commits

Differential Revision: https://reviews.llvm.org/D50282

llvm-svn: 339075
2018-08-06 22:37:49 +00:00
Sam Clegg 47e2b6b29e [WebAssembly] Don't error when --undefined symbols are not found
This matches the behavior of the ELF linker where -u/--undefined
means symbols will get pulled in from archives but won't result
in link error if they are missing.

Also, don't actually great symbol table entries for the undefined
symbols, again matching more closely the ELF linker.

This also results in simplification of the code.

Differential Revision: https://reviews.llvm.org/D50279

llvm-svn: 338938
2018-08-04 00:04:06 +00:00
Sam Clegg f989a92dea [WebAssemlby] Set IsUsedInRegularObj correctly for undefined data symbols
Differential Revision: https://reviews.llvm.org/D49113

llvm-svn: 337314
2018-07-17 19:15:02 +00:00
Sam Clegg b2144058a6 [LTO] Errors in LLVM backend should manifest as lld errors
Differential Revision: https://reviews.llvm.org/D48812

llvm-svn: 336143
2018-07-02 21:01:43 +00:00
Sam Clegg 92c10492de [WebAssembly] Set threadmodel during LTO
Subscribers: dschuff, mehdi_amini, inglorion, jgravelle-google, aheejin, sunfish, steven_wu, llvm-commits

Differential Revision: https://reviews.llvm.org/D48689

llvm-svn: 336118
2018-07-02 16:27:50 +00:00
Sam Clegg 52b2456a95 [WebAssembly] Fix --export of LTO symbols
Differential Revision: https://reviews.llvm.org/D48697

llvm-svn: 335881
2018-06-28 17:21:46 +00:00
Sam Clegg ce004bfe35 [WebAssembly] Add support for bitcode archive members
This change effects the behavior of --export-all. Previously
--export-all would only effect symbols that survived GC. Now
--export-all will prevent any non-local symbols from being GCed.

Differential Revision: https://reviews.llvm.org/D48673

llvm-svn: 335878
2018-06-28 17:04:58 +00:00
Sam Clegg cefbf9aca1 [WebAssembly] LTO: Fix signatures of undefined functions in bitcode
Function symbols that come from bitcode have not signatures.
After LTO when the real symbols are read in we need to make
sure that we set the signature on the existing symbol.
the signature-less undefined functions.

Differential Revision: https://reviews.llvm.org/D48693

llvm-svn: 335875
2018-06-28 16:53:53 +00:00
Sam Clegg c729c1b47d [WebAssembly] Initial support for LTO
Differential Revision: https://reviews.llvm.org/D47162

llvm-svn: 333570
2018-05-30 18:07:52 +00:00