maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity
llvm-project/llvm/unittests
History
Nick Terrell 6814232429 [LLD][ELF] Support --[no-]mmap-output-file with F_no_mmap 
Summary:
Add a flag `F_no_mmap` to `FileOutputBuffer` to support
`--[no-]mmap-output-file` in ELF LLD. LLD currently explicitly ignores
this flag for compatibility with GNU ld and gold.

We need this flag to speed up link time for large binaries in certain
scenarios. When we link some of our larger binaries we find that LLD
takes 50+ GB of memory, which causes memory pressure. The memory
pressure causes the VM to flush dirty pages of the output file to disk.
This is normally okay, since we should be flushing cold pages. However,
when using BtrFS with compression we need to write 128KB at a time when
we flush a page. If any page in that 128KB block is written again, then
it must be flushed a second time, and so on. Since LLD doesn't write
sequentially this causes write amplification. The same 128KB block will
end up being flushed multiple times, causing the linker to many times
more IO than necessary. We've observed 3-5x faster builds with
-no-mmap-output-file when we hit this scenario.

The bad scenario only applies to compressed filesystems, which group
together multiple pages into a single compressed block. I've tested
BtrFS, but the problem will be present for any compressed filesystem
on Linux, since it is caused by the VM.

Silently ignoring --no-mmap-output-file caused a silent regression when
we switched from gold to lld. We pass --no-mmap-output-file to fix this
edge case, but since lld silently ignored the flag we didn't realize it
wasn't being respected.

Benchmark building a 9 GB binary that exposes this edge case. I linked 3
times with --mmap-output-file and 3 times with --no-mmap-output-file and
took the average. The machine has 24 cores @ 2.4 GHz, 112 GB of RAM,
BtrFS mounted with -compress-force=zstd, and an 80% full disk.

| Mode    | Time  |
|---------|-------|
| mmap    | 894 s |
| no mmap | 126 s |

When compression is disabled, BtrFS performs just as well with and
without mmap on this benchmark.

I was unable to reproduce the regression with any binaries in
lld-speed-test.

Reviewed By: ruiu, MaskRay

Differential Revision: https://reviews.llvm.org/D69294
 		2019-10-29 15:49:08 -07:00
..
ADT [APInt] Introduce APIntOps::GetMostSignificantDifferentBit() 2019-10-26 23:20:58 +03:00
Analysis [LOOPGUARD] Remove asserts in getLoopGuardBranch 2019-10-06 16:39:43 +00:00
AsmParser Update the file headers across all of the LLVM projects in the monorepo 2019-01-19 08:50:56 +00:00
BinaryFormat [BinaryFormat] Teach identify_magic about Tapi files. 2019-08-21 21:00:16 +00:00
Bitcode Add AutoUpgrade function to add new address space datalayout string to existing datalayouts. 2019-09-18 22:15:58 +00:00
Bitstream [Bitcode] Move Bitstream to a separate library 2019-07-03 22:40:07 +00:00
CodeGen Recommit "Add a heap alloc site marker field to the ExtraInfo in MachineInstrs" 2019-10-28 16:59:32 -07:00
DebugInfo [Mips] Use appropriate private label prefix based on Mips ABI 2019-10-23 12:24:35 +02:00
Demangle Update more file headers across all of the LLVM projects in the monorepo 2019-01-19 10:56:40 +00:00
ExecutionEngine [Mips] Use appropriate private label prefix based on Mips ABI 2019-10-23 12:24:35 +02:00
FuzzMutate [llvm] Migrate llvm::make_unique to std::make_unique 2019-08-15 15:54:37 +00:00
IR [FPEnv] Teach the IRBuilder about correct use of the strictfp attribute. 2019-10-25 12:57:52 -04:00
LineEditor Update the file headers across all of the LLVM projects in the monorepo 2019-01-19 08:50:56 +00:00
Linker [llvm] Migrate llvm::make_unique to std::make_unique 2019-08-15 15:54:37 +00:00
MC [Mips] Use appropriate private label prefix based on Mips ABI 2019-10-23 12:24:35 +02:00
MI LiveIntervals: Fix handleMoveUp with subreg def moving across a def 2019-10-18 23:24:25 +00:00
Object Add ExceptionStream to llvm::Object::minidump 2019-10-18 14:43:15 +00:00
ObjectYAML Update MinidumpYAML to use minidump::Exception for exception stream 2019-10-18 14:56:19 +00:00
Option Option spell checking: Penalize delimiter flags if input has no argument 2019-05-01 16:45:15 +00:00
Passes Update the file headers across all of the LLVM projects in the monorepo 2019-01-19 08:50:56 +00:00
ProfileData [SampleFDO] Add profile remapping support for profile on-demand loading used 2019-10-18 22:35:20 +00:00
Remarks [Remarks] Fix warning for ambigous `else` behind EXPECT macro 2019-10-16 16:43:34 +00:00
Support [LLD][ELF] Support --[no-]mmap-output-file with F_no_mmap 2019-10-29 15:49:08 -07:00
TableGen [UnitTests] Try and pacify gcc-5 2019-10-05 08:57:17 +00:00
Target [PGO][PGSO] SizeOpts changes. 2019-10-28 12:57:26 -07:00
TextAPI Reland "[TextAPI] Introduce TBDv4" 2019-10-10 04:24:44 +00:00
Transforms [PGO][PGSO] SizeOpts changes. 2019-10-28 12:57:26 -07:00
XRay [llvm] Migrate llvm::make_unique to std::make_unique 2019-08-15 15:54:37 +00:00
tools [Mips][llvm-exegesis] Add a Mips target 2019-10-11 20:26:08 +00:00
CMakeLists.txt [gicombiner] Add a CodeExpander to handle C++ fragments with variable expansion 2019-10-03 19:13:39 +00:00
unittest.cfg.in Add support for unittest inputs. 2018-09-05 23:30:17 +00:00