Similar to qsort, bsearch can be called from non-instrumented
code of glibc. When it happends tls for arguments can be in uninitialized
state.
Unlike to qsort, bsearch does not move data, so we don't need to
check or initialize searched memory or key. Intrumented comparator will
do that on it's own.
Differential Revision: https://reviews.llvm.org/D107387
This fixes support for merging profiles which broke as a consequence
of e50a38840d. The issue was missing
adjustment in merge logic to account for the binary IDs which are
now included in the raw profile just after header.
In addition, this change also:
* Includes the version in module signature that's used for merging
to avoid accidental attempts to merge incompatible profiles.
* Moves the binary IDs size field after version field in the header
as was suggested in the review.
Differential Revision: https://reviews.llvm.org/D107143
This fixes support for merging profiles which broke as a consequence
of e50a38840d. The issue was missing
adjustment in merge logic to account for the binary IDs which are
now included in the raw profile just after header.
In addition, this change also:
* Includes the version in module signature that's used for merging
to avoid accidental attempts to merge incompatible profiles.
* Moves the binary IDs size field after version field in the header
as was suggested in the review.
Differential Revision: https://reviews.llvm.org/D107143
As code diverge from Google style we need
to add more and more exceptions to suppress
conflicts with clang-format and clang-tidy.
As this point it does not provide a additional value.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D107197
This fixes support for merging profiles which broke as a consequence
of e50a38840d. The issue was missing
adjustment in merge logic to account for the binary IDs which are
now included in the raw profile just after header.
In addition, this change also:
* Includes the version in module signature that's used for merging
to avoid accidental attempts to merge incompatible profiles.
* Moves the binary IDs size field after version field in the header
as was suggested in the review.
Differential Revision: https://reviews.llvm.org/D107143
The test contains a race in memset.
The size of reported race depends on how the accessed
memory range split into granules inside of tsan runtime.
The test used to report access of size 8, because presumably
the buffer ended up being aligned to 8 bytes. But after
some unrelated changes this test started to report accesses
of size 1 (presumably .data layout changed), which makes
the test fail.
Guarantee alignment of the buf object explicitly.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107149
The updated lots_of_threads.c test with 300 threads
started running for too long on machines with low
hardware parallelism (e.g. taskset -c 0-1).
On lots of CPUs it finishes in ~2 secs. But with
taskset -c 0-1 it runs for hundreds of seconds
effectively spinning in the barrier in the sleep loop.
We now have the handy futex API in sanitizer_common.
Use it instead of the passive spin loop.
It makes the test run only faster with taskset -c 0-1,
it runs for ~1.5 secs, while with full parallelism
it still runs for ~2 secs (but consumes less CPU time).
Depends on D107131.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D107132
The test contains a race in read/write syscalls.
The size of reported race depends on how the accessed
memory range split into granules inside of tsan runtime.
The test used to report access of size 8, because presumably
the buffer ended up being aligned to 8 bytes. But after
some unrelated changes this test started to report accesses
of size 1 (presumably .data layout changed), which makes
the test fail.
Guarantee alignment of the buf object explicitly.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D107131
Add a test where atomic-release happens while
another thread spins calling load-acquire.
This can expose some interesting interleavings
of release and acquire.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107055
We reliably remove bottom libc-guts frames only on linux/glibc.
Some bots failed on this test showing other bottom frames:
.annobin_libc_start.c libc-start.c (libc.so.6+0x249f4)
generic_start_main.isra.0 libc-start.c (libc.so.6+0x45b0c)
We can't reliably remove all of possible bottom frames.
So remove the assertion for that.
Differential Revision: https://reviews.llvm.org/D107037
Currently __tsan_atomic* functions do FuncEntry/Exit using caller PC
and then use current PC (pointing to __tsan_atomic* itself) during
memory access handling. As the result the top function in reports
involving atomics is __tsan_atomic* and the next frame points to user code.
Remove FuncEntry/Exit in atomic functions and use caller PC
during memory access handling. This removes __tsan_atomic*
from the top of report stacks, so that they point right to user code.
The motivation for this is performance.
Some atomic operations are very hot (mostly loads),
so removing FuncEntry/Exit is beneficial.
This also reduces thread trace consumption (1 event instead of 3).
__tsan_atomic* at the top of the stack is not necessary
and does not add any new information. We already say
"atomic write of size 4", "__tsan_atomic32_store" does not add
anything new.
It also makes reports consistent between atomic and non-atomic
accesses. For normal accesses we say "previous write" and point
to user code; for atomics we say "previous atomic write" and now
also point to user code.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106966
We strip all frames below main but in some cases it may be not enough.
Namely, when main is instrumented but does not call any other instrumented code.
In this case __tsan_func_entry in main obtains PC pointing to __libc_start_main
(as we pass caller PC to __tsan_func_entry), but nothing obtains PC pointing
to main itself (as main does not call any instrumented code).
In such case we will not have main in the stack, and stripping everything
below main won't work.
So strip __libc_start_main explicitly as well.
But keep stripping of main because __libc_start_main is glibc/linux-specific,
so looking for main is more reliable (and usually main is present in stacks).
Depends on D106957.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106958
Caller PC is plain harmful as native caller PC has nothing to do with Java code.
Current PC is not particularly useful and may be somewhat confusing for Java users
as it makes top frame to point to some magical __tsan function.
But obtaining and using these PCs adds runtime cost for every java event.
Remove these PCs. Rely only on official Java frames.
It makes execution faster, code simpler and reports better.
Depends on D106956.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106957
We maintain information about Java allocations,
but for some reason never printed it in reports.
Print it.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106956
1. Add a set of micro benchmarks for memory accesses,
mem* functions and unaligned accesses.
2. Add support for multiple benchmarks in a single binary
(or it would require 12 new benchmark binaries).
3. Remove the "clock growth" machinery,
it affects the current tsan runtime by increasing size of
all vector clocks, but this won't be relevant for the new
tsan runtime.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106961
The current 10 threads is not particularly "lots" and not stressful.
Create 10x300 threads and ensure they all are running at the same time.
Depends on D106953.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D106954
Test that we report the warning for free()
and ensure the test finishes as we usually do with "DONE".
Depends on D106951.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D106952
Compilers don't like attributes in this position:
warning: GCC does not allow 'noinline' attribute in this position on a function definition
error: attributes are not allowed on a function-definition
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D106951
This reverts commit 1e1f752027.
It breaks ignore_noninstrumented_modules=1.
Somehow we did not have any portable tests for this mode before
(only Darwin tests). Add a portable test as well.
Moreover, I think I was too fast uninlining all LibIgnore checks.
For Java, Darwin and OpenMP LibIgnore is always enabled,
so it makes sense to leave it as it was before.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D106855
Add the ability to:
1. tell simctl to wait for debugger when spawning process
2. print the command that is called to launch the process
Reviewed By: delcypher
Differential Revision: https://reviews.llvm.org/D106700
signal(2) and sigaction(2) have defined behaviors for invalid signal number
(EINVAL) and some programs rely on it.
The added test case also reveals that MSAN is too strict in this regard.
Test case passed on x86_64 Linux and AArch64 Linux.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106468
This reverts commit 6b2a96285b.
The ccache builders are still failing. Looks like they need to be updated to
get the llvm-zorg config change in 490633945677656ba75d42ff1ca9d4a400b7b243.
I'll re-apply this as soon as the builders are updated.
This reapplies commit a7733e9556 ("Re-apply
[ORC][ORC-RT] Add initial native-TLV support to MachOPlatform."), and
d4abdefc99 ("[ORC-RT] Rename macho_tlv.x86-64.s
to macho_tlv.x86-64.S (uppercase suffix)").
These patches were reverted in 48aa82cacb while I
investigated bot failures (e.g.
https://lab.llvm.org/buildbot/#/builders/109/builds/18981). The fix was to
disable building of the ORC runtime on buliders using ccache (which is the same
fix used for other compiler-rt projects containing assembly code). This fix was
commited to llvm-zorg in 490633945677656ba75d42ff1ca9d4a400b7b243.
Since d564cfb53c moved
__hwasan_tag_mismatch4 this test has been reporting
a frame 0 of __hwasan_tag_mismatch_v2.
This failure can be seen on our bots:
https://lab.llvm.org/buildbot/#/builders/185/builds/170
Before the change:
#0 0xaaaaba100e40 in main <...>/register-dump-read.c:21:10
After the change:
#0 0xaaaab8494bec in __hwasan_tag_mismatch_v2 <...>/hwasan/hwasan_tag_mismatch_aarch64.S:147
#1 0xaaaab84b4df8 in main <..>/register-dump-read.c:14:10
Update the test to check for a main frame as either frame
0 or frame 1.
This reverts commit d4abdefc99 ("[ORC-RT] Rename
macho_tlv.x86-64.s to macho_tlv.x86-64.S (uppercase suffix)", and
a7733e9556 ("Re-apply "[ORC][ORC-RT] Add initial
native-TLV support to MachOPlatform."), while I investigate failures on
ccache builders (e.g. https://lab.llvm.org/buildbot/#/builders/109/builds/18981)
Reapplies fe1fa43f16, which was reverted in
6d8c63946c, with fixes:
1. Remove .subsections_via_symbols directive from macho_tlv.x86-64.s (it's
not needed here anyway).
2. Return error from pthread_key_create to the MachOPlatform to silence unused
variable warning.
Adds code to LLVM (MachOPlatform) and the ORC runtime to support native MachO
thread local variables. Adding new TLVs to a JITDylib at runtime is supported.
On the LLVM side MachOPlatform is updated to:
1. Identify thread local variables in the LinkGraph and lower them to GOT
accesses to data in the __thread_data or __thread_bss sections.
2. Merge and report the address range of __thread_data and thread_bss sections
to the runtime.
On the ORC runtime a MachOTLVManager class introduced which records the address
range of thread data/bss sections, and creates thread-local instances from the
initial data on demand. An orc-runtime specific tlv_get_addr implementation is
included which saves all register state then calls the MachOTLVManager to get
the address of the requested variable for the current thread.
These have been failing on our bots for a while due to
incomplete backtraces. (you don't get the names of the
functions that did the access, just the reporter frames)
See:
https://lab.llvm.org/buildbot/#/builders/170/builds/180
Adds support for MachO static initializers/deinitializers and eh-frame
registration via the ORC runtime.
This commit introduces cooperative support code into the ORC runtime and ORC
LLVM libraries (especially the MachOPlatform class) to support macho runtime
features for JIT'd code. This commit introduces support for static
initializers, static destructors (via cxa_atexit interposition), and eh-frame
registration. Near-future commits will add support for MachO native
thread-local variables, and language runtime registration (e.g. for Objective-C
and Swift).
The llvm-jitlink tool is updated to use the ORC runtime where available, and
regression tests for the new MachOPlatform support are added to compiler-rt.
Notable changes on the ORC runtime side:
1. The new macho_platform.h / macho_platform.cpp files contain the bulk of the
runtime-side support. This includes eh-frame registration; jit versions of
dlopen, dlsym, and dlclose; a cxa_atexit interpose to record static destructors,
and an '__orc_rt_macho_run_program' function that defines running a JIT'd MachO
program in terms of the jit- dlopen/dlsym/dlclose functions.
2. Replaces JITTargetAddress (and casting operations) with ExecutorAddress
(copied from LLVM) to improve type-safety of address management.
3. Adds serialization support for ExecutorAddress and unordered_map types to
the runtime-side Simple Packed Serialization code.
4. Adds orc-runtime regression tests to ensure that static initializers and
cxa-atexit interposes work as expected.
Notable changes on the LLVM side:
1. The MachOPlatform class is updated to:
1.1. Load the ORC runtime into the ExecutionSession.
1.2. Set up standard aliases for macho-specific runtime functions. E.g.
___cxa_atexit -> ___orc_rt_macho_cxa_atexit.
1.3. Install the MachOPlatformPlugin to scrape LinkGraphs for information
needed to support MachO features (e.g. eh-frames, mod-inits), and
communicate this information to the runtime.
1.4. Provide entry-points that the runtime can call to request initializers,
perform symbol lookup, and request deinitialiers (the latter is
implemented as an empty placeholder as macho object deinits are rarely
used).
1.5. Create a MachO header object for each JITDylib (defining the __mh_header
and __dso_handle symbols).
2. The llvm-jitlink tool (and llvm-jitlink-executor) are updated to use the
runtime when available.
3. A `lookupInitSymbolsAsync` method is added to the Platform base class. This
can be used to issue an async lookup for initializer symbols. The existing
`lookupInitSymbols` method is retained (the GenericIRPlatform code is still
using it), but is deprecated and will be removed soon.
4. JIT-dispatch support code is added to ExecutorProcessControl.
The JIT-dispatch system allows handlers in the JIT process to be associated with
'tag' symbols in the executor, and allows the executor to make remote procedure
calls back to the JIT process (via __orc_rt_jit_dispatch) using those tags.
The primary use case is ORC runtime code that needs to call bakc to handlers in
orc::Platform subclasses. E.g. __orc_rt_macho_jit_dlopen calling back to
MachOPlatform::rt_getInitializers using __orc_rt_macho_get_initializers_tag.
(The system is generic however, and could be used by non-runtime code).
The new ExecutorProcessControl::JITDispatchInfo struct provides the address
(in the executor) of the jit-dispatch function and a jit-dispatch context
object, and implementations of the dispatch function are added to
SelfExecutorProcessControl and OrcRPCExecutorProcessControl.
5. OrcRPCTPCServer is updated to support JIT-dispatch calls over ORC-RPC.
6. Serialization support for StringMap is added to the LLVM-side Simple Packed
Serialization code.
7. A JITLink::allocateBuffer operation is introduced to allocate writable memory
attached to the graph. This is used by the MachO header synthesis code, and will
be generically useful for other clients who want to create new graph content
from scratch.
ptrauth stores info in the address of functions, so it's not the right address we should check if poisoned
rdar://75246928
Differential Revision: https://reviews.llvm.org/D106199
This was fixed in the past for `frexp`, but was not made for `frexpl` & `frexpf` https://github.com/google/sanitizers/issues/321
This patch copies the fix over to `frexpl` because it caused `frexp_interceptor.cpp` test to fail on iPhone and `frexpf` for consistency.
rdar://79652161
Reviewed By: delcypher, vitalybuka
Differential Revision: https://reviews.llvm.org/D104948
Define the address ranges (similar to the C/C++ ones, but with the heap
range merged into the app range) and enable the sanity check.
Reviewed By: dvyukov
Differential Revision: https://reviews.llvm.org/D105629
XFAIL map32bit, define the maximum possible allocation size in
mmap_large.cpp.
Reviewed By: dvyukov
Differential Revision: https://reviews.llvm.org/D105629
These tests depend on TSan seeing the intercepted memcpy(), so they
break when the compiler chooses the builtin version.
Reviewed By: dvyukov
Differential Revision: https://reviews.llvm.org/D105629
The existing one actually failed on the int* p, not on int z (as can be
seen by the fault being 8 bytes rather than 4).
This is also needed to make sure the stack safety analysis does not
classify the alloca as safe.
Reviewed By: hctim
Differential Revision: https://reviews.llvm.org/D105705
This reverts commit 52aeacfbf5.
There isn't full agreement on a path forward yet, but there is agreement that
this shouldn't land as-is. See discussion on https://reviews.llvm.org/D105338
Also reverts unreviewed "[clang] Improve `-Wnull-dereference` diag to be more in-line with reality"
This reverts commit f4877c78c0.
And all the related changes to tests:
This reverts commit 9a0152799f.
This reverts commit 3f7c9cc274.
This reverts commit 329f8197ef.
This reverts commit aa9f58cc2c.
This reverts commit 2df37d5ddd.
This reverts commit a72a441812.
Store to null is deleted, so the test no longer did what it was expecting to do.
Conceal that by creating null pointer in a more elaborate way,
thus retaining original test coverage.
Update the asan_symbolize_script for changes in argparse output
in Python 3.10. The parser output 'options' instead of 'optional
arguments'.
Differential Revision: https://reviews.llvm.org/D105489
The __llvm_prf_names section uses SHF_GNU_RETAIN. However, GNU ld before 2015-10
(https://sourceware.org/bugzilla/show_bug.cgi?id=19161) neither supports it nor
retains __llvm_prf_names according to __start___llvm_prf_names. So --gc-sections
does not work on such old GNU ld.
This is not a problem for gold and sufficiently new lld.
We would find an address with matching tag, only to discover in
ShowCandidate that it's very far away from [stack].
Reviewed By: eugenis
Differential Revision: https://reviews.llvm.org/D105197
If the fault address is at the boundary of memory regions, this could
cause us to segfault otherwise.
Ran test with old compiler_rt to make sure it fails.
Reviewed By: eugenis
Differential Revision: https://reviews.llvm.org/D105032
This change introduces libMutagen/libclang_rt.mutagen.a as a subset of libFuzzer/libclang_rt.fuzzer.a. This library contains only the fuzzing strategies used by libFuzzer to produce new test inputs from provided inputs, dictionaries, and SanitizerCoverage feedback.
Most of this change is simply moving sections of code to one side or the other of the library boundary. The only meaningful new code is:
* The Mutagen.h interface and its implementation in Mutagen.cpp.
* The following methods in MutagenDispatcher.cpp:
* UseCmp
* UseMemmem
* SetCustomMutator
* SetCustomCrossOver
* LateInitialize (similar to the MutationDispatcher's original constructor)
* Mutate_AddWordFromTORC (uses callbacks instead of accessing TPC directly)
* StartMutationSequence
* MutationSequence
* DictionaryEntrySequence
* RecommendDictionary
* RecommendDictionaryEntry
* FuzzerMutate.cpp (which now justs sets callbacks and handles printing)
* MutagenUnittest.cpp (which adds tests of Mutagen.h)
A note on performance: This change was tested with a 100 passes of test/fuzzer/LargeTest.cpp with 1000 runs per pass, both with and without the change. The running time distribution was qualitatively similar both with and without the change, and the average difference was within 30 microseconds (2.240 ms/run vs 2.212 ms/run, respectively). Both times were much higher than observed with the fully optimized system clang (~0.38 ms/run), most likely due to the combination of CMake "dev mode" settings (e.g. CMAKE_BUILD_TYPE="Debug", LLVM_ENABLE_LTO=OFF, etc.). The difference between the two versions built similarly seems to be "in the noise" and suggests no meaningful performance degradation.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D102447
This allows application code checks if origin tracking is on before
printing out traces.
-dfsan-track-origins can be 0,1,2.
The current code only distinguishes 1 and 2 in compile time, but not at runtime.
Made runtime distinguish 1 and 2 too.
Reviewed By: browneee
Differential Revision: https://reviews.llvm.org/D105128
A heap or global buffer that is far away from the faulting address is
unlikely to be the cause, especially if there is a potential
use-after-free as well, so we want to show it after the other
causes.
Reviewed By: eugenis
Differential Revision: https://reviews.llvm.org/D104781
I can't be sure of the cause but I believe these fail
due to to fast unwinding not working on Thumb.
Whatever the case, they have been failing on our bots
for a long time:
https://lab.llvm.org/buildbot/#/builders/170/builds/46
Require fast-unwinder-works for both.
Word on the grapevine was that the committee had some discussion that
ended with unanimous agreement on eliminating relational function pointer comparisons.
We wanted to be bold and just ban all of them cold turkey.
But then we chickened out at the last second and are going for
eliminating just the spaceship overload candidate instead, for now.
See D104680 for reference.
This should be fine and "safe", because the only possible semantic change this
would cause is that overload resolution could possibly be ambiguous if
there was another viable candidate equally as good.
But to save face a little we are going to:
* Issue an "error" for three-way comparisons on function pointers.
But all this is doing really is changing one vague error message,
from an "invalid operands to binary expression" into an
"ordered comparison of function pointers", which sounds more like we mean business.
* Otherwise "warn" that comparing function pointers like that is totally
not cool (unless we are told to keep quiet about this).
Signed-off-by: Matheus Izvekov <mizvekov@gmail.com>
Reviewed By: rsmith
Differential Revision: https://reviews.llvm.org/D104892
on arm64e, pointer auth would catch this access violation before asan.
sign the function pointer so pointer auth will ignore this violation and let asan catch it in this test case.
rdar://79652167
Reviewed By: delcypher
Differential Revision: https://reviews.llvm.org/D104828
The comment says it was flaky in 2016,
but it wasn't possible to debug it back then.
Re-enable the test at least on linux/x86_64.
It will either work, or at least we should
see failure output from lit today.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D104592
Mmap interceptor is not atomic in the sense that it
exposes unmapped shadow for a brief period of time.
This breaks programs that mmap over another mmap
and access the region concurrently.
Don't unmap shadow in the mmap interceptor to fix this.
Just mapping new shadow on top should be enough to zero it.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D104593
These have been broken by https://reviews.llvm.org/D104494.
However, `lib/fuzzer/dataflow/` is unused (?) so addressing this is not a priority.
Added TODOs to re-enable them in the future.
Reviewed By: stephan.yichao.zhao
Differential Revision: https://reviews.llvm.org/D104568
Explain what the given stack trace means before showing it, rather than
only in the paragraph at the end.
Reviewed By: eugenis
Differential Revision: https://reviews.llvm.org/D104523
The default callback instrumentation in x86 LAM mode uses ASLR bits
to randomly choose a tag, and thus has a 1/64 chance of choosing a
stack tag of 0, causing stack tests to fail intermittently. By using
__hwasan_generate_tag to pick tags, we guarantee non-zero tags and
eliminate the test flakiness.
aarch64 doesn't seem to have this problem using thread-local addresses
to pick tags, so perhaps we can remove this workaround once we implement
a similar mechanism for LAM.
Reviewed By: eugenis
Differential Revision: https://reviews.llvm.org/D104470
The current naming scheme adds the `dfs$` prefix to all
DFSan-instrumented functions. This breaks mangling and prevents stack
trace printers and other tools from automatically demangling function
names.
This new naming scheme is mangling-compatible, with the `.dfsan`
suffix being a vendor-specific suffix:
https://itanium-cxx-abi.github.io/cxx-abi/abi.html#mangling-structure
With this fix, demangling utils would work out-of-the-box.
Reviewed By: stephan.yichao.zhao
Differential Revision: https://reviews.llvm.org/D104494
Before: ADDR is located -320 bytes to the right of 1072-byte region
After: ADDR is located 752 bytes inside 1072-byte region
Reviewed By: eugenis, walli99
Differential Revision: https://reviews.llvm.org/D104412
Adds the basic instrumentation needed for stack tagging.
Currently does not support stack short granules or TLS stack histories,
since a different code path is followed for the callback instrumentation
we use.
We may simply wait to support these two features until we switch to
a custom calling convention.
Patch By: xiangzhangllvm, morehouse
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D102901
This mostly follows LLVM's InstrProfReader.cpp error handling.
Previously, attempting to merge corrupted profile data would result in
crashes. See https://crbug.com/1216811#c4.
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D104050
Complete support for fast8:
- amend shadow size and mapping in runtime
- remove fast16 mode and -dfsan-fast-16-labels flag
- remove legacy mode and make fast8 mode the default
- remove dfsan-fast-8-labels flag
- remove functions in dfsan interface only applicable to legacy
- remove legacy-related instrumentation code and tests
- update documentation.
Reviewed By: stephan.yichao.zhao, browneee
Differential Revision: https://reviews.llvm.org/D103745
dfsan does not use sanitizer allocator as others. In practice,
we let it use glibc's allocator since tcmalloc needs more work
to be working with dfsan well. With glibc, we observe large
memory leakage. This could relate to two things:
1) glibc allocator has limitation: for example, tcmalloc can reduce memory footprint 2x easily
2) glibc may call unmmap directly as an internal system call by using system call number. so DFSan has no way to release shadow spaces for those unmmap.
Using sanitizer allocator addresses the above issues
1) its memory management is close to tcmalloc
2) we can register callback when sanitizer allocator calls unmmap, so dfsan can release shadow spaces correctly.
Our experiment with internal server-based application proved that with the change, in a-few-day run, memory usage leakage is close to what tcmalloc does w/o dfsan.
This change mainly follows MSan's code.
1) define allocator callbacks at dfsan_allocator.h|cpp
2) mark allocator APIs to be discard
3) intercept allocator APIs
4) make dfsan_set_label consistent with MSan's SetShadow when setting 0 labels, define dfsan_release_meta_memory when unmap is called
5) add flags about whether zeroing memory after malloc/free. dfsan works at byte-level, so bit-level oparations can cause reading undefined shadow. See D96842. zeroing memory after malloc helps this. About zeroing after free, reading after free is definitely UB, but if user code does so, it is hard to debug an overtainting caused by this w/o running MSan. So we add the flag to help debugging.
This change will be split to small changes for review. Before that, a question is
"this code shares a lot of with MSan, for example, dfsan_allocator.* and dfsan_new_delete.*.
Does it make sense to unify the code at sanitizer_common? will that introduce some
maintenance issue?"
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D101204
The linkage/visibility of `__profn_*` variables are derived
from the profiled functions.
extern_weak => linkonce
available_externally => linkonce_odr
internal => private
extern => private
_ => unchanged
The linkage/visibility of `__profc_*`/`__profd_*` variables are derived from
`__profn_*` with linkage/visibility wrestling for Windows.
The changes can be folded to the following without changing semantics.
```
if (TT.isOSBinFormatCOFF() && !NeedComdat) {
Linkage = GlobalValue::InternalLinkage;
Visibility = GlobalValue::DefaultVisibility;
}
```
That said, I think we can just delete the code block.
An extern/internal function will now use private `__profc_*`/`__profd_*`
variables, instead of internal ones. This saves some symbol table entries.
A non-comdat {linkonce,weak}_odr function will now use hidden external
`__profc_*`/`__profd_*` variables instead of internal ones. There is potential
object file size increase because such symbols need `/INCLUDE:` directives.
However such non-comdat functions are rare (note that non-comdat weak
definitions don't prevent duplicate definition error).
The behavior changes match ELF.
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D103355
We have been seeing this test fail intermittently on our
2 stage AArch64 bot.
As far back as https://lab.llvm.org/buildbot/#/builders/53/builds/2694
Likely due to a lack of resources at certain times on the
shared machine. Up the time limit to give us some more room.
(this limit only applies to the watchdog thread, so if the
test passes then it won't take 20s)
Since https://reviews.llvm.org/D102046 some tests have
been falling back to fast unwinding on our Thumb bot.
This fails because fast unwinding does not work on Thumb.
By adding the extra information we ensure this does not happen
during testing, but the built library can still fast unwind
as a last resort.
Since there are some situations it can work in, like if
eveything is built with clang. During testing we've got gcc
built system libs and clang built tests.
The same change was made for sanitizer-common in
https://reviews.llvm.org/D96337.
Reviewed By: zatrazz
Differential Revision: https://reviews.llvm.org/D103463
DFSan has flags to control flows between pointers and objects referred
by pointers. For example,
a = *p;
L(a) = L(*p) when -dfsan-combine-pointer-labels-on-load = false
L(a) = L(*p) + L(p) when -dfsan-combine-pointer-labels-on-load = true
*p = b;
L(*p) = L(b) when -dfsan-combine-pointer-labels-on-store = false
L(*p) = L(b) + L(p) when -dfsan-combine-pointer-labels-on-store = true
The question is what to do with p += c.
In practice we found many confusing flows if we propagate labels from c
to p. So a new flag works like this
p += c;
L(p) = L(p) when -dfsan-propagate-via-pointer-arithmetic = false
L(p) = L(p) + L(c) when -dfsan-propagate-via-pointer-arithmetic = true
Reviewed-by: gbalats
Differential Revision: https://reviews.llvm.org/D103176
This change introduces libMutagen/libclang_rt.mutagen.a as a subset of libFuzzer/libclang_rt.fuzzer.a. This library contains only the fuzzing strategies used by libFuzzer to produce new test inputs from provided inputs, dictionaries, and SanitizerCoverage feedback.
Most of this change is simply moving sections of code to one side or the other of the library boundary. The only meaningful new code is:
* The Mutagen.h interface and its implementation in Mutagen.cpp.
* The following methods in MutagenDispatcher.cpp:
* UseCmp
* UseMemmem
* SetCustomMutator
* SetCustomCrossOver
* LateInitialize (similar to the MutationDispatcher's original constructor)
* Mutate_AddWordFromTORC (uses callbacks instead of accessing TPC directly)
* StartMutationSequence
* MutationSequence
* DictionaryEntrySequence
* RecommendDictionary
* RecommendDictionaryEntry
* FuzzerMutate.cpp (which now justs sets callbacks and handles printing)
* MutagenUnittest.cpp (which adds tests of Mutagen.h)
A note on performance: This change was tested with a 100 passes of test/fuzzer/LargeTest.cpp with 1000 runs per pass, both with and without the change. The running time distribution was qualitatively similar both with and without the change, and the average difference was within 30 microseconds (2.240 ms/run vs 2.212 ms/run, respectively). Both times were much higher than observed with the fully optimized system clang (~0.38 ms/run), most likely due to the combination of CMake "dev mode" settings (e.g. CMAKE_BUILD_TYPE="Debug", LLVM_ENABLE_LTO=OFF, etc.). The difference between the two versions built similarly seems to be "in the noise" and suggests no meaningful performance degradation.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D102447
This reverts commit 6911114d8c.
Broke the QEMU sanitizer bots due to a missing header dependency. This
actually needs to be fixed on the bot-side, but for now reverting this
patch until I can fix up the bot.
This patch moves -fsanitize=scudo to link the standalone scudo library,
rather than the original compiler-rt based library. This is one of the
major remaining roadblocks to deleting the compiler-rt based scudo,
which should not be used any more. The standalone Scudo is better in
pretty much every way and is much more suitable for production usage.
As well as patching the litmus tests for checking that the
scudo_standalone lib is linked instead of the scudo lib, this patch also
ports all the scudo lit tests to run under scudo standalone.
This patch also adds a feature to scudo standalone that was under test
in the original scudo - that arguments passed to an aligned operator new
were checked that the alignment was a power of two.
Some lit tests could not be migrated, due to the following issues:
1. Features that aren't supported in scudo standalone, like the rss
limit.
2. Different quarantine implementation where the test needs some more
thought.
3. Small bugs in scudo standalone that should probably be fixed, like
the Secondary allocator having a full page on the LHS of an allocation
that only contains the chunk header, so underflows by <= a page aren't
caught.
4. Slight differences in behaviour that's technically correct, like
'realloc(malloc(1), 0)' returns nullptr in standalone, but a real
pointer in old scudo.
5. Some tests that might be migratable, but not easily.
Tests that are obviously not applicable to scudo standalone (like
testing that no sanitizer symbols made it into the DSO) have been
deleted.
After this patch, the remaining work is:
1. Update the Scudo documentation. The flags have changed, etc.
2. Delete the old version of scudo.
3. Patch up the tests in lit-unmigrated, or fix Scudo standalone.
Reviewed By: cryptoad, vitalybuka
Differential Revision: https://reviews.llvm.org/D102543
Handy when testing specific files, already supported in other components.
Example:
cd build; ./bin/llvm-lit ../compiler-rt/test/tsan/ignore_free.cpp
Differential Revision: https://reviews.llvm.org/D103054
This reduces the size of chrome.dll.pdb built with optimizations,
coverage, and line table info from 4,690,210,816 to 2,181,128,192, which
makes it possible to fit under the 4GB limit.
This change can greatly reduce binary size in coverage builds, which do
not need value profiling. IR PGO builds are unaffected. There is a minor
behavior change for frontend PGO.
PGO and coverage both use InstrProfiling to create profile data with
counters. PGO records the address of each function in the __profd_
global. It is used later to map runtime function pointer values back to
source-level function names. Coverage does not appear to use this
information.
Recording the address of every function with code coverage drastically
increases code size. Consider this program:
void foo();
void bar();
inline void inlineMe(int x) {
if (x > 0)
foo();
else
bar();
}
int getVal();
int main() { inlineMe(getVal()); }
With code coverage, the InstrProfiling pass runs before inlining, and it
captures the address of inlineMe in the __profd_ global. This greatly
increases code size, because now the compiler can no longer delete
trivial code.
One downside to this approach is that users of frontend PGO must apply
the -mllvm -enable-value-profiling flag globally in TUs that enable PGO.
Otherwise, some inline virtual method addresses may not be recorded and
will not be able to be promoted. My assumption is that this mllvm flag
is not popular, and most frontend PGO users don't enable it.
Differential Revision: https://reviews.llvm.org/D102818
sleep(1) does not guaranty afterfork order.
Also relative child/parent afterfork order is not important for this test so we
can just avoid checking that.
Reviewed By: dvyukov
Differential Revision: https://reviews.llvm.org/D102810
If there are no counters, an mmap() of the counters section would fail
due to the size argument being too small (EINVAL).
rdar://78175925
Differential Revision: https://reviews.llvm.org/D102735
Override __cxa_atexit and ignore callbacks.
This prevents crashes in a configuration when the symbolizer
is built into sanitizer runtime and consequently into the test process.
LLVM libraries have some global objects destroyed during exit,
so if the test process triggers any bugs after that, the symbolizer crashes.
An example stack trace of such crash:
For the standalone llvm-symbolizer this does not hurt,
we just don't destroy few global objects on exit.
Reviewed By: kda
Differential Revision: https://reviews.llvm.org/D102470
https://reviews.llvm.org/D101681 landed a change to check the testing
configuration which relies on using the `-print-runtime-dir` flag of
clang to determine where the runtime testing library is.
The patch treated not being able to find the path reported by clang
as an error. Unfortunately this seems to break the
`llvm-clang-win-x-aarch64` bot. Either the bot is misconfigured or
clang is reporting a bogus path.
To temporarily unbreak the bot downgrade the fatal error to a warning.
While we're here also print information about the command used to
determine the path to aid debugging.
to a warning.
https://reviews.llvm.org/D101681 introduced a check to make sure the
compiler and compiler-rt were using the same library path when
`COMPILER_RT_TEST_STANDALONE_BUILD_LIBS=ON`, i.e. the developer's
intention is to test the just built libs rather that shipped with the
compiler used for testing.
It seems this broken some bots that are likely misconfigured.
So to unbreak them, for now let's make this a warning so the bot
owners can investigate without breaking their builds.
The path to the runtime libraries used by the compiler under test
is normally identical to the path where just built libraries are
created. However, this is not necessarily the case when doing standalone
builds. This is because the external compiler used by tests may choose
to get its runtime libraries from somewhere else.
When doing standalone builds there are two types of testing we could be
doing:
* Test the just built runtime libraries.
* Test the runtime libraries shipped with the compile under test.
Both types of testing are valid but it confusingly turns out compiler-rt
actually did a mixture of these types of testing.
* The `test/builtins/Unit/` test suite always tested the just built runtime
libraries.
* All other testsuites implicitly use whatever runtime library the
compiler decides to link.
There is no way for us to infer which type of testing the developer
wants so this patch introduces a new
`COMPILER_RT_TEST_STANDALONE_BUILD_LIBS` CMake
option which explicitly declares which runtime libraries should be
tested. If it is `ON` then the just built libraries should be tested,
otherwise the libraries in the external compiler should be tested.
When testing starts the lit test suite queries the compiler used for
testing to see where it will get its runtime libraries from. If these
paths are identical no action is taken (the common case). If the paths
are not identical then we check the value of
`COMPILER_RT_TEST_STANDALONE_BUILD_LIBS` (progated into the config as
`test_standalone_build_libs`) and check if the test suite supports testing in the
requested configuration.
* If we want to test just built libs and the test suite supports it
(currently only `test/builtins/Unit`) then testing proceeds without any changes.
* If we want to test the just built libs and the test suite doesn't
support it we emit a fatal error to prevent the developer from
testing the wrong runtime libraries.
* If we are testing the compiler's built libs then we adjust
`config.compiler_rt_libdir` to point at the compiler's runtime
directory. This makes the `test/builtins/Unit` tests use the
compiler's builtin library. No other changes are required because
all other testsuites implicitly use the compiler's built libs.
To make the above work the
`test_suite_supports_overriding_runtime_lib_path` test suite config
option has been introduced so we can identify what each test suite
supports.
Note all of these checks **have to be performed** when lit runs.
We cannot run the checks at CMake generation time because
multi-configuration build systems prevent us from knowing what the
paths will be.
We could perhaps support `COMPILER_RT_TEST_STANDALONE_BUILD_LIBS` being
`ON` for most test suites (when the runtime library paths differs) in
the future by specifiying a custom compiler resource directory path.
Doing so is out of scope for this patch.
rdar://77182297
Differential Revision: https://reviews.llvm.org/D101681
`-fno-exceptions -fno-asynchronous-unwind-tables` compiled programs don't
produce .eh_frame on Linux and other ELF platforms, so the slow unwinder cannot
print stack traces. Just fall back to the fast unwinder: this allows
-fno-asynchronous-unwind-tables without requiring the sanitizer option
`fast_unwind_on_fatal=1`
Reviewed By: #sanitizers, vitalybuka
Differential Revision: https://reviews.llvm.org/D102046
This removes one of the last dependencies on old Scudo, and should allow
us to delete the old Scudo soon.
Reviewed By: vitalybuka, cryptoad
Differential Revision: https://reviews.llvm.org/D102349
-fsanitize-hwaddress-experimental-aliasing is intended to distinguish
aliasing mode from LAM mode on x86_64. check-hwasan is configured
to use aliasing mode while check-hwasan-lam is configured to use LAM
mode.
The current patch doesn't actually do anything differently in the two
modes. A subsequent patch will actually build the separate runtimes
and use them in each mode.
Currently LAM mode tests must be run in an emulator that
has LAM support. To ensure LAM mode isn't broken by future patches, I
will next set up a QEMU buildbot to run the HWASan tests in LAM.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D102288
We have some significant amount of duplication around
CheckFailed functionality. Each sanitizer copy-pasted
a chunk of code. Some got random improvements like
dealing with recursive failures better. These improvements
could benefit all sanitizers, but they don't.
Deduplicate CheckFailed logic across sanitizers and let each
sanitizer only print the current stack trace.
I've tried to dedup stack printing as well,
but this got me into cmake hell. So let's keep this part
duplicated in each sanitizer for now.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D102221
We already declare subset of annotations in test.h.
But some are duplicated and declared in tests.
Move all annotation declarations to test.h.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D102152
Add a simple test that uses syscall annotations.
Just to ensure at least basic functionality works.
Also factor out annotated syscall wrappers into a separate
header file as they may be useful for future tests.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D102223
This test has two modes - testing reused threads with multiple loops of
batch create/join, and testing new threads with a single loop of
create/join per fork.
The non-reuse variant catches the problem that was fixed in D101881 with
a high probability.
Differential Revision: https://reviews.llvm.org/D101936
This patch does a few cleanup things:
1. The non-standalone scudo has a problem where GWP-ASan allocations
may not meet alignment requirements where Scudo was requested to have
alignment >= 16. Use the new GWP-ASan API to fix this.
2. The standalone variant loses some debugging information inside of
GWP-ASan because we ask GWP-ASan to allocate an aligned size in the
frontend. This means reports end up with 'UaF on a 16-byte allocation'
for a 1-byte allocation with 16-byte alignment. Also use the new API to
fix this.
3. Add post-alloc hooks for GWP-ASan intercepted allocations, and add
stats tracking for GWP-ASan allocations.
4. Add a small test that checks the alignment of the frontend
allocator, so that it can be used under GWP-ASan torture mode.
5. Add GWP-ASan torture mode as a testing configuration to catch these
regressions.
Depends on D94830, D95889.
Reviewed By: cryptoad
Differential Revision: https://reviews.llvm.org/D95884
Address sanitizer can detect stack exhaustion via its SEGV handler, which is
executed on a separate stack using the sigaltstack mechanism. When libFuzzer is
used with address sanitizer, it installs its own signal handlers which defer to
those put in place by the sanitizer before performing additional actions. In the
particular case of a stack overflow, the current setup fails because libFuzzer
doesn't preserve the flag for executing the signal handler on a separate stack:
when we run out of stack space, the operating system can't run the SEGV handler,
so address sanitizer never reports the issue. See the included test for an
example.
This commit fixes the issue by making libFuzzer preserve the SA_ONSTACK flag
when installing its signal handlers; the dedicated signal-handler stack set up
by the sanitizer runtime appears to be large enough to support the additional
frames from the fuzzer.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D101824
The linker suggests using -Wl,-z,notext.
Replaced assert by exit also fixed this.
After renaming, interceptor.c would be used to test interceptors in general by D101204.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D101649
Code patterns like this are common, `#` at the line beginning
(https://google.github.io/styleguide/cppguide.html#Preprocessor_Directives),
one space indentation for if/elif/else directives.
```
#if SANITIZER_LINUX
# if defined(__aarch64__)
# endif
#endif
```
However, currently clang-format wants to reformat the code to
```
#if SANITIZER_LINUX
#if defined(__aarch64__)
#endif
#endif
```
This significantly harms readability in my review. Use `IndentPPDirectives:
AfterHash` to defeat the diagnostic. clang-format will now suggest:
```
#if SANITIZER_LINUX
# if defined(__aarch64__)
# endif
#endif
```
Unfortunately there is no clang-format option using indent with 1 for
just preprocessor directives. However, this is still one step forward
from the current behavior.
Reviewed By: #sanitizers, vitalybuka
Differential Revision: https://reviews.llvm.org/D100238
Before commit "sanitizer_common: introduce kInvalidTid/kMainTid"
asan invalid/unknown thread id was 0xffffff, so presumably we printed "T16777215".
Now it's -1, so we print T-1. Fix the test.
I think the new format is even better, "T-1" clearly looks like something special
rather than a random large number.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D101634
Commit efd254b636 ("tsan: fix deadlock in pthread_atfork callbacks")
fixed another deadlock related to atfork handling.
But builders with DCHECKs enabled reported failures of
pthread_atfork_deadlock2.c and pthread_atfork_deadlock3.c tests
related to the fact that we hold runtime locks on interceptor exit:
https://lab.llvm.org/buildbot/#/builders/70/builds/6727
This issue is somewhat inherent to the current approach,
we indeed execute user code (atfork callbacks) with runtime lock held.
Refactor fork handling to not run user code (atfork callbacks)
with runtime locks held. This change does this by installing
own atfork callbacks during runtime initialization.
Atfork callbacks run in LIFO order, so the expectation is that
our callbacks run last, right before the actual fork.
This way we lock runtime mutexes around fork, but not around
user callbacks.
Extend tests to also install after fork callbacks just to cover
more scenarios. Some tests also started reporting real races
that we previously suppressed.
Also extend tests to cover fork syscall support.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D101517
Fix format specifier.
Fix warnings about non-standard attribute placement.
Make free_race2.c test a bit more interesting:
test access with/without an offset.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D101424
Commit efd254b636 ("tsan: fix deadlock in pthread_atfork callbacks")
fixed another deadlock related to atfork handling.
But builders with DCHECKs enabled reported failures of
pthread_atfork_deadlock2.c and pthread_atfork_deadlock3.c tests
related to the fact that we hold runtime locks on interceptor exit:
https://lab.llvm.org/buildbot/#/builders/70/builds/6727
This issue is somewhat inherent to the current approach,
we indeed execute user code (atfork callbacks) with runtime lock held.
Refactor fork handling to not run user code (atfork callbacks)
with runtime locks held. This change does this by installing
own atfork callbacks during runtime initialization.
Atfork callbacks run in LIFO order, so the expectation is that
our callbacks run last, right before the actual fork.
This way we lock runtime mutexes around fork, but not around
user callbacks.
Extend tests to also install after fork callbacks just to cover
more scenarios. Some tests also started reporting real races
that we previously suppressed.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D101385
We take report/thread_registry locks around fork.
This means we cannot report any bugs in atfork handlers.
We resolved this by enabling per-thread ignores around fork.
This resolved some of the cases, but not all.
The added test triggers a race report from a signal handler
called from atfork callback, we reset per-thread ignores
around signal handlers, so we tried to report it and deadlocked.
But there are more cases: a signal handler can be called
synchronously if it's sent to itself. Or any other report
types would cause deadlocks as well: mutex misuse,
signal handler spoiling errno, etc.
Disable all reports for the duration of fork with
thr->suppress_reports and don't re-enable them around
signal handlers.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D101154
This reapplies 1e1d75b190, which was reverted in ce1a4d5323 due to build
failures.
The unconditional dependencies on clang and llvm-jitlink in
compiler-rt/test/orc/CMakeLists.txt have been removed -- they don't appear to
be necessary, and I suspect they're the cause of the build failures seen
earlier.
Some builders failed with a missing clang dependency. E.g.
CMake Error at /Users/buildslave/jenkins/workspace/clang-stage1-RA/clang-build \
/lib/cmake/llvm/AddLLVM.cmake:1786 (add_dependencies):
The dependency target "clang" of target "check-compiler-rt" does not exist.
Reverting while I investigate.
This reverts commit 1e1d75b190.
Now that page aliasing for x64 has landed, we don't need to worry about
passing tagged pointers to libc, and thus D98875 removed it.
Unfortunately, we still test on aarch64 devices that don't have the
kernel tagged address ABI (https://reviews.llvm.org/D98875#2649269).
All the memory that we pass to the kernel in these tests is from global
variables. Instead of having architecture-specific untagging mechanisms
for this memory, let's just not tag the globals.
Reviewed By: eugenis, morehouse
Differential Revision: https://reviews.llvm.org/D101121
This patch does a few cleanup things:
1. The non-standalone scudo has a problem where GWP-ASan allocations
may not meet alignment requirements where Scudo was requested to have
alignment >= 16. Use the new GWP-ASan API to fix this.
2. The standalone variant loses some debugging information inside of
GWP-ASan because we ask GWP-ASan to allocate an aligned size in the
frontend. This means reports end up with 'UaF on a 16-byte allocation'
for a 1-byte allocation with 16-byte alignment. Also use the new API to
fix this.
3. Add post-alloc hooks for GWP-ASan intercepted allocations, and add
stats tracking for GWP-ASan allocations.
4. Add a small test that checks the alignment of the frontend
allocator, so that it can be used under GWP-ASan torture mode.
5. Add GWP-ASan torture mode as a testing configuration to catch these
regressions.
Depends on D94830, D95889.
Reviewed By: cryptoad
Differential Revision: https://reviews.llvm.org/D95884
The first version of origin tracking tracks only memory stores. Although
this is sufficient for understanding correct flows, it is hard to figure
out where an undefined value is read from. To find reading undefined values,
we still have to do a reverse binary search from the last store in the chain
with printing and logging at possible code paths. This is
quite inefficient.
Tracking memory load instructions can help this case. The main issues of
tracking loads are performance and code size overheads.
With tracking only stores, the code size overhead is 38%,
memory overhead is 1x, and cpu overhead is 3x. In practice #load is much
larger than #store, so both code size and cpu overhead increases. The
first blocker is code size overhead: link fails if we inline tracking
loads. The workaround is using external function calls to propagate
metadata. This is also the workaround ASan uses. The cpu overhead
is ~10x. This is a trade off between debuggability and performance,
and will be used only when debugging cases that tracking only stores
is not enough.
Reviewed By: gbalats
Differential Revision: https://reviews.llvm.org/D100967
This test from @MaskRay comment on D69428. The patch is looking to
break this behavior. If we go with D69428 I hope we will have some
workaround for this test or include explicit test update into the patch.
Reviewed By: MaskRay
Differential Revision: https://reviews.llvm.org/D100906
... so that FreeBSD specific GetTls/glibc specific pthread_self code can be
removed. This also helps FreeBSD arm64/powerpc64 which don't have GetTls
implementation yet.
GetTls is the range of
* thread control block and optional TLS_PRE_TCB_SIZE
* static TLS blocks plus static TLS surplus
On glibc, lsan requires the range to include
`pthread::{specific_1stblock,specific}` so that allocations only referenced by
`pthread_setspecific` can be scanned.
This patch uses `dl_iterate_phdr` to collect TLS blocks. Find the one
with `dlpi_tls_modid==1` as one of the initially loaded module, then find
consecutive ranges. The boundaries give us addr and size.
This allows us to drop the glibc internal `_dl_get_tls_static_info` and
`InitTlsSize`. However, huge glibc x86-64 binaries with numerous shared objects
may observe time complexity penalty, so exclude them for now. Use the simplified
method with non-Android Linux for now, but in theory this can be used with *BSD
and potentially other ELF OSes.
This removal of RISC-V `__builtin_thread_pointer` makes the code compilable with
more compiler versions (added in Clang in 2020-03, added in GCC in 2020-07).
This simplification enables D99566 for TLS Variant I architectures.
Note: as of musl 1.2.2 and FreeBSD 12.2, dlpi_tls_data returned by
dl_iterate_phdr is not desired: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254774
This can be worked around by using `__tls_get_addr({modid,0})` instead
of `dlpi_tls_data`. The workaround can be shared with the workaround for glibc<2.25.
This fixes some tests on Alpine Linux x86-64 (musl)
```
test/lsan/Linux/cleanup_in_tsd_destructor.c
test/lsan/Linux/fork.cpp
test/lsan/Linux/fork_threaded.cpp
test/lsan/Linux/use_tls_static.cpp
test/lsan/many_tls_keys_thread.cpp
test/msan/tls_reuse.cpp
```
and `test/lsan/TestCases/many_tls_keys_pthread.cpp` on glibc aarch64.
The number of sanitizer test failures does not change on FreeBSD/amd64 12.2.
Differential Revision: https://reviews.llvm.org/D98926
Allow test contents to be copied before execution by using
`%ld_flags_rpath_so`, `%ld_flags_rpath_exe`, and `%dynamiclib`
substitutions.
rdar://76302416
Differential Revision: https://reviews.llvm.org/D100240
Mark the test as unsupported to bring the bot online. Could probably be
permanently fixed by using one of the workarounds already present in
compiler-rt.
ASan declares these functions as strongly-defined, which results in
'duplicate symbol' errors when trying to replace them in user code when
linking the runtimes statically.
Reviewed By: eugenis
Differential Revision: https://reviews.llvm.org/D100220
These tests were added in:
1daa48f00559e422c90b
malloc_zero.c and realloc_too_big.c fail when only
leak sanitizer is enabled.
http://lab.llvm.org:8011/#/builders/59/builds/1635
(also in an armv8 32 bit build)
(I would XFAIL them but the same test is run with
address and leak sanitizer enabled and that one does
pass)
This was reverted by f176803ef1 due to
Ubuntu 16.04 x86-64 glibc 2.23 problems.
This commit additionally calls `__tls_get_addr({modid,0})` to work around the
dlpi_tls_data==NULL issues for glibc<2.25
(https://sourceware.org/bugzilla/show_bug.cgi?id=19826)
GetTls is the range of
* thread control block and optional TLS_PRE_TCB_SIZE
* static TLS blocks plus static TLS surplus
On glibc, lsan requires the range to include
`pthread::{specific_1stblock,specific}` so that allocations only referenced by
`pthread_setspecific` can be scanned.
This patch uses `dl_iterate_phdr` to collect TLS blocks. Find the one
with `dlpi_tls_modid==1` as one of the initially loaded module, then find
consecutive ranges. The boundaries give us addr and size.
This allows us to drop the glibc internal `_dl_get_tls_static_info` and
`InitTlsSize` entirely. Use the simplified method with non-Android Linux for
now, but in theory this can be used with *BSD and potentially other ELF OSes.
This simplification enables D99566 for TLS Variant I architectures.
See https://reviews.llvm.org/D93972#2480556 for analysis on GetTls usage
across various sanitizers.
Differential Revision: https://reviews.llvm.org/D98926
With D98926, many_tls_keys_pthread.cpp appears to be working.
On glibc 2.30-0ubuntu2, swapcontext.cpp and Linux/fork_and_leak.cpp work fine
but they strangely fail on clang-cmake-aarch64-full
(https://lab.llvm.org/buildbot/#/builders/7/builds/2240).
Disable them for now.
Note: check-lsan was recently enabled on AArch64 in D98985. A test takes
10+ seconds. We should figure out the bottleneck.
```
/b/sanitizer-x86_64-linux/build/llvm-project/compiler-rt/test/memprof/TestCases/test_terse.cpp:11:11: error: CHECK: expected string not found in input
// CHECK: MIB:[[STACKID:[0-9]+]]/1/40.00/40/40/20.00/20/20/[[AVELIFETIME:[0-9]+]].00/[[AVELIFETIME]]/[[AVELIFETIME]]/0/0/0/0
^
<stdin>:1:1: note: scanning from here
MIB:StackID/AllocCount/AveSize/MinSize/MaxSize/AveAccessCount/MinAccessCount/MaxAccessCount/AveLifetime/MinLifetime/MaxLifetime/NumMigratedCpu/NumLifetimeOverlaps/NumSameAllocCpu/NumSameDeallocCpu
^
<stdin>:4:1: note: possible intended match here
MIB:134217729/1/40.00/40/40/20.00/20/20/7.00/7/7/1/0/0/0
```
Userspace page aliasing allows us to use middle pointer bits for tags
without untagging them before syscalls or accesses. This should enable
easier experimentation with HWASan on x86_64 platforms.
Currently stack, global, and secondary heap tagging are unsupported.
Only primary heap allocations get tagged.
Note that aliasing mode will not work properly in the presence of
fork(), since heap memory will be shared between the parent and child
processes. This mode is non-ideal; we expect Intel LAM to enable full
HWASan support on x86_64 in the future.
Reviewed By: vitalybuka, eugenis
Differential Revision: https://reviews.llvm.org/D98875
Make TSan runtime initialization and finalization hooks work
even if these hooks are not built in the main executable. When these
hooks are defined in another library that is not directly linked against
the TSan runtime (e.g., Swift runtime) we cannot rely on the "strong-def
overriding weak-def" mechanics and have to look them up via `dlsym()`.
Let's also define hooks that are easier to use from C-only code:
```
extern "C" void __tsan_on_initialize();
extern "C" int __tsan_on_finalize(int failed);
```
For now, these will call through to the old hooks. Eventually, we want
to adopt the new hooks downstream and remove the old ones.
This is part of the effort to support Swift Tasks (async/await and
actors) in TSan.
rdar://74256720
Reviewed By: vitalybuka, delcypher
Differential Revision: https://reviews.llvm.org/D98810
Userspace page aliasing allows us to use middle pointer bits for tags
without untagging them before syscalls or accesses. This should enable
easier experimentation with HWASan on x86_64 platforms.
Currently stack, global, and secondary heap tagging are unsupported.
Only primary heap allocations get tagged.
Note that aliasing mode will not work properly in the presence of
fork(), since heap memory will be shared between the parent and child
processes. This mode is non-ideal; we expect Intel LAM to enable full
HWASan support on x86_64 in the future.
Reviewed By: vitalybuka, eugenis
Differential Revision: https://reviews.llvm.org/D98875
Supported ctime_r, fgets, getcwd, get_current_dir_name, gethostname,
getrlimit, getrusage, strcpy, time, inet_pton, localtime_r,
getpwuid_r, epoll_wait, poll, select, sched_getaffinity
Most of them work as calling their non-origin verision directly.
This is a part of https://reviews.llvm.org/D95835.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D98966
Supported strrchr, strrstr, strto*, recvmmsg, recrmsg, nanosleep,
memchr, snprintf, socketpair, sprintf, getocketname, getsocketopt,
gettimeofday, getpeername.
strcpy was added because the test of sprintf need it. It will be
committed by D98966. Please ignore it when reviewing.
This is a part of https://reviews.llvm.org/D95835.
Reviewed By: gbalats
Differential Revision: https://reviews.llvm.org/D99109
`check-lsan` passed on an aarch64-*-linux machine.
Unsupport `many_tls_keys_pthread.cpp` for now: it requires GetTls to include
`specific_1stblock` and `specific` in `struct pthread`.
Differential Revision: https://reviews.llvm.org/D98985
Subsequent patches will implement page-aliasing mode for x86_64, which
will initially only work for the primary heap allocator. We force
callback instrumentation to simplify the initial aliasing
implementation.
Reviewed By: vitalybuka, eugenis
Differential Revision: https://reviews.llvm.org/D98069
If we don't specify the c++ version in these tests, it could cause compile errors because the compiler could default to an older c++
rdar://75247244
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D98913
As reported in D96348 <https://reviews.llvm.org/D96348>, the
`Posix/regex_startend.cpp` test `FAIL`s on Solaris because
`REG_STARTEND` isn't defined. It's a BSD extension not present everywhere.
E.g. AIX doesn't have it, too.
Fixed by wrapping the test in `#ifdef REG_STARTEND`.
Tested on `amd64-pc-solaris2.11`, `sparcv9-sun-solaris2.11`, and
`x86_64-pc-linux-gnu`.
Differential Revision: https://reviews.llvm.org/D98425
On Darwin, MallocNanoZone may log after execv, which messes up this test.
Disable MallocNanoZone for this test since we don't use it anyway with asan.
This environment variable should only affect Darwin and not change behavior on other platforms.
rdar://74992832
Reviewed By: delcypher
Differential Revision: https://reviews.llvm.org/D98735
Right now, when you have an invalid memory address, asan would just crash and does not offer much useful info.
This patch attempted to give a bit more detail on the access.
Differential Revision: https://reviews.llvm.org/D98280
If a log message is triggered between execv and child, this test fails.
In the meantime, disable the test to unblock CI
rdar://74992832
Reviewed By: delcypher
Differential Revision: https://reviews.llvm.org/D98453
Previously, on GLibc systems, the interceptor was calling __compat_regexec
(regexec@GLIBC_2.2.5) insead of the newer __regexec (regexec@GLIBC_2.3.4).
The __compat_regexec strips the REG_STARTEND flag but does not report an
error if other flags are present. This can result in infinite loops for
programs that use REG_STARTEND to find all matches inside a buffer (since
ignoring REG_STARTEND means that the search always starts from the first
character).
The underlying issue is that GLibc's dlsym(RTLD_NEXT, ...) appears to
always return the oldest versioned symbol instead of the default. This
means it does not match the behaviour of dlsym(RTLD_DEFAULT, ...) or the
behaviour documented in the manpage.
It appears a similar issue was encountered with realpath and worked around
in 77ef78a0a5.
See also https://sourceware.org/bugzilla/show_bug.cgi?id=14932 and
https://sourceware.org/bugzilla/show_bug.cgi?id=1319.
Fixes https://github.com/google/sanitizers/issues/1371
Reviewed By: #sanitizers, vitalybuka, marxin
Differential Revision: https://reviews.llvm.org/D96348
Looks like the default options for halt_on_error are different between linux and mac. set it to 0 in the test so the behavior is the same on both platforms.
rdar://75110847
Reviewed By: delcypher
Differential Revision: https://reviews.llvm.org/D98089
Two ASan tests currently `FAIL' on Solaris
AddressSanitizer-i386-sunos :: TestCases/large_func_test.cpp
AddressSanitizer-i386-sunos :: TestCases/use-after-delete.cpp
both for the same reason:
error: no check strings found with prefix 'CHECK-SunOS:'
Fixed by adding the appropriate check strings.
Tested on `amd64-pc-solaris2.11` and `x86_64-pc-linux-gnu`.
Differential Revision: https://reviews.llvm.org/D97931
Previously, this test only ran for mac because platforms have different messaging. This diff enables the test for all posix
rdar://75110847
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D98079
Currently, `print_module_map` is only respected for ubsan if it is ran in tandem with asan. This patch adds support for this flag in standalone mode. I copied the pattern used to implement this for asan.
Also added a common `print_module_map` lit test for Darwin only. Since the print messages are different per platform, we need to write a regex test to cover them. This test is coming in a separate patch
rdar://56135732
Reviewed By: vitalybuka, vsk, delcypher
Differential Revision: https://reviews.llvm.org/D97746
The runtimes build uses variables set by add_lit_testsuite to collect
testsuites from all the runtimes.
Differential Revision: https://reviews.llvm.org/D97913
One ASan test currently `XPASS`es on Solaris:
AddressSanitizer-i386-sunos :: TestCases/Posix/unpoison-alternate-stack.cpp
It was originally `XFAIL`ed in D88501 <https://reviews.llvm.org/D88501>
because `longjmp` from a signal handled is highly unportable, warned
against in XPG7, and was not supported by Solaris `libc` at the time.
However, since then support has been added for some cases including the
current one, so the `XFAIL` can go.
Tested on `amd64-pc-solaris2.11` and `x86_64-pc-linux-gnu`.
Differential Revision: https://reviews.llvm.org/D97933
One ASan test currently `XPASS`es on Solaris:
AddressSanitizer-i386-sunos :: TestCases/Posix/no_asan_gen_globals.c
It was originally `XFAIL`ed in D88218 <https://reviews.llvm.org/D88218>
because Solaris `ld`, unlike GNU `ld`, doesn't strip local labels. Since
then, the integrated assembler has stopped emitting those local labels, so
the difference becomes moot and the `XFAIL` can go.
Tested on `amd64-pc-solaris2.11` and `x86_64-pc-linux-gnu`.
Differential Revision: https://reviews.llvm.org/D97932
We're having flaky failures on this test on the sanitizer slow
buildbot. Not per-run flaky, but it'll be green for a while, then red
for a while. I suspect that changes in codegen are causing the
LLVM_VP_MAX_NUM_VALS_PER_SITE=150 to be above and below the limit
sporadically. The limit on my machine using lld and a non-bootstrapped
compiler is 175, but the bot uses GNU ld and ld.gold at different
points, which could be affecting behaviour.
Change this threshold to LLVM_VP_MAX_NUM_VALS_PER_SITE=130 in order to
try and get it below the failure point, at least for the foreseeable
future.
http://lab.llvm.org:8011/#/builders/37/builds/2744
`__llvm_prf_vnodes` and `__llvm_prf_names` are used by runtime but not
referenced via relocation in the translation unit.
With `-z start-stop-gc` (LLD 13 (D96914); GNU ld 2.37 https://sourceware.org/bugzilla/show_bug.cgi?id=27451),
the linker does not let `__start_/__stop_` references retain their sections.
Place `__llvm_prf_vnodes` and `__llvm_prf_names` in `llvm.used` to make
them retained by the linker.
This patch changes most existing `UsedVars` cases to `CompilerUsedVars`
to reflect the ideal state - if the binary format properly supports
section based GC (dead stripping), `llvm.compiler.used` should be sufficient.
`__llvm_prf_vnodes` and `__llvm_prf_names` are switched to `UsedVars`
since we want them to be unconditionally retained by both compiler and linker.
Behaviors on COFF/Mach-O are not affected.
Reviewed By: davidxl
Differential Revision: https://reviews.llvm.org/D97649
`__llvm_prf_vnodes` and `__llvm_prf_names` are used by runtime but not
referenced via relocation in the translation unit.
With `-z start-stop-gc` (D96914 https://sourceware.org/bugzilla/show_bug.cgi?id=27451),
the linker no longer lets `__start_/__stop_` references retain them.
Place `__llvm_prf_vnodes` and `__llvm_prf_names` in `llvm.used` to make
them retained by the linker.
This patch changes most existing `UsedVars` cases to `CompilerUsedVars`
to reflect the ideal state - if the binary format properly supports
section based GC (dead stripping), `llvm.compiler.used` should be sufficient.
`__llvm_prf_vnodes` and `__llvm_prf_names` are switched to `UsedVars`
since we want them to be unconditionally retained by both compiler and linker.
Behaviors on other COFF/Mach-O are not affected.
Differential Revision: https://reviews.llvm.org/D97649
This reverts commit 680f836c2f.
Disable the non-default-rounding-mode scalbn[f] tests when we're using
the MSVC libraries.
Differential Revision: https://reviews.llvm.org/D91841
This would cause linking errors after https://reviews.llvm.org/D97483
that introduced new prefixes for ABI wrappers with origin tracking mode.
We will renable this after the full origin tracking is checked in.
DFSan at store does store shadow data; store app data; and at load does
load shadow data; load app data.
When an application data is atomic, one overtainting case is
thread A: load shadow
thread B: store shadow
thread B: store app
thread A: load app
If the application address had been used by other flows, thread A reads
previous shadow, causing overtainting.
The change is similar to MSan's solution.
1) enforce ordering of app load/store
2) load shadow after load app; store shadow before shadow app
3) do not track atomic store by reseting its shadow to be 0.
The last one is to address a case like this.
Thread A: load app
Thread B: store shadow
Thread A: load shadow
Thread B: store app
This approach eliminates overtainting as a trade-off between undertainting
flows via shadow data race.
Note that this change addresses only native atomic instructions, but
does not support builtin libcalls yet.
https://llvm.org/docs/Atomics.html#libcalls-atomic
Reviewed-by: morehouse
Differential Revision: https://reviews.llvm.org/D97310
Fix a buffer overrun that can occur when parsing '%c' at the end of a
filename pattern string.
rdar://74571261
Reviewed By: kastiglione
Differential Revision: https://reviews.llvm.org/D97239
Define inline versions of __compiler_rt_fmax* and __compiler_rt_scalbn*
rather than depend on the versions in libm. As with
__compiler_rt_logbn*, these functions are only defined for single,
double, and quad precision (binary128).
Fixes PR32279 for targets using only these FP formats (e.g. Android
on arm/arm64/x86/x86_64).
For single and double precision, on AArch64, use __builtin_fmax[f]
instead of the new inline function, because the builtin expands to the
AArch64 fmaxnm instruction.
Reviewed By: MaskRay
Differential Revision: https://reviews.llvm.org/D91841
Added a lit test that finds its corresponding crash log and checks to make sure it has asn output under `Application Specific Information`.
This required adding two python commands:
- `get_pid_from_output`: takes the output from the asan instrumentation and parses out the process ID
- `print_crashreport_for_pid`: takes in the pid of the process and the file name of the binary that was run and prints the contents of the corresponding crash log.
This test was added in preparation for changing the integration with crash reporter from the old api to the new api, which is implemented in a subsequent commit.
rdar://69767688
Reviewed By: delcypher
Commited by Dan Liew on behalf of Emily Shi.
Differential Revision: https://reviews.llvm.org/D96737
Depending on the order in which lld and compiler-rt projects are
processed by CMake, `TARGET lld` might evaluate to `TRUE` or `FALSE`
even though `lld-available` lit stanza is always set because lld is
being built. We check whether lld project is enabled instead which
is used by other compiler-rt tests.
The ideal solution here would be to use CMake generator expressions,
but those cannot be used for dependencies yet, see:
https://gitlab.kitware.com/cmake/cmake/-/issues/19467
Differential Revision: https://reviews.llvm.org/D97256
__start_/__stop_ references retain C identifier name sections such as
__llvm_prf_*. Putting these into a section group disables this logic.
The ELF section group semantics ensures that group members are retained
or discarded as a unit. When a function symbol is discarded, this allows
allows linker to discard counters, data and values associated with that
function symbol as well.
Note that `noduplicates` COMDAT is lowered to zero-flag section group in
ELF. We only set this for functions that aren't already in a COMDAT and
for those that don't have available_externally linkage since we already
use regular COMDAT groups for those.
Differential Revision: https://reviews.llvm.org/D96757
__start_/__stop_ references retain C identifier name sections such as
__llvm_prf_*. Putting these into a section group disables this logic.
The ELF section group semantics ensures that group members are retained
or discarded as a unit. When a function symbol is discarded, this allows
allows linker to discard counters, data and values associated with that
function symbol as well.
Note that `noduplicates` COMDAT is lowered to zero-flag section group in
ELF. We only set this for functions that aren't already in a COMDAT and
for those that don't have available_externally linkage since we already
use regular COMDAT groups for those.
Differential Revision: https://reviews.llvm.org/D96757
When adding this function in https://reviews.llvm.org/D68794 I did not
notice that internal_prctl has the API of the syscall to prctl rather
than the API of the glibc (posix) wrapper.
This means that the error return value is not necessarily -1 and that
errno is not set by the call.
For InitPrctl this means that the checks do not catch running on a
kernel *without* the required ABI (not caught since I only tested this
function correctly enables the ABI when it exists).
This commit updates the two calls which check for an error condition to
use internal_iserror. That function sets a provided integer to an
equivalent errno value and returns a boolean to indicate success or not.
Tested by running on a kernel that has this ABI and on one that does
not. Verified that running on the kernel without this ABI the current
code prints the provided error message and does not attempt to run the
program. Verified that running on the kernel with this ABI the current
code does not print an error message and turns on the ABI.
This done on an x86 kernel (where the ABI does not exist), an AArch64
kernel without this ABI, and an AArch64 kernel with this ABI.
In order to keep running the testsuite on kernels that do not provide
this new ABI we add another option to the HWASAN_OPTIONS environment
variable, this option determines whether the library kills the process
if it fails to enable the relaxed syscall ABI or not.
This new flag is `fail_without_syscall_abi`.
The check-hwasan testsuite results do not change with this patch on
either x86, AArch64 without a kernel supporting this ABI, and AArch64
with a kernel supporting this ABI.
Differential Revision: https://reviews.llvm.org/D96964
If tsan runtime will try to allocate something during exit handling,
the allocation will fail because there is no VA whatsoever.
It's observed to fail with the following error in some cases:
failed to allocate 0x1000 (4096) bytes of DTLS_NextBlock.
So terminate the process immediately.
Reviewed-in: https://reviews.llvm.org/D96874
This test was found to fail for some of our downstream builds, on
computers where python was not on the default $PATH. Therefore
add a %python substitution to use sys.executable, based on similar
solutions for python calls in tests elsewhere in LLVM.
Differential Revision: https://reviews.llvm.org/D96799
If an app mmaps lots of memory, a user mmap may end up
in the tsan region for traces. Shadow for this range
overlaps with shadow for other user regions.
This causes havok: from false positives to crashes.
Don't leave unmapped holes in the traces region.
Reviewed-in: https://reviews.llvm.org/D96697
Vitaly Buka