- Enable extra coverage counters on Windows.
- Update extra_counters.test to run on Windows also.
- Update TableLookupTest.cpp to include the required pragma/declspec for the extra coverage counters.
Patch By: MichaelSquires
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D106676
Similar to qsort, bsearch can be called from non-instrumented
code of glibc. When it happends tls for arguments can be in uninitialized
state.
Unlike to qsort, bsearch does not move data, so we don't need to
check or initialize searched memory or key. Intrumented comparator will
do that on it's own.
Differential Revision: https://reviews.llvm.org/D107387
This fixes support for merging profiles which broke as a consequence
of e50a38840d. The issue was missing
adjustment in merge logic to account for the binary IDs which are
now included in the raw profile just after header.
In addition, this change also:
* Includes the version in module signature that's used for merging
to avoid accidental attempts to merge incompatible profiles.
* Moves the binary IDs size field after version field in the header
as was suggested in the review.
Differential Revision: https://reviews.llvm.org/D107143
This fixes support for merging profiles which broke as a consequence
of e50a38840d. The issue was missing
adjustment in merge logic to account for the binary IDs which are
now included in the raw profile just after header.
In addition, this change also:
* Includes the version in module signature that's used for merging
to avoid accidental attempts to merge incompatible profiles.
* Moves the binary IDs size field after version field in the header
as was suggested in the review.
Differential Revision: https://reviews.llvm.org/D107143
As code diverge from Google style we need
to add more and more exceptions to suppress
conflicts with clang-format and clang-tidy.
As this point it does not provide a additional value.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D107197
This fixes support for merging profiles which broke as a consequence
of e50a38840d. The issue was missing
adjustment in merge logic to account for the binary IDs which are
now included in the raw profile just after header.
In addition, this change also:
* Includes the version in module signature that's used for merging
to avoid accidental attempts to merge incompatible profiles.
* Moves the binary IDs size field after version field in the header
as was suggested in the review.
Differential Revision: https://reviews.llvm.org/D107143
The test contains a race in memset.
The size of reported race depends on how the accessed
memory range split into granules inside of tsan runtime.
The test used to report access of size 8, because presumably
the buffer ended up being aligned to 8 bytes. But after
some unrelated changes this test started to report accesses
of size 1 (presumably .data layout changed), which makes
the test fail.
Guarantee alignment of the buf object explicitly.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107149
The updated lots_of_threads.c test with 300 threads
started running for too long on machines with low
hardware parallelism (e.g. taskset -c 0-1).
On lots of CPUs it finishes in ~2 secs. But with
taskset -c 0-1 it runs for hundreds of seconds
effectively spinning in the barrier in the sleep loop.
We now have the handy futex API in sanitizer_common.
Use it instead of the passive spin loop.
It makes the test run only faster with taskset -c 0-1,
it runs for ~1.5 secs, while with full parallelism
it still runs for ~2 secs (but consumes less CPU time).
Depends on D107131.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D107132
The test contains a race in read/write syscalls.
The size of reported race depends on how the accessed
memory range split into granules inside of tsan runtime.
The test used to report access of size 8, because presumably
the buffer ended up being aligned to 8 bytes. But after
some unrelated changes this test started to report accesses
of size 1 (presumably .data layout changed), which makes
the test fail.
Guarantee alignment of the buf object explicitly.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D107131
Add a test where atomic-release happens while
another thread spins calling load-acquire.
This can expose some interesting interleavings
of release and acquire.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107055
We reliably remove bottom libc-guts frames only on linux/glibc.
Some bots failed on this test showing other bottom frames:
.annobin_libc_start.c libc-start.c (libc.so.6+0x249f4)
generic_start_main.isra.0 libc-start.c (libc.so.6+0x45b0c)
We can't reliably remove all of possible bottom frames.
So remove the assertion for that.
Differential Revision: https://reviews.llvm.org/D107037
Currently __tsan_atomic* functions do FuncEntry/Exit using caller PC
and then use current PC (pointing to __tsan_atomic* itself) during
memory access handling. As the result the top function in reports
involving atomics is __tsan_atomic* and the next frame points to user code.
Remove FuncEntry/Exit in atomic functions and use caller PC
during memory access handling. This removes __tsan_atomic*
from the top of report stacks, so that they point right to user code.
The motivation for this is performance.
Some atomic operations are very hot (mostly loads),
so removing FuncEntry/Exit is beneficial.
This also reduces thread trace consumption (1 event instead of 3).
__tsan_atomic* at the top of the stack is not necessary
and does not add any new information. We already say
"atomic write of size 4", "__tsan_atomic32_store" does not add
anything new.
It also makes reports consistent between atomic and non-atomic
accesses. For normal accesses we say "previous write" and point
to user code; for atomics we say "previous atomic write" and now
also point to user code.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106966
We strip all frames below main but in some cases it may be not enough.
Namely, when main is instrumented but does not call any other instrumented code.
In this case __tsan_func_entry in main obtains PC pointing to __libc_start_main
(as we pass caller PC to __tsan_func_entry), but nothing obtains PC pointing
to main itself (as main does not call any instrumented code).
In such case we will not have main in the stack, and stripping everything
below main won't work.
So strip __libc_start_main explicitly as well.
But keep stripping of main because __libc_start_main is glibc/linux-specific,
so looking for main is more reliable (and usually main is present in stacks).
Depends on D106957.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106958
Caller PC is plain harmful as native caller PC has nothing to do with Java code.
Current PC is not particularly useful and may be somewhat confusing for Java users
as it makes top frame to point to some magical __tsan function.
But obtaining and using these PCs adds runtime cost for every java event.
Remove these PCs. Rely only on official Java frames.
It makes execution faster, code simpler and reports better.
Depends on D106956.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106957
We maintain information about Java allocations,
but for some reason never printed it in reports.
Print it.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106956
1. Add a set of micro benchmarks for memory accesses,
mem* functions and unaligned accesses.
2. Add support for multiple benchmarks in a single binary
(or it would require 12 new benchmark binaries).
3. Remove the "clock growth" machinery,
it affects the current tsan runtime by increasing size of
all vector clocks, but this won't be relevant for the new
tsan runtime.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106961
The current 10 threads is not particularly "lots" and not stressful.
Create 10x300 threads and ensure they all are running at the same time.
Depends on D106953.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D106954
Test that we report the warning for free()
and ensure the test finishes as we usually do with "DONE".
Depends on D106951.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D106952
Compilers don't like attributes in this position:
warning: GCC does not allow 'noinline' attribute in this position on a function definition
error: attributes are not allowed on a function-definition
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D106951
This reverts commit 1e1f752027.
It breaks ignore_noninstrumented_modules=1.
Somehow we did not have any portable tests for this mode before
(only Darwin tests). Add a portable test as well.
Moreover, I think I was too fast uninlining all LibIgnore checks.
For Java, Darwin and OpenMP LibIgnore is always enabled,
so it makes sense to leave it as it was before.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D106855
Add the ability to:
1. tell simctl to wait for debugger when spawning process
2. print the command that is called to launch the process
Reviewed By: delcypher
Differential Revision: https://reviews.llvm.org/D106700
signal(2) and sigaction(2) have defined behaviors for invalid signal number
(EINVAL) and some programs rely on it.
The added test case also reveals that MSAN is too strict in this regard.
Test case passed on x86_64 Linux and AArch64 Linux.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106468
This reverts commit 6b2a96285b.
The ccache builders are still failing. Looks like they need to be updated to
get the llvm-zorg config change in 490633945677656ba75d42ff1ca9d4a400b7b243.
I'll re-apply this as soon as the builders are updated.
This reapplies commit a7733e9556 ("Re-apply
[ORC][ORC-RT] Add initial native-TLV support to MachOPlatform."), and
d4abdefc99 ("[ORC-RT] Rename macho_tlv.x86-64.s
to macho_tlv.x86-64.S (uppercase suffix)").
These patches were reverted in 48aa82cacb while I
investigated bot failures (e.g.
https://lab.llvm.org/buildbot/#/builders/109/builds/18981). The fix was to
disable building of the ORC runtime on buliders using ccache (which is the same
fix used for other compiler-rt projects containing assembly code). This fix was
commited to llvm-zorg in 490633945677656ba75d42ff1ca9d4a400b7b243.
Since d564cfb53c moved
__hwasan_tag_mismatch4 this test has been reporting
a frame 0 of __hwasan_tag_mismatch_v2.
This failure can be seen on our bots:
https://lab.llvm.org/buildbot/#/builders/185/builds/170
Before the change:
#0 0xaaaaba100e40 in main <...>/register-dump-read.c:21:10
After the change:
#0 0xaaaab8494bec in __hwasan_tag_mismatch_v2 <...>/hwasan/hwasan_tag_mismatch_aarch64.S:147
#1 0xaaaab84b4df8 in main <..>/register-dump-read.c:14:10
Update the test to check for a main frame as either frame
0 or frame 1.
This reverts commit d4abdefc99 ("[ORC-RT] Rename
macho_tlv.x86-64.s to macho_tlv.x86-64.S (uppercase suffix)", and
a7733e9556 ("Re-apply "[ORC][ORC-RT] Add initial
native-TLV support to MachOPlatform."), while I investigate failures on
ccache builders (e.g. https://lab.llvm.org/buildbot/#/builders/109/builds/18981)
Reapplies fe1fa43f16, which was reverted in
6d8c63946c, with fixes:
1. Remove .subsections_via_symbols directive from macho_tlv.x86-64.s (it's
not needed here anyway).
2. Return error from pthread_key_create to the MachOPlatform to silence unused
variable warning.
Adds code to LLVM (MachOPlatform) and the ORC runtime to support native MachO
thread local variables. Adding new TLVs to a JITDylib at runtime is supported.
On the LLVM side MachOPlatform is updated to:
1. Identify thread local variables in the LinkGraph and lower them to GOT
accesses to data in the __thread_data or __thread_bss sections.
2. Merge and report the address range of __thread_data and thread_bss sections
to the runtime.
On the ORC runtime a MachOTLVManager class introduced which records the address
range of thread data/bss sections, and creates thread-local instances from the
initial data on demand. An orc-runtime specific tlv_get_addr implementation is
included which saves all register state then calls the MachOTLVManager to get
the address of the requested variable for the current thread.
These have been failing on our bots for a while due to
incomplete backtraces. (you don't get the names of the
functions that did the access, just the reporter frames)
See:
https://lab.llvm.org/buildbot/#/builders/170/builds/180
Adds support for MachO static initializers/deinitializers and eh-frame
registration via the ORC runtime.
This commit introduces cooperative support code into the ORC runtime and ORC
LLVM libraries (especially the MachOPlatform class) to support macho runtime
features for JIT'd code. This commit introduces support for static
initializers, static destructors (via cxa_atexit interposition), and eh-frame
registration. Near-future commits will add support for MachO native
thread-local variables, and language runtime registration (e.g. for Objective-C
and Swift).
The llvm-jitlink tool is updated to use the ORC runtime where available, and
regression tests for the new MachOPlatform support are added to compiler-rt.
Notable changes on the ORC runtime side:
1. The new macho_platform.h / macho_platform.cpp files contain the bulk of the
runtime-side support. This includes eh-frame registration; jit versions of
dlopen, dlsym, and dlclose; a cxa_atexit interpose to record static destructors,
and an '__orc_rt_macho_run_program' function that defines running a JIT'd MachO
program in terms of the jit- dlopen/dlsym/dlclose functions.
2. Replaces JITTargetAddress (and casting operations) with ExecutorAddress
(copied from LLVM) to improve type-safety of address management.
3. Adds serialization support for ExecutorAddress and unordered_map types to
the runtime-side Simple Packed Serialization code.
4. Adds orc-runtime regression tests to ensure that static initializers and
cxa-atexit interposes work as expected.
Notable changes on the LLVM side:
1. The MachOPlatform class is updated to:
1.1. Load the ORC runtime into the ExecutionSession.
1.2. Set up standard aliases for macho-specific runtime functions. E.g.
___cxa_atexit -> ___orc_rt_macho_cxa_atexit.
1.3. Install the MachOPlatformPlugin to scrape LinkGraphs for information
needed to support MachO features (e.g. eh-frames, mod-inits), and
communicate this information to the runtime.
1.4. Provide entry-points that the runtime can call to request initializers,
perform symbol lookup, and request deinitialiers (the latter is
implemented as an empty placeholder as macho object deinits are rarely
used).
1.5. Create a MachO header object for each JITDylib (defining the __mh_header
and __dso_handle symbols).
2. The llvm-jitlink tool (and llvm-jitlink-executor) are updated to use the
runtime when available.
3. A `lookupInitSymbolsAsync` method is added to the Platform base class. This
can be used to issue an async lookup for initializer symbols. The existing
`lookupInitSymbols` method is retained (the GenericIRPlatform code is still
using it), but is deprecated and will be removed soon.
4. JIT-dispatch support code is added to ExecutorProcessControl.
The JIT-dispatch system allows handlers in the JIT process to be associated with
'tag' symbols in the executor, and allows the executor to make remote procedure
calls back to the JIT process (via __orc_rt_jit_dispatch) using those tags.
The primary use case is ORC runtime code that needs to call bakc to handlers in
orc::Platform subclasses. E.g. __orc_rt_macho_jit_dlopen calling back to
MachOPlatform::rt_getInitializers using __orc_rt_macho_get_initializers_tag.
(The system is generic however, and could be used by non-runtime code).
The new ExecutorProcessControl::JITDispatchInfo struct provides the address
(in the executor) of the jit-dispatch function and a jit-dispatch context
object, and implementations of the dispatch function are added to
SelfExecutorProcessControl and OrcRPCExecutorProcessControl.
5. OrcRPCTPCServer is updated to support JIT-dispatch calls over ORC-RPC.
6. Serialization support for StringMap is added to the LLVM-side Simple Packed
Serialization code.
7. A JITLink::allocateBuffer operation is introduced to allocate writable memory
attached to the graph. This is used by the MachO header synthesis code, and will
be generically useful for other clients who want to create new graph content
from scratch.
ptrauth stores info in the address of functions, so it's not the right address we should check if poisoned
rdar://75246928
Differential Revision: https://reviews.llvm.org/D106199
This was fixed in the past for `frexp`, but was not made for `frexpl` & `frexpf` https://github.com/google/sanitizers/issues/321
This patch copies the fix over to `frexpl` because it caused `frexp_interceptor.cpp` test to fail on iPhone and `frexpf` for consistency.
rdar://79652161
Reviewed By: delcypher, vitalybuka
Differential Revision: https://reviews.llvm.org/D104948
Define the address ranges (similar to the C/C++ ones, but with the heap
range merged into the app range) and enable the sanity check.
Reviewed By: dvyukov
Differential Revision: https://reviews.llvm.org/D105629
XFAIL map32bit, define the maximum possible allocation size in
mmap_large.cpp.
Reviewed By: dvyukov
Differential Revision: https://reviews.llvm.org/D105629
These tests depend on TSan seeing the intercepted memcpy(), so they
break when the compiler chooses the builtin version.
Reviewed By: dvyukov
Differential Revision: https://reviews.llvm.org/D105629
The existing one actually failed on the int* p, not on int z (as can be
seen by the fault being 8 bytes rather than 4).
This is also needed to make sure the stack safety analysis does not
classify the alloca as safe.
Reviewed By: hctim
Differential Revision: https://reviews.llvm.org/D105705
This reverts commit 52aeacfbf5.
There isn't full agreement on a path forward yet, but there is agreement that
this shouldn't land as-is. See discussion on https://reviews.llvm.org/D105338
Also reverts unreviewed "[clang] Improve `-Wnull-dereference` diag to be more in-line with reality"
This reverts commit f4877c78c0.
And all the related changes to tests:
This reverts commit 9a0152799f.
This reverts commit 3f7c9cc274.
This reverts commit 329f8197ef.
This reverts commit aa9f58cc2c.
This reverts commit 2df37d5ddd.
This reverts commit a72a441812.
Store to null is deleted, so the test no longer did what it was expecting to do.
Conceal that by creating null pointer in a more elaborate way,
thus retaining original test coverage.
Update the asan_symbolize_script for changes in argparse output
in Python 3.10. The parser output 'options' instead of 'optional
arguments'.
Differential Revision: https://reviews.llvm.org/D105489
The __llvm_prf_names section uses SHF_GNU_RETAIN. However, GNU ld before 2015-10
(https://sourceware.org/bugzilla/show_bug.cgi?id=19161) neither supports it nor
retains __llvm_prf_names according to __start___llvm_prf_names. So --gc-sections
does not work on such old GNU ld.
This is not a problem for gold and sufficiently new lld.
We would find an address with matching tag, only to discover in
ShowCandidate that it's very far away from [stack].
Reviewed By: eugenis
Differential Revision: https://reviews.llvm.org/D105197
If the fault address is at the boundary of memory regions, this could
cause us to segfault otherwise.
Ran test with old compiler_rt to make sure it fails.
Reviewed By: eugenis
Differential Revision: https://reviews.llvm.org/D105032
This change introduces libMutagen/libclang_rt.mutagen.a as a subset of libFuzzer/libclang_rt.fuzzer.a. This library contains only the fuzzing strategies used by libFuzzer to produce new test inputs from provided inputs, dictionaries, and SanitizerCoverage feedback.
Most of this change is simply moving sections of code to one side or the other of the library boundary. The only meaningful new code is:
* The Mutagen.h interface and its implementation in Mutagen.cpp.
* The following methods in MutagenDispatcher.cpp:
* UseCmp
* UseMemmem
* SetCustomMutator
* SetCustomCrossOver
* LateInitialize (similar to the MutationDispatcher's original constructor)
* Mutate_AddWordFromTORC (uses callbacks instead of accessing TPC directly)
* StartMutationSequence
* MutationSequence
* DictionaryEntrySequence
* RecommendDictionary
* RecommendDictionaryEntry
* FuzzerMutate.cpp (which now justs sets callbacks and handles printing)
* MutagenUnittest.cpp (which adds tests of Mutagen.h)
A note on performance: This change was tested with a 100 passes of test/fuzzer/LargeTest.cpp with 1000 runs per pass, both with and without the change. The running time distribution was qualitatively similar both with and without the change, and the average difference was within 30 microseconds (2.240 ms/run vs 2.212 ms/run, respectively). Both times were much higher than observed with the fully optimized system clang (~0.38 ms/run), most likely due to the combination of CMake "dev mode" settings (e.g. CMAKE_BUILD_TYPE="Debug", LLVM_ENABLE_LTO=OFF, etc.). The difference between the two versions built similarly seems to be "in the noise" and suggests no meaningful performance degradation.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D102447
This allows application code checks if origin tracking is on before
printing out traces.
-dfsan-track-origins can be 0,1,2.
The current code only distinguishes 1 and 2 in compile time, but not at runtime.
Made runtime distinguish 1 and 2 too.
Reviewed By: browneee
Differential Revision: https://reviews.llvm.org/D105128
A heap or global buffer that is far away from the faulting address is
unlikely to be the cause, especially if there is a potential
use-after-free as well, so we want to show it after the other
causes.
Reviewed By: eugenis
Differential Revision: https://reviews.llvm.org/D104781
I can't be sure of the cause but I believe these fail
due to to fast unwinding not working on Thumb.
Whatever the case, they have been failing on our bots
for a long time:
https://lab.llvm.org/buildbot/#/builders/170/builds/46
Require fast-unwinder-works for both.
Word on the grapevine was that the committee had some discussion that
ended with unanimous agreement on eliminating relational function pointer comparisons.
We wanted to be bold and just ban all of them cold turkey.
But then we chickened out at the last second and are going for
eliminating just the spaceship overload candidate instead, for now.
See D104680 for reference.
This should be fine and "safe", because the only possible semantic change this
would cause is that overload resolution could possibly be ambiguous if
there was another viable candidate equally as good.
But to save face a little we are going to:
* Issue an "error" for three-way comparisons on function pointers.
But all this is doing really is changing one vague error message,
from an "invalid operands to binary expression" into an
"ordered comparison of function pointers", which sounds more like we mean business.
* Otherwise "warn" that comparing function pointers like that is totally
not cool (unless we are told to keep quiet about this).
Signed-off-by: Matheus Izvekov <mizvekov@gmail.com>
Reviewed By: rsmith
Differential Revision: https://reviews.llvm.org/D104892
on arm64e, pointer auth would catch this access violation before asan.
sign the function pointer so pointer auth will ignore this violation and let asan catch it in this test case.
rdar://79652167
Reviewed By: delcypher
Differential Revision: https://reviews.llvm.org/D104828
The comment says it was flaky in 2016,
but it wasn't possible to debug it back then.
Re-enable the test at least on linux/x86_64.
It will either work, or at least we should
see failure output from lit today.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D104592
Mmap interceptor is not atomic in the sense that it
exposes unmapped shadow for a brief period of time.
This breaks programs that mmap over another mmap
and access the region concurrently.
Don't unmap shadow in the mmap interceptor to fix this.
Just mapping new shadow on top should be enough to zero it.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D104593
These have been broken by https://reviews.llvm.org/D104494.
However, `lib/fuzzer/dataflow/` is unused (?) so addressing this is not a priority.
Added TODOs to re-enable them in the future.
Reviewed By: stephan.yichao.zhao
Differential Revision: https://reviews.llvm.org/D104568
Explain what the given stack trace means before showing it, rather than
only in the paragraph at the end.
Reviewed By: eugenis
Differential Revision: https://reviews.llvm.org/D104523
The default callback instrumentation in x86 LAM mode uses ASLR bits
to randomly choose a tag, and thus has a 1/64 chance of choosing a
stack tag of 0, causing stack tests to fail intermittently. By using
__hwasan_generate_tag to pick tags, we guarantee non-zero tags and
eliminate the test flakiness.
aarch64 doesn't seem to have this problem using thread-local addresses
to pick tags, so perhaps we can remove this workaround once we implement
a similar mechanism for LAM.
Reviewed By: eugenis
Differential Revision: https://reviews.llvm.org/D104470
The current naming scheme adds the `dfs$` prefix to all
DFSan-instrumented functions. This breaks mangling and prevents stack
trace printers and other tools from automatically demangling function
names.
This new naming scheme is mangling-compatible, with the `.dfsan`
suffix being a vendor-specific suffix:
https://itanium-cxx-abi.github.io/cxx-abi/abi.html#mangling-structure
With this fix, demangling utils would work out-of-the-box.
Reviewed By: stephan.yichao.zhao
Differential Revision: https://reviews.llvm.org/D104494
Before: ADDR is located -320 bytes to the right of 1072-byte region
After: ADDR is located 752 bytes inside 1072-byte region
Reviewed By: eugenis, walli99
Differential Revision: https://reviews.llvm.org/D104412
Adds the basic instrumentation needed for stack tagging.
Currently does not support stack short granules or TLS stack histories,
since a different code path is followed for the callback instrumentation
we use.
We may simply wait to support these two features until we switch to
a custom calling convention.
Patch By: xiangzhangllvm, morehouse
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D102901
This mostly follows LLVM's InstrProfReader.cpp error handling.
Previously, attempting to merge corrupted profile data would result in
crashes. See https://crbug.com/1216811#c4.
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D104050
Complete support for fast8:
- amend shadow size and mapping in runtime
- remove fast16 mode and -dfsan-fast-16-labels flag
- remove legacy mode and make fast8 mode the default
- remove dfsan-fast-8-labels flag
- remove functions in dfsan interface only applicable to legacy
- remove legacy-related instrumentation code and tests
- update documentation.
Reviewed By: stephan.yichao.zhao, browneee
Differential Revision: https://reviews.llvm.org/D103745
dfsan does not use sanitizer allocator as others. In practice,
we let it use glibc's allocator since tcmalloc needs more work
to be working with dfsan well. With glibc, we observe large
memory leakage. This could relate to two things:
1) glibc allocator has limitation: for example, tcmalloc can reduce memory footprint 2x easily
2) glibc may call unmmap directly as an internal system call by using system call number. so DFSan has no way to release shadow spaces for those unmmap.
Using sanitizer allocator addresses the above issues
1) its memory management is close to tcmalloc
2) we can register callback when sanitizer allocator calls unmmap, so dfsan can release shadow spaces correctly.
Our experiment with internal server-based application proved that with the change, in a-few-day run, memory usage leakage is close to what tcmalloc does w/o dfsan.
This change mainly follows MSan's code.
1) define allocator callbacks at dfsan_allocator.h|cpp
2) mark allocator APIs to be discard
3) intercept allocator APIs
4) make dfsan_set_label consistent with MSan's SetShadow when setting 0 labels, define dfsan_release_meta_memory when unmap is called
5) add flags about whether zeroing memory after malloc/free. dfsan works at byte-level, so bit-level oparations can cause reading undefined shadow. See D96842. zeroing memory after malloc helps this. About zeroing after free, reading after free is definitely UB, but if user code does so, it is hard to debug an overtainting caused by this w/o running MSan. So we add the flag to help debugging.
This change will be split to small changes for review. Before that, a question is
"this code shares a lot of with MSan, for example, dfsan_allocator.* and dfsan_new_delete.*.
Does it make sense to unify the code at sanitizer_common? will that introduce some
maintenance issue?"
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D101204
The linkage/visibility of `__profn_*` variables are derived
from the profiled functions.
extern_weak => linkonce
available_externally => linkonce_odr
internal => private
extern => private
_ => unchanged
The linkage/visibility of `__profc_*`/`__profd_*` variables are derived from
`__profn_*` with linkage/visibility wrestling for Windows.
The changes can be folded to the following without changing semantics.
```
if (TT.isOSBinFormatCOFF() && !NeedComdat) {
Linkage = GlobalValue::InternalLinkage;
Visibility = GlobalValue::DefaultVisibility;
}
```
That said, I think we can just delete the code block.
An extern/internal function will now use private `__profc_*`/`__profd_*`
variables, instead of internal ones. This saves some symbol table entries.
A non-comdat {linkonce,weak}_odr function will now use hidden external
`__profc_*`/`__profd_*` variables instead of internal ones. There is potential
object file size increase because such symbols need `/INCLUDE:` directives.
However such non-comdat functions are rare (note that non-comdat weak
definitions don't prevent duplicate definition error).
The behavior changes match ELF.
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D103355
We have been seeing this test fail intermittently on our
2 stage AArch64 bot.
As far back as https://lab.llvm.org/buildbot/#/builders/53/builds/2694
Likely due to a lack of resources at certain times on the
shared machine. Up the time limit to give us some more room.
(this limit only applies to the watchdog thread, so if the
test passes then it won't take 20s)
Since https://reviews.llvm.org/D102046 some tests have
been falling back to fast unwinding on our Thumb bot.
This fails because fast unwinding does not work on Thumb.
By adding the extra information we ensure this does not happen
during testing, but the built library can still fast unwind
as a last resort.
Since there are some situations it can work in, like if
eveything is built with clang. During testing we've got gcc
built system libs and clang built tests.
The same change was made for sanitizer-common in
https://reviews.llvm.org/D96337.
Reviewed By: zatrazz
Differential Revision: https://reviews.llvm.org/D103463
DFSan has flags to control flows between pointers and objects referred
by pointers. For example,
a = *p;
L(a) = L(*p) when -dfsan-combine-pointer-labels-on-load = false
L(a) = L(*p) + L(p) when -dfsan-combine-pointer-labels-on-load = true
*p = b;
L(*p) = L(b) when -dfsan-combine-pointer-labels-on-store = false
L(*p) = L(b) + L(p) when -dfsan-combine-pointer-labels-on-store = true
The question is what to do with p += c.
In practice we found many confusing flows if we propagate labels from c
to p. So a new flag works like this
p += c;
L(p) = L(p) when -dfsan-propagate-via-pointer-arithmetic = false
L(p) = L(p) + L(c) when -dfsan-propagate-via-pointer-arithmetic = true
Reviewed-by: gbalats
Differential Revision: https://reviews.llvm.org/D103176
This change introduces libMutagen/libclang_rt.mutagen.a as a subset of libFuzzer/libclang_rt.fuzzer.a. This library contains only the fuzzing strategies used by libFuzzer to produce new test inputs from provided inputs, dictionaries, and SanitizerCoverage feedback.
Most of this change is simply moving sections of code to one side or the other of the library boundary. The only meaningful new code is:
* The Mutagen.h interface and its implementation in Mutagen.cpp.
* The following methods in MutagenDispatcher.cpp:
* UseCmp
* UseMemmem
* SetCustomMutator
* SetCustomCrossOver
* LateInitialize (similar to the MutationDispatcher's original constructor)
* Mutate_AddWordFromTORC (uses callbacks instead of accessing TPC directly)
* StartMutationSequence
* MutationSequence
* DictionaryEntrySequence
* RecommendDictionary
* RecommendDictionaryEntry
* FuzzerMutate.cpp (which now justs sets callbacks and handles printing)
* MutagenUnittest.cpp (which adds tests of Mutagen.h)
A note on performance: This change was tested with a 100 passes of test/fuzzer/LargeTest.cpp with 1000 runs per pass, both with and without the change. The running time distribution was qualitatively similar both with and without the change, and the average difference was within 30 microseconds (2.240 ms/run vs 2.212 ms/run, respectively). Both times were much higher than observed with the fully optimized system clang (~0.38 ms/run), most likely due to the combination of CMake "dev mode" settings (e.g. CMAKE_BUILD_TYPE="Debug", LLVM_ENABLE_LTO=OFF, etc.). The difference between the two versions built similarly seems to be "in the noise" and suggests no meaningful performance degradation.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D102447
This reverts commit 6911114d8c.
Broke the QEMU sanitizer bots due to a missing header dependency. This
actually needs to be fixed on the bot-side, but for now reverting this
patch until I can fix up the bot.
Vitaly Buka