Currently we effectively duplicate "once" logic for __cxa_guard_acquire
and pthread_once. Unify the implementations.
This is not a no-op change:
- constants used for pthread_once are changed to match __cxa_guard_acquire
(__cxa_guard_acquire constants are tied to ABI, but it does not seem
to be the case for pthread_once)
- pthread_once now also uses PotentiallyBlockingRegion annotations
- __cxa_guard_acquire checks thr->in_ignored_lib to skip user synchronization
It's unclear if these 2 differences are intentional or a mere sloppy inconsistency.
Since all tests still pass, let's assume the latter.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D107359
Atomic functions are semi-hot in profiles.
The CHECKs verify values passed by compiler
and they never fired, so replace them with DCHECKs.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D107373
Similar to qsort, bsearch can be called from non-instrumented
code of glibc. When it happends tls for arguments can be in uninitialized
state.
Unlike to qsort, bsearch does not move data, so we don't need to
check or initialize searched memory or key. Intrumented comparator will
do that on it's own.
Differential Revision: https://reviews.llvm.org/D107387
On macOS the unit tests currently rely on libmalloc being used for
allocations (due to no functioning interceptors) but also having the
ASan/TSan allocator initialized in the same process.
This leads to crashes with the macOS 12.0 libmalloc nano allocator so
disable use of the allocator while running unit tests as a workaround.
rdar://80086125
Differential Revision: https://reviews.llvm.org/D107412
A `Vector` that doesn't require an initial `reserve()` (eg: with a
default, or small enough capacity) can have a constant initializer.
This changes the code in a few places to make that possible:
- mark a few other functions as `constexpr`
- do without any `reinterpret_cast`
- allow to skip `reserve` from `init`
Differential Revision: https://reviews.llvm.org/D107308
mallopt calls are left-over from the times we used
__libc_malloc/__libc_free for internal allocations.
Now we have own internal allocator, so this is not needed.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107342
We currently use ad-hoc spin waiting to synchronize thread creation
and thread start both ways. But spinning tend to degrade ungracefully
under high contention (lots of threads are created at the same time).
Use semaphores for synchronization instead.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107337
Currently unaligned access functions are defined in tsan_interface.cpp
and do a real call to MemoryAccess. This means we have a real call
and no read/write constant propagation.
Unaligned memory access can be quite hot for some programs
(observed on some compression algorithms with ~90% of unaligned accesses).
Move them to tsan_interface_inl.h to avoid the additional call
and enable constant propagation.
Also reorder the actual store and memory access handling for
__sanitizer_unaligned_store callbacks to enable tail calling
in MemoryAccess.
Depends on D107282.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D107283
Add AccessVptr access type.
For now it's converted to the same thr->is_vptr_access,
but later it will be passed directly to ReportRace
and will enable efficient tail calling in MemoryAccess function
(currently __tsan_vptr_update/__tsan_vptr_read can't use
tail calls in MemoryAccess because of the trailing assignment
to thr->is_vptr_access).
Depends on D107276.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D107282
Currently we have MemoryAccess function that accepts
"bool kAccessIsWrite, bool kIsAtomic" and 4 wrappers:
MemoryRead/MemoryWrite/MemoryReadAtomic/MemoryWriteAtomic.
Such scheme with bool flags is not particularly scalable/extendable.
Because of that we did not have Read/Write wrappers for UnalignedMemoryAccess,
and "true, false" or "false, true" at call sites is not very readable.
Moreover, the new tsan runtime will introduce more flags
(e.g. move "freed" and "vptr access" to memory acccess flags).
We can't have 16 wrappers and each flag also takes whole
64-bit register for non-inlined calls.
Introduce AccessType enum that contains bit mask of
read/write, atomic/non-atomic, and later free/non-free,
vptr/non-vptr.
Such scheme is more scalable, more readble, more efficient
(don't consume multiple registers for these flags during calls)
and allows to cover unaligned and range variations of memory
access functions as well.
Also switch from size log to just size.
The new tsan runtime won't have the limitation of supporting
only 1/2/4/8 access sizes, so we don't need the logarithms.
Also add an inline thunk that converts the new interface to the old one.
For inlined calls it should not add any overhead because
all flags/size can be computed as compile time.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D107276
... and rename it to 'warnIfNonZero' to better-reflect what it actually
does.
The goal is to minimize the amount of logic that's conditionally
compiled under '#if __APPLE__'.
Add new fixed-size vector clock for the new tsan runtime.
For now it's unused.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107167
Currently we save the creation stack for sync objects always.
But it's not needed to some sync objects, most notably atomics.
We simply don't use atomic creation stack anywhere.
Allow callers to control saving of the creation stack
and don't save it for atomics.
Depends on D107257.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107258
MetaMap::GetSync shows up in profiles,
so add LIKELY/UNLIKELY annotations.
Depends on D107256.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107257
Don't lock the sync object inside of MetaMap methods.
This has several advantages:
- the new interface does not confuse thread-safety analysis
so we can remove a bunch of NO_THREAD_SAFETY_ANALYSIS attributes
- this allows use of scoped lock objects
- this allows more flexibility, e.g. locking some other mutex
between searching and locking the sync object
Also prefix the methods with GetSync to be consistent with GetBlock method.
Also make interface wrappers inlinable, otherwise we either end up with
2 copies of the method, or with an additional call.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107256
ProcessPendingSignals is called in all interceptors
and user atomic operations. Make the fast-path check
(no pending signals) inlinable.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107217
Some bots started failing with the following error
after changing Alloc to New. Change it back.
ThreadSanitizer: CHECK failed: ((locked[i].recursion)) == ((0))
4 __sanitizer::CheckedMutex::CheckNoLocks()
5 __tsan::ScopedInterceptor::~ScopedInterceptor()
6 memset
7 __tsan::New<__tsan::ExpectRace>()
8 __tsan::AddExpectRace()
9 BenignRaceImpl()
Differential Revision: https://reviews.llvm.org/D107212
Currently we inconsistently use u32 and int for thread ids,
there are also "unique tid" and "os tid" and just lots of other
things identified by integers.
Additionally new tsan runtime will introduce yet another
thread identifier that is very different from current tids.
Similarly for stack IDs, it's easy to confuse u32 with other
integer identifiers. And when a function accepts u32 or a struct
contains u32 field, it's not always clear what it is.
Add Tid and StackID typedefs to make it clear what is what.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107152
We currently build tests without -g, which is quite inconvenient.
Crash stacks don't have line numbers, gdb don't how line numbers either.
Always build tests with -g.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D107168
"Expected" races is a very ancient facility used in tsanv1 tests.
It's not used/needed anymore. Remove it.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D107175
Currently we setup either sigaction signal handler with 3 arguments
or old style signal handler with 1 argument depending on user handler type.
This unnecessarily complicates code. Always setup sigaction handler.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D107186
This fixes support for merging profiles which broke as a consequence
of e50a38840d. The issue was missing
adjustment in merge logic to account for the binary IDs which are
now included in the raw profile just after header.
In addition, this change also:
* Includes the version in module signature that's used for merging
to avoid accidental attempts to merge incompatible profiles.
* Moves the binary IDs size field after version field in the header
as was suggested in the review.
Differential Revision: https://reviews.llvm.org/D107143
This fixes support for merging profiles which broke as a consequence
of e50a38840d. The issue was missing
adjustment in merge logic to account for the binary IDs which are
now included in the raw profile just after header.
In addition, this change also:
* Includes the version in module signature that's used for merging
to avoid accidental attempts to merge incompatible profiles.
* Moves the binary IDs size field after version field in the header
as was suggested in the review.
Differential Revision: https://reviews.llvm.org/D107143
As code diverge from Google style we need
to add more and more exceptions to suppress
conflicts with clang-format and clang-tidy.
As this point it does not provide a additional value.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D107197
Multiple copies of emulated TLS state means inconsistent results when
accessing the same thread-local variable from different shared objects
(https://github.com/android/ndk/issues/1551). Making `__emutls_get_address`
be a weak default visibility symbol should make the dynamic linker
ensure only a single copy gets used at runtime. This is best-effort, but
the more robust approach of putting emulated TLS into its own shared
object would (a) be a much bigger change, and (b) shared objects are
pretty heavyweight, and adding a new one to a space-constrained
environment isn't an easy sell. Given the expected rarity of direct
accesses to emulated TLS variables across different shared objects, the
best-effort approach should suffice.
Reviewed By: danalbert, rprichard
Differential Revision: https://reviews.llvm.org/D107127
This fixes support for merging profiles which broke as a consequence
of e50a38840d. The issue was missing
adjustment in merge logic to account for the binary IDs which are
now included in the raw profile just after header.
In addition, this change also:
* Includes the version in module signature that's used for merging
to avoid accidental attempts to merge incompatible profiles.
* Moves the binary IDs size field after version field in the header
as was suggested in the review.
Differential Revision: https://reviews.llvm.org/D107143
Change `CountersPtr` in `__profd_` to a label difference, which is a link-time
constant. On ELF, when linking a shared object, this requires that `__profc_` is
either private or linkonce/linkonce_odr hidden. On COFF, we need D104564 so that
`.quad a-b` (64-bit label difference) can lower to a 32-bit PC-relative relocation.
```
# ELF: R_X86_64_PC64 (PC-relative)
.quad .L__profc_foo-.L__profd_foo
# Mach-O: a pair of 8-byte X86_64_RELOC_UNSIGNED and X86_64_RELOC_SUBTRACTOR
.quad l___profc_foo-l___profd_foo
# COFF: we actually use IMAGE_REL_AMD64_REL32/IMAGE_REL_ARM64_REL32 so
# the high 32-bit value is zero even if .L__profc_foo < .L__profd_foo
# As compensation, we truncate CountersDelta in the header so that
# __llvm_profile_merge_from_buffer and llvm-profdata reader keep working.
.quad .L__profc_foo-.L__profd_foo
```
(Note: link.exe sorts `.lprfc` before `.lprfd` even if the object writer
has `.lprfd` before `.lprfc`, so we cannot work around by reordering
`.lprfc` and `.lprfd`.)
With this change, a stage 2 (`-DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_BUILD_INSTRUMENTED=IR`)
`ld -pie` linked clang is 1.74% smaller due to fewer R_X86_64_RELATIVE relocations.
```
% readelf -r pie | awk '$3~/R.*/{s[$3]++} END {for (k in s) print k, s[k]}'
R_X86_64_JUMP_SLO 331
R_X86_64_TPOFF64 2
R_X86_64_RELATIVE 476059 # was: 607712
R_X86_64_64 2616
R_X86_64_GLOB_DAT 31
```
The absolute function address (used by llvm-profdata to collect indirect call
targets) can be converted to relative as well, but is not done in this patch.
Differential Revision: https://reviews.llvm.org/D104556
The test contains a race in memset.
The size of reported race depends on how the accessed
memory range split into granules inside of tsan runtime.
The test used to report access of size 8, because presumably
the buffer ended up being aligned to 8 bytes. But after
some unrelated changes this test started to report accesses
of size 1 (presumably .data layout changed), which makes
the test fail.
Guarantee alignment of the buf object explicitly.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107149
Make New<>() a variadic function template and forward any arguments to
the constructor. std::forward<>() is inlined to avoid including
<utility>.
Differential Revision: https://reviews.llvm.org/D107147
We frequenty allocate sizeof(T) memory and call T ctor on that memory
(C++ new keyword effectively). Currently it's quite verbose and
usually takes 2 lines of code.
Add New<T>() helper that does it much more concisely.
Rename internal_free to Free that also sets the pointer to nullptr.
Shorter and safer.
Rename internal_alloc to Alloc, just shorter.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D107085
The updated lots_of_threads.c test with 300 threads
started running for too long on machines with low
hardware parallelism (e.g. taskset -c 0-1).
On lots of CPUs it finishes in ~2 secs. But with
taskset -c 0-1 it runs for hundreds of seconds
effectively spinning in the barrier in the sleep loop.
We now have the handy futex API in sanitizer_common.
Use it instead of the passive spin loop.
It makes the test run only faster with taskset -c 0-1,
it runs for ~1.5 secs, while with full parallelism
it still runs for ~2 secs (but consumes less CPU time).
Depends on D107131.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D107132
The test contains a race in read/write syscalls.
The size of reported race depends on how the accessed
memory range split into granules inside of tsan runtime.
The test used to report access of size 8, because presumably
the buffer ended up being aligned to 8 bytes. But after
some unrelated changes this test started to report accesses
of size 1 (presumably .data layout changed), which makes
the test fail.
Guarantee alignment of the buf object explicitly.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D107131
Android has its own CMAKE_SYSTEM_NAME, but the OS is Linux (Android
target triples look like aarch64-none-linux-android21). The driver will
therefore search for compiler-rt libraries in the "linux" directory and
not the "android" directory, so the default placement of Android
compiler-rt libraries was incorrect. You could fix it by specifying
COMPILER_RT_OS_DIR manually, but it also makes sense to fix the default,
to save others from having to discover and fix the issue for themselves.
Adds magic version header to AllocatorState. This can be used by
out-of-process crash handlers, like Crashpad on Fuchsia, to do offline
reconstruction of GWP-ASan crash metadata.
Crashpad on Fuchsia is intending on dumping the AllocationMetadata pool
and the AllocatorState directly into the minidump. Then, using the
version number, they can unpack the data on serverside using a versioned
unpack tool.
Also add some asserts to make sure the version number gets bumped if the
internal structs get changed.
Reviewed By: eugenis, mcgrathr
Differential Revision: https://reviews.llvm.org/D106690
Rather than throwing an error. This way we can still use files like
hwasan_dynamic_shadow.cpp for other platforms without leading to a
preprocessor error.
Differential Revision: https://reviews.llvm.org/D106979
We call non-inlinable Initialize from all interceptors/syscalls,
but most of the time runtime is already initialized and this just
introduces unnecessary overhead.
Add LazyInitialize that (1) inlinable, (2) does nothing if
.preinit_array is enabled (expected case on Linux).
Depends on D107071.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107072
We are very paranoid with CHECKs in all Java entry points.
These CHECKs were added along with Java support.
At that point it wasn't clear what exactly to expect from JVM part
and if JVM part is correct or not. Thus CHECK paranoia.
These CHECKs never fired in practice, but we pay runtime cost
in every entry point all the time.
Replace CHECKs with DCHECKs.
Depends on D107069.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107071
We used to call Initialize in every Java point.
That was removed in 6563bb53b5 ("tsan: don't use caller/current PC in Java interfaces").
The intention was to add a single Initialize to __tsan_java_init instead.
Do that.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107069
Add intrusive doubly-linked list container template, IList.
It will be used in the new tsan runtime.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107050
Add a test where atomic-release happens while
another thread spins calling load-acquire.
This can expose some interesting interleavings
of release and acquire.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107055
Switch x86_64 requirement for optimized code from SSE3 to SSE4.2.
The new tsan runtime will need few instructions that are only
supported by SSE4:
_mm_max_epu32
_mm_extract_epi8
_mm_insert_epi32
SSE3 was introcued in 2004 and SSE4 in 2006:
https://en.wikipedia.org/wiki/SSE3https://en.wikipedia.org/wiki/SSE4
We are still providing non-optimized C++ version of the code,
so either way it's possible to build tsan runtime for any CPU.
But for Go this will bump strict requirement for -race because
Go contains prebuilt versions and these will be built with -msse4.2.
But requiring a CPU produced at least in 2006 looks reasonable for
a debugging tool (more reasonable than disabling optimizations
for everybody).
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106948
We reliably remove bottom libc-guts frames only on linux/glibc.
Some bots failed on this test showing other bottom frames:
.annobin_libc_start.c libc-start.c (libc.so.6+0x249f4)
generic_start_main.isra.0 libc-start.c (libc.so.6+0x45b0c)
We can't reliably remove all of possible bottom frames.
So remove the assertion for that.
Differential Revision: https://reviews.llvm.org/D107037
This CL modifies the PlatformUnpoisonStacks so that fuchsia can
implement its own logic for unpoisoning the stacks.
For the general case, the behavior is the same as with regular asan: it
will unpoison everything from the current stack pointer until the base
of the stack (stack top).
In some situations, the current stack might not be the same as the
default stack. In those cases, the code will now unpoison the entire
default stack, and will also unpoison the current page of the stack.
Reviewed By: mcgrathr
Differential Revision: https://reviews.llvm.org/D106835
This commit fixes the CFI directives in the crash trampoline so
libunwind can get a backtrace during a crash.
In order to get a backtrace from a libfuzzer crash in fuchsia, we
resume execution in the crashed thread, forcing it to call the
StaticCrashHandler. We do this by setting a "crash trampoline" that has
all the necessary cfi directives for an unwinder to get full backtrace
for that thread.
Due to a bug in libunwind, it was not possible to restore the RSP
pointer, as it was always set to the call frame address (CFA). The
previous version worked around this issue by setting the CFA to the
value of the stack pointer at the point of the crash.
The bug in libunwind is now fixed[0], so I am correcting the CFI
annotations so that the CFA correctly points to the beginning of the
trampoline's call frame.
[0]: https://reviews.llvm.org/D106626
Reviewed By: mcgrathr
Differential Revision: https://reviews.llvm.org/D106725
Currently __tsan_atomic* functions do FuncEntry/Exit using caller PC
and then use current PC (pointing to __tsan_atomic* itself) during
memory access handling. As the result the top function in reports
involving atomics is __tsan_atomic* and the next frame points to user code.
Remove FuncEntry/Exit in atomic functions and use caller PC
during memory access handling. This removes __tsan_atomic*
from the top of report stacks, so that they point right to user code.
The motivation for this is performance.
Some atomic operations are very hot (mostly loads),
so removing FuncEntry/Exit is beneficial.
This also reduces thread trace consumption (1 event instead of 3).
__tsan_atomic* at the top of the stack is not necessary
and does not add any new information. We already say
"atomic write of size 4", "__tsan_atomic32_store" does not add
anything new.
It also makes reports consistent between atomic and non-atomic
accesses. For normal accesses we say "previous write" and point
to user code; for atomics we say "previous atomic write" and now
also point to user code.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106966
Compilers tends to insert memset/memcpy for some struct/array operations,
and these don't play well inside of sanitizer runtimes.
Avoiding these calls was the intention behind internal_memset.
Remove the leftover ={} that can result in memset call.
Reviewed By: vitalybuka, pgousseau
Differential Revision: https://reviews.llvm.org/D106978
We strip all frames below main but in some cases it may be not enough.
Namely, when main is instrumented but does not call any other instrumented code.
In this case __tsan_func_entry in main obtains PC pointing to __libc_start_main
(as we pass caller PC to __tsan_func_entry), but nothing obtains PC pointing
to main itself (as main does not call any instrumented code).
In such case we will not have main in the stack, and stripping everything
below main won't work.
So strip __libc_start_main explicitly as well.
But keep stripping of main because __libc_start_main is glibc/linux-specific,
so looking for main is more reliable (and usually main is present in stacks).
Depends on D106957.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106958
Caller PC is plain harmful as native caller PC has nothing to do with Java code.
Current PC is not particularly useful and may be somewhat confusing for Java users
as it makes top frame to point to some magical __tsan function.
But obtaining and using these PCs adds runtime cost for every java event.
Remove these PCs. Rely only on official Java frames.
It makes execution faster, code simpler and reports better.
Depends on D106956.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106957
We maintain information about Java allocations,
but for some reason never printed it in reports.
Print it.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106956
1. Add a set of micro benchmarks for memory accesses,
mem* functions and unaligned accesses.
2. Add support for multiple benchmarks in a single binary
(or it would require 12 new benchmark binaries).
3. Remove the "clock growth" machinery,
it affects the current tsan runtime by increasing size of
all vector clocks, but this won't be relevant for the new
tsan runtime.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106961
We used to count number of allocations/bytes based on the type
and maybe record them in heap block headers.
But that's all in the past, now it's not used for anything.
Remove the mblock type.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106971
Remove pc argument of ThreadIgnoreEnd, ThreadIgnoreSyncEnd
and AcquireGlobal functions. It's unused and in some places
we don't even have a pc and pass 0 anyway.
Don't confuse readers and don't pretend that pc is needed
and that passing 0 is somehow deficient.
Use simpler convention for ThreadIgnoreBegin and ThreadIgnoreSyncBegin:
accept only pc instread of pc+save_stack. 0 pc means "don't save stack".
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106973
The current 10 threads is not particularly "lots" and not stressful.
Create 10x300 threads and ensure they all are running at the same time.
Depends on D106953.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D106954
Currently the barrier supports only 256 threads,
this does not allow to write reliable tests that use more threads.
Bump max number of threads to 1024 to support writing
good stress tests.
Also replace sched_yield() with usleep(100) on the wait path.
If we write tests that create hundreds of threads (and dozens
of tests can run in parallel), yield would consume massive
amounts of CPU time for spinning.
Depends on D106952.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D106953
Test that we report the warning for free()
and ensure the test finishes as we usually do with "DONE".
Depends on D106951.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D106952
Compilers don't like attributes in this position:
warning: GCC does not allow 'noinline' attribute in this position on a function definition
error: attributes are not allowed on a function-definition
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D106951
Mutex supports reader access, OS blocking, spinning,
portable and smaller than BlockingMutex.
Overall it's supposed to be better than RWMutex/BlockingMutex.
Replace RWMutex/BlockingMutex with Mutex.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D106936
Mutex does not support LINKER_INITIALIZED ctor.
But we used to support it with BlockingMutex.
To prevent potential bugs delete LINKER_INITIALIZED Mutex ctor.
Also mark existing ctor as explicit.
Depends on D106944.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D106945
Mutex does not support LINKER_INITIALIZED support.
As preparation to switching BlockingMutex to Mutex,
proactively replace all BlockingMutex(LINKER_INITIALIZED) to Mutex.
All of these are objects with static storage duration and Mutex ctor
is constexpr, so it should be equivalent.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D106944
This reverts commit 1e1f752027.
It breaks ignore_noninstrumented_modules=1.
Somehow we did not have any portable tests for this mode before
(only Darwin tests). Add a portable test as well.
Moreover, I think I was too fast uninlining all LibIgnore checks.
For Java, Darwin and OpenMP LibIgnore is always enabled,
so it makes sense to leave it as it was before.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D106855
Ensure that libSupport does not carry any static global initializer.
libSupport can be embedded in use cases where we don't want to load all
cl::opt unless we want to parse the command line.
ManagedStatic can be used to enable lazy-initialization of globals.
The -Werror=global-constructors is only added on platform that have
support for the flag and for which std::mutex does not have a global
destructor. This is ensured by having CMake trying to compile a file
with a global mutex before adding the flag to libSupport.
This will just be a call into __sanitizer_fill_shadow defined in the fuchsia source tree. This depends on D105663.
Differential Revision: https://reviews.llvm.org/D105664
This is empty for now, but we will add a check that TBI is enabled once the
tagged pointer ABI for zircon is finalized. This depends on D105667.
Differential Revision: https://reviews.llvm.org/D105668
On Apple platforms the builtins may be built for both arm64 and arm64e.
With Makefile generators separate targets are built using Make sub-invocations.
This causes a race when creating the symlink which may sometimes fail.
Work around this by using a custom target that the builtin targets depend on.
This causes any sub-invocations to depend on the symlinks having been created before.
Mailing list thread: https://lists.llvm.org/pipermail/llvm-dev/2021-July/151822.html
Reviewed By: thakis, steven_wu
Differential Revision: https://reviews.llvm.org/D106305
Add the ability to:
1. tell simctl to wait for debugger when spawning process
2. print the command that is called to launch the process
Reviewed By: delcypher
Differential Revision: https://reviews.llvm.org/D106700
Now that Mutex is blocking there is no point in using BlockingMutex.
Switch to Mutex.
Depends on D106379.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106560
Now that sanitizer_common mutex has feature-parity with tsan mutex,
switch tsan to the sanitizer_common mutex and remove tsan's custom mutex.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D106379
In preparation for replacing tsan Mutex with sanitizer_common Mutex,
which has thread-safety annotations. Thread safety analysis does not
understand MetaMap::GetAndLock which returns a locked sync object.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D106548
This is preparation to switching to the sanitizer_common Mutex.
Without this change after the switch we will start failing
on existing from the runtime with runtime mutexes held.
Previously it worked because CheckNoLocks did not see sanitizer_common mutexes.
Depends on D106547.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D106558
Rename Mutex class in tests to avoid conflicts with sanitizer_common Mutex.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D106547
Copy internal deadlock detector from tsan to sanitizer_common
(with some cosmetic changes).
Tsan version will be deleted in subsequent changes.
This allows us to switch tsan to the sanitizer_common mutex
and remove tsan's mutex.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D106546
This change eliminate the stack frame for the fast path and improves runtime performance.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106505
signal(2) and sigaction(2) have defined behaviors for invalid signal number
(EINVAL) and some programs rely on it.
The added test case also reveals that MSAN is too strict in this regard.
Test case passed on x86_64 Linux and AArch64 Linux.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106468
The current (default) line length is 80 columns.
That's based on old hardware and historical conventions.
There are no existent reasons to keep line length that small,
especially provided that our coding style uses quite lengthy
identifiers. The Linux kernel recently switched to 100,
let's start with 100 as well.
This change intentionally does not re-format code.
Re-formatting is intended to happen incrementally,
or on dir-by-dir basis separately.
Reviewed By: vitalybuka, melver, MaskRay
Differential Revision: https://reviews.llvm.org/D106436
This reverts commit 6b2a96285b.
The ccache builders are still failing. Looks like they need to be updated to
get the llvm-zorg config change in 490633945677656ba75d42ff1ca9d4a400b7b243.
I'll re-apply this as soon as the builders are updated.
This reapplies commit a7733e9556 ("Re-apply
[ORC][ORC-RT] Add initial native-TLV support to MachOPlatform."), and
d4abdefc99 ("[ORC-RT] Rename macho_tlv.x86-64.s
to macho_tlv.x86-64.S (uppercase suffix)").
These patches were reverted in 48aa82cacb while I
investigated bot failures (e.g.
https://lab.llvm.org/buildbot/#/builders/109/builds/18981). The fix was to
disable building of the ORC runtime on buliders using ccache (which is the same
fix used for other compiler-rt projects containing assembly code). This fix was
commited to llvm-zorg in 490633945677656ba75d42ff1ca9d4a400b7b243.
This patch changes the linkage type of a compiler-rt func
(__llvm_write_binary_ids) to fix the sanitizer-windows bot
build issue introduced in change f984ac271.
The issue is as the following:
C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\lib\profile\InstrProfilingInternal.h(201):
error C2496: '__llvm_write_binary_ids': 'selectany' can only be applied
to data items with external linkage
Since d564cfb53c moved
__hwasan_tag_mismatch4 this test has been reporting
a frame 0 of __hwasan_tag_mismatch_v2.
This failure can be seen on our bots:
https://lab.llvm.org/buildbot/#/builders/185/builds/170
Before the change:
#0 0xaaaaba100e40 in main <...>/register-dump-read.c:21:10
After the change:
#0 0xaaaab8494bec in __hwasan_tag_mismatch_v2 <...>/hwasan/hwasan_tag_mismatch_aarch64.S:147
#1 0xaaaab84b4df8 in main <..>/register-dump-read.c:14:10
Update the test to check for a main frame as either frame
0 or frame 1.
Patch "sanitizer_common: modernize SpinMutex" added default
ctor to StaticSpinMutex. But it broke some gcc bots with:
scudo_tsd_exclusive.cpp:25:22: error: non-local variable
‘__scudo::TSD’ declared ‘__thread’ needs dynamic initialization
https://lab.llvm.org/buildbot/#/builders/105/builds/12649
Unfortunatly none of empty ctor {}, no ctor, default constexpr ctor
work for different reasons. So remove StaticSpinMutex ctor
entirely and move deleted copy ctor back to SpinMutex.
Differential Revision: https://reviews.llvm.org/D106424
Some minor improvements:
1. Make StaticSpinMutex non-copyable.
2. Add LIKELY to Lock.
3. Move LockSlow into the .cpp file (now that we have it).
4. The only non-trivial change: use proc_yield(1) instread of proc_yield(10)
with the proportional increase in the number of spin iterations.
Latency of the PAUSE instruction has raised from ~1 cycle to ~100 cycles
in the recent Intel CPUs. So proc_yield(10) is too aggressive backoff.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106350
This reverts commit d4abdefc99 ("[ORC-RT] Rename
macho_tlv.x86-64.s to macho_tlv.x86-64.S (uppercase suffix)", and
a7733e9556 ("Re-apply "[ORC][ORC-RT] Add initial
native-TLV support to MachOPlatform."), while I investigate failures on
ccache builders (e.g. https://lab.llvm.org/buildbot/#/builders/109/builds/18981)
Reapplies fe1fa43f16, which was reverted in
6d8c63946c, with fixes:
1. Remove .subsections_via_symbols directive from macho_tlv.x86-64.s (it's
not needed here anyway).
2. Return error from pthread_key_create to the MachOPlatform to silence unused
variable warning.
Adds code to LLVM (MachOPlatform) and the ORC runtime to support native MachO
thread local variables. Adding new TLVs to a JITDylib at runtime is supported.
On the LLVM side MachOPlatform is updated to:
1. Identify thread local variables in the LinkGraph and lower them to GOT
accesses to data in the __thread_data or __thread_bss sections.
2. Merge and report the address range of __thread_data and thread_bss sections
to the runtime.
On the ORC runtime a MachOTLVManager class introduced which records the address
range of thread data/bss sections, and creates thread-local instances from the
initial data on demand. An orc-runtime specific tlv_get_addr implementation is
included which saves all register state then calls the MachOTLVManager to get
the address of the requested variable for the current thread.
We currently have 3 different mutexes:
- RWMutex
- BlockingMutex
- __tsan::Mutex
RWMutex and __tsan::Mutex are roughly the same,
except that tsan version supports deadlock detection.
BlockingMutex degrades better under heavy contention
from lots of threads (blocks in OS), but much slower
for light contention and has non-portable performance
and has larger static size and is not reader-writer.
Add a new mutex that combines all advantages of these
mutexes: it's reader-writer, has fast non-contended path,
supports blocking to gracefully degrade under higher contention,
has portable size/performance.
For now it's named Mutex2 for incremental submission. The plan is to:
- land this change
- then move deadlock detection logic from tsan
- then rename it to Mutex and remove tsan Mutex
- then typedef RWMutex/BlockingMutex to this mutex
SpinMutex stays as separate type because it has faster fast path:
1 atomic RMW per lock/unlock as compared to 2 for this mutex.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D106231
I don't think the stat subsystem was ever used since tsan
development in 2012. But it adds lots of code and this
effectively dead code needs to be updated if the runtime
code changes, which adds maintanance cost for no benefit.
Normal profiler usually gives enough info and that info
is more trustworthy.
Remove the stats subsystem.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106276
Add pragma line so that errors messages point to the actual
source files rather than to the concatenated gotsan.cpp.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D106275
We need the compiler generated variable to override the weak symbol of
the same name inside the profile runtime, but using LinkOnceODRLinkage
results in weak symbol being emitted in which case the symbol selected
by the linker is going to depend on the order of inputs which can be
fragile.
This change replaces the use of weak definition inside the runtime with
a weak alias. We place the compiler generated symbol inside a COMDAT
group so dead definition can be garbage collected by the linker.
We also disable the use of runtime counter relocation on Darwin since
Mach-O doesn't support weak external references, but Darwin already uses
a different continous mode that relies on overmapping so runtime counter
relocation isn't needed there.
Differential Revision: https://reviews.llvm.org/D105176
For some reason we have 2 switches on arch and add
half of arch flags in one place and half in another.
Merge these 2 switches.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D106274
We obtain the current PC is all interceptors and collectively
common interceptor code contributes to overall slowdown
(in particular cheaper str/mem* functions).
The current way to obtain the current PC involves:
4493e1: e8 3a f3 fe ff callq 438720 <_ZN11__sanitizer10StackTrace12GetCurrentPcEv>
4493e9: 48 89 c6 mov %rax,%rsi
and the called function is:
uptr StackTrace::GetCurrentPc() {
438720: 48 8b 04 24 mov (%rsp),%rax
438724: c3 retq
The new way uses address of a local label and involves just:
44a888: 48 8d 35 fa ff ff ff lea -0x6(%rip),%rsi
I am not switching all uses of StackTrace::GetCurrentPc to GET_CURRENT_PC
because it may lead some differences in produced reports and break tests.
The difference comes from the fact that currently we have PC pointing
to the CALL instruction, but the new way does not yield any code on its own
so the PC points to a random instruction in the function and symbolizing
that instruction can produce additional inlined frames (if the random
instruction happen to relate to some inlined function).
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D106046
This member is now only used when storage is heap-allocated so it does not
need to be const. Dropping 'const' eliminates cast warnings on many builders.
These have been failing on our bots for a while due to
incomplete backtraces. (you don't get the names of the
functions that did the access, just the reporter frames)
See:
https://lab.llvm.org/buildbot/#/builders/170/builds/180
Adds support for MachO static initializers/deinitializers and eh-frame
registration via the ORC runtime.
This commit introduces cooperative support code into the ORC runtime and ORC
LLVM libraries (especially the MachOPlatform class) to support macho runtime
features for JIT'd code. This commit introduces support for static
initializers, static destructors (via cxa_atexit interposition), and eh-frame
registration. Near-future commits will add support for MachO native
thread-local variables, and language runtime registration (e.g. for Objective-C
and Swift).
The llvm-jitlink tool is updated to use the ORC runtime where available, and
regression tests for the new MachOPlatform support are added to compiler-rt.
Notable changes on the ORC runtime side:
1. The new macho_platform.h / macho_platform.cpp files contain the bulk of the
runtime-side support. This includes eh-frame registration; jit versions of
dlopen, dlsym, and dlclose; a cxa_atexit interpose to record static destructors,
and an '__orc_rt_macho_run_program' function that defines running a JIT'd MachO
program in terms of the jit- dlopen/dlsym/dlclose functions.
2. Replaces JITTargetAddress (and casting operations) with ExecutorAddress
(copied from LLVM) to improve type-safety of address management.
3. Adds serialization support for ExecutorAddress and unordered_map types to
the runtime-side Simple Packed Serialization code.
4. Adds orc-runtime regression tests to ensure that static initializers and
cxa-atexit interposes work as expected.
Notable changes on the LLVM side:
1. The MachOPlatform class is updated to:
1.1. Load the ORC runtime into the ExecutionSession.
1.2. Set up standard aliases for macho-specific runtime functions. E.g.
___cxa_atexit -> ___orc_rt_macho_cxa_atexit.
1.3. Install the MachOPlatformPlugin to scrape LinkGraphs for information
needed to support MachO features (e.g. eh-frames, mod-inits), and
communicate this information to the runtime.
1.4. Provide entry-points that the runtime can call to request initializers,
perform symbol lookup, and request deinitialiers (the latter is
implemented as an empty placeholder as macho object deinits are rarely
used).
1.5. Create a MachO header object for each JITDylib (defining the __mh_header
and __dso_handle symbols).
2. The llvm-jitlink tool (and llvm-jitlink-executor) are updated to use the
runtime when available.
3. A `lookupInitSymbolsAsync` method is added to the Platform base class. This
can be used to issue an async lookup for initializer symbols. The existing
`lookupInitSymbols` method is retained (the GenericIRPlatform code is still
using it), but is deprecated and will be removed soon.
4. JIT-dispatch support code is added to ExecutorProcessControl.
The JIT-dispatch system allows handlers in the JIT process to be associated with
'tag' symbols in the executor, and allows the executor to make remote procedure
calls back to the JIT process (via __orc_rt_jit_dispatch) using those tags.
The primary use case is ORC runtime code that needs to call bakc to handlers in
orc::Platform subclasses. E.g. __orc_rt_macho_jit_dlopen calling back to
MachOPlatform::rt_getInitializers using __orc_rt_macho_get_initializers_tag.
(The system is generic however, and could be used by non-runtime code).
The new ExecutorProcessControl::JITDispatchInfo struct provides the address
(in the executor) of the jit-dispatch function and a jit-dispatch context
object, and implementations of the dispatch function are added to
SelfExecutorProcessControl and OrcRPCExecutorProcessControl.
5. OrcRPCTPCServer is updated to support JIT-dispatch calls over ORC-RPC.
6. Serialization support for StringMap is added to the LLVM-side Simple Packed
Serialization code.
7. A JITLink::allocateBuffer operation is introduced to allocate writable memory
attached to the graph. This is used by the MachO header synthesis code, and will
be generically useful for other clients who want to create new graph content
from scratch.
ptrauth stores info in the address of functions, so it's not the right address we should check if poisoned
rdar://75246928
Differential Revision: https://reviews.llvm.org/D106199
Windows bot failed with:
sanitizer_win.cpp.obj : error LNK2019: unresolved external symbol WaitOnAddress referenced in function "void __cdecl __sanitizer::FutexWait(struct __sanitizer::atomic_uint32_t *,unsigned int)" (?FutexWait@__sanitizer@@YAXPEAUatomic_uint32_t@1@I@Z)
sanitizer_win.cpp.obj : error LNK2019: unresolved external symbol WakeByAddressSingle referenced in function "void __cdecl __sanitizer::FutexWake(struct __sanitizer::atomic_uint32_t *,unsigned int)" (?FutexWake@__sanitizer@@YAXPEAUatomic_uint32_t@1@I@Z)
sanitizer_win.cpp.obj : error LNK2019: unresolved external symbol WakeByAddressAll referenced in function "void __cdecl __sanitizer::FutexWake(struct __sanitizer::atomic_uint32_t *,unsigned int)" (?FutexWake@__sanitizer@@YAXPEAUatomic_uint32_t@1@I@Z)
https://lab.llvm.org/buildbot/#/builders/127/builds/14046
According to MSDN we need to link Synchronization.lib:
https://docs.microsoft.com/en-us/windows/win32/api/synchapi/nf-synchapi-waitonaddress
Differential Revision: https://reviews.llvm.org/D106167
This was fixed in the past for `frexp`, but was not made for `frexpl` & `frexpf` https://github.com/google/sanitizers/issues/321
This patch copies the fix over to `frexpl` because it caused `frexp_interceptor.cpp` test to fail on iPhone and `frexpf` for consistency.
rdar://79652161
Reviewed By: delcypher, vitalybuka
Differential Revision: https://reviews.llvm.org/D104948
Semaphore is a portable way to park/unpark threads.
The plan is to use it to implement a portable blocking
mutex in subsequent changes. Semaphore can also be used
to efficiently wait for other things (e.g. we currently
spin to synchronize thread creation and start).
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D106071
After we relocate counters, we no longer need to keep the original copy
around so we can return the memory back to the operating system.
Differential Revision: https://reviews.llvm.org/D104839
Currently we don't lock ScopedErrorReportLock around fork
and it mostly works becuase tsan has own report_mtx that
is locked around fork and tsan reports.
However, sanitizer_common code prints some own reports
which are not protected by tsan's report_mtx. So it's better
to lock ScopedErrorReportLock explicitly.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D106048
The new GET_CURRENT_PC() can lead to spurious top inlined internal frames.
Here are 2 examples from bots, in both cases the malloc is supposed to be
the top frame (#0):
WARNING: ThreadSanitizer: signal-unsafe call inside of a signal
#0 __sanitizer::StackTrace::GetNextInstructionPc(unsigned long)
#1 malloc
Location is heap block of size 99 at 0xbe3800003800 allocated by thread T1:
#0 __sanitizer::StackTrace::GetNextInstructionPc(unsigned long)
#1 malloc
Let's strip these internal top frames from reports.
With other code changes I also observed some top frames
from __tsan::ScopedInterceptor, proactively remove these as well.
Differential Revision: https://reviews.llvm.org/D106081
We obtain the current PC is all interceptors and collectively
common interceptor code contributes to overall slowdown
(in particular cheaper str/mem* functions).
The current way to obtain the current PC involves:
4493e1: e8 3a f3 fe ff callq 438720 <_ZN11__sanitizer10StackTrace12GetCurrentPcEv>
4493e9: 48 89 c6 mov %rax,%rsi
and the called function is:
uptr StackTrace::GetCurrentPc() {
438720: 48 8b 04 24 mov (%rsp),%rax
438724: c3 retq
The new way uses address of a local label and involves just:
44a888: 48 8d 35 fa ff ff ff lea -0x6(%rip),%rsi
I am not switching all uses of StackTrace::GetCurrentPc to GET_CURRENT_PC
because it may lead some differences in produced reports and break tests.
The difference comes from the fact that currently we have PC pointing
to the CALL instruction, but the new way does not yield any code on its own
so the PC points to a random instruction in the function and symbolizing
that instruction can produce additional inlined frames (if the random
instruction happen to relate to some inlined function).
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D106046
Define the address ranges (similar to the C/C++ ones, but with the heap
range merged into the app range) and enable the sanity check.
Reviewed By: dvyukov
Differential Revision: https://reviews.llvm.org/D105629
XFAIL map32bit, define the maximum possible allocation size in
mmap_large.cpp.
Reviewed By: dvyukov
Differential Revision: https://reviews.llvm.org/D105629
Reuse the assembly glue code from sanitizer_common_interceptors.inc and
the handling logic from the __tls_get_addr interceptor.
Reviewed By: dvyukov
Differential Revision: https://reviews.llvm.org/D105629
The kernel supports a full 64-bit VMA, but we can use only 48 bits due
to the limitation imposed by SyncVar::GetId(). So define the address
ranges similar to the other architectures, except that the address
space "tail" needs to be made inaccessible in CheckAndProtect(). Since
it's for only one architecture, don't make an abstraction for this.
Reviewed By: dvyukov
Differential Revision: https://reviews.llvm.org/D105629
These tests depend on TSan seeing the intercepted memcpy(), so they
break when the compiler chooses the builtin version.
Reviewed By: dvyukov
Differential Revision: https://reviews.llvm.org/D105629
s390x requires ThreadRegistry.mtx_.opaque_storage_ to be 4-byte
aligned. Since other architectures may have similar requirements, use
the maximum thread_registry_placeholder alignment from other
sanitizers, which is 64 (LSan).
Reviewed By: dvyukov
Differential Revision: https://reviews.llvm.org/D105629
When running with an old glibc, CollectStaticTlsBlocks() calls
__tls_get_addr() in order to force TLS allocation. This function is not
available on s390 and the code simply does nothing in this case,
so all the resulting static TLS blocks end up being incorrect.
Fix by calling __tls_get_offset() on s390.
Reviewed By: dvyukov
Differential Revision: https://reviews.llvm.org/D105629
setuid(0) hangs on SystemZ under TSan because TSan's BackgroundThread
ignores SIGSETXID. This in turn happens because internal_sigdelset()
messes up the mask bits on big-endian system due to how
__sanitizer_kernel_sigset_t is defined.
Commit d9a1a53b8d ("[ESan] [MIPS] Fix workingset-signal-posix.cpp on
MIPS") fixed this for MIPS by adjusting the __sanitizer_kernel_sigset_t
definition. Generalize this by defining __SANITIZER_KERNEL_NSIG based
on kernel's _NSIG and using uptr[] for __sanitizer_kernel_sigset_t.sig
on all platforms.
Reviewed By: dvyukov
Differential Revision: https://reviews.llvm.org/D105629
Currently ThreadRegistry is overcomplicated because of tsan,
it needs tid quarantine and reuse counters. Other sanitizers
don't need that. It also seems that no other sanitizer now
needs max number of threads. Asan used to need 2^24 limit,
but it does not seem to be needed now. Other sanitizers blindly
copy-pasted that without reasons. Lsan also uses quarantine,
but I don't see why that may be potentially needed.
Add a ThreadRegistry ctor that does not require any sizes
and use it in all sanitizers except for tsan.
In preparation for new tsan runtime, which won't need
any of these parameters as well.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D105713
Dmitry Vyukov