tsan_rtl.cpp is huge and does lots of things.
Move everything related to memory access and tracing
to a separate tsan_rtl_access.cpp file.
No functional changes, only code movement.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D112625
Whenever we call cur_thread_init, we call cur_thread on the next line.
So make cur_thread_init return the current thread directly.
Makes code a bit shorter, does not affect codegen.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D110384
There are 2 reasons to do this:
1. We place hot data in the first cache line of ThreadState,
this assumed that it's cache-line-aligned but we never actually
enforced it (or it was lost at some point).
2. The new vector clock uses vector instructions and requires
data alignment. Later the new vector clock will be embedded in
ThreadState, then ensuring vector clock alignment will be
impossible w/o ThreadState alignment.
Depends on D110519.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D110520
Currently the shadow stack is located in the trace memory mapping.
The new tsan runtime will remove the trace memory mapping.
Move the shadow stack into ThreadState as a preparation step.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D110519
It's only used during race reporting.
There is no point in polluting the main header file with it.
Reviewed By: xgupta
Differential Revision: https://reviews.llvm.org/D110470
Remove nmissed_expected variable.
It's a leftover from removed "expected race" feature and is never incremented.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D110321
tsan_rtl.h is very huge and contains too many things.
Move FastState and Shadow types into a new tsan_shadow.h file.
This also allows to use FastState/Shadow in other header files
without creating circular dependencies (which most likely will
happen today).
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D110272
BackgroundThread function is quite large,
move mem profile initialization into a separate function.
Depends on D110151.
Reviewed By: melver, vitalybuka
Differential Revision: https://reviews.llvm.org/D110152
dlsym calls into dynamic linker which calls malloc and other things.
It's problematic to do it during the actual exit, because
it can happen from a singal handler or from within the runtime
after we reported the first bug, etc.
See https://github.com/google/sanitizers/issues/1440 for an example
(captured in the added test).
Initialize the callbacks during startup instead.
Depends on D110159.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D110166
Add structures for the new trace format,
functions that serialize and add events to the trace
and trace replaying logic.
Differential Revision: https://reviews.llvm.org/D107911
Remove direct uses of Mapping in preperation for removing Mapping type
(which we already don't have for all platforms).
Remove dependence on HAS_48_BIT_ADDRESS_SPACE in preparation for removing it.
As far as I see for Apple/Mac platforms !HAS_48_BIT_ADDRESS_SPACE
simply means SANITIZER_IOS.
Depends on D107741.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107742
Currently we hardcode u64 type for shadow everywhere
and do lots of uptr<->u64* casts. It makes it hard to
change u64 to another type (e.g. u32) and makes it easy
to introduce bugs.
Introduce RawShadow type and use it in MemToShadow, ShadowToMem,
IsShadowMem and throughout the code base as u64 replacement.
This makes it possible to change u64 to something else in future
and generally improves static typing.
Depends on D107481.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D107482
It will be needed in more functions like ReportRace
(the plan is to pass it through MemoryAccess to ReportRace)
and this move will allow to split the huge tsan_rtl.h into parts
(e.g. move FastState/Shadow definitions to a separate header).
Depends on D107465.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107466
Add kAccessExternal memory access flag that denotes
memory accesses with PCs that may have kExternalPCBit set.
In preparation for MemoryAccess refactoring.
Currently unused, but will allow to skip a branch.
Depends on D107464.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107465
Add kAccessFree memory access flag (similar to kAccessVptr).
In preparation for MemoryAccess refactoring.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107464
Add AccessVptr access type.
For now it's converted to the same thr->is_vptr_access,
but later it will be passed directly to ReportRace
and will enable efficient tail calling in MemoryAccess function
(currently __tsan_vptr_update/__tsan_vptr_read can't use
tail calls in MemoryAccess because of the trailing assignment
to thr->is_vptr_access).
Depends on D107276.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D107282
Currently we have MemoryAccess function that accepts
"bool kAccessIsWrite, bool kIsAtomic" and 4 wrappers:
MemoryRead/MemoryWrite/MemoryReadAtomic/MemoryWriteAtomic.
Such scheme with bool flags is not particularly scalable/extendable.
Because of that we did not have Read/Write wrappers for UnalignedMemoryAccess,
and "true, false" or "false, true" at call sites is not very readable.
Moreover, the new tsan runtime will introduce more flags
(e.g. move "freed" and "vptr access" to memory acccess flags).
We can't have 16 wrappers and each flag also takes whole
64-bit register for non-inlined calls.
Introduce AccessType enum that contains bit mask of
read/write, atomic/non-atomic, and later free/non-free,
vptr/non-vptr.
Such scheme is more scalable, more readble, more efficient
(don't consume multiple registers for these flags during calls)
and allows to cover unaligned and range variations of memory
access functions as well.
Also switch from size log to just size.
The new tsan runtime won't have the limitation of supporting
only 1/2/4/8 access sizes, so we don't need the logarithms.
Also add an inline thunk that converts the new interface to the old one.
For inlined calls it should not add any overhead because
all flags/size can be computed as compile time.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D107276
ProcessPendingSignals is called in all interceptors
and user atomic operations. Make the fast-path check
(no pending signals) inlinable.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107217
Currently we inconsistently use u32 and int for thread ids,
there are also "unique tid" and "os tid" and just lots of other
things identified by integers.
Additionally new tsan runtime will introduce yet another
thread identifier that is very different from current tids.
Similarly for stack IDs, it's easy to confuse u32 with other
integer identifiers. And when a function accepts u32 or a struct
contains u32 field, it's not always clear what it is.
Add Tid and StackID typedefs to make it clear what is what.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107152
"Expected" races is a very ancient facility used in tsanv1 tests.
It's not used/needed anymore. Remove it.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D107175
We call non-inlinable Initialize from all interceptors/syscalls,
but most of the time runtime is already initialized and this just
introduces unnecessary overhead.
Add LazyInitialize that (1) inlinable, (2) does nothing if
.preinit_array is enabled (expected case on Linux).
Depends on D107071.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D107072
We maintain information about Java allocations,
but for some reason never printed it in reports.
Print it.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106956
We used to count number of allocations/bytes based on the type
and maybe record them in heap block headers.
But that's all in the past, now it's not used for anything.
Remove the mblock type.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106971
Remove pc argument of ThreadIgnoreEnd, ThreadIgnoreSyncEnd
and AcquireGlobal functions. It's unused and in some places
we don't even have a pc and pass 0 anyway.
Don't confuse readers and don't pretend that pc is needed
and that passing 0 is somehow deficient.
Use simpler convention for ThreadIgnoreBegin and ThreadIgnoreSyncBegin:
accept only pc instread of pc+save_stack. 0 pc means "don't save stack".
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106973
I don't think the stat subsystem was ever used since tsan
development in 2012. But it adds lots of code and this
effectively dead code needs to be updated if the runtime
code changes, which adds maintanance cost for no benefit.
Normal profiler usually gives enough info and that info
is more trustworthy.
Remove the stats subsystem.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D106276
We have some significant amount of duplication around
CheckFailed functionality. Each sanitizer copy-pasted
a chunk of code. Some got random improvements like
dealing with recursive failures better. These improvements
could benefit all sanitizers, but they don't.
Deduplicate CheckFailed logic across sanitizers and let each
sanitizer only print the current stack trace.
I've tried to dedup stack printing as well,
but this got me into cmake hell. So let's keep this part
duplicated in each sanitizer for now.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D102221
Currently we have a bit of a mess related to tids:
- sanitizers re-declare kInvalidTid multiple times
- some call it kUnknownTid
- implicit assumptions that main tid is 0
- asan/memprof claim their tids need to fit into 24 bits,
but this does not seem to be true anymore
- inconsistent use of u32/int to store tids
Introduce kInvalidTid/kMainTid in sanitizer_common
and use them consistently.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D101428
Commit efd254b636 ("tsan: fix deadlock in pthread_atfork callbacks")
fixed another deadlock related to atfork handling.
But builders with DCHECKs enabled reported failures of
pthread_atfork_deadlock2.c and pthread_atfork_deadlock3.c tests
related to the fact that we hold runtime locks on interceptor exit:
https://lab.llvm.org/buildbot/#/builders/70/builds/6727
This issue is somewhat inherent to the current approach,
we indeed execute user code (atfork callbacks) with runtime lock held.
Refactor fork handling to not run user code (atfork callbacks)
with runtime locks held. This change does this by installing
own atfork callbacks during runtime initialization.
Atfork callbacks run in LIFO order, so the expectation is that
our callbacks run last, right before the actual fork.
This way we lock runtime mutexes around fork, but not around
user callbacks.
Extend tests to also install after fork callbacks just to cover
more scenarios. Some tests also started reporting real races
that we previously suppressed.
Also extend tests to cover fork syscall support.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D101517
Commit efd254b636 ("tsan: fix deadlock in pthread_atfork callbacks")
fixed another deadlock related to atfork handling.
But builders with DCHECKs enabled reported failures of
pthread_atfork_deadlock2.c and pthread_atfork_deadlock3.c tests
related to the fact that we hold runtime locks on interceptor exit:
https://lab.llvm.org/buildbot/#/builders/70/builds/6727
This issue is somewhat inherent to the current approach,
we indeed execute user code (atfork callbacks) with runtime lock held.
Refactor fork handling to not run user code (atfork callbacks)
with runtime locks held. This change does this by installing
own atfork callbacks during runtime initialization.
Atfork callbacks run in LIFO order, so the expectation is that
our callbacks run last, right before the actual fork.
This way we lock runtime mutexes around fork, but not around
user callbacks.
Extend tests to also install after fork callbacks just to cover
more scenarios. Some tests also started reporting real races
that we previously suppressed.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D101385
We take report/thread_registry locks around fork.
This means we cannot report any bugs in atfork handlers.
We resolved this by enabling per-thread ignores around fork.
This resolved some of the cases, but not all.
The added test triggers a race report from a signal handler
called from atfork callback, we reset per-thread ignores
around signal handlers, so we tried to report it and deadlocked.
But there are more cases: a signal handler can be called
synchronously if it's sent to itself. Or any other report
types would cause deadlocks as well: mutex misuse,
signal handler spoiling errno, etc.
Disable all reports for the duration of fork with
thr->suppress_reports and don't re-enable them around
signal handlers.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D101154
realeaseAcquire() is a new function added to TSan in support of the Go data-race detector.
It's semantics is:
void ThreadClock::releaseAcquire(SyncClock *sc) const {
for (int i = 0; i < kMaxThreads; i++) {
tmp = clock[i];
clock[i] = max(clock[i], sc->clock[i]);
sc->clock[i] = tmp;
}
}
For context see: https://go-review.googlesource.com/c/go/+/220419
Reviewed-in: https://reviews.llvm.org/D76322
Author: dfava (Daniel Fava)
Generally we ignore interceptors coming from called_from_lib-suppressed libraries.
However, we must not ignore critical interceptors like e.g. pthread_create,
otherwise runtime will lost track of threads.
pthread_detach is one of these interceptors we should not ignore as it affects
thread states and behavior of pthread_join which we don't ignore as well.
Currently we can produce very obscure false positives. For more context see:
https://groups.google.com/forum/#!topic/thread-sanitizer/ecH2P0QUqPs
The added test captures this pattern.
While we are here rename ThreadTid to ThreadConsumeTid to make it clear that
it's not just a "getter", it resets user_id to 0. This lead to confusion recently.
Reviewed in https://reviews.llvm.org/D74828
I verified that the test is red without the interceptors.
rdar://40334350
Reviewed By: kubamracek, vitalybuka
Differential Revision: https://reviews.llvm.org/D66616
llvm-svn: 371439
Switch `LongJmp` over to lookup JmpBuf via plain old (unmangled) SP.
This makes the computation of mangled SPs in the TSan assembly files
unnecessary, which will be cleaned up in follow-up revisions.
Reviewed By: dvyukov
Differential Revision: https://reviews.llvm.org/D63942
llvm-svn: 364818
They need to have same AddressSpaceView and MapUnmapCallback.
Reviewers: eugenis
Subscribers: kubamracek, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D61168
llvm-svn: 359719
Originally this code as added for 64-bit platform and was never changed.
Add static_assert to make sure that we have correct map on all platforms.
llvm-svn: 359269
Dmitry Vyukov