Extend `llvm-profdata` to read in a `.proflite` file and also a debug info file to generate a normal `.profdata` profile. This reduces the binary size by 8.4% when building an instrumented Clang binary without value profiling (164 MB vs 179 MB).
This work is part of the "lightweight instrumentation" RFC: https://groups.google.com/g/llvm-dev/c/r03Z6JoN7d4
Reviewed By: kyulee
Differential Revision: https://reviews.llvm.org/D114566
There were changes made to the linux version of this test that were not made for darwin
(see https://reviews.llvm.org/D115837) and this caused downstream failures.
Adding comment to this test to remind people to edit interface_symbols_darwin.cpp.
There is the reverse of this comment in the darwin file to remind us to edit the linux version already.
Differential Revision: https://reviews.llvm.org/D115899
LSan (`ASAN_OPTIONS=detect_leaks=1`) is supported on macOS, but disabled
by default on Darwin (`SANITIZER_MAC`):
```
COMMON_FLAG(bool, detect_leaks, !SANITIZER_MAC, "Enable memory leak detection.")
```
We enable it here for ASan tests to prevent regressions (per comment).
However, LSan is not supported for the iOS simulator and the tests fail
when it is enabled.
Make this "Is macOS?" check more precise since the current one (`Darwin
&& x86_64`) has two issues:
* Includes the simulators
* Excludes macOS on Apple Silicon
This will allow us to (re)enable simulator testing on Green dragon to
give open source better feedback about sanitizer changes:
https://green.lab.llvm.org
rdar://86529234
Differential Revision: https://reviews.llvm.org/D115816
This test would hang when the system ran out of resources and we fail to
create all 300 threads.
Differential Revision: https://reviews.llvm.org/D115845
Adds -L<search-path> and -l<library> options that are analogous to ld's
versions.
Each instance of -L<search-path> or -l<library> will apply to the most recent
-jd option on the command line (-jd <name> creates a JITDylib with the given
name). Library names will match against JITDylibs first, then llvm-jitlink will
look through the search paths for files named <search-path>/lib<library>.dylib
or <search-path>/lib<library>.a.
The default "main" JITDylib will link against all JITDylibs created by -jd
options, and all JITDylibs will link against the process symbols (unless
-no-process-symbols is specified).
The -dlopen option is renamed -preload, and will load dylibs into the JITDylib
for the ORC runtime only.
The effect of these changes is to make it easier to describe a non-trivial
program layout to llvm-jitlink for testing purposes. E.g. the following
invocation describes a program consisting of three JITDylibs: "main" (created
implicitly) containing main.o, "Foo" containing foo1.o and foo2.o, and linking
against library "bar" (not a JITDylib, so it must be a .dylib or .a on disk)
and "Baz" (which is a JITDylib), and "Baz" containing baz.o.
llvm-jitlink \
main.o \
-jd Foo foo1.o foo2.o -L${HOME}/lib -lbar -lBaz
-jd Baz baz.o
This change moves optimized callbacks from each .o file to compiler-rt. Instead of using code generation it uses direct assembly implementation. Please note that the 'or' version is not implemented and it will produce unresolved external if somehow 'or' version is requested.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D114558
This change switches tsan to the new runtime which features:
- 2x smaller shadow memory (2x of app memory)
- faster fully vectorized race detection
- small fixed-size vector clocks (512b)
- fast vectorized vector clock operations
- unlimited number of alive threads/goroutimes
Depends on D112602.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D112603
Currently the test calls dlclose in the thread
concurrently with the main thread calling a function
from the dynamic library. This is not good.
Wait for the main thread to call the function
before calling dlclose.
Depends on D115612.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D115613
The test contains a race and checks that it's detected.
But the race may not be detected since we are doing aggressive flushes
and if the state flush happens between racing accesses, tsan won't
detect the race). So return 1 to make the test deterministic
regardless of the race.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D115612
This change moves optimized callbacks from each .o file to compiler-rt. Instead of using code generation it uses direct assembly implementation. Please note that the 'or' version is not implemented and it will produce unresolved external if somehow 'or' version is requested.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D114558
This change moves optimized callbacks from each .o file to compiler-rt. Instead of using code generation it uses direct assembly implementation. Please note that the 'or' version is not implemented and it will produce unresolved external if somehow 'or' version is requested.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D114558
The test has been flaky for years, and I think we should remove it to
eliminate noise on the buildbot.
Neither me nor dokyungs have been able to fully deflake the test, and it
tests a non-default Entropic flag.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D115453
This reverts commit e5c2a46c5e as this
change introduced a linker error when building sanitizer runtimes:
ld.lld: error: undefined symbol: __sanitizer::internal_start_thread(void* (*)(void*), void*)
>>> referenced by sanitizer_stackdepot.cpp:133 (compiler-rt/lib/sanitizer_common/sanitizer_stackdepot.cpp:133)
>>> compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommonSymbolizer.x86_64.dir/sanitizer_stackdepot.cpp.obj:(__sanitizer::(anonymous namespace)::CompressThread::NewWorkNotify())
This change switches tsan to the new runtime which features:
- 2x smaller shadow memory (2x of app memory)
- faster fully vectorized race detection
- small fixed-size vector clocks (512b)
- fast vectorized vector clock operations
- unlimited number of alive threads/goroutimes
Depends on D112602.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D112603
This change moves optimized callbacks from each .o file to compiler-rt. Instead of using code generation it uses direct assembly implementation. Please note that the 'or' version is not implemented and it will produce unresolved external if somehow 'or' version is requested.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D114558
We call UnmapShadow before the actual munmap, at that point we don't yet
know if the provided address/size are sane. We can't call UnmapShadow
after the actual munmap becuase at that point the memory range can
already be reused for something else, so we can't rely on the munmap
return value to understand is the values are sane.
While calling munmap with insane values (non-canonical address, negative
size, etc) is an error, the kernel won't crash. We must also try to not
crash as the failure mode is very confusing (paging fault inside of the
runtime on some derived shadow address).
Such invalid arguments are observed on Chromium tests:
https://bugs.chromium.org/p/chromium/issues/detail?id=1275581
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D114944
The added test demonstrates loading a dynamic library with static TLS.
Such static TLS is a hack that allows a dynamic library to have faster TLS,
but it can be loaded only iff all threads happened to allocate some excess
of static TLS space for whatever reason. If it's not the case loading fails with:
dlopen: cannot load any more object with static TLS
We used to produce a false positive because dlopen will write into TLS
of all existing threads to initialize/zero TLS region for the loaded library.
And this appears to be racing with initialization of TLS in the thread
since we model a write into the whole static TLS region (we don't what part
of it is currently unused):
WARNING: ThreadSanitizer: data race (pid=2317365)
Write of size 1 at 0x7f1fa9bfcdd7 by main thread:
0 memset
1 init_one_static_tls
2 __pthread_init_static_tls
[[ this is where main calls dlopen ]]
3 main
Previous write of size 8 at 0x7f1fa9bfcdd0 by thread T1:
0 __tsan_tls_initialization
Fix this by ignoring accesses during dlopen.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D114953
Larger blocks are more convenient for compressions.
Blocks are allocated with MmapNoReserveOrDie to save some memory.
Also it's 15% faster on StackDepotBenchmarkSuite
Depends on D114464.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D114488
Sometimes stacks for at_exit callbacks don't include any of the user functions/files.
For example, a race with a global std container destructor will only contain
the container type name and our at_exit_wrapper function. No signs what global variable
this is.
Remember and include in reports the function that installed the at_exit callback.
This should give glues as to what variable is being destroyed.
Depends on D114606.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D114607
Add a test for a common C++ bug when a global object is destroyed
while background threads still use it.
Depends on D114604.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D114605
This change switches tsan to the new runtime which features:
- 2x smaller shadow memory (2x of app memory)
- faster fully vectorized race detection
- small fixed-size vector clocks (512b)
- fast vectorized vector clock operations
- unlimited number of alive threads/goroutimes
Depends on D112602.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D112603
We currently use a wrong value for heap block
(only works for C++, but not for Java).
Use the correct value (we already computed it before, just forgot to use).
Depends on D114593.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D114595
Add a basic test that checks races between vector/non-vector
read/write accesses of different sizes/offsets in different orders.
This gives coverage of __tsan_read/write16 callbacks.
Depends on D114591.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D114592
Vector SSE accesses make compiler emit __tsan_[unaligned_]read/write16 callbacks.
Make it possible to test these.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D114591
The test tries to provoke internal allocator to be locked during fork
and then force the child process to use the internal allocator.
This test sometimes deadlocks with the new tsan runtime.
Depends on D114514.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D114515
Test size larger than clear_shadow_mmap_threshold,
which is handled differently.
Depends on D114348.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D114366
This change switches tsan to the new runtime which features:
- 2x smaller shadow memory (2x of app memory)
- faster fully vectorized race detection
- small fixed-size vector clocks (512b)
- fast vectorized vector clock operations
- unlimited number of alive threads/goroutimes
Differential Revision: https://reviews.llvm.org/D112603
We dropped the printing of live on exit blocks in rG1243cef245f6 -
the commit changed the insertOrMerge logic. Remove the message since it
is no longer needed (all live blocks are inserted into the hashmap)
before serializing/printing the profile. Furthermore, the original
intent was to capture evicted blocks so it wasn't entirely correct.
Also update the binary format test invocation to remove the redundant
print_text directive now that it is the default.
Differential Revision: https://reviews.llvm.org/D114285
This change switches tsan to the new runtime which features:
- 2x smaller shadow memory (2x of app memory)
- faster fully vectorized race detection
- small fixed-size vector clocks (512b)
- fast vectorized vector clock operations
- unlimited number of alive threads/goroutimes
Depends on D112602.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D112603
All runtime callbacks must be non-instrumented with the new tsan runtime
(it's now more picky with respect to recursion into runtime).
Disable instrumentation in Darwin tests as we do in all other tests now.
Differential Revision: https://reviews.llvm.org/D114348
Add a fork test that models what happens on Mac
where fork calls malloc/free inside of our atfork
callbacks.
Reviewed By: vitalybuka, yln
Differential Revision: https://reviews.llvm.org/D114250
The new test started failing on bots with:
CHECK failed: tsan_rtl.cpp:327 "((addr + size)) <= ((TraceMemEnd()))"
(0xf06200e03010, 0xf06200000000) (tid=4073872)
https://lab.llvm.org/buildbot#builders/179/builds/1761
This is a latent bug in aarch64 virtual address space layout,
there is not enough address space to fit traces for all threads.
But since the trace space is going away with the new tsan runtime
(D112603), disable the test.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D113990
Use of gethostent provokes caching of some resources inside of libc.
They are freed in __libc_thread_freeres very late in thread lifetime,
after our ThreadFinish. __libc_thread_freeres calls free which
previously crashed in malloc hooks.
Fix it by setting ignore_interceptors for finished threads,
which in turn prevents malloc hooks.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D113989
pthread_setname_np does linear search over all thread descriptors
to map pthread_t to the thread descriptor. This has O(N^2) complexity
and becomes much worse in the new tsan runtime that keeps all ever
existed threads in the thread registry.
Replace linear search with direct access if pthread_setname_np
is called for the current thread (a very common case).
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D113916
Similar to how the other swift sections are registered by the ORC
runtime's macho platform, add the __swift5_types section, which contains
type metadata. Add a simple test that demonstrates that the swift
runtime recognized the registered types.
rdar://85358530
Differential Revision: https://reviews.llvm.org/D113811
This change switches tsan to the new runtime which features:
- 2x smaller shadow memory (2x of app memory)
- faster fully vectorized race detection
- small fixed-size vector clocks (512b)
- fast vectorized vector clock operations
- unlimited number of alive threads/goroutimes
Depends on D112602.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D112603
Start the background thread only after fork, but not after clone.
For fork we did this always and it's known to work (or user code has adopted).
But if we do this for the new clone interceptor some code (sandbox2) fails.
So model we used to do for years and don't start the background thread after clone.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D113744
The compiler does not recognize HACKY_CALL as a call
(we intentionally hide it from the compiler so that it can
compile non-leaf functions as leaf functions).
To compensate for that hacky call thunk saves and restores
all caller-saved registers. However, it saves only
general-purposes registers and does not save XMM registers.
This is a latent bug that was masked up until a recent "NFC" commit
d736002e90 ("tsan: move memory access functions to a separate file"),
which allowed more inlining and exposed the 10-year bug.
Save and restore caller-saved XMM registers (all) as well.
Currently the bug manifests as e.g. frexp interceptor messes the
return value and the added test fails with:
i=8177 y=0.000000 exp=4
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D113742
Some compiler-rt tests are inherently incompatible with VE because..
* No consistent denormal support on VE. We skip denormal fp inputs in builtin tests.
* `madvise` unsupported on VE.
* Instruction alignment requirements.
Reviewed By: phosek
Differential Revision: https://reviews.llvm.org/D113093
Set the default memprof serialization format as binary. 9 tests are
updated to use print_text=true. Also fixed an issue with concatenation
of default and test specified options (missing separator).
Differential Revision: https://reviews.llvm.org/D113617
This change implements the raw binary format discussed in
https://lists.llvm.org/pipermail/llvm-dev/2021-September/153007.html
Summary of changes
* Add a new memprof option to choose binary or text (default) format.
* Add a rawprofile library which serializes the MIB map to profile.
* Add a unit test for rawprofile.
* Mark sanitizer procmaps methods as virtual to be able to mock them.
* Extend memprof_profile_dump regression test.
Differential Revision: https://reviews.llvm.org/D113317
The existing implementation uses a cache + eviction based scheme to
record heap profile information. This design was adopted to ensure a
constant memory overhead (due to fixed number of cache entries) along
with incremental write-to-disk for evictions. We find that since the
number to entries to track is O(unique-allocation-contexts) the overhead
of keeping all contexts in memory is not very high. On a clang workload,
the max number of unique allocation contexts was ~35K, median ~11K.
For each context, we (currently) store 64 bytes of data - this amounts
to 5.5MB (max). Given the low overheads for a complex workload, we can
simplify the implementation by using a hashmap without eviction.
Other changes:
* Memory map is dumped at the end rather than startup. The relative
order in the profile dump is unchanged since we no longer have evicted
entries at runtime.
* Added a test to check meminfoblocks are merged.
Differential Revision: https://reviews.llvm.org/D111676
Clone does not exist on Mac.
There are chances it will break on other OSes.
Enable it incrementally starting with Linux only,
other OSes can enable it later as needed.
Reviewed By: melver, thakis
Differential Revision: https://reviews.llvm.org/D113693
gtest uses clone for death tests and it needs the same
handling as fork to prevent deadlock (take runtime mutexes
before and release them after).
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D113677
Currently, SANITIZER_COMMON_SUPPORTED_OS is being used to enable many libraries.
Unfortunately this makes it impossible to selectively disable a library based on the OS.
This patch removes this limitation by adding a separate list of supported OSs for the lsan, ubsan, ubsan_minimal, and stats libraries.
Reviewed By: delcypher
Differential Revision: https://reviews.llvm.org/D113444
Entropic scheduling with exec-time option can be misled, if inputs
on the right path to become crashing inputs accidentally take more
time to execute before it's added to the corpus. This patch, by letting
more of such inputs added to the corpus (four inputs of size 7 to 10,
instead of a single input of size 2), reduces possibilities of being
influenced by timing flakiness.
A longer-term fix could be to reduce timing flakiness in the fuzzer;
one way could be to execute inputs multiple times and take average of
their execution time before they are added to the corpus.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D113544
add tracing for loads and stores.
The primary goal is to have more options for data-flow-guided fuzzing,
i.e. use data flow insights to perform better mutations or more agressive corpus expansion.
But the feature is general puspose, could be used for other things too.
Pipe the flag though clang and clang driver, same as for the other SanitizerCoverage flags.
While at it, change some plain arrays into std::array.
Tests: clang flags test, LLVM IR test, compiler-rt executable test.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D113447
I recently spent some extra time debugging a false positive because I
didn't realize the "real" tag was in the short granule. Adding the
short tag here makes it more obvious that we could be dealing with a
short granule.
Reviewed By: hctim, eugenis
Differential Revision: https://reviews.llvm.org/D112949
Previously we only applied it to the first one, which could allow
subsequent global tags to exceed the valid number of bits.
Reviewed By: hctim
Differential Revision: https://reviews.llvm.org/D112853
D112630 ("sanitizer_common: fix up onprint.cpp test")
added O_CREAT, but we also need O_TRUNC b/c the file
may not exist, or may exist as well.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D112788
Add `__c11_atomic_fetch_nand` builtin to language extensions and support `__atomic_fetch_nand` libcall in compiler-rt.
Reviewed By: theraven
Differential Revision: https://reviews.llvm.org/D112400
Enables the arm64 MachO platform, adds basic tests, and implements the
missing TLV relocations and runtime wrapper function. The TLV
relocations are just handled as GOT accesses.
rdar://84671534
Differential Revision: https://reviews.llvm.org/D112656
Commit D112602 ("sanitizer_common: tighten on_print hook test")
changed fopen to open in this test. fopen created the file
if if does not exist, but open does not. This was unnoticed
during local testing because lit is not hermetic and reuses
files from previous runs, but it started failing on bots.
Fix the open call.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D112630
The new tsan runtime does not support arbitrary forms
of recursing into the runtime from hooks.
Disable instrumentation of the hook and use write instead
of fwrite (calls malloc internally).
The new version still recurses (write is intercepted),
but does not fail now (the issue at hand was malloc).
Depends on D112601.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D112602
This is a test-only failure. The test wrongly assumes that this gets us
a tagged pointer:
```
NSObject* num1 = @7;
assert(isTaggedPtr(num1));
```
However, on newer deployment targets that have “const data support” we
get a “normal” pointer to constant object.
Radar-Id: rdar://problem/83217293
PPC64 bot failed with the following error.
The buildbot output is not particularly useful,
but looking at other similar tests, it seems
that there is something broken in free stacks on PPC64.
Use the same hack as other tests use to expect
an additional stray frame.
/home/buildbots/ppc64le-clang-lnt-test/clang-ppc64le-lnt/llvm/compiler-rt/test/tsan/free_race3.c:28:11: error: CHECK: expected string not found in input
// CHECK: Previous write of size 4 at {{.*}} by thread T1{{.*}}:
^
<stdin>:13:9: note: scanning from here
#1 main /home/buildbots/ppc64le-clang-lnt-test/clang-ppc64le-lnt/llvm/compiler-rt/test/tsan/free_race3.c:17:3 (free_race3.c.tmp+0x1012fab8)
^
<stdin>:17:2: note: possible intended match here
ThreadSanitizer: reported 1 warnings
^
Input file: <stdin>
Check file: /home/buildbots/ppc64le-clang-lnt-test/clang-ppc64le-lnt/llvm/compiler-rt/test/tsan/free_race3.c
-dump-input=help explains the following input dump.
Input was:
<<<<<<
.
.
.
8: Previous write of size 4 at 0x7ffff4d01ab0 by thread T1:
9: #0 Thread /home/buildbots/ppc64le-clang-lnt-test/clang-ppc64le-lnt/llvm/compiler-rt/test/tsan/free_race3.c:8:10 (free_race3.c.tmp+0x1012f9dc)
10:
11: Thread T1 (tid=3222898, finished) created by main thread at:
12: #0 pthread_create /home/buildbots/ppc64le-clang-lnt-test/clang-ppc64le-lnt/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1001:3 (free_race3.c.tmp+0x100b9040)
13: #1 main /home/buildbots/ppc64le-clang-lnt-test/clang-ppc64le-lnt/llvm/compiler-rt/test/tsan/free_race3.c:17:3 (free_race3.c.tmp+0x1012fab8)
check:28'0 X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
14:
check:28'0 ~
15: SUMMARY: ThreadSanitizer: data race /home/buildbots/ppc64le-clang-lnt-test/clang-ppc64le-lnt/llvm/compiler-rt/test/tsan/free_race3.c:19:3 in main
check:28'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
16: ==================
check:28'0 ~~~~~~~~~~~~~~~~~~~
17: ThreadSanitizer: reported 1 warnings
check:28'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
check:28'1 ? possible intended match
>>>>>>
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D112444
Add a test where a race with free is called during the free itself
(we only have tests where a race with free is caught during the other memory acces).
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D112433
Based on post-commit review discussion on
2bd8493847 with Richard Smith.
Other uses of forcing HasEmptyPlaceHolder to false seem OK to me -
they're all around pointer/reference types where the pointer/reference
token will appear at the rightmost side of the left side of the type
name, so they make nested types (eg: the "int" in "int *") behave as
though there is a non-empty placeholder (because the "*" is essentially
the placeholder as far as the "int" is concerned).
This was originally committed in 277623f4d5
Reverted in f9ad1d1c77 due to breakages
outside of clang - lldb seems to have some strange/strong dependence on
"char [N]" versus "char[N]" when printing strings (not due to that name
appearing in DWARF, but probably due to using clang to stringify type
names) that'll need to be addressed, plus a few other odds and ends in
other subprojects (clang-tools-extra, compiler-rt, etc).
Allows us to use the small code model when we disable relocation
relaxation.
Reviewed By: eugenis
Differential Revision: https://reviews.llvm.org/D111344
On newer glibc, this test detects an extra match somewhere under
pthread_getattr_np. This results in Thread: lines getting spread out in
the report and failing to match the CHECKs.
Fix the CHECKs to allow this possibility.
Reviewed By: fmayer
Differential Revision: https://reviews.llvm.org/D111841
When LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=on
Asan-i386-calls-Dynamic-Test and Asan-i386-inline-Dynamic-Test fail to
run on a x86_64 host. This is because asan's unit test lit files are
configured once, rather than per target arch as with the non-unit
tests. LD_LIBRARY_PATH ends up incorrect, and the tests try linking
against the x86_64 runtime which fails.
This changes the unit test CMake machinery to configure the default
and dynamic unit tests once per target arch, similar to the other asan
tests. Then the fix from https://reviews.llvm.org/D108859 is adapted
to the unit test Lit files with some modifications.
Fixes PR52158.
Differential Revision: https://reviews.llvm.org/D111756
We are running `ls -lh` in gcov-execlp.c test in Posix folder.
However `-h` is not a POSIX option,ls on some POSIX system (eg: AIX)
may not support it.
This patch remove this option to avoid break.
Reviewed By: anhtuyen
Differential Revision: https://reviews.llvm.org/D111807
Test sometimes fails on buildbot (after two non-Origins executions):
/usr/bin/ld: warning: Cannot export local symbol 'dfsan_flush'
RSS at start: 4620, after mmap: 107020, after mmap+set label: 209424, after fixed map: 4624, after another mmap+set label: 209424, after munmap: 4624
/usr/bin/ld: warning: Cannot export local symbol 'dfsan_flush'
RSS at start: 4620, after mmap: 107020, after mmap+set label: 209424, after fixed map: 4624, after another mmap+set label: 209424, after munmap: 4624
/usr/bin/ld: warning: Cannot export local symbol 'dfsan_flush'
RSS at start: 4620, after mmap: 107020, after mmap+set label: 317992, after fixed map: 10792, after another mmap+set label: 317992, after munmap: 10792
release_shadow_space.c.tmp: /b/sanitizer-x86_64-linux/build/llvm-project/compiler-rt/test/dfsan/release_shadow_space.c:91: int main(int, char **): Assertion `after_fixed_mmap <= before + delta' failed.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D111522
There is a bug reported at https://bugs.llvm.org/show_bug.cgi?id=48938
After looking through the glibc, I found the `atexit(f)` is the same as `__cxa_atexit(f, NULL, NULL)`. In orc runtime, we identify different JITDylib by their dso_handle value, so that a NULL dso_handle is invalid. So in this patch, I added a `PlatformJDDSOHandle` to ELFNixRuntimeState, and functions which are registered by atexit will be registered at PlatformJD.
Reviewed By: lhames
Differential Revision: https://reviews.llvm.org/D111413
TestCases/stress_dtls.c was failing when we ran memprof tests for the first
time. The test checks that __tls_get_addr is not in the output for the last
run when it is possible for the interceptor __interceptor___tls_get_addr to
be in the output from stack dumps. The test actually intends to check that
the various __tls_get_addr reports don't get emitted when intercept_tls_get_addr=0.
This updates the test to also check for the following `:` and preceding `==`
which should ignore the __interceptor___tls_get_addr interceptor.
Differential Revision: https://reviews.llvm.org/D111192
When using a static libunwind, the check_memcpy.c can fail because it checks
that tsan intercepted all memcpy/memmoves in the final binary. Though if the
static libunwind is not instrumented, then this will fail because it may contain
regular memcpy/memmoves.
This adds a new REQUIRES check for ensuring that this test won't run unless a
dynamic libunwind.so is provided.
Differential Revision: https://reviews.llvm.org/D111194
On Ubuntu Focal x13 is used by something in the process of calling
sched_yield. Causing the test to fail depending on when the thread
is stopped.
Adding x14 works around this and the test passes consistently.
Not switching to only x14 because that could make other platforms
fail. With both we'll always find at least one and even if both
values are present we'll only get one report.
Reviewed By: oontvoo, vitalybuka
Differential Revision: https://reviews.llvm.org/D110931
These runtime callbacks are supposed to be non-instrumented,
we can't handle runtime recursion well, nor can we afford
explicit recursion checks in the hot functions (memory access,
function entry/exit).
It used to work (not crash), but it won't work with the new runtime.
Mark all runtime callbacks as non-instrumented.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D111157
Use of space as a separator for options is problematic for wrapper
scripts (i.e. implementations of `%run`) that have to marshall
environment variables to target different than the host.
Rather than requiring every implementation of `%run` to support spaces
in `TSAN_OPTIONS` it is simpler to fix this single test case.
rdar://83637067
Differential Revision: https://reviews.llvm.org/D110967
Previously for mem* intrinsics we only incremented the access count for
the first word in the range. However, after thinking it through I think
it makes more sense to record an access for every word in the range.
This better matches the behavior of inlined memory intrinsics, and also
allows better analysis of utilization at a future date.
Differential Revision: https://reviews.llvm.org/D110799
ad890aa232 landed a test without
using the `%run` prefix which means the test fails to run for
platforms that need it (e.g. iOS simulators).
This patch adds the `%run` prefix. While we're here also split
the single `RUN` line into two to make debugging easier.
rdar://83637296
Differential Revision: https://reviews.llvm.org/D110734
check-orc-rt had no cmake target dependency on orc or llvm-jitlink, which
could lead to regression test failures in compiler-rt. This patch should
fix the issue.
Patch by Jack Andersen (jackoalan@gmail.com). Thanks Jack!
Reviewed By: lhames
Differential Revision: https://reviews.llvm.org/D110659
Some tests with binary IDs would fail with error: no profile can be merged.
This is because raw profiles could have unaligned headers when emitting binary
IDs. This means padding should be emitted after binary IDs are emitted to
ensure everything else is aligned. This patch adds padding after each binary ID
to ensure the next binary ID size is 8-byte aligned. This also adds extra
checks to ensure we aren't reading corrupted data when printing binary IDs.
Differential Revision: https://reviews.llvm.org/D110365
Currently detection of races with TLS/stack initialization
is broken because we imitate the write before thread initialization,
so it's modelled with a wrong thread/epoch.
Fix that and add a test.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D110538
This adds REQUIRES: shared_cxxabi to a bunch of tests that would fail if this
weak reference in sanitizer common was undefined. This is necessary in cases
where libc++abi.a is statically linked in. Because there is no strong reference
to __cxa_demangle in compiler-rt, then if libc++abi is linked in via a static
archive, then the linker will not extract the archive member that would define
that weak symbol. This causes a handful of tests to fail because this leads to
the symbolizer printing mangled symbols where tests expect them demangled.
Technically, this feature is WAI since sanitizer runtimes shouldn't fail if
this symbol isn't resolved, and linking statically means you wouldn't need to
link in all of libc++abi. As a workaround, we can simply make it a requirement
that these tests use shared libc++abis.
Differential Revision: https://reviews.llvm.org/D109639
The stress test does various assorted things
(memory accesses, function calls, atomic operations,
thread creation/join, intercepted libc calls)
in multiple threads just to stress various parts
of the runtime.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D110416
Add a test for __tsan_flush_memory() and for background
flushing of the runtime memory.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D110409
This test specifically checks that profiles are not mergeable if there's a
change in the CounterPtr in the profile header. The test manually changes
CounterPtr by explicitly calling memset on some offset into the profile file.
This test would fail if binary IDs were emitted because the offset calculation
does not take into account the binary ID sizes.
This patch updates the test to use types provided in profile/InstrProfData.inc
to make it more resistant to profile layout changes.
Differential Revision: https://reviews.llvm.org/D110277
Add a test for a trace corner case that lead to a bug
in experimental runtime replacement.
Since it passes with the current runtime it makes sense
to submit it on its own.
Depends on D110264.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D110265
intercept-rethrow-exception.cc fails when running runtimes tests if linking in
a hermetic libc++abi. This is because if libc++abi is used, then asan expects
to intercept __cxa_rethrow_primary_exception on linux, which should unpoison the
stack. If we statically link in libc++abi though, it will contain a strong
definition for __cxa_rethrow_primary_exception which wins over the weakly
defined interceptor provided by asan, causing the test to fail by not unpoisoning
the stack on the exception being thrown.
It's likely no one has encountered this before and possible that upstream tests
opt for dynamically linking where the interceptor can work properly. An ideal
long term solution would be to update the interceptor and libc++[abi] APIs to
work for this case, but that will likely take a long time to work out. In the
meantime, since the test isn't necessarily broken, we can just add another
REQUIRES check to make sure that it's only run if we aren't statically linking
in libc++abi.
Differential Revision: https://reviews.llvm.org/D109938
Switch Java heap move to the new scheme required for the new tsan runtime.
Instead of copying the shadow we reset the destination range.
The new v3 trace contains addresses of accesses, so we cannot simply copy the shadow.
This can lead to false negatives, but cannot lead to false positives.
Depends on D110159.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D110190
dlsym calls into dynamic linker which calls malloc and other things.
It's problematic to do it during the actual exit, because
it can happen from a singal handler or from within the runtime
after we reported the first bug, etc.
See https://github.com/google/sanitizers/issues/1440 for an example
(captured in the added test).
Initialize the callbacks during startup instead.
Depends on D110159.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D110166
When setting the report path, recursively create the directory as
needed. This brings the profile path support for memprof on par with
normal PGO. The code was largely cloned from __llvm_profile_recursive_mkdir
in compiler-rt/lib/profile/InstrProfilingUtil.c.
Differential Revision: https://reviews.llvm.org/D109794
The current code writes the pc-table at the process startup,
which may happen before the common_flags() are initialized.
Move writing to the process end.
This is consistent with how we write the counters and avoids the problem with the uninitalized flags.
Add prints if verbosity>=1.
Reviewed By: kostik
Differential Revision: https://reviews.llvm.org/D110119
When running tests like SanitizerCommon-asan-x86_64-Linux :: Linux/crypt_r.cpp,
it may attempt to use the host header crypt.h rather than a sysroot header.
This is significant in the event where struct crypt_data defined on host is
different from the sysroot used to make the sanitizer runtime libraries. This
can result in logical differences between the expected size/layout of struct
crypt_data known by sanitizers and the strict crypt_data provided by the host crypt.h.
Since tests should still use the CMAKE_SYSROOT, this ensures that CMAKE_SYSROOT
is propagated to compiler-rt tests.
Differential Revision: https://reviews.llvm.org/D109796
We hit some undefined symbol errors to 128-bit floating point functions when linking this test.
ld.lld: error: undefined symbol: __multf3
>>> referenced by strtof128_l.o:(round_and_return) in archive /usr/lib/x86_64-linux-gnu/libc.a
>>> referenced by strtof128_l.o:(round_and_return) in archive /usr/lib/x86_64-linux-gnu/libc.a
>>> referenced by strtof128_l.o:(round_and_return) in archive /usr/lib/x86_64-linux-gnu/libc.a
>>> referenced 4 more times
>>> did you mean: __muldf3
>>> defined in: /usr/local/google/home/leonardchan/llvm-monorepo/llvm-build-1-master-fuchsia-toolchain/lib/clang/14.0.0/lib/x86_64-unknown-linux-gnu/libclang_rt.builtins.a
Host libc expects these to be defined, and compiler-rt will only define these
for certain platforms (see definition for CRT_LDBL_128BIT). Since we likely
can't do anything about the host libc, we can at least restrict the test to
check that these functions are supported.
Differential Revision: https://reviews.llvm.org/D109709
On x86_64-unknown-linux-gnu, `-m32` tests set LD_LIBRARY_PATH to
`config.compiler_rt_libdir` (`$build/lib/clang/14.0.0/lib/x86_64-unknown-linux-gnu`)
instead of i386-unknown-linux-gnu, so `-shared-libsan` executables
cannot find their runtime (e.g. `TestCases/replaceable_new_delete.cpp`).
Detect -m32 and -m64 in config.target_cflags, and adjust `config.compiler_rt_libdir`.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D108859
This patch use the same way as the https://reviews.llvm.org/rGfe1fa43f16beac1506a2e73a9f7b3c81179744eb to handle the thread local variable.
It allocates 2 * pointerSize space in GOT to represent the thread key and data address. Instead of using the _tls_get_addr function, I customed a function __orc_rt_elfnix_tls_get_addr to get the address of thread local varible. Currently, this is a wip patch, only one TLS relocation R_X86_64_TLSGD is supported and I need to add the corresponding test cases.
To allocate the TLS descriptor in GOT, I need to get the edge kind information in PerGraphGOTAndPLTStubBuilder, So I add a `Edge::Kind K` argument in some functions in PerGraphGOTAndPLTStubBuilder.h. If it is not suitable, I can think further to solve this problem.
Differential Revision: https://reviews.llvm.org/D109293
Add integration tests for dyld interposition: DYLD_LIBRARY_PATH and
DYLD_INSERT_LIBRARIES.
DYLD_INSERT_LIBRARIES is also relevant for TSan thread
finalization/destruction sequence in the presence of additional pthread
introspection hooks (libBacktraceRecording.dylib for Xcode 'Queue
Debugging' feature).
rdar://78739125
Differential Revision: https://reviews.llvm.org/D109332
Previously the test was failing on platforms where `long` was less than
64-bits wide (e.g. older WatchOS simulators and arm64_32) because the
`padding` field was too small.
The test currently relies on the `my_object->isa` being scribbled or
left unmodified after `my_object` is freed. However, this was not the
case because the `isa` pointer intersected with
`ChunkHeader::free_context_id`. `free_context_id` starts at the
beginning of user memory but it only initialized once the memory is
freed. This caused the `isa` pointer to change after it was freed
leading to the test crashing.
To fix this the `padding` field has been made explicitly 64-bits wide
(same size as `ChunkHeader::free_context_id`).
rdar://75806757
Differential Revision: https://reviews.llvm.org/D109409
I found that the initial corpus allocation of fork mode has certain defects.
I designed a new initial corpus allocation strategy based on size grouping.
This method can give more energy to the small seeds in the corpus and
increase the throughput of the test.
Fuzzbench data (glibfuzzer is -fork_corpus_groups=1):
https://www.fuzzbench.com/reports/experimental/2021-08-05-parallel/index.html
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D105084
Extend the existing single-pass algorithm for `Merger::Merge` with an algorithm that gives better results. This new implementation can be used with a new **set_cover_merge=1** flag.
This greedy set cover implementation gives a substantially smaller final corpus (40%-80% less testcases) while preserving the same features/coverage. At the same time, the execution time penalty is not that significant (+50% for ~1M corpus files and far less for smaller corpora). These results were obtained by comparing several targets with varying size corpora.
Change `Merger::CrashResistantMergeInternalStep` to collect all features from each file and not just unique ones. This is needed for the set cover algorithm to work correctly. The implementation of the algorithm in `Merger::SetCoverMerge` uses a bitvector to store features that are covered by a file while performing the pass. Collisions while indexing the bitvector are ignored similarly to the fuzzer.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D105284
We've been seeing this test return 31 instead of 32 for the "functions"
line in this test on our AArch64 bots.
One possible cause is some of the children not finishing in time
before the llvm-profdata commands are run, if the machine is heavily loaded.
Wait for all the children to finish before exiting the parent.
Reviewed By: zequanwu
Differential Revision: https://reviews.llvm.org/D109222
This is important as with exceptions enabled, non-POD allocas often have
two lifetime ends: the exception handler, and the normal one.
Reviewed By: eugenis
Differential Revision: https://reviews.llvm.org/D108365
Similar to D97585.
D25456 used `S_ATTR_LIVE_SUPPORT` to ensure the data variable will be retained
or discarded as a unit with the counter variable, so llvm.compiler.used is
sufficient. It allows ld to dead strip unneeded profc and profd variables.
Reviewed By: vsk
Differential Revision: https://reviews.llvm.org/D105445
[ubsan] warn inside the sigaction interceptor if static linking is suspected, and continue instead of crashing on null deref
Reviewed By: kostik
Differential Revision: https://reviews.llvm.org/D109081
Newer Xcode toolchains ship a new otool implementation that prints out
section contents in a slightly different way than otool-classic. Specify
"-V" to otool to get the expected test output.
Differential Revision: https://reviews.llvm.org/D108929
Replace D107203, because __llvm_profile_set_file_object is usually used when the
process doesn't have permission to open/create file. That patch trying to copy
from old profile to new profile contradicts with the usage.
Differential Revision: https://reviews.llvm.org/D108242
In that case it is very likely that there will be a tag mismatch anyway.
We handle the case that the pointer belongs to neither of the allocators
by getting a nullptr from allocator.GetBlockBegin.
Reviewed By: hctim, eugenis
Differential Revision: https://reviews.llvm.org/D108383
Fixes a regression when the allocator is disabled, and a dirty
allocation is re-used. This only occurs when the allocator is disabled,
so a test-only fix, but still necessary.
Reviewed By: eugenis
Differential Revision: https://reviews.llvm.org/D108650
Set SA_SIGINFO only if we set sighandler, or we can set the flag, and
return it as 'old' without actual sigaction set.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D108616
We recently enabled crt for powerpc in
https://reviews.llvm.org/rGb7611ad0b16769d3bf172e84fa9296158f8f1910.
And we started to see some unexpected error message when running
check-runtimes.
eg:
https://lab.llvm.org/buildbot/#/builders/57/builds/9488/steps/6/logs/stdio
line 100 - 103:
"
clang-14: error: unknown argument: '-m64 -fno-function-sections'
clang-14: error: unknown argument: '-m64 -fno-function-sections'
clang-14: error: unknown argument: '-m64 -fno-function-sections'
clang-14: error: unknown argument: '-m64 -fno-function-sections'
"
Looks like we shouldn't strip the space at the beginning,
or else the command line passed to subprocess won't work well.
Reviewed By: phosek, MaskRay
Differential Revision: https://reviews.llvm.org/D108329
The shadow for a short granule is stored in the last byte of the
granule. Currently, if there's a tail-overwrite report (a
buffer-overflow-write in uninstrumented code), we report the shadow byte
as a mismatch against the magic.
Fix this bug by slapping the shadow into the expected value. This also
makes sure that if the uninstrumented WRITE does clobber the shadow
byte, it reports the shadow was actually clobbered as well.
Reviewed By: eugenis, fmayer
Differential Revision: https://reviews.llvm.org/D107938
This change adds support to ORCv2 and the Orc runtime library for static
initializers, C++ static destructors, and exception handler registration for
ELF-based platforms, at present Linux and FreeBSD on x86_64. It is based on the
MachO platform and runtime support introduced in bb5f97e3ad.
Patch by Peter Housel. Thanks very much Peter!
Reviewed By: lhames
Differential Revision: https://reviews.llvm.org/D108081
Before this change we were locking the StackDepot in the fork()
interceptor. This results in a deadlock when allocator functions are
used in a pthread_atfork() callback.
Instead, set up a pthread_atfork() callback at init that locks/unlocks
both StackDepot and the allocator. Since our callback is set up very
early, the pre-fork callback is executed late, and both post-fork ones
are executed early, which works perfect for us.
Differential Revision: https://reviews.llvm.org/D108063
Tsan's check_memcpy.c test was disabled under debug because it failed.
But it points to real issues and does not help to just disable it.
I tried to enable it and see what fail and the first hit was default ctor for:
struct ChainedOriginDepotDesc {
u32 here_id;
u32 prev_id;
};
initializing these fields to 0's help partially,
but compiler still emits memset before calling ctor.
I did not try to see what's the next failure, because if it fails
on such small structs, it won't be realistic to fix everything
and keep working.
Compile runtimes with -O1 under debug instead.
It seems to fix all current failures. At least I run check-tsan
under clang/gcc x debug/non-debug and all combinations passed.
-O1 does not usually use too aggressive optimizations
and sometimes even makes debugging easier because machine code
is not exceedingly verbose.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D107962
New ports in glibc typically don't define ELF_INITFINI, so
DT_INIT/DT_FINI support is disabled.
(rhel ppc64le likely patches their glibc this way as well.)
musl can disable DT_INIT/DT_FINI via -DNO_LEGACY_INITFINI.
So we cannot guarantee ctor()/dtor() will be printed.
This fails with:
/tmp/FlagsTest-5761bc.o: In function `sancov.module_ctor_8bit_counters':
FlagsTest.cpp:(.text.sancov.module_ctor_8bit_counters[sancov.module_ctor_8bit_counters]+0x14): undefined reference to `__start___sancov_cntrs'
FlagsTest.cpp:(.text.sancov.module_ctor_8bit_counters[sancov.module_ctor_8bit_counters]+0x18): undefined reference to `__stop___sancov_cntrs'
<...>
Since https://reviews.llvm.org/D107374. However the changes
there don't seem to be the real fault so xfail while I look into it.
Ellis Hoag
gtt1995