Summary:
A number of testcases in TSAN are designed to deal with intermittent problems
not exist in all executions of the tested program. A script called deflake.bash
runs the executable up to 10 times to deal with the intermittent nature of the tests.
The purpose of this patch is to parameterize the hard-coded threshold above via
--cmake_variables=-DTSAN_TEST_DEFLAKE_THRESHOLD=SomeIntegerValue
When this cmake var is not set, the default value of 10 will be used.
Reviewer: dvyukov (Dmitry Vyukov), eugenis (Evgenii Stepanov), rnk (Reid Kleckner), hubert.reinterpretcast (Hubert Tong), vitalybuka (Vitaly Buka)
Reviewed By: vitalybuka (Vitaly Buka)
Subscribers: mgorny (Michal Gorny), jfb (JF Bastien), steven.zhang (qshanz), llvm-commits (Mailing List llvm-commits), Sanitizers
Tag: LLVM, Sanitizers
Differential Revision: https://reviews.llvm.org/D73707
The interceptor uses thread-local variables, which (until very recently)
are emu-tls. An access to such variable may call malloc which can
deadlock the runtime library.
For CleanseCrashInput, discards stdout output anyway since it is not used.
These changes are to defend against aggressive PID recycle on windows to reduce the chance of contention on files.
Using pipe instead of file also workaround the problem that when the
process is spawned by llvm-lit, the aborted process keeps a handle to the
output file such that the output file can not be removed. This will
cause random test failures.
https://devblogs.microsoft.com/oldnewthing/20110107-00/?p=11803
Reviewers: kcc, vitalybuka
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D73329
Add an optional table lookup after the existing logarithm computation
for MidSize < Size <= MaxSize during size -> class lookups. The lookup is
O(1) due to indexing a precomputed (via constexpr) table based on a size
table. Switch to this approach for the Android size class maps.
Other approaches considered:
- Binary search was found to have an unacceptable (~30%) performance cost.
- An approach using NEON instructions (see older version of D73824) was found
to be slightly slower than this approach on newer SoCs but significantly
slower on older ones.
By selecting the values in the size tables to minimize wastage (for example,
by passing the malloc_info output of a target program to the included
compute_size_class_config program), we can increase the density of allocations
at a small (~0.5% on bionic malloc_sql_trace as measured using an identity
table) performance cost.
Reduces RSS on specific Android processes as follows (KB):
Before After
zygote (median of 50 runs) 26836 26792 (-0.2%)
zygote64 (median of 50 runs) 30384 30076 (-1.0%)
dex2oat (median of 3 runs) 375792 372952 (-0.8%)
I also measured the amount of whole-system idle dirty heap on Android by
rebooting the system and then running the following script repeatedly until
the results were stable:
for i in $(seq 1 50); do grep -A5 scudo: /proc/*/smaps | grep Pss: | cut -d: -f2 | awk '{s+=$1} END {print s}' ; sleep 1; done
I did this 3 times both before and after this change and the results were:
Before: 365650, 356795, 372663
After: 344521, 356328, 342589
These results are noisy so it is hard to make a definite conclusion, but
there does appear to be a significant effect.
On other platforms, increase the sizes of all size classes by a fixed offset
equal to the size of the allocation header. This has also been found to improve
density, since it is likely for allocation sizes to be a power of 2, which
would otherwise waste space by pushing the allocation into the next size class.
Differential Revision: https://reviews.llvm.org/D73824
This lets us remove two pointer indirections (one by removing the pointer,
and another by making the AllocatorPtr declaration hidden) in the C++ wrappers.
Differential Revision: https://reviews.llvm.org/D74356
Summary:
Instead of hand-crafting an offset into the structure returned by
dlopen(3) to get at the link map, use the documented API. This is
described in dlinfo(3): by calling it with `RTLD_DI_LINKMAP`, the
dynamic linker ensures the right address is returned.
This is a recommit of 92e267a94d, with
dlinfo(3) expliclity being referenced only for FreeBSD, non-Android
Linux, NetBSD and Solaris. Other OSes will have to add their own
implementation.
Reviewers: devnexen, emaste, MaskRay, krytarowski
Reviewed By: krytarowski
Subscribers: krytarowski, vitalybuka, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73990
Summary:
The refactoring has caused a failure in
http://lab.llvm.org:8011/builders/clang-ppc64le-linux/builds/29265
The idea of failing the symbolization when the symbolizer bufer is too small
was incorrect. The symbolizer can be invoked for other frames that may fit into
the buffer and get symbolized.
Reviewers: vitalybuka, eugenis
Subscribers: dberris, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D74343
Summary:
Instead of hand-crafting an offset into the structure returned by
dlopen(3) to get at the link map, use the documented API. This is
described in dlinfo(3): by calling it with `RTLD_DI_LINKMAP`, the
dynamic linker ensures the right address is returned.
Reviewers: devnexen, emaste, MaskRay, krytarowski
Reviewed By: krytarowski
Subscribers: krytarowski, vitalybuka, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73990
Summary:
Nothing critical, just a few potential improvements I've noticed while reading
the code:
- return `false` when symbolizer buffer is too small to read all data
- invert some conditions to reduce indentation
- prefer `nullptr` over `0` for pointers; init some pointers on stack;
- remove minor code duplication
Reviewers: eugenis, vitalybuka
Subscribers: dberris, #sanitizers, llvm-commits, kcc
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D74137
Summary:
This tweaks some behaviors of the allocator wrt 32-bit, notably
tailoring the size-class map.
I had to remove a `printStats` from `__scudo_print_stats` since when
within Bionic they share the same slot so they can't coexist at the
same time. I have to find a solution for that later, but right now we
are not using the Svelte configuration.
Reviewers: rengolin
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D74178
EXCLUDE_FROM_ALL means something else for add_lit_testsuite as it does
for something like add_executable. Distinguish between the two by
renaming the variable and making it an argument to add_lit_testsuite.
Differential revision: https://reviews.llvm.org/D74168
Summary:
New shard out of the Check() function for GWP-ASan uses
android_set_abort_message. This is happily present on bionic Android,
but not Android for glibc host x86. Fix up to use the weak definition
always, so we don't have to worry.
Reviewers: eugenis
Reviewed By: eugenis
Subscribers: #sanitizers, llvm-commits, pcc, cferris
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D74142
Summary:
GWP-ASan's PRNG didn't use Initial-Exec TLS. Fix that to ensure that we don't
have infinite recursion, and also that we don't allocate a DTV on Android when
GWP-ASan is touched.
Test coverage ensuring that the sample counter is UINT32_MAX for an
uninitialised GWP-ASan is provided by gwp_asan/tests/late_init.cpp.
Reviewers: pcc, cferris
Reviewed By: pcc
Subscribers: #sanitizers, llvm-commits, rprichard, eugenis
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D74135
The class is only used in SizeClassAllocator32 in 64-bit mode, but we don't
use that class in 64-bit mode.
Differential Revision: https://reviews.llvm.org/D74099
Summary:
Forewarning: This patch looks big in #LOC changed. I promise it's not that bad, it just moves a lot of content from one file to another. I've gone ahead and left inline comments on Phabricator for sections where this has happened.
This patch:
1. Introduces the crash handler API (crash_handler_api.h).
2. Moves information required for out-of-process crash handling into an AllocatorState. This is a trivially-copied POD struct that designed to be recovered from a deceased process, and used by the crash handler to create a GWP-ASan report (along with the other trivially-copied Metadata struct).
3. Implements the crash handler API using the AllocatorState and Metadata.
4. Adds tests for the crash handler.
5. Reimplements the (now optionally linked by the supporting allocator) in-process crash handler (i.e. the segv handler) using the new crash handler API.
6. Minor updates Scudo & Scudo Standalone to fix compatibility.
7. Changed capitalisation of errors (e.g. /s/Use after free/Use After Free).
Reviewers: cryptoad, eugenis, jfb
Reviewed By: eugenis
Subscribers: merge_guards_bot, pcc, jfb, dexonsmith, mgorny, cryptoad, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73557
Summary:
I tried to move the `madvise` calls outside of one of the secondary
mutexes, but this backfired. There is situation when a low release
interval is set combined with secondary pressure that leads to a race:
a thread can get a block from the cache, while another thread is
`madvise`'ing that block, resulting in a null header.
I changed the secondary race test so that this situation would be
triggered, and moved the release into the cache mutex scope.
Reviewers: cferris, pcc, eugenis, hctim, morehouse
Subscribers: jfb, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D74072
XRay builds uses llvm-config to obtain the ldflags and libs and then
passes those to CMake. Unfortunately, this breaks on Windows because
CMake tries to interpret backslashes followed by certain characters
as flags. We need to rewrite these into forward slashes that are used
by CMake (even on Windows).
Differential Revision: https://reviews.llvm.org/D73523
Summary:
`sigaltstack` interception is implemented D73816. This updated OpenBSD and Solaris bits to fix the build errors.
Reviewers: eugenis
Reviewed By: eugenis
Subscribers: fedor.sergeev, krytarowski, emaste, eugenis, dberris, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Patch by Igor Sugak.
Differential Revision: https://reviews.llvm.org/D73976
By subtracting 1 from Size at the beginning we can simplify the
subsequent calculations. This also saves 4 instructions on aarch64
and 9 instructions on x86_64, but seems to be perf neutral.
Differential Revision: https://reviews.llvm.org/D73936
Summary:
Fix parsing of mangled stack trace lines where the address has been
replaced with "0x", literally.
Reviewers: vitalybuka
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73823
Summary:
An implementation for `sigaltstack` to make its side effect be visible to MSAN.
```
ninja check-msan
```
Reviewers: vitalybuka, eugenis
Reviewed By: eugenis
Subscribers: dberris, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73816
Patch by Igor Sugak.
As a result of recent changes to the Android size classes, the malloc_free_loop
benchmark started exhausting the 8192 size class at 32768 iterations. To avoid
this problem (and to make the test more realistic), change the benchmark to
use a variety of size classes.
Differential Revision: https://reviews.llvm.org/D73918
Summary:
This patch allows for late initialisation of the GWP-ASan allocator. Previously, if late initialisation occurred, the sample counter was never updated, meaning we would end up having to wait for 2^32 allocations before getting a sampled allocation.
Now, we initialise the sampling mechanism in init() as well. We require init() to be called single-threaded, so this isn't a problem.
Reviewers: eugenis
Reviewed By: eugenis
Subscribers: merge_guards_bot, mgorny, #sanitizers, llvm-commits, cferris
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73896
The configuration for -fPIC in the builtins library when built standalone
is unconditional, stating that the flags would "normally be added... by
the llvm cmake step"
This is untrue, as the llvm cmake step checks LLVM_ENABLE_PIC, which allows
a client to turn off -fPIC.
I've added an option when compiler-rt builtins are configured standalone, such
as when built as part of the LLVM runtimes system, to guard the application of
-fPIC for users that want it.
Patch By: JamesNagurne
Differential Revision: https://reviews.llvm.org/D72950
Summary:
This changes a couple of parameters in the default Android config to
address some performance and memory footprint issues (well to be closer
to the default Bionic allocator numbers).
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73750
Summary:
The Secondary's cache needs to be released when the Combined's
`releaseToOS` function is called (via `M_PURGE`) for example,
which this CL adds.
Additionally, if doing a forced release, we'll release the
transfer batch class as well since now we can do that.
There is a couple of other house keeping changes as well:
- read the page size only once in the Secondary Cache `store`
- remove the interval check for `CanRelease`: we are going to
make that configurable via `mallopt` so this needs not be
set in stone there.
Reviewers: cferris, hctim, pcc, eugenis
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73730
Summary: `.cmd` is interpreted as script in windows console.
Reviewers: davidxl, rnk
Reviewed By: davidxl
Differential Revision: https://reviews.llvm.org/D73327
Summary:
A couple of seemingly innocuous changes ended up having a large impact
on the 32-bit performance. I still have to make those configurable at
some point, but right now it will have to do.
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73658
Summary:
Since commit c299d1981d scudo
standalone can't be built without including sys/auxv.h.
I do not have that file on my system, and my builds have failed
when trying to simply build "all" runtimes. Assuming that "all"
means "all possible given the current environment" we need to
guard the setting of COMPILER_RT_HAS_SCUDO_STANDALONE=TRUE by
first checking if sys/auxv.h can be found.
Reviewers: pcc, cryptoad
Reviewed By: pcc
Subscribers: mgorny, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73631
mlockall and munlockall were introduced in Android API 17, so avoid
referencing them on prior versions.
Differential Revision: https://reviews.llvm.org/D73515
Summary:
Zygote & children's stderr is lost, so use Bionic's provided allocation
free syslog function for `outputRaw`. Get rid of the mutex as it's not
vital and could cause issues with `fork`.
Reviewers: cferris, pcc, eugenis, hctim, morehouse
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73561
Summary:
This CL changes multiple things to improve performance (notably on
Android).We introduce a cache class for the Secondary that is taking
care of this mechanism now.
The changes:
- change the Secondary "freelist" to an array. By keeping free secondary
blocks linked together through their headers, we were keeping a page
per block, which isn't great. Also we know touch less pages when
walking the new "freelist".
- fix an issue with the freelist getting full: if the pattern is an ever
increasing size malloc then free, the freelist would fill up and
entries would not be used. So now we empty the list if we get to many
"full" events;
- use the global release to os interval option for the secondary: it
was too costly to release all the time, particularly for pattern that
are malloc(X)/free(X)/malloc(X). Now the release will only occur
after the selected interval, when going through the deallocate path;
- allow release of the `BatchClassId` class: it is releasable, we just
have to make sure we don't mark the batches containing batches
pointers as free.
- change the default release interval to 1s for Android to match the
current Bionic allocator configuration. A patch is coming up to allow
changing it through `mallopt`.
- lower the smallest class that can be released to `PageSize/64`.
Reviewers: cferris, pcc, eugenis, morehouse, hctim
Subscribers: phosek, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73507
Summary:
This is an Android-specific interface for iterating over all live
allocations in a memory range.
Reviewers: hctim, cferris
Subscribers: mgorny, mgrang, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73305
This is a small refactoring to prepare for porting LSan to Fuchsia.
Factor out parts of lsan_thread.{cpp,h} that don't apply to Fuchsia.
Since existing supported systems are POSIX-based, the affected code
is moved to lsan_posix.{cpp.h}.
Patch By: mcgrathr
Differential Revision: https://reviews.llvm.org/D73309
This is a small refactoring to prepare for porting LSan to Fuchsia.
On Fuchsia, the system supplies a unified API for suspending threads and
enumerating roots from OS-specific places like thread state and global data
ranges. So its LockStuffAndStopTheWorld implementation will make specific
callbacks for all the OS-specific root collection work before making the
common callback that includes the actual leak-checking logic.
Patch By: mcgrathr
Differential Revision: https://reviews.llvm.org/D72988
Summary:
* Implement enable() and disable() in GWP-ASan.
* Setup atfork handler.
* Improve test harness sanity and re-enable GWP-ASan in Scudo.
Scudo_standalone disables embedded GWP-ASan as necessary around fork().
Standalone GWP-ASan sets the atfork handler in init() if asked to. This
requires a working malloc(), therefore GWP-ASan initialization in Scudo
is delayed to the post-init callback.
Test harness changes are about setting up a single global instance of
the GWP-ASan allocator so that pthread_atfork() does not create
dangling pointers.
Test case shamelessly stolen from D72470.
Reviewers: cryptoad, hctim, jfb
Subscribers: mgorny, jfb, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73294
Summary:
In order to do this `FUZZER_SUPPORTED_OS` had to be pulled out of
`lib/fuzzer/CMakeLists.txt` and into the main config so we can use it
from the `test/fuzzer/CMakeList.txt`. `FUZZER_SUPPORTED_OS` currently
has the same value of `SANITIZER_COMMON_SUPPORTED_OS` which preserves
the existing behaviour but this allows us in the future to adjust the
supported platforms independent of `SANITIZER_COMMON_SUPPORTED_OS`. This
mirrors the other sanitizers.
For non-Apple platforms `FUZZER_SUPPORTED_OS` is not defined and
surprisingly this was the behaviour before this patch because
`SANITIZER_COMMON_SUPPORTED_OS` was actually empty. This appears to
not matter right now because the functions that take an `OS` as an
argument seem to ignore it on non-Apple platforms.
While this change tries to be NFC it is technically not because we
now generate an iossim config whereas previously we didn't. This seems
like the right thing to do because the build system was configured to
compile LibFuzzer for iossim but previously we weren't generating a lit
test config for it. The device/simulator testing configs don't run by
default anyway so this shouldn't break testing.
This change relies on the get_capitalized_apple_platform() function
added in a previous commit.
rdar://problem/58798733
Reviewers: kubamracek, yln
Subscribers: mgorny, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73243
Summary:
The previous code hard-coded platform names but compiler-rt's CMake
build system actually already knows which Apple platforms TSan supports.
This change uses this information to enumerate the different Apple
platforms.
This change relies on the `get_capitalized_apple_platform()` function
added in a previous commit.
rdar://problem/58798733
Reviewers: kubamracek, yln
Subscribers: mgorny, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73238
Summary:
The previous code hard-coded platform names but compiler-rt's CMake
build system actually already knows which Apple platforms ASan supports.
This change uses this information to enumerate the different Apple
platforms.
rdar://problem/58798733
Reviewers: kubamracek, yln
Subscribers: mgorny, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73232
Summary:
there is an ongoing work on interchangeable custom mutators
(https://github.com/google/clusterfuzz/pull/1333/files#r367706283)
and having some sort of signalling from libFuzzer that it has loaded
a custom mutator would be helpful.
The initial idea was to make the mutator to print something, but given
the anticipated variety of different mutators, it does not seem possible
to make all of them print the same message to signal their execution.
Reviewers: kcc, metzman
Reviewed By: metzman
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73136
Summary:
Unity is making irresponsible assumptions as to how clumped up memory
should be. With larger regions, we break those, resulting in errors
like:
"Using memoryadresses from more that 16GB of memory"
This is unfortunately one of those situations where we have to bend to
existing code because we doubt it's going to change any time soon.
128MB should be enough, but we could be flirting with OOMs in the
higher class sizes.
Reviewers: cferris, eugenis, hctim, morehouse, pcc
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73143
Fix clear_cache_test to work on NetBSD with PaX MPROTECT enabled, that
is when creating W+X mmaps is prohibited. Use the recommended solution:
create two mappings for the same memory area, make one of them RW, while
the other RX. Copy the function into the RW area but run it from the RX
area.
In order to implement this, I've split the pointer variables to
'write_buffer' and 'execution_buffer'. Both are separate pointers
on NetBSD, while they have the same value on other systems.
I've also split the memcpy_f() into two: new memcpy_f() that only takes
care of copying memory and discards the (known) result of memcpy(),
and realign_f() that applies ARM realignment to the given pointer.
Again, there should be no difference on non-NetBSD systems but on NetBSD
copying is done on write_buffer, while realignment on pointer
to the execution_buffer.
I have tested this change on NetBSD and Linux.
Differential Revision: https://reviews.llvm.org/D72578
Summary:
In some configuration, `sched_getaffinity` can fail. Some reasons for
that being the lack of `CAP_SYS_NICE` capability or some syscall
filtering and so on.
This should not be fatal to the allocator, so in this situation, we
will fallback to the `MaxTSDCount` value specified in the allocator
configuration.
Reviewers: cferris, eugenis, hctim, morehouse, pcc
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73055
This is an alternative to the continous mode that was implemented in
D68351. This mode relies on padding and the ability to mmap a file over
the existing mapping which is generally only available on POSIX systems
and isn't suitable for other platforms.
This change instead introduces the ability to relocate counters at
runtime using a level of indirection. On every counter access, we add a
bias to the counter address. This bias is stored in a symbol that's
provided by the profile runtime and is initially set to zero, meaning no
relocation. The runtime can mmap the profile into memory at abitrary
location, and set bias to the offset between the original and the new
counter location, at which point every subsequent counter access will be
to the new location, which allows updating profile directly akin to the
continous mode.
The advantage of this implementation is that doesn't require any special
OS support. The disadvantage is the extra overhead due to additional
instructions required for each counter access (overhead both in terms of
binary size and performance) plus duplication of counters (i.e. one copy
in the binary itself and another copy that's mmapped).
Differential Revision: https://reviews.llvm.org/D69740
Summary:
This commit modifies the way `ExecuteCommand` works in fuchsia by adding
special logic to handle `/dev/null`.
The FuzzerCommand interface does not have a way to "discard" the output,
so other parts of the code just set the output file to `getDevNull()`.
The problem is that fuchsia does not have a named file that is
equivalent to `/dev/null`, so opening that file just fails.
This commit detects whether the specified output file is `getDevNull`,
and if that's the case, it will not copy the file descriptor for stdout
in the spawned process.
NOTE that modifying `FuzzerCommand` to add a "discardOutput" function
involves a significant refactor of all the other platforms, as they all
rely on the `toString()` method of `FuzzerCommand`.
This allows libfuzzer in fuchsia to run with `fork=1`, as the merge
process (`FuzzerMerge.cpp`) invoked `ExecuteCommand` with `/dev/null` as the
output.
Reviewers: aarongreen, phosek
Reviewed By: aarongreen
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D72894
When the hardware and operating system support the ARM Memory Tagging
Extension, tag primary allocation granules with a random tag. The granules
either side of the allocation are tagged with tag 0, which is normally
excluded from the set of tags that may be selected randomly. Memory is
also retagged with a random tag when it is freed, and we opportunistically
reuse the new tag when the block is reused to reduce overhead. This causes
linear buffer overflows to be caught deterministically and non-linear buffer
overflows and use-after-free to be caught probabilistically.
This feature is currently only enabled for the Android allocator
and depends on an experimental Linux kernel branch available here:
https://github.com/pcc/linux/tree/android-experimental-mte
All code that depends on the kernel branch is hidden behind a macro,
ANDROID_EXPERIMENTAL_MTE. This is the same macro that is used by the Android
platform and may only be defined in non-production configurations. When the
userspace interface is finalized the code will be updated to use the stable
interface and all #ifdef ANDROID_EXPERIMENTAL_MTE will be removed.
Differential Revision: https://reviews.llvm.org/D70762
Summary:
The IBM XL compiler uses `-qfuncsect` for `-ffunction-sections`.
The comment about sanitizers and `-f[no-]function-sections` is corrected
also, as it is pertinent to this patch.
The sanitizer-related use of `-fno-function-sections` is associated with
powerpc64le, a target for which there is an IBM XL compiler, so that use
is updated in this patch to apply `-qnofuncsect` in case a build using
the XL compiler is viable on that platform.
This patch has been verified with the XL compiler on AIX only.
Reviewers: daltenty, stevewan
Reviewed By: daltenty
Subscribers: mgorny, steven.zhang, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D72335
Summary:
fork() wasn't well (or at all) supported in Scudo. This materialized
in deadlocks in children.
In order to properly support fork, we will lock the allocator pre-fork
and unlock it post-fork in parent and child. This is done via a
`pthread_atfork` call installing the necessary handlers.
A couple of things suck here: this function allocates - so this has to
be done post initialization as our init path is not reentrance, and it
doesn't allow for an extra pointer - so we can't pass the allocator we
are currently working with.
In order to work around this, I added a post-init template parameter
that gets executed once the allocator is initialized for the current
thread. Its job for the C wrappers is to install the atfork handlers.
I reorganized a bit the impacted area and added some tests, courtesy
of cferris@ that were deadlocking prior to this fix.
Subscribers: jfb, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D72470
The executable acquires an advisory record lock (`fcntl(fd, F_SETLKW, *)`) on a profile file.
Merge pool size >= 10 may be beneficial when the concurrency is large.
Also fix a small problem about snprintf. It can cause the filename to be truncated after %m.
Reviewed By: davidxl
Differential Revision: https://reviews.llvm.org/D71970
__sanitizer_stat_init is called for the executable first, then the
shared object. In WriterModuleReport(), the information for the shared
object will be recorded first. It'd be nice to get rid of the order
requirement of static constructors. (This should make .ctors platforms
work.)
Summary:
Qsort interceptor suppresses all checks by unpoisoning the data in the
wrapper of a comparator function, and then unpoisoning the output array
as well.
This change adds an explicit run of the comparator on all elements of
the input array to catch any sanitizer bugs.
Reviewers: vitalybuka
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71780
Rather than handling zlib handling manually, use `find_package` from CMake
to find zlib properly. Use this to normalize the `LLVM_ENABLE_ZLIB`,
`HAVE_ZLIB`, `HAVE_ZLIB_H`. Furthermore, require zlib if `LLVM_ENABLE_ZLIB` is
set to `YES`, which requires the distributor to explicitly select whether
zlib is enabled or not. This simplifies the CMake handling and usage in
the rest of the tooling.
This restores 68a235d07f,
e6c7ed6d21. The problem with the windows
bot is a need for clearing the cache.
This reverts commit 68a235d07f.
This commit broke the clang-x64-windows-msvc build bot and a follow-up
commit did not fix it. Reverting to fix the bot.
Rather than handling zlib handling manually, use `find_package` from CMake
to find zlib properly. Use this to normalize the `LLVM_ENABLE_ZLIB`,
`HAVE_ZLIB`, `HAVE_ZLIB_H`. Furthermore, require zlib if `LLVM_ENABLE_ZLIB` is
set to `YES`, which requires the distributor to explicitly select whether
zlib is enabled or not. This simplifies the CMake handling and usage in
the rest of the tooling.
Pass -static so that clang will not pass -Wl,--dynamic-linker,... to the
linker. The test is not expected to run under a ld.so. (Technically it
works under a ld.so but glibc expects to see a PT_DYNAMIC. lld
intentionally does not follow GNU ld's complex rules regarding
PT_DYNAMIC.)
This allows commit 1417558e4a to be
relanded.
This reverts commit 7a9ebe9512, and
dependent commit 54c5224203, which
disables qsort interception for some iOS platforms.
After this change, the -Nolibc sanitizer common test binary crashes on
startup on my regular Linux workstation, as well as on our bots:
https://ci.chromium.org/p/chromium/builders/try/linux_upload_clang/740
********************
Failing Tests (1):
SanitizerCommon-Unit ::
./Sanitizer-x86_64-Test/SanitizerCommon.NolibcMain
Loading it up in gdb shows that it crashes during relocation processing,
which suggests that some glibc loader versions do not support the
THREADLOCAL data added in this interceptor.
Summary:
Add support for NetBSD 9.0 and newer versions of interceptors
operating on struct statvfs: fstatvfs, fstatvfs1, getmntinfo,
getvfsstat, statvfs, statvfs1.
The default promoted interceptors are for NetBSD 9.99.26. Older
ones (currently 9.0) are kept in a new NetBSD specific file:
/sanitizer_common_interceptors_netbsd_compat.inc. This file
defines compat interceptors and mangles `INIT_*` macros,
concatenating the current interceptors and the compat ones.
This redefinition is not elegant, but it avoids preprocessor madness.
Define struct_statvfs90_sz for the compat purposes.
Reviewers: mgorny, kcc, vitalybuka, joerg
Reviewed By: mgorny
Subscribers: dberris, llvm-commits, #sanitizers
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71700
as it's failing the netbsd specific linter parts of the sanitizer linter:
llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors_netbsd_compat.inc:23: Lines should be <= 80 characters long [whitespace/line_length]
llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_netbsd.cpp:2450: Do not use variable-length arrays. Use an appropriately named ('k' followed by CamelCase) compile-time constant for the size.
This reverts commit 78f714f824.
Summary:
Add support for NetBSD 9.0 and newer versions of interceptors
operating on struct statvfs: fstatvfs, fstatvfs1, getmntinfo,
getvfsstat, statvfs, statvfs1.
The default promoted interceptors are for NetBSD 9.99.26. Older
ones (currently 9.0) are kept in a new NetBSD specific file:
/sanitizer_common_interceptors_netbsd_compat.inc. This file
defines compat interceptors and mangles `INIT_*` macros,
concatenating the current interceptors and the compat ones.
This redefinition is not elegant, but it avoids preprocessor madness.
Define struct_statvfs90_sz for the compat purposes.
Reviewers: mgorny, kcc, vitalybuka, joerg
Reviewed By: mgorny
Subscribers: dberris, llvm-commits, #sanitizers
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71700
This change breaks LLVM bootstrap with ASan and MSan.
FAILED: lib/ToolDrivers/llvm-lib/Options.inc
OptParser.td:137:1: error: Option is equivalent to
def INPUT : Option<[], "<input>", KIND_INPUT>;
^
OptParser.td:137:1: error: Other defined here
def INPUT : Option<[], "<input>", KIND_INPUT>;
This reverts commit caa48a6b88.
Building the sanitizers for watchOS currently fails with
sanitizer_common_interceptors.inc:9656:8: error: thread-local storage is not supported for the current target
static THREADLOCAL SIZE_T qsort_size;
I've also speculatively disabled QSORT interception for tvOS to unblock
failing builds. I'll ask someone with more sanitizer knowledge to check
after the holidays.
Summary:
Qsort interceptor suppresses all checks by unpoisoning the data in the
wrapper of a comparator function, and then unpoisoning the output array
as well.
This change adds an explicit run of the comparator on all elements of
the input array to catch any sanitizer bugs.
Reviewers: vitalybuka
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71780
Summary:
This fixes qsort-related false positives with glibc-2.27.
I'm not entirely sure why they did not show up with the earlier
versions; the code seems similar enough.
Reviewers: vitalybuka
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71740
Recent versions of the iOS simulator require that a "simulator device"
is booted before we can use `simctl spawn` (see iossim_run.py) to start
processes.
We can use `simctl bootstatus` to ensure that the simulator device
is booted before we run any tests via lit. The `-b` option starts the
device if necessary.
Reviewed By: delcypher
Differential Revision: https://reviews.llvm.org/D71449
Summary:
Some Android builds that we are interested in define `__BIONIC__`
but not `__ANDROID__`, so expand `SCUDO_ANDROID` to encompass those.
Reviewers: cferris, hctim, pcc, eugenis, morehouse
Subscribers: krytarowski, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71772
Summary:
Qsort interceptor suppresses all checks by unpoisoning the data in the
wrapper of a comparator function, and then unpoisoning the output array
as well.
This change adds an explicit run of the comparator on all elements of
the input array to catch any sanitizer bugs.
Reviewers: vitalybuka
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71780
Summary:
This fixes qsort-related false positives with glibc-2.27.
I'm not entirely sure why they did not show up with the earlier
versions; the code seems similar enough.
Reviewers: vitalybuka
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71740
Introduce a new %run_nomprotect substitution to run tests that do not
work with MPROTECT enabled. This uses paxctl via a wrapper on NetBSD,
and evaluates to plain %run on other systems.
Differential Revision: https://reviews.llvm.org/D71513
Summary:
In order to implement `malloc_{enable|disable}` we were just disabling
(or really locking) the Primary and the Secondary. That meant that
allocations could still be serviced from the TSD as long as the cache
wouldn't have to be filled from the Primary.
This wasn't working out for Android tests, so this change implements
registry disabling (eg: locking) so that `getTSDAndLock` doesn't
return a TSD if the allocator is disabled. This also means that the
Primary doesn't have to be disabled in this situation.
For the Shared Registry, we loop through all the TSDs and lock them.
For the Exclusive Registry, we add a `Disabled` boolean to the Registry
that forces `getTSDAndLock` to use the Fallback TSD instead of the
thread local one. Disabling the Registry is then done by locking the
Fallback TSD and setting the boolean in question (I don't think this
needed an atomic variable but I might be wrong).
I clang-formatted the whole thing as usual hence the couple of extra
whiteline changes in this CL.
Reviewers: cferris, pcc, hctim, morehouse, eugenis
Subscribers: jfb, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71719
Add a missing %run substitution to fread_fwrite test. This fixes
the test on NetBSD where %run disables ASLR as necessary for MSAN
to function.
Differential Revision: https://reviews.llvm.org/D71623
Summary: Also add an error case when targetting an unimplement architecture.
Subscribers: nemanjai, jsji, shchenz, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71670
Summary:
Remove REQUIRES-ANY alias lit directive since it is hardly used and can
be easily implemented using an OR expression using REQUIRES. Fixup
remaining testcases still using REQUIRES-ANY.
Reviewers: probinson, jdenny, gparker42
Reviewed By: gparker42
Subscribers: eugenis, asb, rbar, johnrusso, simoncook, sabuasal, niosHD, delcypher, jrtc27, zzheng, edward-jones, rogfer01, MartinMosbeck, brucehoult, the_o, PkmX, jocewei, lenary, s.egerton, pzheng, sameer.abuasal, apazos, luismarques, cfe-commits, #sanitizers, llvm-commits
Tags: #llvm, #clang, #sanitizers
Differential Revision: https://reviews.llvm.org/D71408
Construction of InternalMmapVector is often followed by a call to
reserve(), which may result in immediate reallocation of the memory
for the internal storage. This patch delays that allocation until
it is really needed.
Differential Revision: https://reviews.llvm.org/D71342
Summary:
This is needed because on some platforms we can't install signal
handlers and so the application just traps (i.e. crashes) rather than being intercepted
by ASan's signal handler which in the default Darwin config doesn't
exit with a crashing exit code.
rdar://problem/57984547
Reviewers: yln, kubamracek, jfb
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71573
This skips calling `pthread_self` when `main_thread_identity` hasn't
been initialized yet. `main_thread_identity` is only ever assigned in
`__tsan::InitializePlatform`. This change should be relatively safe; we
are not changing behavior other than skipping the call to `pthread_self`
when `main_thread_identity == 0`.
rdar://57822138
Reviewed By: kubamracek
Differential Revision: https://reviews.llvm.org/D71559
Summary:
When running the tests on a Ubuntu 18.04 machine this test is crashing for
me inside the runtime linker. My guess is that it is trying to save more
registers (possibly large vector ones) and the current stack space is not
sufficient.
Reviewers: samsonov, kcc, eugenis
Reviewed By: eugenis
Subscribers: eugenis, merge_guards_bot, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71461
Turns out that gtest in LLVM is only 1.8.0 (the newest version 1.10.0)
supports the GTEST_SKIP() macro, and apparently I didn't build w/o
GWP-ASan.
Should fix the GN bot, as well as any bots that may spuriously break on
platforms where the code wasn't correctly ifdef'd out as well.
Summary:
Adds GWP-ASan to Scudo standalone. Default parameters are pulled across from the
GWP-ASan build. No backtrace support as of yet.
Reviewers: cryptoad, eugenis, pcc
Reviewed By: cryptoad
Subscribers: merge_guards_bot, mgorny, #sanitizers, llvm-commits, cferris, vlad.tsyrklevich, pcc
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71229
At some point cpplint.py became very noisy during a build spewing
a few hundred lines of "Done processing..." even with SILENT=1 in
cmake. This attempts to redirect the stdout of "Done processing" to
the task log along with any errors.
Tested by this with and without SILENT=1 to check things.
Differential Revision: https://reviews.llvm.org/D71402
Reviewed By: eugenis
Summary:
The motivation for this change is to have a distinguisher in libFuzzer
that would let the runner know whether multistep merge is supported or not by
a particular fuzz target binary. Otherwise, multistep merge fails to execute
with older version of libFuzzer, and there is no way to verify that easily.
Reviewers: kcc
Subscribers: dberris, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71423
Summary:
Add basic support for emscripten.
This enables libFuzzer to build (using build.sh) for emscripten and fuzz
a target compiled with
-fsanitize-coverage=inline-8bit-counters.
Basic fuzzing and bug finding work with this commit.
RSS limit and timeouts will not work because they depend on system
functions that are not implemented/widely supported in emscripten.
Reviewers: kcc, vitalybuka, hctim
Reviewed By: hctim
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71285
Summary:
Counters can be flushed in a multi-threaded context for example when the process is forked in different threads (https://github.com/llvm/llvm-project/blob/master/llvm/lib/Transforms/Instrumentation/GCOVProfiling.cpp#L632-L663).
In order to avoid pretty bad things, a critical section is needed around the flush.
We had a lot of crashes in this code in Firefox CI when we switched to clang for linux ccov builds and those crashes disappeared with this patch.
Reviewers: marco-c, froydnj, dmajor, davidxl, vsk
Reviewed By: marco-c, dmajor
Subscribers: ahatanak, froydnj, dmajor, dberris, jfb, #sanitizers, llvm-commits, sylvestre.ledru
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D70910
This flaky test that I added really gives our CI a lot of headaches.
Although I was never able to reproduce this locally, it sporadically
hangs/fails on our bots. I decided to silently pass the test whenever
we are unable to setup the proper test condition after 10 retries. This
is of course suboptimal and a last recourse. Please let me know if you
know how to test this better.
rdar://57844626
When writing out a profile, avoid allocating a page on the stack for the
purpose of writing out zeroes, as some embedded environments do not have
enough stack space to accomodate this.
Instead, use a small, fixed-size zero buffer that can be written
repeatedly.
For a synthetic file with >100,000 functions, I did not measure a
significant difference in profile write times. We are removing a
page-length zero-fill `memset()` in favor of several smaller buffered
`fwrite()` calls: in practice, I am not sure there is much of a
difference. The performance impact is only expected to affect the
continuous sync mode (%c) -- zero padding is less than 8 bytes in all
other cases.
rdar://57810014
Differential Revision: https://reviews.llvm.org/D71323
MmapOrDie allocates memory multiple to page size. LowLevelAllocator
should use all that memory for the internal buffer because there are
chances that subsequent requests may be small enough to fit in that
space.
Differential Revision: https://reviews.llvm.org/D71275
Instead of testing whether the pointer is aligned, just align it
unconditionally and compare it to the original pointer.
This moves the computation of UserPtr up to before we start preparing the
header, so that the memory tagging code will be able to read the original
header containing the bounds of the previous allocation before it gets
potentially clobbered by the pointer realignment code.
Differential Revision: https://reviews.llvm.org/D71292
With tag-on-free we will need to get the chunk of a deallocated block. Change
getChunkFromBlock() so that it doesn't check that the chunk is allocated,
and move the check into the caller, so that it can be reused for this purpose.
Differential Revision: https://reviews.llvm.org/D71291
Summary:
The function was only defined for x86 and arm families, which ends
up being an issue for PPC in g3.
Define the function, simply returning `false` for "other"
architectures.
Reviewers: hctim, pcc, cferris, eugenis, vitalybuka
Subscribers: kristof.beyls, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71223
Summary:
A script to symbolize hwasan reports after the fact using unstripped
binaries. Supports stack-based reports. Requires llvm-symbolizer
(addr2line is not an option).
Reviewers: pcc, hctim
Subscribers: mgorny, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71148
Summary:
GWP-ASan test currently fail on 32-bit platforms, as some of the pointers are
larger than `uintptr_t` on 32-bit platforms. Fix up all those instances.
Also add an uncompress varint test where the result is an underflow.
Furthermore, allow multi-init for testing. Each gtest when running
`check-gwp_asan` apparently runs in its own instance, but when integrating
these tests into Android, this behaviour isn't the same. We remove the
global multi-init check here, to allow for testing to work elsewhere, and we're
not really worried about multi-init anyway as it's part of our contract with
the allocator.
Reviewers: eugenis, vlad.tsyrklevich
Reviewed By: eugenis
Subscribers: #sanitizers, llvm-commits, pcc
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71121
Summary:
Counters can be flushed in a multi-threaded context for example when the process is forked in different threads (https://github.com/llvm/llvm-project/blob/master/llvm/lib/Transforms/Instrumentation/GCOVProfiling.cpp#L632-L663).
In order to avoid pretty bad things, a critical section is needed around the flush.
We had a lot of crashes in this code in Firefox CI when we switched to clang for linux ccov builds and those crashes disappeared with this patch.
Reviewers: marco-c, froydnj, dmajor, davidxl
Reviewed By: marco-c, dmajor
Subscribers: froydnj, dmajor, dberris, jfb, #sanitizers, llvm-commits, sylvestre.ledru
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D70910
Use a new %run wrapper for ASAN/MSAN/TSAN tests that calls paxctl
in order to disable ASLR on the test executables. This makes it
possible to test sanitizers on systems where ASLR is enabled by default.
Differential Revision: https://reviews.llvm.org/D70958
The test ScudoWrappersCTest.Realloc expects realloc of memalign to work on
Android, but this relies on dealloc_type_mismatch being set to false. Commit
0d3d4d3b0 caused us to start setting it to true in the C wrapper tests,
which broke the test. Set it to the correct value on Android.
Differential Revision: https://reviews.llvm.org/D71078
The Android headers don't provide a declaration of valloc or pvalloc, so we
need to declare them ourselves.
Differential Revision: https://reviews.llvm.org/D71077
Summary:
Some versions of memcpy mark pointer arguments as __nonnull, that triggers UBSan
errors even when the length passed is 0.
Reviewers: manojgupta, metzman
Subscribers: dberris, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71031
[compiler-rt] FDP: assert that num_bytes_to_consume == 0 when size == 0.
Revise the coverage mapping format to reduce binary size by:
1. Naming function records and marking them `linkonce_odr`, and
2. Compressing filenames.
This shrinks the size of llc's coverage segment by 82% (334MB -> 62MB)
and speeds up end-to-end single-threaded report generation by 10%. For
reference the compressed name data in llc is 81MB (__llvm_prf_names).
Rationale for changes to the format:
- With the current format, most coverage function records are discarded.
E.g., more than 97% of the records in llc are *duplicate* placeholders
for functions visible-but-not-used in TUs. Placeholders *are* used to
show under-covered functions, but duplicate placeholders waste space.
- We reached general consensus about giving (1) a try at the 2017 code
coverage BoF [1]. The thinking was that using `linkonce_odr` to merge
duplicates is simpler than alternatives like teaching build systems
about a coverage-aware database/module/etc on the side.
- Revising the format is expensive due to the backwards compatibility
requirement, so we might as well compress filenames while we're at it.
This shrinks the encoded filenames in llc by 86% (12MB -> 1.6MB).
See CoverageMappingFormat.rst for the details on what exactly has
changed.
Fixes PR34533 [2], hopefully.
[1] http://lists.llvm.org/pipermail/llvm-dev/2017-October/118428.html
[2] https://bugs.llvm.org/show_bug.cgi?id=34533
Differential Revision: https://reviews.llvm.org/D69471
Summary:
In order to be compliant with tcmalloc's extension ownership
determination function, we have to expose a function that will
say if a chunk was allocated by us.
As to whether or not this has security consequences: someone
able to call this function repeatedly could use it to determine
secrets (cookie) or craft a valid header. So this should not be
exposed directly to untrusted user input.
Add related tests.
Additionally clang-format caught a few things to change.
Reviewers: hctim, pcc, cferris, eugenis, vitalybuka
Subscribers: JDevlieghere, jfb, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D70908
Summary:
Implicit Conversion Sanitizer is *almost* feature complete.
There aren't *that* much unsanitized things left,
two major ones are increment/decrement (this patch) and bit fields.
As it was discussed in
[[ https://bugs.llvm.org/show_bug.cgi?id=39519 | PR39519 ]],
unlike `CompoundAssignOperator` (which is promoted internally),
or `BinaryOperator` (for which we always have promotion/demotion in AST)
or parts of `UnaryOperator` (we have promotion/demotion but only for
certain operations), for inc/dec, clang omits promotion/demotion
altogether, under as-if rule.
This is technically correct: https://rise4fun.com/Alive/zPgD
As it can be seen in `InstCombineCasts.cpp` `canEvaluateTruncated()`,
`add`/`sub`/`mul`/`and`/`or`/`xor` operators can all arbitrarily
be extended or truncated:
901cd3b3f6/llvm/lib/Transforms/InstCombine/InstCombineCasts.cpp (L1320-L1334)
But that has serious implications:
1. Since we no longer model implicit casts, do we pessimise
their AST representation and everything that uses it?
2. There is no demotion, so lossy demotion sanitizer does not trigger :]
Now, i'm not going to argue about the first problem here,
but the second one **needs** to be addressed. As it was stated
in the report, this is done intentionally, so changing
this in all modes would be considered a penalization/regression.
Which means, the sanitization-less codegen must not be altered.
It was also suggested to not change the sanitized codegen
to the one with demotion, but i quite strongly believe
that will not be the wise choice here:
1. One will need to re-engineer the check that the inc/dec was lossy
in terms of `@llvm.{u,s}{add,sub}.with.overflow` builtins
2. We will still need to compute the result we would lossily demote.
(i.e. the result of wide `add`ition/`sub`traction)
3. I suspect it would need to be done right here, in sanitization.
Which kinda defeats the point of
using `@llvm.{u,s}{add,sub}.with.overflow` builtins:
we'd have two `add`s with basically the same arguments,
one of which is used for check+error-less codepath and other one
for the error reporting. That seems worse than a single wide op+check.
4. OR, we would need to do that in the compiler-rt handler.
Which means we'll need a whole new handler.
But then what about the `CompoundAssignOperator`,
it would also be applicable for it.
So this also doesn't really seem like the right path to me.
5. At least X86 (but likely others) pessimizes all sub-`i32` operations
(due to partial register stalls), so even if we avoid promotion+demotion,
the computations will //likely// be performed in `i32` anyways.
So i'm not really seeing much benefit of
not doing the straight-forward thing.
While looking into this, i have noticed a few more LLVM middle-end
missed canonicalizations, and filed
[[ https://bugs.llvm.org/show_bug.cgi?id=44100 | PR44100 ]],
[[ https://bugs.llvm.org/show_bug.cgi?id=44102 | PR44102 ]].
Those are not specific to inc/dec, we also have them for
`CompoundAssignOperator`, and it can happen for normal arithmetics, too.
But if we take some other path in the patch, it will not be applicable
here, and we will have most likely played ourselves.
TLDR: front-end should emit canonical, easy-to-optimize yet
un-optimized code. It is middle-end's job to make it optimal.
I'm really hoping reviewers agree with my personal assessment
of the path this patch should take..
This originally landed in 9872ea4ed1
but got immediately reverted in cbfa237892
because the assertion was faulty. That fault ended up being caused
by the enum - while there will be promotion, both types are unsigned,
with same width. So we still don't need to sanitize non-signed cases.
So far. Maybe the assert will tell us this isn't so.
Fixes [[ https://bugs.llvm.org/show_bug.cgi?id=44054 | PR44054 ]].
Refs. https://github.com/google/sanitizers/issues/940
Reviewers: rjmccall, erichkeane, rsmith, vsk
Reviewed By: erichkeane
Subscribers: mehdi_amini, dexonsmith, cfe-commits, #sanitizers, llvm-commits, aaron.ballman, t.p.northover, efriedma, regehr
Tags: #llvm, #clang, #sanitizers
Differential Revision: https://reviews.llvm.org/D70539
This test was previously effectively doing:
P = malloc(X); write X bytes to P; P = realloc(P, X - Y); P = realloc(P, X)
and expecting that all X bytes stored to P would still be identical after
the final realloc.
This happens to be true for the current scudo implementation of realloc,
but is not guaranteed to be true by the C standard ("Any bytes in the new
object beyond the size of the old object have indeterminate values.").
This implementation detail will change with the new memory tagging support,
which unconditionally zeros newly allocated granules when memory tagging
is enabled. Fix this by limiting the number of bytes that we test to the
minimum size that we realloc the allocation to.
Differential Revision: https://reviews.llvm.org/D70761
The macros INLINE and COMPILER_CHECK always expand to the same thing (inline
and static_assert respectively). Both expansions are standards compliant C++
and are used consistently in the rest of LLVM, so let's improve consistency
with the rest of LLVM by replacing them with the expansions.
Differential Revision: https://reviews.llvm.org/D70793
Otherwise, we will hit a use-after-free when testing multiple instances of
the same allocator on the same thread. This only recently became a problem
with D70552 which caused us to run both ScudoCombinedTest.BasicCombined and
ScudoCombinedTest.ReleaseToOS on the unit tests' main thread.
Differential Revision: https://reviews.llvm.org/D70760
Shadow memory (and short granules) are not prepended with memory
address and arrow at the end of line is removed.
Differential Revision: https://reviews.llvm.org/D70707
Summary:
This CL makes unit tests compatible with Fuchsia's zxtest. This
required a few changes here and there, but also unearthed some
incompatibilities that had to be addressed.
A header is introduced to allow to account for the zxtest/gtest
differences, some `#if SCUDO_FUCHSIA` are used to disable incompatible
code (the 32-bit primary, or the exclusive TSD).
It also brought to my attention that I was using
`__scudo_default_options` in different tests, which ended up in a
single binary, and I am not sure how that ever worked. So move
this to the main cpp.
Additionally fully disable the secondary freelist on Fuchsia as we do
not track VMOs for secondary allocations, so no release possible.
With some modifications to Scudo's BUILD.gn in Fuchsia:
```
[==========] 79 tests from 23 test cases ran (10280 ms total).
[ PASSED ] 79 tests
```
Reviewers: mcgrathr, phosek, hctim, pcc, eugenis, cferris
Subscribers: srhines, jfb, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D70682
See PR43425:
https://bugs.llvm.org/show_bug.cgi?id=43425
When writing profile data on Windows we were opening profile file with
exclusive read/write access.
In case we are trying to write to the file from multiple processes
simultaneously, subsequent calls to CreateFileA would return
INVALID_HANDLE_VALUE.
To fix this, I changed to open without exclusive access and then take a
lock.
Patch by Michael Holman!
Differential revision: https://reviews.llvm.org/D70330
The asssertion that was added does not hold,
breaks on test-suite/MultiSource/Applications/SPASS/analyze.c
Will reduce the testcase and revisit.
This reverts commit 9872ea4ed1, 870f3542d3.
Summary:
Implicit Conversion Sanitizer is *almost* feature complete.
There aren't *that* much unsanitized things left,
two major ones are increment/decrement (this patch) and bit fields.
As it was discussed in
[[ https://bugs.llvm.org/show_bug.cgi?id=39519 | PR39519 ]],
unlike `CompoundAssignOperator` (which is promoted internally),
or `BinaryOperator` (for which we always have promotion/demotion in AST)
or parts of `UnaryOperator` (we have promotion/demotion but only for
certain operations), for inc/dec, clang omits promotion/demotion
altogether, under as-if rule.
This is technically correct: https://rise4fun.com/Alive/zPgD
As it can be seen in `InstCombineCasts.cpp` `canEvaluateTruncated()`,
`add`/`sub`/`mul`/`and`/`or`/`xor` operators can all arbitrarily
be extended or truncated:
901cd3b3f6/llvm/lib/Transforms/InstCombine/InstCombineCasts.cpp (L1320-L1334)
But that has serious implications:
1. Since we no longer model implicit casts, do we pessimise
their AST representation and everything that uses it?
2. There is no demotion, so lossy demotion sanitizer does not trigger :]
Now, i'm not going to argue about the first problem here,
but the second one **needs** to be addressed. As it was stated
in the report, this is done intentionally, so changing
this in all modes would be considered a penalization/regression.
Which means, the sanitization-less codegen must not be altered.
It was also suggested to not change the sanitized codegen
to the one with demotion, but i quite strongly believe
that will not be the wise choice here:
1. One will need to re-engineer the check that the inc/dec was lossy
in terms of `@llvm.{u,s}{add,sub}.with.overflow` builtins
2. We will still need to compute the result we would lossily demote.
(i.e. the result of wide `add`ition/`sub`traction)
3. I suspect it would need to be done right here, in sanitization.
Which kinda defeats the point of
using `@llvm.{u,s}{add,sub}.with.overflow` builtins:
we'd have two `add`s with basically the same arguments,
one of which is used for check+error-less codepath and other one
for the error reporting. That seems worse than a single wide op+check.
4. OR, we would need to do that in the compiler-rt handler.
Which means we'll need a whole new handler.
But then what about the `CompoundAssignOperator`,
it would also be applicable for it.
So this also doesn't really seem like the right path to me.
5. At least X86 (but likely others) pessimizes all sub-`i32` operations
(due to partial register stalls), so even if we avoid promotion+demotion,
the computations will //likely// be performed in `i32` anyways.
So i'm not really seeing much benefit of
not doing the straight-forward thing.
While looking into this, i have noticed a few more LLVM middle-end
missed canonicalizations, and filed
[[ https://bugs.llvm.org/show_bug.cgi?id=44100 | PR44100 ]],
[[ https://bugs.llvm.org/show_bug.cgi?id=44102 | PR44102 ]].
Those are not specific to inc/dec, we also have them for
`CompoundAssignOperator`, and it can happen for normal arithmetics, too.
But if we take some other path in the patch, it will not be applicable
here, and we will have most likely played ourselves.
TLDR: front-end should emit canonical, easy-to-optimize yet
un-optimized code. It is middle-end's job to make it optimal.
I'm really hoping reviewers agree with my personal assessment
of the path this patch should take..
Fixes [[ https://bugs.llvm.org/show_bug.cgi?id=44054 | PR44054 ]].
Reviewers: rjmccall, erichkeane, rsmith, vsk
Reviewed By: erichkeane
Subscribers: mehdi_amini, dexonsmith, cfe-commits, #sanitizers, llvm-commits, aaron.ballman, t.p.northover, efriedma, regehr
Tags: #llvm, #clang, #sanitizers
Differential Revision: https://reviews.llvm.org/D70539
Summary:
As mentioned in D69104, glibc changed ABI recently with the [[ https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=2f959dfe849e0646e27403f2e4091536496ac0f0| 2f959dfe ]] change.
D69104 dealt with just 32-bit ARM, but that is just one of the many affected architectures.
E.g. x86_64, i?86, riscv64, sparc 32-bit, s390 31-bit are affected too (and various others).
This patch instead of adding a long list of further architectures that wouldn't be checked ever next to arm 32-bit changes the structures to match the 2.31 layout and performs the checking on Linux for ipc_perm mode position/size only on non-Linux or on Linux with glibc 2.31 or later. I think this matches what is done for aarch64 already.
If needed, we could list architectures that haven't changed ABI (e.g. powerpc), so that they would be checked even with older glibcs. AFAIK sanitizers don't actually use ipc_perm.mode and
so all they care about is the size and alignment of the whole structure.
Note, s390 31-bit and arm 32-bit big-endian changed ABI even further, there will now be shmctl with old symbol version and shmctl@@GLIBC_2.31 which will be incompatible. I'm afraid this isn't really solvable unless the sanitizer libraries are symbol versioned and use matching symbol versions to glibc symbols for stuff they intercept, plus use dlvsym.
This patch doesn't try to address that.
Patch by Jakub Jelinek.
Reviewers: kcc, eugenis, dvyukov
Reviewed By: eugenis
Subscribers: jyknight, kristof.beyls, fedor.sergeev, simoncook, PkmX, s.egerton, steven.zhang, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D70662
Implements __fixtfti builtin for PowerPC. This builtin converts a
long double (IBM double-double) to a signed int128. The conversion relies on
the unsigned conversion of the absolute value of the long double.
Tests included for both positive and negative long doubles.
Patch By: Baptiste Saleil
Differential Revision: https://reviews.llvm.org/D69730
Summary:
When platforms use their own `LIKELY()` definitions, it can be quite
troublesome to ensure they don't conflict with the GWP-ASan internal
definitions. Just force the GWP_ASAN_ prefix to help this issue.
Reviewers: eugenis
Reviewed By: eugenis
Subscribers: #sanitizers, llvm-commits, cferris, pcc
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D70683
Summary:
cferris@ found an issue where calling `releaseToOS` prior to any other
heap operation would lead to a crash, due to the allocator not being
properly initialized (it was discovered via `mallopt`).
The fix is to call `initThreadMaybe` prior to calling `releaseToOS` for
the Primary.
Add a test that crashes prior to fix.
Reviewers: hctim, cferris, pcc, eugenis
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D70552
On Darwin, part of the profile runtime is included in the builtin
library. f35032e changed the location of InstrProfData.inc but the
builtin build for Darwin hasn't been updated to include the new
path which causes a breakage when building for Darwin. This change
addresses this breakage.
This header fragment is useful on its own for any consumer that wants
to use custom instruction profile runtime with the LLVM instrumentation.
The concrete use case is in Fuchsia's kernel where we want to use
instruction profile instrumentation, but we cannot use the compiler-rt
runtime because it's not designed for use in the kernel environment.
This change allows installing this header as part of compiler-rt.
Differential Revision: https://reviews.llvm.org/D64532
This header fragment is useful on its own for any consumer that wants
to use custom instruction profile runtime with the LLVM instrumentation.
The concrete use case is in Fuchsia's kernel where we want to use
instruction profile instrumentation, but we cannot use the compiler-rt
runtime because it's not designed for use in the kernel environment.
This change allows installing this header as part of compiler-rt.
Differential Revision: https://reviews.llvm.org/D64532
Summary:
This commit moves the `DiscardOutput` function in FuzzerIO to
FuzzerUtil, so fuchsia can have its own specialized version.
In fuchsia, accessing `/dev/null` is not supported, and there's nothing
similar to a file that discards everything that is written to it. The
way of doing something similar in fuchsia is by using `fdio_null_create`
and binding that to a file descriptor with `fdio_bind_to_fd`.
This change should fix one of the issues with the `-close_fd_mask` flag
in libfuzzer, in which closing stdout was not working due to
`fopen("/dev/null", "w")` returning `NULL`.
Reviewers: kcc, aarongreen
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D69593
Summary:
This commit fixes part of the issues with stack unwinding in fuchsia for
arm64 and x86_64. It consists of multiple fixes:
(1) The cfa_offset calculation was wrong, instead of pointing to the
previous stack pointer, it was pointing to the current one. It worked in
most of the cases because the crashing functions already had a
prologue and had their cfa information relative to another register. The
fix consists on adding a constant that can be used to calculate the
crashing function's stack pointer, and base all the cfi information
relative to that offset.
(2) (arm64) Due to errors with the syntax for the dwarf information, most
of the `OP_NUM` macros were not working. The problem was that they were
referred to as `r##NUM` (like `r14`), when it should have been `x##num`
(like `x14`), or even without the x.
(3) (arm64) The link register was being considered a part of the main
registers (`r30`), when in the real struct it has its own field. Given
that the link register is in the same spot in the struct as r[30] would be,
and that C++ doesn't care about anything, the calculation was still correct.
(4) (x86_64) The stack doesn't need to be aligned to 16 bytes when we
jump to the trampoline function, but it needs to be before performing
call instructions. Encoding that logic in cfi information was tricky, so
we decided to make the cfa information relative to `rbp` and align `rsp`.
Note that this could have been done using another register directly,
but it seems cleaner to make a new fake stack frame.
There are some other minor changes like adding a `brk 1` instruction in
arm64 to make sure that we never return to the crash trampoline (similar to
what we do in x86_64).
Sadly this commit does not fix unwinding for all use cases for arm64.
Crashing functions that do not add information related to the return column in
their cfi information will fail to unwind due to a bug in libunwinder.
Reviewers: mcgrathr, jakehehrlich, phosek, kcc, aarongreen
Subscribers: aprantl, kristof.beyls, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D69579
Summary:
This commit fixes the check in the return value from the `DuplicateFile`
function, which returns a new file descriptor. `DuplicateFile` can
return 0 if that file descriptor is available (for example, if stdin has
already been closed).
In particular, this could cause a bug with the `-close_fd_mask` flag in
some platforms: just call the fuzzer with stdin closed and the
`-close_fd_mask=2` flag, and stderr will not be muted.
Example fuzzer:
```
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* Data, size_t Size) {
fprintf(stderr, "STDERR\n");
fprintf(stdout, "STDOUT\n");
return 0;
}
```
Invocation (muting both stderr and stdout):
```
./test -close_fd_mask=3 -runs=1 0<&-
INFO: Seed: 1155116940
INFO: Loaded 1 modules (1 inline 8-bit counters): 1 [0x48b020, 0x48b021),
INFO: Loaded 1 PC tables (1 PCs): 1 [0x478dc8,0x478dd8),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
STDERR
INFO: A corpus is not provided, starting from an empty corpus
STDERR
Done 2 runs in 0 second(s)
```
Reviewers: mcgrathr, jakehehrlich, phosek, kcc, aarongreen
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D68775
Summary:
A few small improvements and optimizations:
- when refilling the free list, push back the last batch and return
the front one: this allows to keep the allocations towards the front
of the region;
- instead of using 48 entries in the shuffle array, use a multiple of
`MaxNumCached`;
- make the maximum number of batches to create on refil a constant;
ultimately it should be configurable, but that's for later;
- `initCache` doesn't need to zero out the cache, it's already done.
- it turns out that when using `||` or `&&`, the compiler is adamant
on adding a short circuit for every part of the expression. Which
ends up making somewhat annoying asm with lots of test and
conditional jump. I am changing that to bitwise `|` or `&` in two
place so that the generated code looks better. Added comments since
it might feel weird to people.
This yields to some small performance gains overall, nothing drastic
though.
Reviewers: hctim, morehouse, cferris, eugenis
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D70452
Darwin lazily saves the AVX512 context on first use [1]: instead of checking
that it already does to figure out if the OS supports AVX512, trust that
the kernel will do the right thing and always assume the context save
support is available.
[1] https://github.com/apple/darwin-xnu/blob/xnu-4903.221.2/osfmk/i386/fpu.c#L174
Reviewers: ab, RKSimon, craig.topper
Reviewed By: craig.topper
Subscribers: dberris, JDevlieghere, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D70454
When we're not building libc's allocator, just use a regular TLS variable. This
lets the unit tests pass on Android devices whose libc uses Scudo. Otherwise
libc's copy of Scudo and the unit tests' copy will both try to use the same
TLS slot, in likely incompatible ways.
This requires using ELF TLS, so start passing -fno-emulated-tls when building
the library and the unit tests on Android.
Differential Revision: https://reviews.llvm.org/D70472
Continuous mode is not yet supported on Fuchsia/Windows, however an
error should not be reported unless the user attempted to actually
enable continuous mode.
Summary:
The sanitizer symbolizers support printing the function offset
(difference between pc and function start) of a stackframe using the
`%q` format specifier.
Unfortunately this didn't actually work because neither the atos
or dladdr symbolizer set the `AddressInfo::function_offset` field.
This patch teaches both symbolizers to try to compute the function
offset. In the case of the atos symbolizer, atos might not report the
function offset (e.g. it reports a source location instead) so in this
case it fallsback to using `dladdr()` to compute the function offset.
Two test cases are included.
rdar://problem/56695185
Reviewers: kubamracek, yln
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D69549
Make it possible to use online profile merging ("%m" mode) with
continuous sync ("%c" mode).
To implement this, the merged profile is locked in the runtime
initialization step and either a) filled out for the first time or b)
checked for compatibility. Then, the profile can simply be mmap()'d with
MAP_SHARED set. With the mmap() in place, counter updates from every
process which uses an image are mapped onto the same set of physical
pages assigned by the filesystem cache. After the mmap() is set up, the
profile is unlocked.
Differential Revision: https://reviews.llvm.org/D69586
Bump default value for `-miphoneos-version-min=x` to 9.0 (from 8.0).
TSan tests that use thread local storage `__thread` (e.g., tls_race.cpp)
are failing to link for the iOS simulator (arch=x86_64 and
-miphoneos-version-min=8.0) on internal/master (will be submitted to
Glacier train):
```
ld: targeted OS version does not support use of thread local variables in _main for architecture x86_64
```
iOS 9.0 was released 2015.
Bump default value for `SANITIZER_MIN_OSX_VERSION` to 10.10 (from 10.9).
TSan does not work on macOS 10.9 and a nice error message is preferable
to an "unreferenced symbol" error when loading the TSan runtime.
We could try to only bump the deployment target for TSan, but we would
have to invest into adding support for this to our CMake build and it
does not seem worth it. macOS 10.10 was released in 2014.
rdar://31335781
coverage-fork.cpp uses `fork()` which requires additional permissions
in the iOS simulator sandbox. We cannot use `sandbox-exec` to grant
these permissions since this is a Posix (not Darwin) test.
This is a patch to support D66328, which was reverted until this lands.
Enable a compiler-rt test that used to fail previously with D66328.
Differential Revision: https://reviews.llvm.org/D67283
Summary:
`SCUDO_DEBUG` was not enabled for unit tests, meaning the `DCHECK`s
were never tripped. While turning this on, I discovered that a few
of those not-exercised checks were actually wrong. This CL addresses
those incorrect checks.
Not that to work in tests `CHECK_IMPL` has to explicitely use the
`scudo` namespace. Also changes a C cast to a C++ cast.
Reviewers: hctim, pcc, cferris, eugenis, vitalybuka
Subscribers: mgorny, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D70276
Summary:
Previously it wasn't obvious what the default value of various sanitizer
options were. A very close approximation of the "default values" for the
options are the current value of the options at the time of printing the
help output.
In the case that no other options are provided then the current values
are the default values (apart from `help`).
```
ASAN_OPTIONS=help=1 ./program
```
This patch causes the current option values to be printed when the
`help` output is enabled. The original intention for this patch was to append
`(Default: <value>)` to an option's help text. However because this
is technically wrong (and misleading) I've opted to append
`(Current Value: <value>)` instead.
When trying to implement a way of displaying the default value of the
options I tried another solution where the default value used in `*.inc` files
were used to create compile time strings that where used when printing
the help output. This solution was not satisfactory for several reasons:
* Stringifying the default values with the preprocessor did not work very
well in several cases. Some options contain boolean operators which no
amount of macro expansion can get rid of.
* It was much more invasive than this patch. Every sanitizer had to be changed.
* The settings of `__<sanitizer>_default_options()` are ignored.
For those reasons I opted for the solution in this patch.
rdar://problem/42567204
Reviewers: kubamracek, yln, kcc, dvyukov, vitalybuka, cryptoad, eugenis, samsonov
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D69546
Split out the logic to get the size of a merged profile and to do a
compatibility check. This can be shared with both the continuous+merging
mode implementation, as well as the runtime-allocated counters
implementation planned for Fuchsia.
Lifted out of D69586.
Differential Revision: https://reviews.llvm.org/D70135
A few years back a similar change was made for getenv since neither function is supported on the PS4 platform.
Recently, commit d889d1e added a call to setenv in compiler-rt which was causing linking errors because the symbol was not found. This fixes that issue by putting in a shim similar to how we previously dealt with the lack of getenv.
Differential Revision: https://reviews.llvm.org/D70033
Summary:
This patch fixes two problems with the crtbegin.c as written:
1. In do_init, register_frame_info is not guarded by a #define, but in
do_fini, deregister_frame_info is guarded by #ifndef
CRT_HAS_INITFINI_ARRAY. Thus when CRT_HAS_INITFINI_ARRAY is not
defined, frames are registered but then never deregistered.
The frame registry mechanism builds a linked-list from the .so's
static variable do_init.object, and when the .so is unloaded, this
memory becomes invalid and should be deregistered.
Further, libgcc's crtbegin treats the frame registry as independent
from the initfini array mechanism.
This patch fixes this by adding a new #define,
"EH_USE_FRAME_INFO_REGISTRY", which is set by the cmake option
COMPILER_RT_CRT_USE_EH_FRAME_REGISTRY Currently, do_init calls
register_frame_info, and then calls the binary's constructors. This
allows constructors to safely use libunwind. However, do_fini calls
deregister_frame_info and then calls the binary's destructors. This
prevents destructors from safely using libunwind.
This patch also switches that ordering, so that destructors can safely
use libunwind. As it happens, this is a fairly common scenario for
thread sanitizer.
Summary: Bionic was modified to have all function names consistent. Modify the code and get rid of the special case for bionic since it's no longer needed.
Reviewers: cryptoad
Reviewed By: cryptoad
Subscribers: srhines, llvm-commits, #sanitizers
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D70031
__fixunstfti converts a long double (IBM double-double) to an unsigned 128 bit
integer. This patch enables it to handle a previously unhandled case in which
a negative low double may impact the result of the conversion.
Collaborated with @masoud.ataei and @renenkel.
Patch By: Baptiste Saleil
Differential Revision: https://reviews.llvm.org/D69193
Fallout from:
[clang] Report sanitizer blacklist as a dependency in cc1
Default blacklists are now passed via -fsanitize-system-blacklist from driver to cc1.
https://bugs.llvm.org/show_bug.cgi?id=43722
GWP-ASan didn't include SANITIZER_COMMON_CFLAGS, and thus would produce
LLVM bitcode files, when compiler-rt is generally built without LTO.
This allows crtbegin and crtend to be built, allowing RISC-V
to no longer rely on implementations from libgcc.
Differential revision: https://reviews.llvm.org/D68393
D69405 causes failure if running LIT when the compiler was built without lld.
Patch by Anh Tuyen Tran (anhtuyen)
Differential Revision: https://reviews.llvm.org/D69685
This was an experiment made possible by a non-standard feature of the
Android dynamic loader.
It required introducing a flag to tell the compiler which ABI was being
targeted.
This flag is no longer needed, since the generated code now works for
both ABI's.
We leave that flag untouched for backwards compatibility. This also
means that if we need to distinguish between targeted ABI's again
we can do that without disturbing any existing workflows.
We leave a comment in the source code and mention in the help text to
explain this for any confused person reading the code in the future.
Patch by Matthew Malcomson
Differential Revision: https://reviews.llvm.org/D69574
This commit added use of a Windows API in InstrProfilingPort.h.
When _MSC_VER is defined (for MSVC), windows.h is already included
earlier in the same header (for atomics), but MinGW, the gcc
atomics builtins are used instead. Therefore explicitly include
windows.h here, where the API is used.
Use the printf macros from inttypes.h to sidestep -Wformat issues:
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:425:14: error: format specifies type 'long long' but the argument has type 'off_t' (aka 'long') [-Werror,-Wformat]
CurrentFileOffset, PageSize);
^~~~~~~~~~~~~~~~~
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/profile/InstrProfilingPort.h:114:50: note: expanded from macro 'PROF_ERR'
fprintf(stderr, "LLVM Profile Error: " Format, __VA_ARGS__);
~~~~~~ ^~~~~~~~~~~
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:461:41: error: format specifies type 'unsigned long long' but the argument has type 'uint64_t' (aka 'unsigned long') [-Werror,-Wformat]
strerror(errno), CountersBegin, PageAlignedCountersLength, Fileno,
^~~~~~~~~~~~~~~~~~~~~~~~~
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/profile/InstrProfilingPort.h:114:50: note: expanded from macro 'PROF_ERR'
fprintf(stderr, "LLVM Profile Error: " Format, __VA_ARGS__);
~~~~~~ ^~~~~~~~~~~
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/profile/InstrProfilingFile.c:462:9: error: format specifies type 'unsigned long long' but the argument has type 'uint64_t' (aka 'unsigned long') [-Werror,-Wformat]
FileOffsetToCounters);
^~~~~~~~~~~~~~~~~~~~
/var/lib/buildbot/sanitizer-buildbot6/sanitizer-x86_64-linux-android/build/llvm-project/compiler-rt/lib/profile/InstrProfilingPort.h:114:50: note: expanded from macro 'PROF_ERR'
fprintf(stderr, "LLVM Profile Error: " Format, __VA_ARGS__);
VLAs in C appear to not work on Windows, so use COMPILER_RT_ALLOCA:
C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\lib\profile\InstrProfilingWriter.c(264): error C2057: expected constant expression
C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\lib\profile\InstrProfilingWriter.c(264): error C2466: cannot allocate an array of constant size 0
C:\b\slave\sanitizer-windows\llvm-project\compiler-rt\lib\profile\InstrProfilingWriter.c(264): error C2133: 'Zeroes': unknown size
Add support for continuously syncing profile counter updates to a file.
The motivation for this is that programs do not always exit cleanly. On
iOS, for example, programs are usually killed via a signal from the OS.
Running atexit() handlers after catching a signal is unreliable, so some
method for progressively writing out profile data is necessary.
The approach taken here is to mmap() the `__llvm_prf_cnts` section onto
a raw profile. To do this, the linker must page-align the counter and
data sections, and the runtime must ensure that counters are mapped to a
page-aligned offset within a raw profile.
Continuous mode is (for the moment) incompatible with the online merging
mode. This limitation is lifted in https://reviews.llvm.org/D69586.
Continuous mode is also (for the moment) incompatible with value
profiling, as I'm not sure whether there is interest in this and the
implementation may be tricky.
As I have not been able to test extensively on non-Darwin platforms,
only Darwin support is included for the moment. However, continuous mode
may "just work" without modification on Linux and some UNIX-likes. AIUI
the default value for the GNU linker's `--section-alignment` flag is set
to the page size on many systems. This appears to be true for LLD as
well, as its `no_nmagic` option is on by default. Continuous mode will
not "just work" on Fuchsia or Windows, as it's not possible to mmap() a
section on these platforms. There is a proposal to add a layer of
indirection to the profile instrumentation to support these platforms.
rdar://54210980
Differential Revision: https://reviews.llvm.org/D68351
Summary:
cferris@ found an issue due to the new Secondary free list behavior
and unfortunately it's completely my fault. The issue is twofold:
- I lost track of the (major) fact that the Combined assumes that
all chunks returned by the Secondary are zero'd out apprioriately
when dealing with `ZeroContents`. With the introduction of the
freelist, it's no longer the case as there can be a small portion
of memory between the header and the next page boundary that is
left untouched (the rest is zero'd via release). So the next time
that block is returned, it's not fully zero'd out.
- There was no test that would exercise that behavior :(
There are several ways to fix this, the one I chose makes the most
sense to me: we pass `ZeroContents` to the Secondary's `allocate`
and it zero's out the block if requested and it's coming from the
freelist. The prevents an extraneous `memset` in case the block
comes from `map`. Another possbility could have been to `memset`
in `deallocate`, but it's probably overzealous as all secondary
blocks don't need to be zero'd out.
Add a test that would have found the issue prior to fix.
Reviewers: morehouse, hctim, cferris, pcc, eugenis, vitalybuka
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D69675
Summary:
xcodebuild does not work unless XCode is installed whereas xcrun also work
when only the Command Line Tools are installed. Unlike the check for the
version (D69610), this did not cause an erro for me since the fallback to
/usr/include for the OSX sysroot worked.
Reviewers: yln, delcypher
Reviewed By: yln
Subscribers: dberris, mgorny, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D69659
Summary:
Starting with 8a5bfbe6db (D68292) this file
unconditionally uses xcodebuild to get the SDK version. On my system this
always fails with
`xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory '/Library/Developer/CommandLineTools' is a command line tools instance`
Reviewers: delcypher, yln
Reviewed By: delcypher, yln
Subscribers: dberris, mgorny, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D69610
Summary:
Sometimes an allocation stack trace is not very informative. Provide a
way to replace it with a stack trace of the user's choice.
Reviewers: pcc, kcc
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D69208
Summary:
Previously the CMake code looked for filepaths of the form
`<arch>/<filename>` as an indication that `<arch>/<filename>` provided a
specialization of a top-level file `<filename>`. For powerpc there was a
bug because the powerpc specialized implementations lived in `ppc/` but
the architectures were `powerpc64` and `powerpc64le` which meant that
CMake was looking for files at `powerpc64/<filename>` and
`powerpc64le/<filename>`.
The result of this is that for powerpc the builtins library contained a
duplicate symbol for `divtc3` because it had the generic implementation
and the specialized version in the built static library.
Although we could just add similar code to what there is for arm (i.e.
compute `${_arch}`) to fix this, this is extremely error prone (until
r375150 no error was raised). Instead this patch takes a different
approach that removes looking for the architecture name entirely.
Instead this patch uses the convention that a source file in a
sub-directory might be a specialization of a generic implementation and
if a source file of the same name (ignoring extension) exists at the
top-level then it is the corresponding generic implementation. This
approach is much simpler because it doesn't require keeping track of
different architecture names.
This convention already existed in repository but previously it was
implicit. This change makes it explicit.
This patch is motivated by wanting to revert r375162 which worked around
the powerpc bug found when r375150 landed.
Once it lands we should revert r375162.
Reviewers: phosek, beanz, compnerd, shiva0217, amyk, rupprecht, kongyi, mstorsjo, t.p.northover, weimingz, jroelofs, joerg, sidneym
Subscribers: nemanjai, mgorny, kristof.beyls, jsji, shchenz, steven.zhang, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D69189
Summary:
The flag allows the user to specify a maximum allocation size that the
sanitizers will honor. Any larger allocations will return nullptr or
crash depending on allocator_may_return_null.
Reviewers: kcc, eugenis
Reviewed By: kcc, eugenis
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D69576
Summary:
The secondary allocator is slow, because we map and unmap each block
on allocation and deallocation.
While I really like the security benefits of such a behavior, this
yields very disappointing performance numbers on Android for larger
allocation benchmarks.
So this change adds a free list to the secondary, that will hold
recently deallocated chunks, and (currently) release the extraneous
memory. This allows to save on some memory mapping operations on
allocation and deallocation. I do not think that this lowers the
security of the secondary, but can increase the memory footprint a
little bit (RSS & VA).
The maximum number of blocks the free list can hold is templatable,
`0U` meaning that we fallback to the old behavior. The higher that
number, the higher the extra memory footprint.
I added default configurations for all our platforms, but they are
likely to change in the near future based on needs and feedback.
Reviewers: hctim, morehouse, cferris, pcc, eugenis, vitalybuka
Subscribers: mgorny, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D69570
Summary:
The hwasan interceptor ABI doesn't have interceptors for longjmp and setjmp.
This patch introduces them.
We require the size of the jmp_buf on the platform to be at least as large as
the jmp_buf in our implementation. To enforce this we compile
hwasan_type_test.cpp that ensures a compile time failure if this is not true.
Tested on both GCC and clang using an AArch64 virtual machine.
Reviewers: eugenis, kcc, pcc, Sanatizers
Reviewed By: eugenis, Sanatizers
Tags: #sanatizers, #llvm
Differential Revision: https://reviews.llvm.org/D69045
Patch By: Matthew Malcomson <matthew.malcomson@arm.com>
Summary:
When the ABI namespace isn't a reserved identifier, we were issuing a
warning, but this should have been an error since the beginning. This
commit enforces that the ABI namespace is a reserved identifier, and
changes the ABI namespace used by LibFuzzer.
Reviewers: phosek, EricWF
Subscribers: mgorny, christof, jkorous, dexonsmith, #sanitizers, libcxx-commits, llvm-commits
Tags: #sanitizers, #libc, #llvm
Differential Revision: https://reviews.llvm.org/D69408
Summary:
Apparently during the review of D69265, and my flailing around with
git, a somewhat important line disappeared.
On top of that, there was no test exercising that code path, and
while writing the follow up patch I intended to write, some `CHECK`s
were failing.
Re-add the missing line, and add a test that fails without said line.
Reviewers: hctim, morehouse, pcc, cferris
Reviewed By: hctim
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D69529
Do not add an lld dependency when this target does not exist. In this
case the system installation of lld is used (or whatever is detected
with -fuse-ld=lld by default).
Summary:
This is a clean patch using the last diff of D69265, but using git
instead of svn, since svn went ro and arc was making my life harded
than it needed to be.
I was going to introduce a couple more lists and realized that our
lists are currently a bit all over the place. While we have a singly
linked list type relatively well defined, we are using doubly linked
lists defined on the fly for the stats and for the secondary blocks.
This CL adds a doubly linked list object, reorganizing the singly list
one to extract as much of the common code as possible. We use this
new type in the stats and the secondary. We also reorganize the list
tests to benefit from this consolidation.
There are a few side effect changes such as using for iterator loops
that are, in my opinion, cleaner in a couple of places.
Reviewers: hctim, morehouse, pcc, cferris
Reviewed By: hctim
Subscribers: jfb, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D69516
Use new control bits CTR_EL0.DIC and CTR_EL0.IDC to discover the d-cache
cleaning and i-cache invalidation requirements for instruction-to-data
coherence. This matches the behavior in the latest libgcc.
Author: Shaokun Zhang <zhangshaokun@hisilicon.com>
Reviewed By: peter.smith
Differential Revision: https://reviews.llvm.org/D69247
Melanie Blower