Commit Graph

12582 Commits

Author SHA1 Message Date
Max Moroz 076475713c [libFuzzer] Fix denominator in the "inputs have DFT" log line.
Summary:
The number of "inputs have the Data Flow Trace" cannot be greater than
the number of inputs touching the focus function. The existing message is rather
confusing as the same log would mention a greater total number of traces a few
lines above.

Reviewers: kcc, metzman

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D74779
2020-02-19 11:05:14 -08:00
Jordan Rupprecht 3f7d0e7e31 [NFC][compiler-rt][test] Fully qualify string -> std::string 2020-02-19 08:59:26 -08:00
Daniel Fava 9b91bcf6c6 tsan: removing redundant loop in ThreadClock::release()
The removed loop clears reused for entries at the tail of a SyncClock.
The loop is redundant since those entries were already cleared by the
immediately preceding loop, which iterates over all entries in the
SyncClock (including the tail entries).
2020-02-19 17:30:00 +01:00
Kostya Kortchinsky fc69967a4b [scudo][standalone] Shift some data from dynamic to static
Summary:
Most of our larger data is dynamically allocated (via `map`) but it
became an hindrance with regard to init time, for a cost to benefit
ratio that is not great. So change the `TSD`s, `RegionInfo`, `ByteMap`
to be static.

Additionally, for reclaiming, we used mapped & unmapped a buffer each
time, which is costly. It turns out that we can have a static buffer,
and that there isn't much contention on it.

One of the other things changed here, is that we hard set the number
of TSDs on Android to the maximum number, as there could be a
situation where cores are put to sleep and we could miss some.

Subscribers: mgorny, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D74696
2020-02-18 09:38:50 -08:00
Dan Liew f4141367d0 [compiler-rt] Make various Apple lit substitutions work correctly for other Apple platforms.
This change makes the following lit substitutions expand to the correct
thing for macOS, iOS, tvOS, and watchOS.

%darwin_min_target_with_full_runtime_arc_support
%macos_min_target_10_11

rdar://problem/59463146
2020-02-14 13:53:54 -08:00
Dan Liew a7018e8a2e [compiler-rt] Disable building LibFuzzer for WatchOS and the corresponding simulator.
rdar://problem/59466685
2020-02-14 13:53:54 -08:00
Dan Liew f5951bdc3a [compiler-rt] Provide a lit config variable that defines the minimum deployment OS version flag appropriate for the configured OS.
This patch defines `config.apple_platform_min_deployment_target_flag`
in the ASan, LibFuzzer, TSan, and UBSan lit test configs.

rdar://problem/59463146
2020-02-14 13:53:54 -08:00
Dan Liew 8337fb00e6 [compiler-rt] Add `DARWIN_osx_MIN_VER_FLAG` variable to match the other Apple platforms.
rdar://problem/59463146
2020-02-14 13:53:53 -08:00
Dan Liew 86dceb9bcd [compiler-rt] Use the correct minimum version flag for simulators on Apple
platforms.

For the simulators this changes the following compilation flags (
used both for building the runtime and lit tests).

iOS simulator: `-miphoneos-version-min` -> `-mios-simulator-version-min`
watchOS simulator: `-mwatchos-version-min` -> `-mwatchos-simulator-version-min`
tvOS simulator: `-mtvos-version-min` -> `-mtvos-simulator-version-min`

rdar://problem/59463146
2020-02-14 13:53:53 -08:00
Christopher Ferris 5f91c7b980 [scudo][standalone] Allow setting release to OS
Summary:
Add a method to set the release to OS value as the system runs,
and allow this to be set differently in the primary and the secondary.
Also, add a default value to use for primary and secondary. This
allows Android to have a default that is different for
primary/secondary.

Update mallopt to support setting the release to OS value.

Reviewers: pcc, cryptoad

Reviewed By: cryptoad

Subscribers: cryptoad, jfb, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D74448
2020-02-14 12:57:34 -08:00
Kostya Kortchinsky ef7488ef20 [GWP-ASan] Silence gcc error
Summary: It complains about reaching the end of a non-void returning function.

Reviewers: eugenis, hctim, morehouse

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D74578
2020-02-13 13:08:37 -08:00
Kostya Kortchinsky 21695710cf [scudo][standalone] Workaround for full regions on Android
Summary:
Due to Unity, we had to reduce our region sizes, but in some rare
situations, some programs (mostly tests AFAICT) manage to fill up
a region for a given size class.

So this adds a workaround for that attempts to allocate the block
from the immediately larger size class, wasting some memory but
allowing the application to keep going.

Reviewers: pcc, eugenis, cferris, hctim, morehouse

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D74567
2020-02-13 12:54:25 -08:00
Melanie Blower a0d913a1ac Fix regression due to reviews.llvm.org/D74436 by adding option ffp-contract=off to RUN line 2020-02-12 19:05:18 -08:00
Mitch Phillips 91e194d1ff [GWP-ASan] [NFC] Change enum from ANDROID->BIONIC. 2020-02-12 16:06:59 -08:00
Yuanfang Chen 4caeb62e51 [Fuzzer] Rename ExecuteCommandWithPopen to ExecuteCommandNon-Fushsia target will keep using popen/pclose implementation. OnFuchsia, Two-args version of `ExecuteCommand` is a simple wrapper of theone-arg version. (Hopefully) Fix D73329 build on Fuchsia. 2020-02-12 16:03:55 -08:00
Mitch Phillips 5f2a74c87a [GWP-ASan] Update alignment on Android.
Summary:
Android has different alignment requirements. You can read more about
them here
(https://cs.android.com/android/platform/superproject/+/master:bionic/tests/malloc_test.cpp;l=808),
but the general gist is that for malloc(x <= 8), we do malloc(8), and
for everything else, we do 16-byte alignment.

Reviewers: eugenis, morehouse, cferris

Reviewed By: eugenis, morehouse

Subscribers: #sanitizers, llvm-commits, pcc

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D74364
2020-02-12 15:24:58 -08:00
Anh Tuyen Tran dadc214e4d Title: [TSAN] Parameterize the hard-coded threshold of deflake in tsan test
Summary:
A number of testcases in TSAN are designed to deal with intermittent problems
not exist in all executions of the tested program. A script called deflake.bash
runs the executable up to 10 times to deal with the intermittent nature of the tests.

The purpose of this patch is to parameterize the hard-coded threshold above via
--cmake_variables=-DTSAN_TEST_DEFLAKE_THRESHOLD=SomeIntegerValue
When this cmake var is not set, the default value of 10 will be used.

Reviewer: dvyukov (Dmitry Vyukov), eugenis (Evgenii Stepanov), rnk (Reid Kleckner), hubert.reinterpretcast (Hubert Tong), vitalybuka (Vitaly Buka)

Reviewed By: vitalybuka (Vitaly Buka)

Subscribers: mgorny (Michal Gorny), jfb (JF Bastien), steven.zhang (qshanz), llvm-commits (Mailing List llvm-commits), Sanitizers

Tag: LLVM, Sanitizers

Differential Revision: https://reviews.llvm.org/D73707
2020-02-12 15:51:57 +00:00
Sam Clegg 2f172d8d3c [compiler-rt] Compile __powitf2 under wasm
See https://github.com/emscripten-core/emscripten/issues/10374
See https://reviews.llvm.org/D74274

Differential Revision: https://reviews.llvm.org/D74275
2020-02-11 17:35:07 -08:00
Max Moroz 20a604d3f5 [compiler-rt] FuzzedDataProvider: add ConsumeData and method.
Reviewers: metzman

Subscribers: dberris, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D74359
2020-02-11 13:46:24 -08:00
Peter Collingbourne 87303fd917 scudo: Fix various test failures, mostly on 32-bit.
Differential Revision: https://reviews.llvm.org/D74429
2020-02-11 12:18:35 -08:00
Peter Collingbourne 515e19ae7b Fix errors/warnings in scudo build. 2020-02-11 08:37:37 -08:00
Kamil Rytarowski b664321ff1 [compiler-rt] Restrict sanitizer_linux.cpp dlinfo(3) to FreeBSD
FreeBSD is currently the only user in this file.
Accessing this symbol on Linux does not work as is.
2020-02-11 12:10:03 +01:00
Evgenii Stepanov f69c83645b [asan] Disable qsort interceptor on Android.
The interceptor uses thread-local variables, which (until very recently)
are emu-tls. An access to such variable may call malloc which can
deadlock the runtime library.
2020-02-10 18:26:18 -08:00
Yuanfang Chen 4f3c3bbbf8 Reland "[NFC][libFuzzer] Prefix TempPath with string showing the work it is doing."
With fix (somehow one hunk is missed).
2020-02-10 18:23:39 -08:00
Yuanfang Chen b1c7623982 Revert "[NFC][libFuzzer] Prefix TempPath with string showing the work it is doing."
This reverts commit 8a29cb4421.

fuzzer-linux bot has failure because of this.
2020-02-10 18:06:17 -08:00
Peter Collingbourne e79c3b4c2d scudo: Fix Android build.
Differential Revision: https://reviews.llvm.org/D74366
2020-02-10 17:01:10 -08:00
Yuanfang Chen 8a29cb4421 [NFC][libFuzzer] Prefix TempPath with string showing the work it is doing. 2020-02-10 16:45:10 -08:00
Yuanfang Chen 85515c7fd5 [libFuzzer] communicate through pipe to subprocess for MinimizeCrashInput
For CleanseCrashInput, discards stdout output anyway since it is not used.

These changes are to defend against aggressive PID recycle on windows to reduce the chance of contention on files.

Using pipe instead of file also workaround the problem that when the
process is spawned by llvm-lit, the aborted process keeps a handle to the
output file such that the output file can not be removed. This will
cause random test failures.

https://devblogs.microsoft.com/oldnewthing/20110107-00/?p=11803

Reviewers: kcc, vitalybuka

Reviewed By: vitalybuka

Differential Revision: https://reviews.llvm.org/D73329
2020-02-10 16:45:10 -08:00
Peter Collingbourne 9068766b9a scudo: Add a dump of primary allocation sizes to malloc_info output.
This will be useful for optimizing the size class map.

Differential Revision: https://reviews.llvm.org/D74098
2020-02-10 15:05:27 -08:00
Peter Collingbourne 041547eb4e scudo: Table driven size classes for Android allocator.
Add an optional table lookup after the existing logarithm computation
for MidSize < Size <= MaxSize during size -> class lookups. The lookup is
O(1) due to indexing a precomputed (via constexpr) table based on a size
table. Switch to this approach for the Android size class maps.

Other approaches considered:
- Binary search was found to have an unacceptable (~30%) performance cost.
- An approach using NEON instructions (see older version of D73824) was found
  to be slightly slower than this approach on newer SoCs but significantly
  slower on older ones.

By selecting the values in the size tables to minimize wastage (for example,
by passing the malloc_info output of a target program to the included
compute_size_class_config program), we can increase the density of allocations
at a small (~0.5% on bionic malloc_sql_trace as measured using an identity
table) performance cost.

Reduces RSS on specific Android processes as follows (KB):

                             Before  After
zygote (median of 50 runs)    26836  26792 (-0.2%)
zygote64 (median of 50 runs)  30384  30076 (-1.0%)
dex2oat (median of 3 runs)   375792 372952 (-0.8%)

I also measured the amount of whole-system idle dirty heap on Android by
rebooting the system and then running the following script repeatedly until
the results were stable:

for i in $(seq 1 50); do grep -A5 scudo: /proc/*/smaps | grep Pss: | cut -d: -f2 | awk '{s+=$1} END {print s}' ; sleep 1; done

I did this 3 times both before and after this change and the results were:

Before: 365650, 356795, 372663
After:  344521, 356328, 342589

These results are noisy so it is hard to make a definite conclusion, but
there does appear to be a significant effect.

On other platforms, increase the sizes of all size classes by a fixed offset
equal to the size of the allocation header. This has also been found to improve
density, since it is likely for allocation sizes to be a power of 2, which
would otherwise waste space by pushing the allocation into the next size class.

Differential Revision: https://reviews.llvm.org/D73824
2020-02-10 14:59:49 -08:00
Peter Collingbourne 681773f291 scudo: Instead of exporting a pointer to the allocator, export the allocator directly. NFCI.
This lets us remove two pointer indirections (one by removing the pointer,
and another by making the AllocatorPtr declaration hidden) in the C++ wrappers.

Differential Revision: https://reviews.llvm.org/D74356
2020-02-10 14:57:54 -08:00
Dimitry Andric 52f2df1ecd [Sanitizers] Get link map on FreeBSD and NetBSD via documented API
Summary:
Instead of hand-crafting an offset into the structure returned by
dlopen(3) to get at the link map, use the documented API.  This is
described in dlinfo(3): by calling it with `RTLD_DI_LINKMAP`, the
dynamic linker ensures the right address is returned.

This is a recommit of 92e267a94d, with
dlinfo(3) expliclity being referenced only for FreeBSD, non-Android
Linux, NetBSD and Solaris.  Other OSes will have to add their own
implementation.

Reviewers: devnexen, emaste, MaskRay, krytarowski

Reviewed By: krytarowski

Subscribers: krytarowski, vitalybuka, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73990
2020-02-10 23:43:20 +01:00
Dimitry Andric 480eea4e45 Revert "[Sanitizers] Get link map on FreeBSD via documented API"
This reverts commit 92e267a94d, as it
appears Android is missing dlinfo(3).
2020-02-10 23:26:33 +01:00
Max Moroz 405093045f [compiler-rt] Follow up fix for the refactoring in https://reviews.llvm.org/D74137.
Summary:
The refactoring has caused a failure in
http://lab.llvm.org:8011/builders/clang-ppc64le-linux/builds/29265

The idea of failing the symbolization when the symbolizer bufer is too small
was incorrect. The symbolizer can be invoked for other frames that may fit into
the buffer and get symbolized.

Reviewers: vitalybuka, eugenis

Subscribers: dberris, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D74343
2020-02-10 10:28:06 -08:00
Dimitry Andric 92e267a94d [Sanitizers] Get link map on FreeBSD via documented API
Summary:
Instead of hand-crafting an offset into the structure returned by
dlopen(3) to get at the link map, use the documented API.  This is
described in dlinfo(3): by calling it with `RTLD_DI_LINKMAP`, the
dynamic linker ensures the right address is returned.

Reviewers: devnexen, emaste, MaskRay, krytarowski

Reviewed By: krytarowski

Subscribers: krytarowski, vitalybuka, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73990
2020-02-10 19:21:19 +01:00
Max Moroz 5ad62d3b7f [compiler-rt] Some clean up / refactoring in sanitizer_symbolizer_libcdep.cpp.
Summary:
Nothing critical, just a few potential improvements I've noticed while reading
the code:
- return `false` when symbolizer buffer is too small to read all data
- invert some conditions to reduce indentation
- prefer `nullptr` over `0` for pointers; init some pointers on stack;
- remove minor code duplication

Reviewers: eugenis, vitalybuka

Subscribers: dberris, #sanitizers, llvm-commits, kcc

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D74137
2020-02-10 06:50:59 -08:00
Kostya Kortchinsky fe6e77f6fb [scudo][standalone] 32-bit improvement
Summary:
This tweaks some behaviors of the allocator wrt 32-bit, notably
tailoring the size-class map.

I had to remove a `printStats` from `__scudo_print_stats` since when
within Bionic they share the same slot so they can't coexist at the
same time. I have to find a solution for that later, but right now we
are not using the Svelte configuration.

Reviewers: rengolin

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D74178
2020-02-07 11:16:48 -08:00
Jonas Devlieghere 4fe839ef3a [CMake] Rename EXCLUDE_FROM_ALL and make it an argument to add_lit_testsuite
EXCLUDE_FROM_ALL means something else for add_lit_testsuite as it does
for something like add_executable. Distinguish between the two by
renaming the variable and making it an argument to add_lit_testsuite.

Differential revision: https://reviews.llvm.org/D74168
2020-02-06 15:33:18 -08:00
Dimitry Andric 924c9030a8 [compiler-rt] clang-format FreeBSD-specific sanitizer sources
Formatting only, no functional change intended.
2020-02-06 22:48:08 +01:00
Mitch Phillips 8d19af685c [GWP-ASan] Use weak abort message definition.
Summary:
New shard out of the Check() function for GWP-ASan uses
android_set_abort_message. This is happily present on bionic Android,
but not Android for glibc host x86. Fix up to use the weak definition
always, so we don't have to worry.

Reviewers: eugenis

Reviewed By: eugenis

Subscribers: #sanitizers, llvm-commits, pcc, cferris

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D74142
2020-02-06 10:36:25 -08:00
Mitch Phillips 25de3f98b8 [GWP-ASan] Fix PRNG to use IE TLS.
Summary:
GWP-ASan's PRNG didn't use Initial-Exec TLS. Fix that to ensure that we don't
have infinite recursion, and also that we don't allocate a DTV on Android when
GWP-ASan is touched.

Test coverage ensuring that the sample counter is UINT32_MAX for an
uninitialised GWP-ASan is provided by gwp_asan/tests/late_init.cpp.

Reviewers: pcc, cferris

Reviewed By: pcc

Subscribers: #sanitizers, llvm-commits, rprichard, eugenis

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D74135
2020-02-06 10:08:23 -08:00
Peter Collingbourne dab7bdad04 scudo: Delete unused class ScudoByteMap. NFCI.
The class is only used in SizeClassAllocator32 in 64-bit mode, but we don't
use that class in 64-bit mode.

Differential Revision: https://reviews.llvm.org/D74099
2020-02-06 09:30:24 -08:00
Mitch Phillips 0bfc489093 [GWP-ASan] Fix unused variables from crash handler + clang-format
Summary: NFC - See title

Reviewers: eugenis

Reviewed By: eugenis

Subscribers: merge_guards_bot, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D74100
2020-02-05 17:31:32 -08:00
Mitch Phillips 0c3b2986ac [GWP-ASan] Change crash handler constant.
Constant is too large to fit into uintptr_t on 32-bit.
2020-02-05 16:46:51 -08:00
Mitch Phillips a62586846f [GWP-ASan] Crash Handler API.
Summary:
Forewarning: This patch looks big in #LOC changed. I promise it's not that bad, it just moves a lot of content from one file to another. I've gone ahead and left inline comments on Phabricator for sections where this has happened.

This patch:
 1. Introduces the crash handler API (crash_handler_api.h).
 2. Moves information required for out-of-process crash handling into an AllocatorState. This is a trivially-copied POD struct that designed to be recovered from a deceased process, and used by the crash handler to create a GWP-ASan report (along with the other trivially-copied Metadata struct).
 3. Implements the crash handler API using the AllocatorState and Metadata.
 4. Adds tests for the crash handler.
 5. Reimplements the (now optionally linked by the supporting allocator) in-process crash handler (i.e. the segv handler) using the new crash handler API.
 6. Minor updates Scudo & Scudo Standalone to fix compatibility.
 7. Changed capitalisation of errors (e.g. /s/Use after free/Use After Free).

Reviewers: cryptoad, eugenis, jfb

Reviewed By: eugenis

Subscribers: merge_guards_bot, pcc, jfb, dexonsmith, mgorny, cryptoad, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73557
2020-02-05 15:39:17 -08:00
Kostya Kortchinsky a9d5f8989d [scudo][standalone] Fix a race in the secondary release
Summary:
I tried to move the `madvise` calls outside of one of the secondary
mutexes, but this backfired. There is situation when a low release
interval is set combined with secondary pressure that leads to a race:
a thread can get a block from the cache, while another thread is
`madvise`'ing that block, resulting in a null header.

I changed the secondary race test so that this situation would be
triggered, and moved the release into the cache mutex scope.

Reviewers: cferris, pcc, eugenis, hctim, morehouse

Subscribers: jfb, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D74072
2020-02-05 11:02:51 -08:00
Petr Hosek 7800310cb5 [CMake][compiler-rt] Replace Windows backslashes with CMake ones
XRay builds uses llvm-config to obtain the ldflags and libs and then
passes those to CMake. Unfortunately, this breaks on Windows because
CMake tries to interpret backslashes followed by certain characters
as flags. We need to rewrite these into forward slashes that are used
by CMake (even on Windows).

Differential Revision: https://reviews.llvm.org/D73523
2020-02-04 22:35:31 -08:00
Evgenii Stepanov f48c1f5085 [compiler-rt] fix OpenBSD and Solaris build with sigaltstack interception
Summary:
`sigaltstack` interception is implemented D73816. This updated OpenBSD and Solaris bits to fix the build errors.

Reviewers: eugenis

Reviewed By: eugenis

Subscribers: fedor.sergeev, krytarowski, emaste, eugenis, dberris, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Patch by Igor Sugak.

Differential Revision: https://reviews.llvm.org/D73976
2020-02-04 15:50:59 -08:00
Kamil Rytarowski b79b7674eb [compiler-rt] [netbsd] Fix build after "implement sigaltstack interception"
Fallout after: https://reviews.llvm.org/D73816
2020-02-04 23:31:58 +01:00
Dimitry Andric 8a1f4feb1b [compiler-rt] Fix sanitizer_common build for FreeBSD
This was broken by 28c91219c7, which added `struct_stack_t_sz` only to
sanitizer_platform_limits_posix.{cpp,h}.
2020-02-04 22:51:07 +01:00
Evgenii Stepanov dd921cb89b [sanitizer] Add missing declarations for sigaltstack syscall wrappers.
Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73987
2020-02-04 13:10:43 -08:00
Peter Collingbourne f7de7084f4 scudo: Simplify getClassIdBySize() logic. NFCI.
By subtracting 1 from Size at the beginning we can simplify the
subsequent calculations. This also saves 4 instructions on aarch64
and 9 instructions on x86_64, but seems to be perf neutral.

Differential Revision: https://reviews.llvm.org/D73936
2020-02-04 09:32:27 -08:00
Evgenii Stepanov 6da6153759 hwasan_symbolize: allow 0x in the address field
Summary:
Fix parsing of mangled stack trace lines where the address has been
replaced with "0x", literally.

Reviewers: vitalybuka

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73823
2020-02-03 18:23:48 -08:00
Evgenii Stepanov 0dc634babf hwasan_symbolize: warn about missing symbols at most once per library
Reviewers: vitalybuka

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73822
2020-02-03 18:23:48 -08:00
Evgenii Stepanov 28c91219c7 [compiler-rt] implement sigaltstack interception
Summary:
An implementation for `sigaltstack` to make its side effect be visible to MSAN.

```
ninja check-msan
```

Reviewers: vitalybuka, eugenis

Reviewed By: eugenis

Subscribers: dberris, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73816

Patch by Igor Sugak.
2020-02-03 16:28:47 -08:00
Peter Collingbourne 47cda0cb36 scudo: Use more size classes in the malloc_free_loop benchmarks.
As a result of recent changes to the Android size classes, the malloc_free_loop
benchmark started exhausting the 8192 size class at 32768 iterations. To avoid
this problem (and to make the test more realistic), change the benchmark to
use a variety of size classes.

Differential Revision: https://reviews.llvm.org/D73918
2020-02-03 13:18:25 -08:00
Mitch Phillips 0d6fccb460 [GWP-ASan] Allow late initialisation if single-threaded.
Summary:
This patch allows for late initialisation of the GWP-ASan allocator. Previously, if late initialisation occurred, the sample counter was never updated, meaning we would end up having to wait for 2^32 allocations before getting a sampled allocation.

Now, we initialise the sampling mechanism in init() as well. We require init() to be called single-threaded, so this isn't a problem.

Reviewers: eugenis

Reviewed By: eugenis

Subscribers: merge_guards_bot, mgorny, #sanitizers, llvm-commits, cferris

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73896
2020-02-03 12:34:26 -08:00
Max Moroz ad7b908b4e [libFuzzer] Make dataflow and focus functions more user friendly.
Summary:
- Fail loudly if SetFocusFunction failed when it should not. For more info see
  - https://github.com/google/oss-fuzz/issues/3311
  - https://github.com/google/sanitizers/issues/1190
- Fail loudly if CollectDataFlow is called without seed corpus.

Reviewers: kcc, metzman

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73813
2020-02-03 08:36:03 -08:00
Petr Hosek c96eeebca8 [CMake] compiler-rt: Add COMPILER_RT_BUILTINS_ENABLE_PIC
The configuration for -fPIC in the builtins library when built standalone
is unconditional, stating that the flags would "normally be added... by
the llvm cmake step"

This is untrue, as the llvm cmake step checks LLVM_ENABLE_PIC, which allows
a client to turn off -fPIC.

I've added an option when compiler-rt builtins are configured standalone, such
as when built as part of the LLVM runtimes system, to guard the application of
-fPIC for users that want it.

Patch By: JamesNagurne

Differential Revision: https://reviews.llvm.org/D72950
2020-01-31 15:57:18 -08:00
Kostya Kortchinsky 64cb77b946 [scudo][standalone] Change default Android config
Summary:
This changes a couple of parameters in the default Android config to
address some performance and memory footprint issues (well to be closer
to the default Bionic allocator numbers).

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73750
2020-01-31 14:46:35 -08:00
Kamil Rytarowski 3a200f3f2e [compiler-rt] Fix build on NetBSD 9.99.44
Fix build on >= 9.99.44 after the removal of urio(4).
Add compat code for the device as NetBSD-9.0 is supported.
2020-01-31 14:57:20 +01:00
Kostya Kortchinsky 654f5d6845 [scudo][standalone] Release secondary memory on purge
Summary:
The Secondary's cache needs to be released when the Combined's
`releaseToOS` function is called (via `M_PURGE`) for example,
which this CL adds.

Additionally, if doing a forced release, we'll release the
transfer batch class as well since now we can do that.

There is a couple of other house keeping changes as well:
- read the page size only once in the Secondary Cache `store`
- remove the interval check for `CanRelease`: we are going to
  make that configurable via `mallopt` so this needs not be
  set in stone there.

Reviewers: cferris, hctim, pcc, eugenis

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73730
2020-01-30 13:22:45 -08:00
Yuanfang Chen 29181e5426 [compiler-rt][profile] fix test/instrprof-set-filename.c on windows
Summary: `.cmd` is interpreted as script in windows console.

Reviewers: davidxl, rnk

Reviewed By: davidxl

Differential Revision: https://reviews.llvm.org/D73327
2020-01-29 15:00:56 -08:00
Kostya Kortchinsky 6cb830de6e [scudo][standalone] Revert some perf-degrading changes
Summary:
A couple of seemingly innocuous changes ended up having a large impact
on the 32-bit performance. I still have to make those configurable at
some point, but right now it will have to do.

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73658
2020-01-29 14:00:48 -08:00
Bjorn Pettersson f4ca2ac822 [scudo] Skip building scudo standalone if sys/auxv.h can't be found
Summary:
Since commit c299d1981d scudo
standalone can't be built without including sys/auxv.h.
I do not have that file on my system, and my builds have failed
when trying to simply build "all" runtimes. Assuming that "all"
means "all possible given the current environment" we need to
guard the setting of COMPILER_RT_HAS_SCUDO_STANDALONE=TRUE by
first checking if sys/auxv.h can be found.

Reviewers: pcc, cryptoad

Reviewed By: pcc

Subscribers: mgorny, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73631
2020-01-29 18:18:59 +01:00
Shoaib Meenai d8f6950828 [asan] Fix test compilation on Android API <= 17
mlockall and munlockall were introduced in Android API 17, so avoid
referencing them on prior versions.

Differential Revision: https://reviews.llvm.org/D73515
2020-01-28 14:36:19 -08:00
Roland McGrath 90a10f00ff [lsan] Support LeakSanitizer runtime on Fuchsia
Support LeakSanitizer runtime on Fuchsia.

Patch By: mcgrathr

Differential Revision: https://reviews.llvm.org/D72887
2020-01-28 11:34:53 -08:00
Kostya Kortchinsky a1f6ff2681 [scudo][standalone] Fix Android logging
Summary:
Zygote & children's stderr is lost, so use Bionic's provided allocation
free syslog function for `outputRaw`. Get rid of the mutex as it's not
vital and could cause issues with `fork`.

Reviewers: cferris, pcc, eugenis, hctim, morehouse

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73561
2020-01-28 11:03:39 -08:00
Kostya Kortchinsky 993e3c9269 [scudo][standalone] Secondary & general other improvements
Summary:
This CL changes multiple things to improve performance (notably on
Android).We introduce a cache class for the Secondary that is taking
care of this mechanism now.

The changes:
- change the Secondary "freelist" to an array. By keeping free secondary
  blocks linked together through their headers, we were keeping a page
  per block, which isn't great. Also we know touch less pages when
  walking the new "freelist".
- fix an issue with the freelist getting full: if the pattern is an ever
  increasing size malloc then free, the freelist would fill up and
  entries would not be used. So now we empty the list if we get to many
  "full" events;
- use the global release to os interval option for the secondary: it
  was too costly to release all the time, particularly for pattern that
  are malloc(X)/free(X)/malloc(X). Now the release will only occur
  after the selected interval, when going through the deallocate path;
- allow release of the `BatchClassId` class: it is releasable, we just
  have to make sure we don't mark the batches containing batches
  pointers as free.
- change the default release interval to 1s for Android to match the
  current Bionic allocator configuration. A patch is coming up to allow
  changing it through `mallopt`.
- lower the smallest class that can be released to `PageSize/64`.

Reviewers: cferris, pcc, eugenis, morehouse, hctim

Subscribers: phosek, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73507
2020-01-28 07:28:55 -08:00
Petr Hosek 2533bc2361 Revert "[lsan] Support LeakSanitizer runtime on Fuchsia"
This reverts commit d59e3429f3.
2020-01-28 00:24:41 -08:00
Roland McGrath d59e3429f3 [lsan] Support LeakSanitizer runtime on Fuchsia
Support LeakSanitizer runtime on Fuchsia.

Patch By: mcgrathr

Differential Revision: https://reviews.llvm.org/D72887
2020-01-27 23:35:06 -08:00
Evgenii Stepanov 46044a6900 [gwp-asan] Implement malloc_iterate.
Summary:
This is an Android-specific interface for iterating over all live
allocations in a memory range.

Reviewers: hctim, cferris

Subscribers: mgorny, mgrang, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73305
2020-01-27 14:37:33 -08:00
Roland McGrath 38fd1806a1 [lsan] Factor pthread-specific assumptions out of thread tracking code
This is a small refactoring to prepare for porting LSan to Fuchsia.
Factor out parts of lsan_thread.{cpp,h} that don't apply to Fuchsia.
Since existing supported systems are POSIX-based, the affected code
is moved to lsan_posix.{cpp.h}.

Patch By: mcgrathr

Differential Revision: https://reviews.llvm.org/D73309
2020-01-24 16:55:11 -08:00
Roland McGrath aae707cd88 [lsan] Expose Frontier object to OS-specific LockStuffAndStopTheWorld callback
This is a small refactoring to prepare for porting LSan to Fuchsia.
On Fuchsia, the system supplies a unified API for suspending threads and
enumerating roots from OS-specific places like thread state and global data
ranges. So its LockStuffAndStopTheWorld implementation will make specific
callbacks for all the OS-specific root collection work before making the
common callback that includes the actual leak-checking logic.

Patch By: mcgrathr

Differential Revision: https://reviews.llvm.org/D72988
2020-01-24 16:53:35 -08:00
Roland McGrath 81b700e302 [sanitizer_common] Implement MemoryMappingLayout for Fuchsia
This is needed to port lsan to Fuchsia.

Patch By: mcgrathr

Differential Revision: https://reviews.llvm.org/D72886
2020-01-24 16:35:43 -08:00
Mitch Phillips e1440f594c [GWP-ASan] Add names to anonymous mappings.
Summary:
Adds names to anonymous GWP-ASan mappings. This helps Android with debugging
via. /proc/maps, as GWP-ASan-allocated mappings are now easily identifyable.

Reviewers: eugenis, cferris

Reviewed By: eugenis

Subscribers: merge_guards_bot, #sanitizers, llvm-commits, cryptoad, pcc

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73373
2020-01-24 15:02:02 -08:00
Evgenii Stepanov 596d06145a [GWP-ASan] enable/disable and fork support.
Summary:
* Implement enable() and disable() in GWP-ASan.
* Setup atfork handler.
* Improve test harness sanity and re-enable GWP-ASan in Scudo.

Scudo_standalone disables embedded GWP-ASan as necessary around fork().
Standalone GWP-ASan sets the atfork handler in init() if asked to. This
requires a working malloc(), therefore GWP-ASan initialization in Scudo
is delayed to the post-init callback.

Test harness changes are about setting up a single global instance of
the GWP-ASan allocator so that pthread_atfork() does not create
dangling pointers.

Test case shamelessly stolen from D72470.

Reviewers: cryptoad, hctim, jfb

Subscribers: mgorny, jfb, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73294
2020-01-24 13:53:26 -08:00
Evgenii Stepanov 966b5182ba Change internal_start_thread arguments to match pthread_create.
This avoids a CFI-unfriendly function pointer type cast in
internal_start_thread.
2020-01-23 13:15:16 -08:00
Dan Liew 9d9b470e69 [CMake] Refactor iOS simulator/device test configuration generation code for LibFuzzer.
Summary:
In order to do this `FUZZER_SUPPORTED_OS` had to be pulled out of
`lib/fuzzer/CMakeLists.txt` and into the main config so we can use it
from the `test/fuzzer/CMakeList.txt`. `FUZZER_SUPPORTED_OS` currently
has the same value of `SANITIZER_COMMON_SUPPORTED_OS` which preserves
the existing behaviour but this allows us in the future to adjust the
supported platforms independent of `SANITIZER_COMMON_SUPPORTED_OS`. This
mirrors the other sanitizers.

For non-Apple platforms `FUZZER_SUPPORTED_OS` is not defined and
surprisingly this was the behaviour before this patch because
`SANITIZER_COMMON_SUPPORTED_OS` was actually empty. This appears to
not matter right now because the functions that take an `OS` as an
argument seem to ignore it on non-Apple platforms.

While this change tries to be NFC it is technically not because we
now generate an iossim config whereas previously we didn't. This seems
like the right thing to do because the build system was configured to
compile LibFuzzer for iossim but previously we weren't generating a lit
test config for it. The device/simulator testing configs don't run by
default anyway so this shouldn't break testing.

This change relies on the get_capitalized_apple_platform() function
added in a previous commit.

rdar://problem/58798733

Reviewers: kubamracek, yln

Subscribers: mgorny, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73243
2020-01-23 12:44:00 -08:00
Dan Liew 06569361d0 [CMake][NFC] Refactor iOS simulator/device test configuration generation code for TSan.
Summary:
The previous code hard-coded platform names but compiler-rt's CMake
build system actually already knows which Apple platforms TSan supports.

This change uses this information to enumerate the different Apple
platforms.

This change relies on the `get_capitalized_apple_platform()` function
added in a previous commit.

rdar://problem/58798733

Reviewers: kubamracek, yln

Subscribers: mgorny, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73238
2020-01-23 12:44:00 -08:00
Dan Liew 1262745060 [CMake][NFC] Refactor iOS simulator/device test configuration generation code for ASan.
Summary:
The previous code hard-coded platform names but compiler-rt's CMake
build system actually already knows which Apple platforms ASan supports.

This change uses this information to enumerate the different Apple
platforms.

rdar://problem/58798733

Reviewers: kubamracek, yln

Subscribers: mgorny, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73232
2020-01-23 12:44:00 -08:00
Max Moroz 15f1d5d144 [libFuzzer] Add INFO output when LLVMFuzzerCustomMutator is found.
Summary:
there is an ongoing work on interchangeable custom mutators
(https://github.com/google/clusterfuzz/pull/1333/files#r367706283)
and having some sort of signalling from libFuzzer that it has loaded
a custom mutator would be helpful.

The initial idea was to make the mutator to print something, but given
the anticipated variety of different mutators, it does not seem possible
to make all of them print the same message to signal their execution.

Reviewers: kcc, metzman

Reviewed By: metzman

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73136
2020-01-22 12:56:16 -08:00
Kostya Kortchinsky 990acd450c [scudo][standalone] Reduce Android region sizes to 128MB
Summary:
Unity is making irresponsible assumptions as to how clumped up memory
should be. With larger regions, we break those, resulting in errors
like:

"Using memoryadresses from more that 16GB of memory"

This is unfortunately one of those situations where we have to bend to
existing code because we doubt it's going to change any time soon.

128MB should be enough, but we could be flirting with OOMs in the
higher class sizes.

Reviewers: cferris, eugenis, hctim, morehouse, pcc

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73143
2020-01-22 09:42:07 -08:00
Michał Górny 3215f7c7a8 [compiler-rt] [builtins] Fix clear_cache_test to work with MPROTECT
Fix clear_cache_test to work on NetBSD with PaX MPROTECT enabled, that
is when creating W+X mmaps is prohibited.  Use the recommended solution:
create two mappings for the same memory area, make one of them RW, while
the other RX.  Copy the function into the RW area but run it from the RX
area.

In order to implement this, I've split the pointer variables to
'write_buffer' and 'execution_buffer'.  Both are separate pointers
on NetBSD, while they have the same value on other systems.

I've also split the memcpy_f() into two: new memcpy_f() that only takes
care of copying memory and discards the (known) result of memcpy(),
and realign_f() that applies ARM realignment to the given pointer.
Again, there should be no difference on non-NetBSD systems but on NetBSD
copying is done on write_buffer, while realignment on pointer
to the execution_buffer.

I have tested this change on NetBSD and Linux.

Differential Revision: https://reviews.llvm.org/D72578
2020-01-22 17:04:54 +01:00
Kostya Kortchinsky 561fa84477 [scudo][standalone] Allow sched_getaffinity to fail
Summary:
In some configuration, `sched_getaffinity` can fail. Some reasons for
that being the lack of `CAP_SYS_NICE` capability or some syscall
filtering and so on.

This should not be fatal to the allocator, so in this situation, we
will fallback to the `MaxTSDCount` value specified in the allocator
configuration.

Reviewers: cferris, eugenis, hctim, morehouse, pcc

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D73055
2020-01-21 11:18:18 -08:00
Peter Collingbourne 9b9c68a2d6 hwasan: Remove dead code. NFCI.
Differential Revision: https://reviews.llvm.org/D72896
2020-01-17 15:12:38 -08:00
Petr Hosek d3db13af7e [profile] Support counter relocation at runtime
This is an alternative to the continous mode that was implemented in
D68351. This mode relies on padding and the ability to mmap a file over
the existing mapping which is generally only available on POSIX systems
and isn't suitable for other platforms.

This change instead introduces the ability to relocate counters at
runtime using a level of indirection. On every counter access, we add a
bias to the counter address. This bias is stored in a symbol that's
provided by the profile runtime and is initially set to zero, meaning no
relocation. The runtime can mmap the profile into memory at abitrary
location, and set bias to the offset between the original and the new
counter location, at which point every subsequent counter access will be
to the new location, which allows updating profile directly akin to the
continous mode.

The advantage of this implementation is that doesn't require any special
OS support. The disadvantage is the extra overhead due to additional
instructions required for each counter access (overhead both in terms of
binary size and performance) plus duplication of counters (i.e. one copy
in the binary itself and another copy that's mmapped).

Differential Revision: https://reviews.llvm.org/D69740
2020-01-17 15:02:23 -08:00
Marco Vanotti 44aaca3de4 [libFuzzer] Allow discarding output in ExecuteCommand in Fuchsia.
Summary:
This commit modifies the way `ExecuteCommand` works in fuchsia by adding
special logic to handle `/dev/null`.

The FuzzerCommand interface does not have a way to "discard" the output,
so other parts of the code just set the output file to `getDevNull()`.
The problem is that fuchsia does not have a named file that is
equivalent to `/dev/null`, so opening that file just fails.

This commit detects whether the specified output file is `getDevNull`,
and if that's the case, it will not copy the file descriptor for stdout
in the spawned process.

NOTE that modifying `FuzzerCommand` to add a "discardOutput" function
involves a significant refactor of all the other platforms, as they all
rely on the `toString()` method of `FuzzerCommand`.

This allows libfuzzer in fuchsia to run with `fork=1`, as the merge
process (`FuzzerMerge.cpp`) invoked `ExecuteCommand` with `/dev/null` as the
output.

Reviewers: aarongreen, phosek

Reviewed By: aarongreen

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D72894
2020-01-17 12:15:46 -08:00
Peter Collingbourne c299d1981d scudo: Add initial memory tagging support.
When the hardware and operating system support the ARM Memory Tagging
Extension, tag primary allocation granules with a random tag. The granules
either side of the allocation are tagged with tag 0, which is normally
excluded from the set of tags that may be selected randomly. Memory is
also retagged with a random tag when it is freed, and we opportunistically
reuse the new tag when the block is reused to reduce overhead. This causes
linear buffer overflows to be caught deterministically and non-linear buffer
overflows and use-after-free to be caught probabilistically.

This feature is currently only enabled for the Android allocator
and depends on an experimental Linux kernel branch available here:
https://github.com/pcc/linux/tree/android-experimental-mte

All code that depends on the kernel branch is hidden behind a macro,
ANDROID_EXPERIMENTAL_MTE. This is the same macro that is used by the Android
platform and may only be defined in non-production configurations. When the
userspace interface is finalized the code will be updated to use the stable
interface and all #ifdef ANDROID_EXPERIMENTAL_MTE will be removed.

Differential Revision: https://reviews.llvm.org/D70762
2020-01-16 13:27:49 -08:00
Hubert Tong e429f24ed8 [CMake] Enable -qfuncsect when building with IBM XL
Summary:
The IBM XL compiler uses `-qfuncsect` for `-ffunction-sections`.

The comment about sanitizers and `-f[no-]function-sections` is corrected
also, as it is pertinent to this patch.

The sanitizer-related use of `-fno-function-sections` is associated with
powerpc64le, a target for which there is an IBM XL compiler, so that use
is updated in this patch to apply `-qnofuncsect` in case a build using
the XL compiler is viable on that platform.

This patch has been verified with the XL compiler on AIX only.

Reviewers: daltenty, stevewan

Reviewed By: daltenty

Subscribers: mgorny, steven.zhang, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D72335
2020-01-15 10:45:50 -05:00
Kostya Kortchinsky 9ef6faf496 [scudo][standalone] Fork support
Summary:
fork() wasn't well (or at all) supported in Scudo. This materialized
in deadlocks in children.

In order to properly support fork, we will lock the allocator pre-fork
and unlock it post-fork in parent and child. This is done via a
`pthread_atfork` call installing the necessary handlers.

A couple of things suck here: this function allocates - so this has to
be done post initialization as our init path is not reentrance, and it
doesn't allow for an extra pointer - so we can't pass the allocator we
are currently working with.

In order to work around this, I added a post-init template parameter
that gets executed once the allocator is initialized for the current
thread. Its job for the C wrappers is to install the atfork handlers.

I reorganized a bit the impacted area and added some tests, courtesy
of cferris@ that were deadlocking prior to this fix.

Subscribers: jfb, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D72470
2020-01-14 07:51:48 -08:00
Fangrui Song c5b94ea265 [profile] Support merge pool size >= 10
The executable acquires an advisory record lock (`fcntl(fd, F_SETLKW, *)`) on a profile file.
Merge pool size >= 10 may be beneficial when the concurrency is large.

Also fix a small problem about snprintf. It can cause the filename to be truncated after %m.

Reviewed By: davidxl

Differential Revision: https://reviews.llvm.org/D71970
2020-01-12 00:27:18 -08:00
Lirong Yuan a5a6fd3f95 Summary: update macro for OFF_T so that sanitizer works on AARCH64.
Reviewers: vitalybuka, eugenis, MaskRay

Reviewed By: eugenis, MaskRay

Subscribers: MaskRay, kristof.beyls, #sanitizers, llvm-commits, jkz, scw

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D72367
2020-01-10 13:34:49 -08:00
Fangrui Song 7f1026a752 [cfi][test] cross-dso/stats.cpp: don't assume the order of static constructors
__sanitizer_stat_init is called for the executable first, then the
shared object. In WriterModuleReport(), the information for the shared
object will be recorded first. It'd be nice to get rid of the order
requirement of static constructors. (This should make .ctors platforms
work.)
2020-01-08 15:50:51 -08:00
Evgenii Stepanov 7ba4595c86 [msan] Fix underflow in qsort interceptor. 2020-01-06 16:12:52 -08:00
Evgenii Stepanov b5e7f95cfb [msan] Check qsort input.
Summary:
Qsort interceptor suppresses all checks by unpoisoning the data in the
wrapper of a comparator function, and then unpoisoning the output array
as well.

This change adds an explicit run of the comparator on all elements of
the input array to catch any sanitizer bugs.

Reviewers: vitalybuka

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D71780
2020-01-06 15:07:02 -08:00
Kamil Rytarowski 0da15ff318 [compiler-rt] [netbsd] Switch to syscall for ThreadSelfTlsTcb()
This change avoids using internal, namespaced per-CPU calls that
are not a stable interface to read the TSL base value.
2020-01-03 02:40:52 +01:00
Saleem Abdulrasool abb0075306 build: reduce CMake handling for zlib
Rather than handling zlib handling manually, use `find_package` from CMake
to find zlib properly. Use this to normalize the `LLVM_ENABLE_ZLIB`,
`HAVE_ZLIB`, `HAVE_ZLIB_H`. Furthermore, require zlib if `LLVM_ENABLE_ZLIB` is
set to `YES`, which requires the distributor to explicitly select whether
zlib is enabled or not. This simplifies the CMake handling and usage in
the rest of the tooling.

This restores 68a235d07f,
e6c7ed6d21.  The problem with the windows
bot is a need for clearing the cache.
2020-01-02 11:19:12 -08:00
Sterling Augustine 108daf7611 Check for aarch64 when not using .init_arrays
Reviewers: vvereschaka

Subscribers: kristof.beyls, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D72005
2020-01-02 10:10:49 -08:00
James Henderson e406cca5f9 Revert "build: reduce CMake handling for zlib"
This reverts commit 68a235d07f.

This commit broke the clang-x64-windows-msvc build bot and a follow-up
commit did not fix it. Reverting to fix the bot.
2020-01-02 16:02:10 +00:00
Saleem Abdulrasool 68a235d07f build: reduce CMake handling for zlib
Rather than handling zlib handling manually, use `find_package` from CMake
to find zlib properly. Use this to normalize the `LLVM_ENABLE_ZLIB`,
`HAVE_ZLIB`, `HAVE_ZLIB_H`. Furthermore, require zlib if `LLVM_ENABLE_ZLIB` is
set to `YES`, which requires the distributor to explicitly select whether
zlib is enabled or not. This simplifies the CMake handling and usage in
the rest of the tooling.
2020-01-01 16:36:59 -08:00
Fangrui Song e7853a5ce2 [CMake] Fix lld detection after D69685
D69685 actually broke lld detection for my build (probably due to CMake
processing order).

Before:

```
build projects/compiler-rt/lib/sanitizer_common/tests/Sanitizer-x86_64-Test-Nolibc: ... bin/clang || ...
```

After:

```
build projects/compiler-rt/lib/sanitizer_common/tests/Sanitizer-x86_64-Test-Nolibc: ... bin/clang bin/lld || ...
```

Differential Revision: https://reviews.llvm.org/D71950
2019-12-28 13:05:12 -08:00
Fangrui Song 33a1b3d8fc [sanitizer] Link Sanitizer-x86_64-Test-Nolibc with -static
Pass -static so that clang will not pass -Wl,--dynamic-linker,... to the
linker. The test is not expected to run under a ld.so. (Technically it
works under a ld.so but glibc expects to see a PT_DYNAMIC. lld
intentionally does not follow GNU ld's complex rules regarding
PT_DYNAMIC.)

This allows commit 1417558e4a to be
relanded.
2019-12-27 15:25:08 -08:00
Sterling Augustine 1d891a32cf Support powerpc and sparc when building without init_array.
Summary: Support powerpc and sparc when building without init_array.

Reviewers: rdhindsa, gribozavr

Subscribers: jyknight, nemanjai, fedor.sergeev, jsji, shchenz, steven.zhang, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D71944
2019-12-27 13:02:11 -08:00
Reid Kleckner ef7a659c21 Reland "[msan] Intercept qsort, qsort_r."
This reverts commit 8fcce5ac73.

I spoke too soon, the revert does not actually cause the startup crash
to go away.
2019-12-27 11:29:00 -08:00
Reid Kleckner 8fcce5ac73 Revert "[msan] Intercept qsort, qsort_r."
This reverts commit 7a9ebe9512, and
dependent commit 54c5224203, which
disables qsort interception for some iOS platforms.

After this change, the -Nolibc sanitizer common test binary crashes on
startup on my regular Linux workstation, as well as on our bots:
https://ci.chromium.org/p/chromium/builders/try/linux_upload_clang/740

 ********************
  Failing Tests (1):
       SanitizerCommon-Unit ::
       ./Sanitizer-x86_64-Test/SanitizerCommon.NolibcMain

Loading it up in gdb shows that it crashes during relocation processing,
which suggests that some glibc loader versions do not support the
THREADLOCAL data added in this interceptor.
2019-12-27 11:24:07 -08:00
Kamil Rytarowski 84afd9c536 [compiler-rt] [netbsd] Add support for versioned statvfs interceptors
Summary:
Add support for NetBSD 9.0 and newer versions of interceptors
operating on struct statvfs: fstatvfs, fstatvfs1, getmntinfo,
getvfsstat, statvfs, statvfs1.

The default promoted interceptors are for NetBSD 9.99.26. Older
ones (currently 9.0) are kept in a new NetBSD specific file:
/sanitizer_common_interceptors_netbsd_compat.inc. This file
defines compat interceptors and mangles `INIT_*` macros,
concatenating the current interceptors and the compat ones.
This redefinition is not elegant, but it avoids preprocessor madness.

Define struct_statvfs90_sz for the compat purposes.

Reviewers: mgorny, kcc, vitalybuka, joerg

Reviewed By: mgorny

Subscribers: dberris, llvm-commits, #sanitizers

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D71700
2019-12-27 18:50:14 +01:00
Eric Christopher 8f9304f559 Temporarily Revert "[compiler-rt] [netbsd] Add support for versioned statvfs interceptors"
as it's failing the netbsd specific linter parts of the sanitizer linter:

llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors_netbsd_compat.inc:23:  Lines should be <= 80 characters long  [whitespace/line_length]
llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_netbsd.cpp:2450:  Do not use variable-length arrays.  Use an appropriately named ('k' followed by CamelCase) compile-time constant for the size.

This reverts commit 78f714f824.
2019-12-26 18:40:20 -08:00
Kamil Rytarowski 78f714f824 [compiler-rt] [netbsd] Add support for versioned statvfs interceptors
Summary:
Add support for NetBSD 9.0 and newer versions of interceptors
operating on struct statvfs: fstatvfs, fstatvfs1, getmntinfo,
getvfsstat, statvfs, statvfs1.

The default promoted interceptors are for NetBSD 9.99.26. Older
ones (currently 9.0) are kept in a new NetBSD specific file:
/sanitizer_common_interceptors_netbsd_compat.inc. This file
defines compat interceptors and mangles `INIT_*` macros,
concatenating the current interceptors and the compat ones.
This redefinition is not elegant, but it avoids preprocessor madness.

Define struct_statvfs90_sz for the compat purposes.

Reviewers: mgorny, kcc, vitalybuka, joerg

Reviewed By: mgorny

Subscribers: dberris, llvm-commits, #sanitizers

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D71700
2019-12-27 01:15:39 +01:00
Evgenii Stepanov 04926e67fd Revert "[msan] Check qsort input."
This change breaks LLVM bootstrap with ASan and MSan.

FAILED: lib/ToolDrivers/llvm-lib/Options.inc
OptParser.td:137:1: error: Option is equivalent to
def INPUT : Option<[], "<input>", KIND_INPUT>;
^
OptParser.td:137:1: error: Other defined here
def INPUT : Option<[], "<input>", KIND_INPUT>;

This reverts commit caa48a6b88.
2019-12-26 12:29:48 -08:00
Florian Hahn 54c5224203 [compiler-rt] Disable QSORT interception on watchOS and tvOS.
Building the sanitizers for watchOS currently fails with
    sanitizer_common_interceptors.inc:9656:8: error: thread-local storage is not supported for the current target
    static THREADLOCAL SIZE_T qsort_size;

I've also speculatively disabled QSORT interception for tvOS to unblock
failing builds. I'll ask someone with more sanitizer knowledge to check
after the holidays.
2019-12-26 10:39:29 +01:00
Kamil Rytarowski aff6c9db9a [compiler-rt] [netbsd] Correct the fallback definition of PT_LWPNEXT
Fixes build on NetBSD 9.0.
2019-12-25 18:21:48 +01:00
Kamil Rytarowski 52b5fe5f45 [compiler-rt] [netbsd] Define _RTLD_SOURCE to fix build
The TLS base (LWP private pointer) functions are namespaced and
hidden i.e. inside the _RTLD_SOURCE namespace.
2019-12-25 04:09:50 +01:00
Kamil Rytarowski 4b8232d4f0 [compiler-rt] Adapt for ptrace(2) changes in NetBSD-9.99.30
Enable compat support for now legacy PT_LWPINFO.
Support PT_LWPSTATUS and PT_LWPNEXT.
2019-12-24 20:34:58 +01:00
Kamil Rytarowski fc356dcc11 [compiler-rt] Adapt stop-the-world for ptrace changes in NetBSD-9.99.30
Handle PT_LWPNEXT for newer kernels and keep PT_LWPINFO for older ones.
2019-12-24 20:33:54 +01:00
Evgenii Stepanov caa48a6b88 [msan] Check qsort input.
Summary:
Qsort interceptor suppresses all checks by unpoisoning the data in the
wrapper of a comparator function, and then unpoisoning the output array
as well.

This change adds an explicit run of the comparator on all elements of
the input array to catch any sanitizer bugs.

Reviewers: vitalybuka

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D71780
2019-12-23 11:34:49 -08:00
Evgenii Stepanov 7a9ebe9512 [msan] Intercept qsort, qsort_r.
Summary:
This fixes qsort-related false positives with glibc-2.27.
I'm not entirely sure why they did not show up with the earlier
versions; the code seems similar enough.

Reviewers: vitalybuka

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D71740
2019-12-23 11:34:49 -08:00
Jonas Devlieghere 60236fedc9 Revert "[msan] Check qsort input." and "[msan] Intercept qsort, qsort_r."
Temporarily revert the qsort changes because they fail to build on bots
that build with modules:

> error: thread-local storage is not supported for the current
> target (iossim)

http://green.lab.llvm.org/green/job/clang-stage2-Rthinlto/1820/console
http://green.lab.llvm.org/green/view/LLDB/job/lldb-cmake/4983/console

This reverts commit ddf897fc80.
This reverts commit 07861e955d.
2019-12-20 21:34:35 -08:00
Julian Lettner d1783db9db [iOS sim] Ensure simulator device is booted in iossim_prepare.py
Recent versions of the iOS simulator require that a "simulator device"
is booted before we can use `simctl spawn` (see iossim_run.py) to start
processes.

We can use `simctl bootstatus` to ensure that the simulator device
is booted before we run any tests via lit.  The `-b` option starts the
device if necessary.

Reviewed By: delcypher

Differential Revision: https://reviews.llvm.org/D71449
2019-12-20 13:44:01 -08:00
Michał Górny e73c662b9e [compiler-rt] [test] Disable MPROTECT for XRay tests on NetBSD 2019-12-20 22:08:01 +01:00
Kostya Kortchinsky 0fd6f19025 [scudo][standalone] Support __BIONIC__
Summary:
Some Android builds that we are interested in define `__BIONIC__`
but not `__ANDROID__`, so expand `SCUDO_ANDROID` to encompass those.

Reviewers: cferris, hctim, pcc, eugenis, morehouse

Subscribers: krytarowski, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D71772
2019-12-20 12:59:33 -08:00
Evgenii Stepanov ddf897fc80 [msan] Check qsort input.
Summary:
Qsort interceptor suppresses all checks by unpoisoning the data in the
wrapper of a comparator function, and then unpoisoning the output array
as well.

This change adds an explicit run of the comparator on all elements of
the input array to catch any sanitizer bugs.

Reviewers: vitalybuka

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D71780
2019-12-20 12:41:57 -08:00
Evgenii Stepanov 07861e955d [msan] Intercept qsort, qsort_r.
Summary:
This fixes qsort-related false positives with glibc-2.27.
I'm not entirely sure why they did not show up with the earlier
versions; the code seems similar enough.

Reviewers: vitalybuka

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D71740
2019-12-20 12:27:09 -08:00
Michał Górny 190b9110c2 [compiler-rt] [test] Disable MPROTECT on two builtin tests
Introduce a new %run_nomprotect substitution to run tests that do not
work with MPROTECT enabled.  This uses paxctl via a wrapper on NetBSD,
and evaluates to plain %run on other systems.

Differential Revision: https://reviews.llvm.org/D71513
2019-12-20 17:50:53 +01:00
Kostya Kortchinsky 77e906ac78 [scudo][standalone] Implement TSD registry disabling
Summary:
In order to implement `malloc_{enable|disable}` we were just disabling
(or really locking) the Primary and the Secondary. That meant that
allocations could still be serviced from the TSD as long as the cache
wouldn't have to be filled from the Primary.

This wasn't working out for Android tests, so this change implements
registry disabling (eg: locking) so that `getTSDAndLock` doesn't
return a TSD if the allocator is disabled. This also means that the
Primary doesn't have to be disabled in this situation.

For the Shared Registry, we loop through all the TSDs and lock them.
For the Exclusive Registry, we add a `Disabled` boolean to the Registry
that forces `getTSDAndLock` to use the Fallback TSD instead of the
thread local one. Disabling the Registry is then done by locking the
Fallback TSD and setting the boolean in question (I don't think this
needed an atomic variable but I might be wrong).

I clang-formatted the whole thing as usual hence the couple of extra
whiteline changes in this CL.

Reviewers: cferris, pcc, hctim, morehouse, eugenis

Subscribers: jfb, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D71719
2019-12-20 06:52:13 -08:00
Michał Górny 20345707b2 [compiler-rt] [test] Add missing %run to fread_fwrite MSAN test
Add a missing %run substitution to fread_fwrite test.  This fixes
the test on NetBSD where %run disables ASLR as necessary for MSAN
to function.

Differential Revision: https://reviews.llvm.org/D71623
2019-12-19 20:28:00 +01:00
Dmitri Gribenko 0109efe751 Revert "Support powerpc when builing without init_array."
This reverts commit 5789e83ded. It broke
the build on aarch64.
2019-12-19 11:25:14 +01:00
Kamil Rytarowski 3a189bac9b [compiler-rt] Enable SANITIZER_CAN_USE_PREINIT_ARRAY on NetBSD
.preinit_array is supported since 9.0.
2019-12-19 03:23:51 +01:00
Kamil Rytarowski d15241ed2c [compiler-rt] [fuzzer] Enable LSan in libFuzzer tests on NetBSD 2019-12-19 01:16:14 +01:00
Kamil Rytarowski 7e8541f3df [compiler-rt] [fuzzer] Include stdarg.h for va_list
Fixes build on NetBSD after 139e216e66.
2019-12-19 01:11:39 +01:00
Sterling Augustine 5789e83ded Support powerpc when builing without init_array.
Summary: Also add an error case when targetting an unimplement architecture.

Subscribers: nemanjai, jsji, shchenz, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D71670
2019-12-18 13:37:08 -08:00
Kamil Rytarowski 3f96686700 Correct version check 9.9.26 -> 9.99.26 2019-12-18 21:10:33 +01:00
Mitch Phillips 2f4c3c565a [NFC] Fix sanitizer lint problem that's breaking Android bot.
Fixes the lint issue introduced by
e14ff22694
2019-12-18 08:47:31 -08:00
Michał Górny d3d1ca14ce [compiler-rt] [test] Disable ASLR on UBSAN+[AMT]SAN tests on NetBSD 2019-12-18 17:21:15 +01:00
Michał Górny 9102a59084 [compiler-rt] [test] Disable ASLR on LSAN+ASAN tests on NetBSD 2019-12-18 17:21:14 +01:00
Michał Górny d372576d8a [compiler-rt] [test] Disable ASLR for fuzzer tests on NetBSD 2019-12-18 16:51:31 +01:00
Kamil Rytarowski e14ff22694 [compiler-rt] Sync NetBSD ioctl definitions with 9.99.26
Document the minimal version supported to 9.0.
2019-12-18 15:44:01 +01:00
Thomas Preud'homme ddd0bb8dba [lit] Remove lit's REQUIRES-ANY directive
Summary:
Remove REQUIRES-ANY alias lit directive since it is hardly used and can
be easily implemented using an OR expression using REQUIRES. Fixup
remaining testcases still using REQUIRES-ANY.

Reviewers: probinson, jdenny, gparker42

Reviewed By: gparker42

Subscribers: eugenis, asb, rbar, johnrusso, simoncook, sabuasal, niosHD, delcypher, jrtc27, zzheng, edward-jones, rogfer01, MartinMosbeck, brucehoult, the_o, PkmX, jocewei, lenary, s.egerton, pzheng, sameer.abuasal, apazos, luismarques, cfe-commits, #sanitizers, llvm-commits

Tags: #llvm, #clang, #sanitizers

Differential Revision: https://reviews.llvm.org/D71408
2019-12-17 10:36:36 +00:00
Igor Kudrin a57adc7a0b [sanitizer] Construct InternalMmapVector without memory allocation.
Construction of InternalMmapVector is often followed by a call to
reserve(), which may result in immediate reallocation of the memory
for the internal storage. This patch delays that allocation until
it is really needed.

Differential Revision: https://reviews.llvm.org/D71342
2019-12-17 15:03:23 +07:00
Dan Liew e2bcc33b50 Change `asan/TestCases/Darwin/malloc_zone-protected.cpp` to call abort so that it **always crashes**.
Summary:
This is needed because on some platforms we can't install signal
handlers and so the application just traps (i.e. crashes) rather than being intercepted
by ASan's signal handler which in the default Darwin config doesn't
exit with a crashing exit code.

rdar://problem/57984547

Reviewers: yln, kubamracek, jfb

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D71573
2019-12-16 16:37:50 -08:00
Julian Lettner f9a706a36a [TSan][Darwin] Avoid calling pthread_self() before libpthread is initialized
This skips calling `pthread_self` when `main_thread_identity` hasn't
been initialized yet.  `main_thread_identity` is only ever assigned in
`__tsan::InitializePlatform`.  This change should be relatively safe; we
are not changing behavior other than skipping the call to `pthread_self`
when `main_thread_identity == 0`.

rdar://57822138

Reviewed By: kubamracek

Differential Revision: https://reviews.llvm.org/D71559
2019-12-16 13:10:51 -08:00
Vedant Kumar ed83942bc0 [ubsan] Enable testing of .m files
This allows `.m` test files, like the existing Misc/bool.m, to be
tested.
2019-12-13 14:55:29 -08:00
Alex Richardson 9cb7a1be2a [LSAN] Increase stack space for guard-page.c test
Summary:
When running the tests on a Ubuntu 18.04 machine this test is crashing for
me inside the runtime linker. My guess is that it is trying to save more
registers (possibly large vector ones) and the current stack space is not
sufficient.

Reviewers: samsonov, kcc, eugenis

Reviewed By: eugenis

Subscribers: eugenis, merge_guards_bot, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D71461
2019-12-13 21:40:03 +00:00
Mitch Phillips d6c445ea69 [NFC] Guard scudo_standalone's optional dependency on GWP-ASan behind flags. 2019-12-13 10:55:53 -08:00
Mitch Phillips a00cd6dfea [GWP-ASan] [Scudo] ifdef entire GWP-ASan tests.
Turns out that gtest in LLVM is only 1.8.0 (the newest version 1.10.0)
supports the GTEST_SKIP() macro, and apparently I didn't build w/o
GWP-ASan.

Should fix the GN bot, as well as any bots that may spuriously break on
platforms where the code wasn't correctly ifdef'd out as well.
2019-12-13 09:45:28 -08:00
Mitch Phillips ed4618edb3 [Scudo] [GWP-ASan] Add GWP-ASan to Scudo Standalone.
Summary:
Adds GWP-ASan to Scudo standalone. Default parameters are pulled across from the
GWP-ASan build. No backtrace support as of yet.

Reviewers: cryptoad, eugenis, pcc

Reviewed By: cryptoad

Subscribers: merge_guards_bot, mgorny, #sanitizers, llvm-commits, cferris, vlad.tsyrklevich, pcc

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D71229
2019-12-13 09:09:41 -08:00
Julian Lettner cdb4560557 [iOS sim] Simplify iossim_run.py script 2019-12-12 17:34:08 -08:00
Eric Christopher 3ecfe64fe3 [sanitizers] Redirect stdout and stderr to TASK_LOG
At some point cpplint.py became very noisy during a build spewing
a few hundred lines of "Done processing..." even with SILENT=1 in
cmake. This attempts to redirect the stdout of "Done processing" to
the task log along with any errors.

Tested by this with and without SILENT=1 to check things.

Differential Revision: https://reviews.llvm.org/D71402
Reviewed By: eugenis
2019-12-12 13:11:16 -08:00
Max Moroz 926fa4088c [compiler-rt] libFuzzer: update -merge_control_file= help message.
Summary:
The motivation for this change is to have a distinguisher in libFuzzer
that would let the runner know whether multistep merge is supported or not by
a particular fuzz target binary. Otherwise, multistep merge fails to execute
with older version of libFuzzer, and there is no way to verify that easily.

Reviewers: kcc

Subscribers: dberris, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D71423
2019-12-12 11:09:40 -08:00
Jonathan Metzman 23bee0b0cf [fuzzer] Add basic support for emscripten.
Summary:
Add basic support for emscripten.

This enables libFuzzer to build (using build.sh) for emscripten and fuzz
a target compiled with
-fsanitize-coverage=inline-8bit-counters.

Basic fuzzing and bug finding work with this commit.
RSS limit and timeouts will not work because they depend on system
functions that are not implemented/widely supported in emscripten.

Reviewers: kcc, vitalybuka, hctim

Reviewed By: hctim

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D71285
2019-12-12 08:56:47 -08:00