Commit Graph

9434 Commits

Author SHA1 Message Date
Kostya Kortchinsky f7b1489ffc [scudo][standalone] Get statistics in a char buffer
Summary:
Following up on D68471, this CL introduces some `getStats` APIs to
gather statistics in char buffers (`ScopedString` really) instead of
printing them out right away. Ultimately `printStats` will just
output the buffer, but that allows us to potentially do some work
on the intermediate buffer, and can be used for a `mallocz` type
of functionality. This allows us to pretty much get rid of all the
`Printf` calls around, but I am keeping the function in for
debugging purposes.

This changes the existing tests to use the new APIs when required.

I will add new tests as suggested in D68471 in another CL.

Reviewers: morehouse, hctim, vitalybuka, eugenis, cferris

Reviewed By: morehouse

Subscribers: delcypher, #sanitizers, llvm-commits

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D68653

llvm-svn: 374173
2019-10-09 15:09:28 +00:00
David Carlier d80c2520d9 [builtins] Unbreak build on FreeBSD armv7 after D60351
headers include reordering.

Reviewers: phosek, echristo

Reviewed-By: phosek

Differential Revsion: https://reviews.llvm.org/D68045

llvm-svn: 374070
2019-10-08 15:45:35 +00:00
Evgeniy Stepanov 2e2c934762 [msan] Add interceptors: crypt, crypt_r.
Reviewers: vitalybuka

Subscribers: srhines, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D68431

llvm-svn: 373993
2019-10-08 00:00:30 +00:00
Vitaly Buka 2fdec42a16 [tsan, go] fix Go windows build
Summary:
Don't use weak exports when building tsan into a shared library for Go. gcc can't handle the pragmas used to make the weak references.

Include files that have been added since the last update to build.bat. (We should really find a better way to list all the files needed.)

Add windows version defines (WINVER and _WIN32_WINNT) to get AcquireSRWLockExclusive and ReleaseSRWLockExclusive defined.

Define GetProcessMemoryInfo to use the kernel32 version. This is kind of a hack, the windows header files should do this translation for us. I think we're not in the right family partition (we're using Desktop, but that translation only happens for App and System partitions???), but hacking the family partition seems equally gross and I have no idea what the consequences of that might be.

Patch by Keith Randall.

Reviewers: dvyukov, vitalybuka

Reviewed By: vitalybuka

Subscribers: jfb, delcypher, #sanitizers, llvm-commits

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D68599

llvm-svn: 373984
2019-10-07 23:11:07 +00:00
Vitaly Buka 2059105637 [tsan, go] break commands into multiple lines
Summary: Patch by Keith Randall.

Reviewers: dvyukov, vitalybuka

Subscribers: delcypher, jfb, #sanitizers, llvm-commits

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D68596

llvm-svn: 373983
2019-10-07 23:04:16 +00:00
Vitaly Buka 9917c76107 [sanitizer] Print SIGTRAP for corresponding signal
Reviewers: eugenis, jfb

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D68603

llvm-svn: 373979
2019-10-07 22:43:19 +00:00
Vitaly Buka 87dd968849 [tsan] Don't delay SIGTRAP handler
Reviewers: eugenis, jfb

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D68604

llvm-svn: 373978
2019-10-07 22:43:17 +00:00
Kostya Kortchinsky bebdab63e8 [scudo][standalone] Correct releaseToOS behavior
Summary:
There was an issue in `releaseToOSMaybe`: one of the criteria to
decide if we should proceed with the release was wrong. Namely:

```
const uptr N = Sci->Stats.PoppedBlocks - Sci->Stats.PushedBlocks;
if (N * BlockSize < PageSize)
  return; // No chance to release anything.
```

I meant to check if the amount of bytes in the free list was lower
than a page, but this actually checks if the amount of **in use** bytes
was lower than a page.

The correct code is:

```
const uptr BytesInFreeList =
  Region->AllocatedUser -
  (Region->Stats.PoppedBlocks - Region->Stats.PushedBlocks) * BlockSize;
if (BytesInFreeList < PageSize)
  return 0; // No chance to release anything.
```

Consequences of the bug:
- if a class size has less than a page worth of in-use bytes (allocated
  or in a cache), reclaiming would not occur, whatever the amount of
  blocks in the free list; in real world scenarios this is unlikely to
  happen and be impactful;
- if a class size had less than a page worth of free bytes (and enough
  in-use bytes, etc), then reclaiming would be attempted, with likely
  no result. This means the reclaiming was overzealous at times.

I didn't have a good way to test for this, so I changed the prototype
of the function to return the number of bytes released, allowing to
get the information needed. The test added fails with the initial
criteria.

Another issue is that `ReleaseToOsInterval` can actually be 0, meaning
we always try to release (side note: it's terrible for performances).
so change a `> 0` check to `>= 0`.

Additionally, decrease the `CanRelease` threshold to `PageSize / 32`.
I still have to make that configurable but I will do it at another time.

Finally, rename some variables in `printStats`: I feel like "available"
was too ambiguous, so change it to "total".

Reviewers: morehouse, hctim, eugenis, vitalybuka, cferris

Reviewed By: morehouse

Subscribers: delcypher, #sanitizers, llvm-commits

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D68471

llvm-svn: 373930
2019-10-07 17:37:39 +00:00
Kostya Kortchinsky dc802dbef2 [scudo][standalone] Make malloc_info return a minimal XML
Summary:
Initially, our malloc_info was returning ENOTSUP, but Android would
rather have it return successfully and write a barebone XML to the
stream, so we will oblige.

Add an associated test.

Reviewers: cferris, morehouse, hctim, eugenis, vitalybuka

Reviewed By: morehouse

Subscribers: delcypher, #sanitizers, llvm-commits

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D68427

llvm-svn: 373754
2019-10-04 15:46:34 +00:00
Vitaly Buka 161229eb1b compiler-rt: use __GLIBC_PREREQ for SANITIZER_INTERCEPT_GETRANDOM
Summary: Fixes https://github.com/google/oss-fuzz/issues/2836

Reviewers: eugenis

Subscribers: dberris, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D68178

llvm-svn: 373681
2019-10-04 00:38:07 +00:00
Vitaly Buka 1fff2881a8 compiler-rt: move all __GLIBC_PREREQ into own header file
Reviewers: eugenis

Subscribers: dberris, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D68177

llvm-svn: 373632
2019-10-03 17:46:07 +00:00
Matt Morehouse 629cad1266 Move SanitizerInitializeUnwinder outside anonymous namespace.
Fixes the Android build breakage introduced in r373528.

llvm-svn: 373539
2019-10-02 22:57:37 +00:00
Matt Morehouse 4d39410b1e [sanitizer_common] Declare __sanitizer_on_print in Windows interception.
Speculative fix for Windows bot.

llvm-svn: 373532
2019-10-02 22:00:29 +00:00
Matt Morehouse 62f190ce9a [sanitizer_common] Add __sanitizer_on_print to interface list.
Should fix the current Windows buildbot failure.

llvm-svn: 373530
2019-10-02 21:21:18 +00:00
Vitaly Buka d39e7e2cf1 [compiler-rt] Use GetNextInstructionPc in signal handlers
Summary:
All other stack trace callers assume that PC contains return address.
HWAsan already use GetNextInstructionPc in similar code.

PR43339

Reviewers: eugenis, kcc, jfb

Subscribers: dexonsmith, dberris, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D68313

llvm-svn: 373529
2019-10-02 21:20:37 +00:00
Vitaly Buka 59bf75caab NFC: add namespace {}
llvm-svn: 373528
2019-10-02 21:20:32 +00:00
Matt Morehouse e55c442b1b [sanitizer_common] Rename OnPrint to __sanitizer_on_print.
Summary:
https://reviews.llvm.org/D28596 exposed OnPrint in the global namespace,
which can cause collisions with user-defined OnPrint() functions.

Reviewers: vitalybuka, dvyukov

Reviewed By: vitalybuka, dvyukov

Subscribers: llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D67987

llvm-svn: 373518
2019-10-02 20:13:21 +00:00
Matt Morehouse 1c8e05110c [libFuzzer] Remove lazy counters.
Summary: Lazy counters haven't improved performance for large fuzz targets.

Reviewers: kcc

Reviewed By: kcc

Subscribers: llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D67476

llvm-svn: 373403
2019-10-01 22:49:06 +00:00
Vitaly Buka 3ee619c112 Revert "compiler-rt: use __GLIBC_PREREQ for SANITIZER_INTERCEPT_GETRANDOM"
Revert "compiler-rt: move all __GLIBC_PREREQ into own header file"

"move all __GLIBC_PREREQ" breaks build on some bots

This reverts commit 2d75ee9373.
This reverts commit 7a6461fcc2.

llvm-svn: 373367
2019-10-01 18:03:11 +00:00
Vitaly Buka 7a6461fcc2 compiler-rt: use __GLIBC_PREREQ for SANITIZER_INTERCEPT_GETRANDOM
Summary: Fixes https://github.com/google/oss-fuzz/issues/2836

Reviewers: eugenis

Subscribers: dberris, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D68178

llvm-svn: 373284
2019-10-01 00:58:29 +00:00
Vitaly Buka 2d75ee9373 compiler-rt: move all __GLIBC_PREREQ into own header file
Reviewers: eugenis

Subscribers: dberris, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D68177

llvm-svn: 373283
2019-10-01 00:58:28 +00:00
Vitaly Buka f74f5f2568 Rename tsan_interceptors.cpp into tsan_interceptors_posix.cpp
Summary:
It's needed to use __GLIBC_PREREQ from <features.h>

tsan didn't let us to include <features.h> by using --sysroot=. to disable system includes on
anything that is not named as "tsan*posix*", "tsan*mac*", "tsan*linux*".
See compiler-rt/lib/tsan/CMakeLists.txt

Reviewers: eugenis, dvyukov, kcc

Reviewed By: kcc

Subscribers: mgorny, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D68176

llvm-svn: 373282
2019-10-01 00:58:26 +00:00
Evgeniy Stepanov 72131161a4 [msan] Intercept __getrlimit.
Summary:
This interceptor is useful on its own, but the main purpose of this
change is to intercept libpthread initialization on linux/glibc in
order to run __msan_init before any .preinit_array constructors.

We used to trigger on pthread_initialize_minimal -> getrlimit(), but
that call has changed to __getrlimit at some point.

Reviewers: vitalybuka, pcc

Subscribers: jfb, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D68168

llvm-svn: 373239
2019-09-30 17:49:48 +00:00
Mitch Phillips d1e222e552 [libFuzzer] Dump trace and provide correct msg for overwritten input.
Summary:
Now crashes with a stacktrace and uses 'overwrites-const-input' as the error
message instead of 'out-of-memory'.

Reviewers: morehouse, Dor1s

Reviewed By: morehouse, Dor1s

Subscribers: #sanitizers, llvm-commits, metzman, Dor1s

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D68067

llvm-svn: 373130
2019-09-27 22:04:36 +00:00
Peter Collingbourne c336557f02 hwasan: Compatibility fixes for short granules.
We can't use short granules with stack instrumentation when targeting older
API levels because the rest of the system won't understand the short granule
tags stored in shadow memory.

Moreover, we need to be able to let old binaries (which won't understand
short granule tags) run on a new system that supports short granule
tags. Such binaries will call the __hwasan_tag_mismatch function when their
outlined checks fail. We can compensate for the binary's lack of support
for short granules by implementing the short granule part of the check in
the __hwasan_tag_mismatch function. Unfortunately we can't do anything about
inline checks, but I don't believe that we can generate these by default on
aarch64, nor did we do so when the ABI was fixed.

A new function, __hwasan_tag_mismatch_v2, is introduced that lets code
targeting the new runtime avoid redoing the short granule check. Because tag
mismatches are rare this isn't important from a performance perspective; the
main benefit is that it introduces a symbol dependency that prevents binaries
targeting the new runtime from running on older (i.e. incompatible) runtimes.

Differential Revision: https://reviews.llvm.org/D68059

llvm-svn: 373035
2019-09-27 01:02:10 +00:00
Mitch Phillips da3cf61654 [libFuzzer] [NFC] Fix grammar error with "it's"
llvm-svn: 372937
2019-09-26 00:54:30 +00:00
Joachim Protze 886a4ff977 [TSAN] Add read/write range interface functions with PC
Adding annotation function variants __tsan_write_range_pc and
__tsan_read_range_pc to annotate ranged access to memory while providing a
program counter for the access.

Differential Revision: https://reviews.llvm.org/D66885

llvm-svn: 372730
2019-09-24 11:19:02 +00:00
Dmitry Vyukov 88a5bba7b5 sanitizer_common: fix freebsd build error
Variable flags is not used. Remove it.

Suggested-by: randall77 (Keith Randall)
Review: https://reviews.llvm.org/D67928
llvm-svn: 372698
2019-09-24 08:27:51 +00:00
Rumeet Dhindsa 1605eb1c1c Fix int to bool errors exposed due to r372612.
Differential Revision: https://reviews.llvm.org/D67937
M    lib/builtins/fp_add_impl.inc
M    lib/builtins/fp_lib.h
M    lib/builtins/fp_trunc_impl.inc

llvm-svn: 372684
2019-09-24 02:59:02 +00:00
Jinsong Ji e2af0e5ee8 [compiler-rt] Fix lint check failure on comments
This fixes buildbot failures for https://reviews.llvm.org/rL372459.
(at least on PowerPC/Z )

The fix is generated by running clang-format on the error lines only.

llvm-svn: 372511
2019-09-22 15:31:03 +00:00
Kamil Rytarowski 1b58389428 Add __lsan::ScopedInterceptorDisabler for strerror(3)
Summary:
strerror(3) on NetBSD uses internally TSD with a destructor that is never
fired for exit(3). It's correctly called for pthread_exit(3) scenarios.

This is a case when a leak on exit(3) is expected, unavoidable and harmless.

Reviewers: joerg, vitalybuka, dvyukov, mgorny

Reviewed By: vitalybuka

Subscribers: dmgreen, kristof.beyls, jfb, llvm-commits, #sanitizers

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D67337

llvm-svn: 372461
2019-09-21 07:45:02 +00:00
Kamil Rytarowski 8827047551 Stop tracking atexit/__cxa_atexit/pthread_atfork allocations in LSan/NetBSD
Summary:
The atexit(3) and __cxa_atexit() calls allocate internally memory and free on exit,
after executing all callback. This causes false positives as DoLeakCheck() is called
from the atexit handler. In the LSan/ASan tests there are strict checks triggering
false positives here.

Intercept all atexit(3) and __cxa_atexit() calls and disable LSan when calling the
real functions.

Stop tracing allocations in pthread_atfork(3) funtions, as there are performed
internal allocations that are not freed for the time of running StopTheWorld()
code. This avoids false-positives.

The same changes have to be replicated in the ASan and LSan runtime.

Non-NetBSD OSs are not tested and this code is restricted to NetBSD only.

Reviewers: dvyukov, joerg, mgorny, vitalybuka, eugenis

Reviewed By: vitalybuka

Subscribers: jfb, llvm-commits, #sanitizers

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D67331

llvm-svn: 372459
2019-09-21 07:30:42 +00:00
Evgeniy Stepanov f1b6bd403d [lsan] Fix deadlock in dl_iterate_phdr.
Summary:
Do not grab the allocator lock before calling dl_iterate_phdr. This may
cause a lock order inversion with (valid) user code that uses malloc
inside a dl_iterate_phdr callback.

Reviewers: vitalybuka, hctim

Subscribers: jfb, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D67738

llvm-svn: 372348
2019-09-19 19:52:57 +00:00
Dmitry Vyukov d97865e530 tsan: allow the Go runtime to return multiple stack frames for a single PC
This fix allows tsan to report stack traces correctly even in the
presence of mid-stack inlining by the Go compiler.

See https://go-review.googlesource.com/c/go/+/195781 for the Go runtime side of this change.

Author: randall77 (Keith Randall)
Reviewed: https://reviews.llvm.org/D67671
llvm-svn: 372205
2019-09-18 09:18:04 +00:00
David Bolvansky 9a14ee81d2 Remove asan test for strncat(x, y, 0)
llvm-svn: 372143
2019-09-17 17:17:30 +00:00
David Bolvansky 5abd6f46ae [ASAN] Adjust asan tests due to new optimizations
llvm-svn: 372141
2019-09-17 17:07:31 +00:00
Jake Ehrlich 4b23c24bc8 [libFuzzer] Always print DSO map on Fuchsia libFuzzer launch
Fuchsia doesn't have /proc/id/maps, so it relies on the kernel logging system
to provide the DSO map to be able to symbolize in the context of ASLR. The DSO
map is logged automatically on Fuchsia when encountering a crash or writing to
the sanitizer log for the first time in a process. There are several cases
where libFuzzer doesn't encounter a crash, e.g. on timeouts, OOMs, and when
configured to print new PCs as they become covered, to name a few. Therefore,
this change always writes to the sanitizer log on startup to ensure the DSO map
is available in the log.

Author: aarongreen
Differential Revision: https://reviews.llvm.org/D66233

llvm-svn: 372056
2019-09-17 00:34:41 +00:00
Jian Cai 155a43edb0 [compiler-rt][crt] make test case nontrivial in check_cxx_section_exists
Summary:
.init_array gets optimized away when building with -O2 and as a result,
check_cxx_section_exists failed to pass -DCOMPILER_RT_HAS_INITFINI_ARRAY
when building crtbegin.o and crtend.o, which causes binaries linked with
them encounter segmentation fault. See https://crbug.com/855759 for
details. This change prevents .init_array section to be optimized away
even with -O2 or higher optimization level.

Subscribers: dberris, mgorny, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D67628

llvm-svn: 372038
2019-09-16 21:47:47 +00:00
Max Moroz d0f63f83e7 [libFuzzer] Remove unused version of FuzzedDataProvider.h.
Summary: The actual version lives in compiler-rt/include/fuzzer/.

Reviewers: Dor1s

Reviewed By: Dor1s

Subscribers: delcypher, #sanitizers, llvm-commits

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D67623

llvm-svn: 371997
2019-09-16 15:00:21 +00:00
Kamil Rytarowski 02519fc7a6 Add getauxval() compat for NetBSD
Summary:
getauxval() is not available on NetBSD and there is no a direct equivalent.

Add a function that implements the same semantics with NetBSD internals.

Reorder the GetPageSize() functions to prefer the sysctl approach for NetBSD.
It no longer makes a difference which approach is better. Avoid changing
conditional code path.

Reviewers: vitalybuka, dvyukov, mgorny, joerg

Reviewed By: vitalybuka

Subscribers: llvm-commits, #sanitizers

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D67329

llvm-svn: 371758
2019-09-12 18:57:58 +00:00
Vitaly Buka 23bbeb52f3 [compiler-rt] cpplint of inc files in background
llvm-svn: 371705
2019-09-12 02:20:37 +00:00
Vitaly Buka d2af368aee [compiler-rt] Remove some cpplint filters
llvm-svn: 371704
2019-09-12 02:20:36 +00:00
Vitaly Buka a3cd67c007 [compiler-rt] Better lint output for .inc files
llvm-svn: 371702
2019-09-12 01:35:09 +00:00
Vitaly Buka c0fa632236 Remove NOLINTs from compiler-rt
llvm-svn: 371687
2019-09-11 23:19:48 +00:00
Vitaly Buka 92002bd75b Fix mac build
llvm-svn: 371680
2019-09-11 22:19:18 +00:00
Vitaly Buka 48eb4a27d1 Update compiler-rt cpplint.py
adb3500107

llvm-svn: 371675
2019-09-11 21:33:06 +00:00
Kostya Kortchinsky 161cca266a [scudo][standalone] Android related improvements
Summary:
This changes a few things to improve memory footprint and performances
on Android, and fixes a test compilation error:
- add `stdlib.h` to `wrappers_c_test.cc` to address
  https://bugs.llvm.org/show_bug.cgi?id=42810
- change Android size class maps, based on benchmarks, to improve
  performances and lower the Svelte memory footprint. Also change the
  32-bit region size for said configuration
- change the `reallocate` logic to reallocate in place for sizes larger
  than the original chunk size, when they still fit in the same block.
  This addresses patterns from `memory_replay` dumps like the following:
```
202: realloc 0xb48fd000 0xb4930650 12352
202: realloc 0xb48fd000 0xb48fd000 12420
202: realloc 0xb48fd000 0xb48fd000 12492
202: realloc 0xb48fd000 0xb48fd000 12564
202: realloc 0xb48fd000 0xb48fd000 12636
202: realloc 0xb48fd000 0xb48fd000 12708
202: realloc 0xb48fd000 0xb48fd000 12780
202: realloc 0xb48fd000 0xb48fd000 12852
202: realloc 0xb48fd000 0xb48fd000 12924
202: realloc 0xb48fd000 0xb48fd000 12996
202: realloc 0xb48fd000 0xb48fd000 13068
202: realloc 0xb48fd000 0xb48fd000 13140
202: realloc 0xb48fd000 0xb48fd000 13212
202: realloc 0xb48fd000 0xb48fd000 13284
202: realloc 0xb48fd000 0xb48fd000 13356
202: realloc 0xb48fd000 0xb48fd000 13428
202: realloc 0xb48fd000 0xb48fd000 13500
202: realloc 0xb48fd000 0xb48fd000 13572
202: realloc 0xb48fd000 0xb48fd000 13644
202: realloc 0xb48fd000 0xb48fd000 13716
202: realloc 0xb48fd000 0xb48fd000 13788
...
```
  In this situation we were deallocating the old chunk, and
  allocating a new one for every single one of those, but now we can
  keep the same chunk (we just updated the header), which saves some
  heap operations.

Reviewers: hctim, morehouse, vitalybuka, eugenis, cferris, rengolin

Reviewed By: morehouse

Subscribers: srhines, delcypher, #sanitizers, llvm-commits

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D67293

llvm-svn: 371628
2019-09-11 14:48:41 +00:00
Max Moroz f054067f27 [libFuzzer] Make -merge=1 to reuse coverage information from the control file.
Summary:
This change allows to perform corpus merging in two steps. This is useful when
the user wants to address the following two points simultaneously:

1) Get trustworthy incremental stats for the coverage and corpus size changes
    when adding new corpus units.
2) Make sure the shorter units will be preferred when two or more units give the
    same unique signal (equivalent to the `REDUCE` logic).

This solution was brainstormed together with @kcc, hopefully it looks good to
the other people too. The proposed use case scenario:

1) We have a `fuzz_target` binary and `existing_corpus` directory.
2) We do fuzzing and write new units into the `new_corpus` directory.
3) We want to merge the new corpus into the existing corpus and satisfy the
    points mentioned above.
4) We create an empty directory `merged_corpus` and run the first merge step:

    `
    ./fuzz_target -merge=1 -merge_control_file=MCF ./merged_corpus ./existing_corpus
    `

    this provides the initial stats for `existing_corpus`, e.g. from the output:

    `
    MERGE-OUTER: 3 new files with 11 new features added; 11 new coverage edges
    `

5) We recreate `merged_corpus` directory and run the second merge step:

    `
    ./fuzz_target -merge=1 -merge_control_file=MCF ./merged_corpus ./existing_corpus ./new_corpus
    `

    this provides the final stats for the merged corpus, e.g. from the output:

    `
    MERGE-OUTER: 6 new files with 14 new features added; 14 new coverage edges
    `

Alternative solutions to this approach are:

A) Store precise coverage information for every unit (not only unique signal).
B) Execute the same two steps without reusing the control file.

Either of these would be suboptimal as it would impose an extra disk or CPU load
respectively, which is bad given the quadratic complexity in the worst case.

Tested on Linux, Mac, Windows.

Reviewers: morehouse, metzman, hctim, kcc

Reviewed By: morehouse

Subscribers: JDevlieghere, delcypher, mgrang, #sanitizers, llvm-commits, kcc

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D66107

llvm-svn: 371620
2019-09-11 14:11:08 +00:00
Dmitri Gribenko 57256af307 Revert "clang-misexpect: Profile Guided Validation of Performance Annotations in LLVM"
This reverts commit r371584. It introduced a dependency from compiler-rt
to llvm/include/ADT, which is problematic for multiple reasons.

One is that it is a novel dependency edge, which needs cross-compliation
machinery for llvm/include/ADT (yes, it is true that right now
compiler-rt included only header-only libraries, however, if we allow
compiler-rt to depend on anything from ADT, other libraries will
eventually get used).

Secondly, depending on ADT from compiler-rt exposes ADT symbols from
compiler-rt, which would cause ODR violations when Clang is built with
the profile library.

llvm-svn: 371598
2019-09-11 09:16:17 +00:00
Petr Hosek 394a8ed8f1 clang-misexpect: Profile Guided Validation of Performance Annotations in LLVM
This patch contains the basic functionality for reporting potentially
incorrect usage of __builtin_expect() by comparing the developer's
annotation against a collected PGO profile. A more detailed proposal and
discussion appears on the CFE-dev mailing list
(http://lists.llvm.org/pipermail/cfe-dev/2019-July/062971.html) and a
prototype of the initial frontend changes appear here in D65300

We revised the work in D65300 by moving the misexpect check into the
LLVM backend, and adding support for IR and sampling based profiles, in
addition to frontend instrumentation.

We add new misexpect metadata tags to those instructions directly
influenced by the llvm.expect intrinsic (branch, switch, and select)
when lowering the intrinsics. The misexpect metadata contains
information about the expected target of the intrinsic so that we can
check against the correct PGO counter when emitting diagnostics, and the
compiler's values for the LikelyBranchWeight and UnlikelyBranchWeight.
We use these branch weight values to determine when to emit the
diagnostic to the user.

A future patch should address the comment at the top of
LowerExpectIntrisic.cpp to hoist the LikelyBranchWeight and
UnlikelyBranchWeight values into a shared space that can be accessed
outside of the LowerExpectIntrinsic pass. Once that is done, the
misexpect metadata can be updated to be smaller.

In the long term, it is possible to reconstruct portions of the
misexpect metadata from the existing profile data. However, we have
avoided this to keep the code simple, and because some kind of metadata
tag will be required to identify which branch/switch/select instructions
are influenced by the use of llvm.expect

Patch By: paulkirth
Differential Revision: https://reviews.llvm.org/D66324

llvm-svn: 371584
2019-09-11 01:09:16 +00:00
Max Moroz 9508738cd1 [UBSan] Do not overwrite the default print_summary sanitizer option.
Summary:
This option is true by default in sanitizer common. The default
false value was added a while ago without any reasoning in
524e934112

so, presumably it's safe to remove for consistency.

Reviewers: hctim, samsonov, morehouse, kcc, vitalybuka

Reviewed By: hctim, samsonov, vitalybuka

Subscribers: delcypher, #sanitizers, llvm-commits, kcc

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D67193

llvm-svn: 371442
2019-09-09 19:30:48 +00:00
Julian Lettner fc910c507e [TSan] Add interceptors for mach_vm_[de]allocate
I verified that the test is red without the interceptors.

rdar://40334350

Reviewed By: kubamracek, vitalybuka

Differential Revision: https://reviews.llvm.org/D66616

llvm-svn: 371439
2019-09-09 18:57:32 +00:00
Ed Maste 1a3dd638c4 compiler-rt: use fp_t instead of long double, for consistency
Most builtins accepting or returning long double use the fp_t typedef.
Change the remaining few cases to do so.

Differential Revision:	https://reviews.llvm.org/D35034

llvm-svn: 371400
2019-09-09 13:50:20 +00:00
Kamil Rytarowski c8afbf3d95 Do not intercept malloc_usable_size on NetBSD
llvm-svn: 371334
2019-09-08 15:00:13 +00:00
Yi Kong 33b8a55329 Revert "Revert "[builtins] Rounding mode support for addxf3/subxf3""
Test failure fixed.

This reverts commit e204d244ba.

llvm-svn: 371003
2019-09-05 01:05:05 +00:00
Craig Topper 5465875e93 [X86] Add support for avx512bf16 for __builtin_cpu_supports and compiler-rt's cpu indicator.
llvm-svn: 370915
2019-09-04 16:01:43 +00:00
Ed Maste 1b6ee802c6 compiler-rt: use 64-bit time_t for all FreeBSD archs except i386
At present only i386 has 32-bit time_t on FreeBSD.

Reviewed by:	dim
Differential Revision:	https://reviews.llvm.org/D66758

llvm-svn: 370756
2019-09-03 13:44:47 +00:00
Ed Maste 14cf2b20ca compiler-rt: use more __sanitizer_time_t on FreeBSD
A few structs were using long for time_t members.  NFC.

Reviewed by:	devnexen
Differential Revision:	https://reviews.llvm.org/D66756

llvm-svn: 370755
2019-09-03 13:41:51 +00:00
Vitaly Buka 342197e6fc [sanitizer] Support monorepo layout in symbolizer build script
llvm-svn: 370288
2019-08-29 02:02:03 +00:00
Vitaly Buka f6b9126890 [sanitizer] Simplify COMPILER_RT setup in lint script
llvm-svn: 370277
2019-08-28 23:52:21 +00:00
Julian Lettner d3136661ba [sanitizer_common] Close superfluous file descriptors in spawned process
Use attribute flag `POSIX_SPAWN_CLOEXEC_DEFAULT` in the call to
`posix_spawn`.

If this flag is set, then only file descriptors explicitly described by
the file_actions argument are available in the spawned process; all of
the other file descriptors are automatically closed in the spawned
process.

POSIX_SPAWN_CLOEXEC_DEFAULT is an Apple-specific extension.

llvm-svn: 370121
2019-08-27 22:12:26 +00:00
Mitch Phillips ae56e593b9 Add GWP-ASan fuzz target to compiler-rt/tools.
Summary:
@eugenis to approve addition of //compiler-rt/tools.
@pree-jackie please confirm that this WFY.

D66494 introduced the GWP-ASan stack_trace_compressor_fuzzer. Building fuzz
targets in compiler-rt is a new affair, and has some challenges:
- If the host compiler doesn't have compiler-rt, the -fsanitize=fuzzer may not
  be able to link against `libclang_rt.fuzzer*`.
- Things in compiler-rt generally aren't built when you want to build with
  sanitizers using `-DLLVM_USE_SANITIZER`. This tricky to work around, so
  we create the new tools directory so that we can build fuzz targets with
  sanitizers. This has the added bonus of fixing the problem above as well, as
  we can now just guard the fuzz target build to only be done with
  `-DLLVM_USE_SANITIZE_COVERAGE=On`.

Reviewers: eugenis, pree-jackie

Reviewed By: eugenis, pree-jackie

Subscribers: dberris, mgorny, #sanitizers, llvm-commits, eugenis, pree-jackie, lebedev.ri, vitalybuka, morehouse

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D66776

llvm-svn: 370094
2019-08-27 18:28:07 +00:00
Alexander Richardson 677c6ddd33 [asan_symbolize] Fix broken pipe handling for python 2.7
I D65322 I added a check for BrokenPipeError. However, python 2.7 doesn't
have BrokenPipeError. To be python 2.7 and 3 compatible we need to catch
IOError instead and check for errno == errno.EPIPE.

llvm-svn: 370025
2019-08-27 07:27:41 +00:00
Vitaly Buka 09fcec7c0a NFC: clang-format r370008 to suppress lint errors
llvm-svn: 370023
2019-08-27 06:59:57 +00:00
David Carlier 5058dd0f49 Fix buildbot
llvm-svn: 370011
2019-08-27 04:24:19 +00:00
David Carlier 4797f68b15 [Sanitizer] Using huge page on FreeBSD for shadow mapping
- Unless explicit configuration, using FreeBSD super pages feature for shadow mapping.
- asan only for now.

Reviewers: dim, emaste, vitalybuka

Reviewed By: vitalybuka

Differential Revision: https://reviews.llvm.org/D65851

llvm-svn: 370008
2019-08-27 04:02:19 +00:00
Alexander Richardson 2511b5a463 [asan_symbolize] Attempt to fix build-bot failure after latest change
llvm-svn: 369929
2019-08-26 17:11:25 +00:00
Alexander Richardson e320db434e [asan_symbolize] Avoid blocking when llvm-symbolizer is installed as addr2line
Summary:
Currently, llvm-symbolizer will print -1 when presented with -1 and not
print a second line. In that case we will block for ever trying to read
the file name. This also happens for non-existent files, in which case GNU
addr2line exits immediate, but llvm-symbolizer does not (see
https://llvm.org/PR42754). While touching these lines, I also added some
more debug logging to help diagnose this and potential future issues.

Reviewers: kcc, eugenis, glider, samsonov

Reviewed By: eugenis

Subscribers: kubamracek, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D65322

llvm-svn: 369924
2019-08-26 16:22:04 +00:00
Bjorn Pettersson 140f06f1b5 [GWP_ASAN] Avoid using VERSION_GREATER_EQUAL in cmake files
This is a fixup for r369823 which introduced the use of
VERSION_GREATER_EQUAL in the cmake config for gwp_asan.

Minimum supported version of cmake in LLVM is 3.4.3 and
VERSION_GREATER_EQUAL was not introduced until later
versions of cmake.

llvm-svn: 369891
2019-08-26 11:02:36 +00:00
Peter Collingbourne b2e7b85468 hwasan: Align n_namesz and n_descsz to 4 when reading notes.
There is no requirement for the producer of a note to include the note
alignment in these fields. As a result we can end up missing the HWASAN note
if one of the other notes in the binary has the alignment missing.

Differential Revision: https://reviews.llvm.org/D66692

llvm-svn: 369826
2019-08-23 23:33:26 +00:00
Mitch Phillips 27d69b2f4f [GWP-ASan] Split options_parser and backtrace_sanitizer_common.
Summary:
optional/options_parser and optional/backtrace_sanitizer_common are logically
separate components. They both use sanitizer-common to power their
functionality, but there was an unstated implicit dependency that in order for
backtrace_sanitizer_common to function correctly, one had to also use
options_parser.

This was because options_parser called __sanitizer::InitialiseCommonFlags. This
is a requirement for backtrace_sanitizer_common to work, as the sanitizer
unwinder uses the sanitizer_common flags and will SEGV on a null page if
they're not initialised correctly.

This patch removes this hidden dependency. You can now use
backtrace_sanitizer_common without the requirements of options_parser.

This patch also makes the GWP-ASan unit tests only have a soft dependency on
sanitizer-common. The unit tests previously explicitly used
__sanitizer::Printf, which is now provided under
tests/optional/printf_sanitizer_common. This allows Android to build the unit
tests using their own signal-safe printf().

Reviewers: eugenis

Reviewed By: eugenis

Subscribers: srhines, mgorny, #sanitizers, llvm-commits, vlad.tsyrklevich, morehouse

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D66684

llvm-svn: 369825
2019-08-23 23:23:48 +00:00
Mitch Phillips 05bc1468c2 Fix stack_trace_compressor builds for Clang < 6.0
Summary:
Clang 4.* doesn't supply -fsanitize=fuzzer, and Clang 5.* doesn't supply
-fsanitize=fuzzer-no-link. Generally, in LLVM, fuzz targets are added through
the add_llvm_fuzzer build rule, which can't be used in compiler-rt (as it has
to be able to be standalone built).

Instead of adding tooling to add a dummy main (which kind of defeats the
purpose of these fuzz targets), we instead build the fuzz target only when the
Clang version is >= 6.*.

Reviewers: tejohnson

Subscribers: mgorny, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D66682

llvm-svn: 369823
2019-08-23 23:13:18 +00:00
Peter Collingbourne f7ca57468a Move a break into the correct place. NFCI.
Should silence new C fallthrough warning.

llvm-svn: 369813
2019-08-23 21:27:56 +00:00
Peter Collingbourne 21a1814417 hwasan: Untag unwound stack frames by wrapping personality functions.
One problem with untagging memory in landing pads is that it only works
correctly if the function that catches the exception is instrumented.
If the function is uninstrumented, we have no opportunity to untag the
memory.

To address this, replace landing pad instrumentation with personality function
wrapping. Each function with an instrumented stack has its personality function
replaced with a wrapper provided by the runtime. Functions that did not have
a personality function to begin with also get wrappers if they may be unwound
past. As the unwinder calls personality functions during stack unwinding,
the original personality function is called and the function's stack frame is
untagged by the wrapper if the personality function instructs the unwinder
to keep unwinding. If unwinding stops at a landing pad, the function is
still responsible for untagging its stack frame if it resumes unwinding.

The old landing pad mechanism is preserved for compatibility with old runtimes.

Differential Revision: https://reviews.llvm.org/D66377

llvm-svn: 369721
2019-08-23 01:28:44 +00:00
Taewook Oh e03f34dd09 [sanitizer] Resubmit D66620 from monorepo
Summary: https://reviews.llvm.org/D66620 is accepted but was based on the multi-repo setup, so I was not able to `arc patch` it. Resubmit the diff under monorepo

Committed on behalf of @sugak (Igor Sugak)

Reviewers: sugak

Subscribers: #sanitizers, llvm-commits, vitalybuka

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D66624

llvm-svn: 369716
2019-08-22 23:34:08 +00:00
David Carlier f2003f4cb8 Fixing buildbot due to style.
llvm-svn: 369711
2019-08-22 21:53:21 +00:00
David Carlier e2ed800d62 [Sanitizer] checks ASLR on FreeBSD
- Especially MemorySanitizer fails if those sysctl configs are enabled.

Reviewers: vitalybuka, emaste, dim

Reviewed By: dim

Differential Revision: https://reviews.llvm.org/D66582

llvm-svn: 369708
2019-08-22 21:36:35 +00:00
Matthew G McGovern 2eaeba6f15 LibFuzzer support for 32bit MSVC
This fixes the two build errors when trying to compile LibFuzzer for
32bit with MSVC.

    - authored by Max Shavrick (mxms at microsoft)

llvm-svn: 369704
2019-08-22 20:44:34 +00:00
Petr Hosek 028b5499ff Revert "[GWP-ASan] Remove c++ standard lib dependency."
This reverts commit r369606: this doesn't addressed the underlying
problem and it's not the correct solution.

llvm-svn: 369623
2019-08-22 07:03:38 +00:00
Petr Hosek 3dd2258519 [GWP-ASan] Remove c++ standard lib dependency.
Remove c++ standard library dependency for now for @phosek. They have a
complicated build system that breaks with the fuzzer target here.

Also added a todo to remedy later.

Differential Revision: https://reviews.llvm.org/D66568

llvm-svn: 369606
2019-08-22 00:22:56 +00:00
Julian Lettner 894abb46f8 [TSan] #include header instead of forward declaring interceptees
llvm-svn: 369601
2019-08-21 23:42:06 +00:00
Mitch Phillips c776f3f3c2 [GWP-ASan] Add public-facing documentation [6].
Summary:
Note: Do not submit this documentation until Scudo support is reviewed and submitted (should be #[5]).

See D60593 for further information.

This patch introduces the public-facing documentation for GWP-ASan, as well as updating the definition of one of the options, which wasn't properly merged. The document describes the design and features of GWP-ASan, as well as how to use GWP-ASan from both a user's standpoint, and development documentation for supporting allocators.

Reviewers: jfb, morehouse, vlad.tsyrklevich

Reviewed By: morehouse, vlad.tsyrklevich

Subscribers: kcc, dexonsmith, kubamracek, cryptoad, jfb, #sanitizers, llvm-commits, vlad.tsyrklevich, morehouse

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D62875

llvm-svn: 369552
2019-08-21 17:53:51 +00:00
Mitch Phillips f9269b5396 [GWP-ASan] Build stack_trace_compressor_fuzzer.
Summary:
Flips the switch to build stack_trace_compressor_fuzzer. This was recently
temporarily disabled in rL369079 as it was breaking the sanitizer buildbots.

My diagnosis of the problem is that on clang-only bootstrap builds, we build
gwp_asan before libfuzzer. This causes a discrepancy when the clang driver
attempts to link libclang_rt.fuzzer* as CMake doesn't see a dependency there.

I've (hopefully) fixed the issue by adding a direct dependency for the fuzz
target so CMake can resolve the build order properly. As part of this, the
libFuzzer 'fuzzer' target has to be discovered before the declaration of the
fuzz target.

pcc@ for mild review + notification as buildcop.

Reviewers: pcc

Reviewed By: pcc

Subscribers: mgorny, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D66494

llvm-svn: 369551
2019-08-21 17:52:51 +00:00
Nico Weber 98f800dabe compiler-rt: Fix warning if COMPILER_RT_HAS_FCNTL_LCK is 0
Fixes "warning: implicit declaration of function 'flock' is invalid in C99"
for flock().

llvm-svn: 369534
2019-08-21 15:24:25 +00:00
Sam McCall a451156bb6 reland [gtest] Fix printing of StringRef and SmallString in assert messages.
Renames GTEST_NO_LLVM_RAW_OSTREAM -> GTEST_NO_LLVM_SUPPORT and guards
the new features behind it.

This reverts commit a063bcf3ef5a879adbe9639a3c187d876eee0e66.

llvm-svn: 369527
2019-08-21 13:56:29 +00:00
Vitaly Buka 93a3cbc746 Revert r369472 and r369441
check-sanitizer does not work on Linux

llvm-svn: 369495
2019-08-21 05:06:21 +00:00
Sebastian Pop 5a7bba09ac [AArch64][asan] fix typo in AsanStats::Print
This created an infinite loop that timed out several build bots while
executing the test in compiler-rt/test/asan/TestCases/atexit_stats.cpp

Differential Revision: https://reviews.llvm.org/D60243

llvm-svn: 369472
2019-08-20 23:28:05 +00:00
Sebastian Pop 63487bfec9 [AArch64] Speed-up leak and address sanitizers on AArch64 for 48-bit VMA
This patch fixes https://github.com/google/sanitizers/issues/703
On a Graviton-A1 aarch64 machine with 48-bit VMA,
the time spent in LSan and ASan reduced from 2.5s to 0.01s when running

clang -fsanitize=leak compiler-rt/test/lsan/TestCases/sanity_check_pure_c.c && time ./a.out
clang -fsanitize=address compiler-rt/test/lsan/TestCases/sanity_check_pure_c.c && time ./a.out

With this patch, LSan and ASan create both the 32 and 64 allocators and select
at run time between the two allocators following a global variable that is
initialized at init time to whether the allocator64 can be used in the virtual
address space.

Differential Revision: https://reviews.llvm.org/D60243

llvm-svn: 369441
2019-08-20 20:54:05 +00:00
Mitch Phillips 8f5e1755ca [GWP-ASan] Fix typos.
Summary:
Fix two spelling typos and de-indent a guarded #define so that it's
consistent with clang-format.

Reviewers: vitalybuka

Reviewed By: vitalybuka

Subscribers: #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D66311

llvm-svn: 369433
2019-08-20 20:16:11 +00:00
Kostya Kortchinsky 3e5360f194 [scudo][standalone] Fix malloc_iterate
Summary:
cferris's Bionic tests found an issue in Scudo's `malloc_iterate`.

We were inclusive of both boundaries, which resulted in a `Block` that
was located on said boundary to be possibly accounted for twice, or
just being accounted for while iterating on regions that are not ours
(usually the unmapped ones in between Primary regions).

The fix is to exclude the upper boundary in `iterateOverChunks`, and
add a regression test.

This additionally corrects a typo in a comment, and change the 64-bit
Primary iteration function to not assume that `BatchClassId` is 0.

Reviewers: cferris, morehouse, hctim, vitalybuka, eugenis

Reviewed By: hctim

Subscribers: delcypher, #sanitizers, llvm-commits

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D66231

llvm-svn: 369400
2019-08-20 16:17:08 +00:00
Hans Wennborg 9e8b011195 [compiler-rt][crt] Pass -fno-lto in check_cxx_section_exists
Otherwise it doesn't work when building with -DLLVM_ENABLE_LTO=thin

(We hit this in Chromium in
https://bugs.chromium.org/p/chromium/issues/detail?id=966403)

llvm-svn: 369336
2019-08-20 07:41:14 +00:00
Julian Lettner def061e6f0 [TSan] Rename file with libdispatch interceptors
llvm-svn: 369314
2019-08-20 00:04:19 +00:00
Peter Collingbourne 4340c2cf8f Build symbolizer runtime with C++14.
Should hopefully fix sanitizer-x86_64-linux bot.

llvm-svn: 369290
2019-08-19 18:43:52 +00:00
Julian Lettner e644d5e230 [TSan] Rename file to make it clear that it defines interceptors
Rename file `tsan_libdispatch.cpp -> tsan_libdispatch_interceptors.cpp`
to make it clear that it's main purpose is defining interceptors.

llvm-svn: 369289
2019-08-19 18:41:20 +00:00
David Carlier 949f190810 [Sanitizer] arc4random interception on Mac
Reviewers: yln,vitalybuka

Reviewed By: yln

Differential Revision: https://reviews.llvm.org/D66391

llvm-svn: 369285
2019-08-19 18:12:15 +00:00
Petr Hosek edf504f503 [Fuchsia] Create the VMO during initialization, not during exit
We want to avoid doing expensive work during atexit since the process
might be terminated before we can publish the VMO and write out the
symbolizer markup, so move the VMO creation to the initialization
phase and only write data during the atexit phase.

Differential Revision: https://reviews.llvm.org/D66323

llvm-svn: 369180
2019-08-17 00:54:22 +00:00
Julian Lettner 0c2f26d664 Revert "[TSan] Don't guard #include <xpc/xpc.h>"
This reverts commit 8191585b36.

llvm-svn: 369165
2019-08-16 22:58:13 +00:00
Julian Lettner 4a9b747bfb [TSan] Add interceptors for os_unfair_lock
llvm-svn: 369164
2019-08-16 22:41:25 +00:00
Julian Lettner 8191585b36 [TSan] Don't guard #include <xpc/xpc.h>
The xpc_connection_* APIs that we are intercepting are available
starting at macOS 10.7. This is old enough so that we don't need to
guard them.

llvm-svn: 369150
2019-08-16 20:43:09 +00:00
Mitch Phillips effc28c139 Disable stack_trace_compressor_fuzzer.
Should hopefully fix the remainder of the buildbot issues. Just disabling this
for now with a comment that I'm working on it. Can actually fix the real problem
when I'm at a real computer.

llvm-svn: 369079
2019-08-16 02:03:33 +00:00
Mitch Phillips 4660ea953e Moved binary off add_llvm_executable.
Used add_executable instead, as this allows a standalone compiler-rt to build,
as the add_llvm_executable build target isn't accessible in a standalone CRT
preparation.

llvm-svn: 369071
2019-08-15 23:50:05 +00:00
Mitch Phillips 4be5d53a33 Re-instate 369051.
Looks like I accidentally reverted r369051 to the old
CMake-version-specific flag when committing 369055.

llvm-svn: 369067
2019-08-15 23:28:33 +00:00
Mitch Phillips 2fb4e754c6 Guard fuzzer build behind Clang-only flags.
Should fix sanitizer buildbots and any one else who's building
compiler-rt using gcc.

llvm-svn: 369055
2019-08-15 22:15:46 +00:00
Mitch Phillips fdee340bea Remove CMake >= v3.13 target_link_options.
Instead, use set_target_properties.

llvm-svn: 369051
2019-08-15 21:34:13 +00:00
Mitch Phillips be8a2f7565 [GWP-ASan] Implement stack frame compression.
Summary:
This patch introduces stack frame compression to GWP-ASan. Each stack frame is
variable-length integer encoded as the difference between frame[i] and
frame[i - 1]. Furthermore, we use zig-zag encoding on the difference to ensure
that negative differences are also encoded into a relatively small number of
bytes.

Examples of what the compression looks like can be seen in
`gwp_asan/tests/compression.cpp`.

This compression can reduce the memory consumption cost of stack traces by
~50%.

Reviewers: vlad.tsyrklevich

Reviewed By: vlad.tsyrklevich

Subscribers: mgorny, #sanitizers, llvm-commits, eugenis, morehouse

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D66189

llvm-svn: 369048
2019-08-15 21:09:09 +00:00
Julian Lettner bb99437f6d [sanitizer_common] Always use posix_spawn on Darwin
On Darwin we have two external symbolizers: atos and llvm-symbolizer.
atos was changed to use posix_spawn (instead of fork+execv) in a
previous commit [1]. Let's use posix_spawn for llvm-symbolizer as well.

Our hope is that eventually we can transition to posix_spawn on other
platforms too.

[1] 399408a92f

llvm-svn: 369021
2019-08-15 17:25:10 +00:00
Jonas Devlieghere 8903433bb2 Re-land "[compiler-rt] Migrate llvm::make_unique to std::make_unique"
With the compiler-rt check for C++14 updated in r368960, this should now
be fine to land.

llvm-svn: 369009
2019-08-15 14:57:44 +00:00
Jonas Devlieghere d65f37dbcc Revert "[compiler-rt] Migrate llvm::make_unique to std::make_unique"
The X-ray unit tests in compiler-rt are overriding the C++ version by
explicitly passing -std=c++11 in the compiler invocation. This poses a
problem as these tests are including LLVM headers that can now use C++14
features. I'm temporarily reverting this as I investigate the correct
solution.

llvm-svn: 368952
2019-08-15 03:02:57 +00:00
Julian Lettner 399408a92f [sanitizer_common] Replace forkpty with posix_spawn on Darwin
On Darwin, we currently use forkpty to communicate with the "atos"
symbolizer. There are several problems that fork[pty] has, e.g. that
after fork, interceptors are still active and this sometimes causes
crashes or hangs. This is especially problematic for TSan, which uses
interceptors for OS-provided locks and mutexes, and even Libc functions
use those.

This patch replaces forkpty with posix_spawn on Darwin. Since
posix_spawn doesn't fork (at least on Darwin), the interceptors are not
a problem. Another benefit is that we'll handle post-fork failures (e.g.
sandbox disallows "exec") gracefully now.

Related revisions and previous attempts that were blocked by or had to
be revered due to test failures:
https://reviews.llvm.org/D48451
https://reviews.llvm.org/D40032

Reviewed By: kubamracek

Differential Revision: https://reviews.llvm.org/D65253

llvm-svn: 368947
2019-08-15 00:18:55 +00:00
Jonas Devlieghere 5edd6843fa [compiler-rt] Migrate llvm::make_unique to std::make_unique
Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.

Differential revision: https://reviews.llvm.org/D66259

llvm-svn: 368946
2019-08-15 00:06:49 +00:00
Kostya Kortchinsky 2be59170d4 [scudo][standalone] Add more stats to mallinfo
Summary:
Android requires additional stats in mallinfo. While we can provide
right away the number of bytes mapped (Primary+Secondary), there was
no way to get the number of free bytes (only makes sense for the
Primary since the Secondary unmaps everything on deallocation).

An approximation could be `StatMapped - StatAllocated`, but since we
are mapping in `1<<17` increments for the 64-bit Primary, it's fairly
inaccurate.

So we introduce `StatFree` (note it's `Free`, not `Freed`!), which
keeps track of the amount of Primary blocks currently unallocated.

Reviewers: cferris, eugenis, vitalybuka, hctim, morehouse

Reviewed By: morehouse

Subscribers: delcypher, #sanitizers, llvm-commits

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D66112

llvm-svn: 368866
2019-08-14 16:04:01 +00:00
Mitch Phillips 352d1b59c0 [GWP-ASan] Update backtrace function signature.
Summary:
Updates the function signature and comments for backtracing (and printing
backtraces). This update brings GWP-ASan in line with future requirements for
stack frame compression, wherein the length of the trace is provided
explicitly, rather than relying on nullptr-termination.

Reviewers: vlad.tsyrklevich

Reviewed By: vlad.tsyrklevich

Subscribers: #sanitizers, llvm-commits, morehouse

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D66099

llvm-svn: 368619
2019-08-12 21:36:44 +00:00
Max Moroz 74cec618f3 [libFuzzer] Merge: print feature coverage number as well.
Summary:
feature coverage is a useful signal that is available during the merge
process, but was not printed previously.

Output example:

```
$ ./fuzzer -use_value_profile=1 -merge=1 new_corpus/ seed_corpus/
INFO: Seed: 1676551929
INFO: Loaded 1 modules   (2380 inline 8-bit counters): 2380 [0x90d180, 0x90dacc), 
INFO: Loaded 1 PC tables (2380 PCs): 2380 [0x684018,0x68d4d8), 
MERGE-OUTER: 180 files, 78 in the initial corpus
MERGE-OUTER: attempt 1
INFO: Seed: 1676574577
INFO: Loaded 1 modules   (2380 inline 8-bit counters): 2380 [0x90d180, 0x90dacc), 
INFO: Loaded 1 PC tables (2380 PCs): 2380 [0x684018,0x68d4d8), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
MERGE-INNER: using the control file '/tmp/libFuzzerTemp.111754.txt'
MERGE-INNER: 180 total files; 0 processed earlier; will process 180 files now
#1	pulse  cov: 134 ft: 330 exec/s: 0 rss: 37Mb
#2	pulse  cov: 142 ft: 462 exec/s: 0 rss: 38Mb
#4	pulse  cov: 152 ft: 651 exec/s: 0 rss: 38Mb
#8	pulse  cov: 152 ft: 943 exec/s: 0 rss: 38Mb
#16	pulse  cov: 520 ft: 2783 exec/s: 0 rss: 39Mb
#32	pulse  cov: 552 ft: 3280 exec/s: 0 rss: 41Mb
#64	pulse  cov: 576 ft: 3641 exec/s: 0 rss: 50Mb
#78	LOADED cov: 602 ft: 3936 exec/s: 0 rss: 88Mb
#128	pulse  cov: 611 ft: 3996 exec/s: 0 rss: 93Mb
#180	DONE   cov: 611 ft: 4016 exec/s: 0 rss: 155Mb
MERGE-OUTER: succesfull in 1 attempt(s)
MERGE-OUTER: the control file has 39741 bytes
MERGE-OUTER: consumed 0Mb (37Mb rss) to parse the control file
MERGE-OUTER: 9 new files with 80 new features added; 9 new coverage edges
```

Reviewers: hctim, morehouse

Reviewed By: morehouse

Subscribers: delcypher, #sanitizers, llvm-commits, kcc

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D66030

llvm-svn: 368617
2019-08-12 20:21:27 +00:00
Nico Weber d2e493c337 Fix Wnewline-eof after r368598
llvm-svn: 368613
2019-08-12 19:57:17 +00:00
Dan Liew c3b93bed29 [asan_symbolize] Fix bug where the frame counter was not incremented.
Summary:
This bug occurred when a plug-in requested that a binary not be
symbolized while the script is trying to symbolize a stack frame. In
this case `self.frame_no` would not be incremented. This would cause
subsequent stack frames that are symbolized to be incorrectly numbered.

To fix this `get_symbolized_lines()` has been modified to take an
argument that indicates whether the stack frame counter should
incremented. In `process_line_posix()` `get_symbolized_lines(None, ...)`
is now used in in the case where we don't want to symbolize a line so
that we can keep the frame counter increment in a single function.

A test case is included. The test uses a dummy plugin that always asks
`asan_symbolize.py` script to not symbolize the first binary that the
script asks about. Prior to the patch this would cause the output to
script to look something like

```
  #0 0x0
  #0 0x0 in do_access
  #1 0x0 in main
```

This is the second attempt at landing this patch. The first (r368373)
failed due to failing some android bots and so was reverted in r368472.
The new test is now disabled for Android. It turns out that the patch
also fails for iOS too so it is also disabled for that family of
platforms too.

rdar://problem/49476995

Reviewers: kubamracek, yln, samsonov, dvyukov, vitalybuka

Subscribers: #sanitizers, llvm-commits

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D65495

llvm-svn: 368603
2019-08-12 18:51:25 +00:00
Matthew G McGovern 38a1aa117f [builtins] MSVC warning disable for clean build
- https://reviews.llvm.org/D66023
    - amended for ifdef/if gcc errors in previous verison

llvm-svn: 368598
2019-08-12 18:08:44 +00:00
Kostya Kortchinsky 8095449e68 [scudo][standalone] Minor corrections
Summary:
Few corrections with no functional change:
- replacing `%zd` with `%zu` all around: the values are unsigned
- prefer `MAP_ANONYMOUS` to `MAP_ANON` (it's deprecated)
- remove the unused `enum LinkerInitialized`
- mark a parameter as `UNUSED` in Fuchsia's `getRandom`
- correct the casing of a variable and use `nullptr` instead of 0 for
  pointers in `list.h`
- reorder some `typedef` to be consistent between `signed` and
  `unsigned`

Reviewers: eugenis, vitalybuka, morehouse, hctim

Reviewed By: vitalybuka, morehouse

Subscribers: delcypher, #sanitizers, llvm-commits

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D65660

llvm-svn: 368585
2019-08-12 15:35:09 +00:00
David Carlier 8d1646bf66 [Sanitizer] Reenable getusershell interception
and disabling it forAndroid.

Reviewers: krytarowski, vitalybuka

Reviewed By: krytarowski

Differential Revision: https://reviews.llvm.org/D66027

llvm-svn: 368504
2019-08-10 03:35:54 +00:00
Julian Lettner 5ef4b190d9 [Sanitizer][Darwin] Add interceptor for malloc_zone_from_ptr
Ensure that malloc_default_zone and malloc_zone_from_ptr return the
sanitizer-installed malloc zone even when MallocStackLogging (MSL) is
requested. This prevents crashes in certain situations. Note that the
sanitizers and MSL cannot be used together. If both are enabled, MSL
functionality is essentially deactivated since it only hooks the default
allocator which is replaced by a custom sanitizer allocator.

rdar://53686175

Reviewed By: kubamracek

Differential Revision: https://reviews.llvm.org/D65990

llvm-svn: 368492
2019-08-09 21:46:32 +00:00
Eric Christopher 11c1847237 Revert "[sanitizers] MSVC warning disable for clean build" and follow-up that tried to fix the build as it's still broken.
This reverts commit 368476 and 368480.

llvm-svn: 368481
2019-08-09 20:43:36 +00:00
Martin Storsjo 96a2b25bcb Fix compilation after SVN r368476
That revision broke compilation with this error:

lib/builtins/fixunsxfdi.c:13:2: error: unterminated conditional directive
 #if !_ARCH_PPC

llvm-svn: 368480
2019-08-09 20:36:00 +00:00
Matthew G McGovern 8e2842cc85 [sanitizers] MSVC warning disable for clean build
- https://reviews.llvm.org/D66023

llvm-svn: 368476
2019-08-09 20:09:46 +00:00
Mitch Phillips cace571c91 Revert "[asan_symbolize] Fix bug where the frame counter was not incremented."
This reverts commit 52a36fae2a.

This commit broke the sanitizer_android buildbot. See comments at
https://reviews.llvm.org/rL368373 for more details.

llvm-svn: 368472
2019-08-09 19:36:41 +00:00
Max Moroz 3653aeeffa [libFuzzer] Merge: print stats after reading the output corpus dir.
Summary:
The purpose is to be able to extract the number of new edges added to
the original (i.e. output) corpus directory after doing the merge. Use case
example: in ClusterFuzz, we do merge after every fuzzing session, to avoid
uploading too many corpus files, and we also record coverage stats at that
point. Having a separate line indicating stats after reading the initial output
corpus directory would make the stats extraction easier for both humans and
parsing scripts.

Context: https://github.com/google/clusterfuzz/issues/802.

Reviewers: morehouse, hctim

Reviewed By: hctim

Subscribers: delcypher, #sanitizers, llvm-commits, kcc

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D66020

llvm-svn: 368461
2019-08-09 18:20:53 +00:00
Benjamin Kramer 8819a734ae [sanitizer] Update symbolizer/scripts/global_symbols.txt
llvm-svn: 368440
2019-08-09 14:03:30 +00:00
Dan Liew 52a36fae2a [asan_symbolize] Fix bug where the frame counter was not incremented.
Summary:
This bug occurred when a plug-in requested that a binary not be
symbolized while the script is trying to symbolize a stack frame. In
this case `self.frame_no` would not be incremented. This would cause
subsequent stack frames that are symbolized to be incorrectly numbered.

To fix this `get_symbolized_lines()` has been modified to take an
argument that indicates whether the stack frame counter should
incremented. In `process_line_posix()` `get_symbolized_lines(None, ...)`
is now used in in the case where we don't want to symbolize a line so
that we can keep the frame counter increment in a single function.

A test case is included. The test uses a dummy plugin that always asks
`asan_symbolize.py` script to not symbolize the first binary that the
script asks about. Prior to the patch this would cause the output to
script to look something like

```
  #0 0x0
  #0 0x0 in do_access
  #1 0x0 in main
```

rdar://problem/49476995

Reviewers: kubamracek, yln, samsonov, dvyukov, vitalybuka

Subscribers: #sanitizers, llvm-commits

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D65495

llvm-svn: 368373
2019-08-09 00:52:07 +00:00
Peter Collingbourne 3a9059effb hwasan: Add a code model check for tagged globals.
See D65364 for the code model requirements for tagged globals. Because
of the relocations used these requirements cannot be checked at link
time so they must be checked at runtime.

Differential Revision: https://reviews.llvm.org/D65968

llvm-svn: 368351
2019-08-08 21:40:00 +00:00
Max Moroz df3b465c9c [compiler-rt] Add ConsumeProbability and ConsumeFloatingPoint methods to FDP.
Summary:
Also slightly cleaned up the comments and changed the header's extension
back to `.h` as per comments on https://reviews.llvm.org/D65812.

New methods added:

* `ConsumeProbability` returns [0.0, 1.0] by consuming an unsigned integer value
   from the input data and dividing that value by the integer's max value.
* `ConsumeFloatingPointInRange` returns a floating point value in the given
   range. Relies on `ConsumeProbability` method. This method does not have the
   limitation of `std::uniform_real_distribution` that requires the given range
   to be <= the floating point type's max. If the range is too large, this
   implementation will additionally call `ConsumeBool` to decide whether the
   result will be in the first or the second half of the range.
* `ConsumeFloatingPoint` returns a floating point value in the range
  `[std::numeric_limits<T>::lowest(), std::numeric_limits<T>::min()]`.

Tested on Linux, Mac, Windows.


Reviewers: morehouse

Reviewed By: morehouse

Subscribers: kubamracek, mgorny, dberris, delcypher, #sanitizers, llvm-commits

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D65905

llvm-svn: 368331
2019-08-08 19:49:37 +00:00
Kamil Rytarowski 42ae6fecd4 Sync ioctl(2) list with NetBSD 9.99.3
Register 36 new ioctl(2) calls.

Enable NVMM for amd64 as the API has been stabilized.

llvm-svn: 368247
2019-08-08 02:21:44 +00:00
Kamil Rytarowski 72c940417a Restrict the NetBSD ASan TSD fallback to !ASAN_DYNAMIC
The fallback to the alternative implementation of TSD with TLS
is only needed for the static version of ASan for NetBSD.

The same code cannot be reused for the dynamic version of ASan as
TLS breaks and TSD code works.

llvm-svn: 368219
2019-08-07 21:56:43 +00:00
Alexander Richardson 0b168ffdc3 [TSAN] Fix tsan on FreeBSD after D54889
Summary:
It appears that since https://reviews.llvm.org/D54889, BackgroundThread()
crashes immediately because cur_thread()-> will return a null pointer
which is then dereferenced. I'm not sure why I only see this issue on
FreeBSD and not Linux since it should also be unintialized on other platforms.

Reviewers: yuri, dvyukov, dim, emaste

Subscribers: kubamracek, krytarowski, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D65705

llvm-svn: 368103
2019-08-06 22:30:48 +00:00
Peter Collingbourne 0930643ff6 hwasan: Instrument globals.
Globals are instrumented by adding a pointer tag to their symbol values
and emitting metadata into a special section that allows the runtime to tag
their memory when the library is loaded.

Due to order of initialization issues explained in more detail in the comments,
shadow initialization cannot happen during regular global initialization.
Instead, the location of the global section is marked using an ELF note,
and we require libc support for calling a function provided by the HWASAN
runtime when libraries are loaded and unloaded.

Based on ideas discussed with @evgeny777 in D56672.

Differential Revision: https://reviews.llvm.org/D65770

llvm-svn: 368102
2019-08-06 22:07:29 +00:00
David Carlier 074a4c6274 [Sanitizer] little typo
llvm-svn: 368093
2019-08-06 21:46:01 +00:00
David Carlier 906e727972 [Sanitizer] Linux explicitally migrate shadow mapping to Transparent Huge Page
in madvise mode, the shadow pages will be migrated only via madvise explicit calls.

Reviewers: vitalybuka

Reviewed By: vitalybuka

Differential Revision: https://reviews.llvm.org/D65775

llvm-svn: 368090
2019-08-06 21:30:03 +00:00
Max Moroz 6c619aadc4 [compiler-rt] Rename FuzzedDataProvider.h to .hpp and other minor changes.
Summary:
.hpp makes more sense for this header as it's C++ only, plus it
contains the actual implementation.

Reviewers: Dor1s

Reviewed By: Dor1s

Subscribers: kubamracek, dberris, mgorny, delcypher, #sanitizers, llvm-commits

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D65812

llvm-svn: 368054
2019-08-06 16:02:39 +00:00
Vitaly Buka ac9ee01fcb [compiler-rt] Implement getrandom interception
Summary:
Straightforward implementation of `getrandom` syscall and libc
hooks.

Test Plan: Local MSAN failures caused by uninstrumented `getrandom`
calls stop failing.

Patch by Andrew Krieger.

Reviewers: eugenis, vitalybuka

Reviewed By: vitalybuka

Subscribers: srhines, kubamracek, dberris, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D65551

llvm-svn: 367999
2019-08-06 08:41:53 +00:00
Puyan Lotfi 1b3718e609 [compiler-rt] Reverting r367962 due to Fuchsia bot build-breakage.
llvm-svn: 367990
2019-08-06 08:08:07 +00:00
Puyan Lotfi c9d90d5002 [compiler-rt] Appending COMPILER_RT_LIBCXX_PATH -isystem include for xray (3)
Third landing attempt: Added "if (HAVE_LIBCXX)" to keep Green Dragon green.

Haven't found a better way to pass the libcxx include path for building
compiler-rt with libcxx; this seems to be missing only for xray.

Differential Revision: https://reviews.llvm.org/D65307

llvm-svn: 367962
2019-08-06 00:34:34 +00:00
Yi Kong 295d4b7727 Build libfuzzer libcxx-static with PIC
r356153 changed default build option of static libcxx to no PIC. We now
need to explicitly specify CMAKE_POSITION_INDEPENDENT_CODE to get PIC
libcxx.

Differential Revision: https://reviews.llvm.org/D65773

llvm-svn: 367943
2019-08-05 22:55:17 +00:00
Eric Christopher 1d73e228db BMI2 support is indicated in bit eight of EBX, not nine.
See Intel SDM, Vol 2A, Table 3-8:
https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-2a-manual.pdf#page=296

Differential Revision: https://reviews.llvm.org/D65766

llvm-svn: 367929
2019-08-05 21:25:59 +00:00
David Carlier 1d92925a5a [Sanitizer] Linux refactor shadow huge page mode handling
Disabling Transparent huge page mode refactored in one function.

Reviewers: vitalybuka

Reviewed By: vitalybuka

Differential Revision: https://reviews.llvm.org/D65771

llvm-svn: 367925
2019-08-05 20:54:36 +00:00
Max Moroz f1b0a93e3a [compiler-rt] Move FDP to include/fuzzer/FuzzedDataProvider.h for easier use.
Summary:
FuzzedDataProvider is a helper class for writing fuzz targets that fuzz
multple inputs simultaneously. The header is supposed to be used for fuzzing
engine agnostic fuzz targets (i.e. the same target can be used with libFuzzer,
AFL, honggfuzz, and other engines). The common thing though is that fuzz targets
are typically compiled with clang, as it provides all sanitizers as well as
different coverage instrumentation modes. Therefore, making this FDP class a
part of the compiler-rt installation package would make it easier to develop
and distribute fuzz targets across different projects, build systems, etc.
Some context also available in https://github.com/google/oss-fuzz/pull/2547.

This CL does not delete the header from `lib/fuzzer/utils` directory in order to
provide the downstream users some time for a smooth migration to the new
header location.

Reviewers: kcc, morehouse

Reviewed By: morehouse

Subscribers: lebedev.ri, kubamracek, dberris, mgorny, delcypher, #sanitizers, llvm-commits

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D65661

llvm-svn: 367917
2019-08-05 19:55:52 +00:00
Nico Weber bb7ad98a47 Follow-up for r367863 and r367656
llvm-svn: 367888
2019-08-05 16:50:56 +00:00
Nico Weber 673dc3d4a0 compiler-rt: Rename cc files below test/asan to cpp
See r367803 and similar other changes.

llvm-svn: 367887
2019-08-05 16:48:12 +00:00
Nico Weber f3750a4420 Try to fix OOB tests more on Windows after r367642
See PR42868 for more details.

The affected list of tests is:

Failing Tests (8):
    AddressSanitizer-Unit :: ./Asan-x86_64-calls-Test.exe/AddressSanitizer.LargeOOBRightTest
    AddressSanitizer-Unit :: ./Asan-x86_64-calls-Test.exe/AddressSanitizer.OOBRightTest
    AddressSanitizer-Unit :: ./Asan-x86_64-calls-Test.exe/AddressSanitizer.OOB_char
    AddressSanitizer-Unit :: ./Asan-x86_64-calls-Test.exe/AddressSanitizer.OOB_int
    AddressSanitizer-Unit :: ./Asan-x86_64-inline-Test.exe/AddressSanitizer.LargeOOBRightTest
    AddressSanitizer-Unit :: ./Asan-x86_64-inline-Test.exe/AddressSanitizer.OOBRightTest
    AddressSanitizer-Unit :: ./Asan-x86_64-inline-Test.exe/AddressSanitizer.OOB_char
    AddressSanitizer-Unit :: ./Asan-x86_64-inline-Test.exe/AddressSanitizer.OOB_int

llvm-svn: 367874
2019-08-05 15:10:15 +00:00
Nico Weber 997d626de6 Try to fix OOB tests on at least Windows after r367642
gtest's built-in regex engine doesn't support (). Looks like it's not
needed, just remove it.

See PR42868 for more details.

llvm-svn: 367873
2019-08-05 15:04:20 +00:00
Nico Weber c4310f921d compiler-rt: Rename .cc file in test/dfsan to cpp
See r367849 et al.

llvm-svn: 367854
2019-08-05 13:19:28 +00:00
Fangrui Song 97ccf6b8c1 compiler-rt: Rename .cc file in test/lsan to .cpp
Like r367463, but for test/lsan.

llvm-svn: 367803
2019-08-05 07:04:42 +00:00
Rainer Orth f98a153c8d [sanitizer_common][tests] Fix SanitizerCommon-Unit :: ./Sanitizer-*-Test/SanitizerCommon.PthreadDestructorIterations on Solaris
SanitizerCommon.PthreadDestructorIterations currently FAILs on Solaris:

  [ RUN      ] SanitizerCommon.PthreadDestructorIterations
  /vol/llvm/src/compiler-rt/local/lib/sanitizer_common/tests/sanitizer_posix_test.cc:58: Failure
  Value of: destructor_executed
    Actual: true
  Expected: false
  [  FAILED  ] SanitizerCommon.PthreadDestructorIterations (1 ms)

It turns out that destructor is called 4 times after the first call to SpawnThread, but
5 times after the second.  While PTHREAD_DESTRUCTOR_ITERATIONS is 4 in
<limits.h>, the Solaris pthread_key_create(3C) man page documents

  If, after all the destructors have been called for all keys  with  non-
  null  values,  there  are  still  some  keys  with non-null values, the
  process will be repeated. POSIX requires that this process be  executed
  at   least   PTHREAD_DESTRUCTOR_ITERATIONS  times.  Solaris  calls  the
  destructors repeatedly until all values with associated destructors are
  NULL. Destructors that set new values can cause an infinite loop.

The patch adjusts the test case to allow for this.

Tested on x86_64-pc-solaris2.11.

Differential Revision: https://reviews.llvm.org/D65055

llvm-svn: 367705
2019-08-02 18:55:22 +00:00
Fangrui Song bcaeed49cb compiler-rt: Rename .cc file in test/tsan to .cpp
Like r367463, but for test/tsan.

llvm-svn: 367656
2019-08-02 07:18:07 +00:00
Vitaly Buka 74296c99b3 Fix flaky test caused by PR42868
llvm-svn: 367642
2019-08-02 02:27:04 +00:00
Nico Weber 4a1a113a99 Remove a few straggler ".cc"s in compiler-rt/lib
llvm-svn: 367589
2019-08-01 17:53:25 +00:00
Nico Weber 267d63f80a compiler-rt: Call a generated c++ file in hwasan .cpp instead of .cc
llvm-svn: 367572
2019-08-01 14:48:16 +00:00
Nico Weber 3fa3831830 compiler-rt: Rename .cc file in lib/scudo/standalone/tests to .cpp
Like r367463, but for scudo/standalone/tests.

With this, all files in compiler-rt/lib have extension cpp.

llvm-svn: 367569
2019-08-01 14:38:49 +00:00
Nico Weber 6d46ebefb7 compiler-rt: Rename .cc file in lib/scudo/standalone to .cpp
Like r367463, but for scudo/standalone.

llvm-svn: 367568
2019-08-01 14:36:38 +00:00
Nico Weber d11b16e1fe compiler-rt: Rename .cc file in lib/tsan/{benchmarks,dd,go} to .cpp
Like r367463, but for tsan/{benchmarks,dd,go}.

The files benchmarks aren't referenced in the build anywhere and where added
in 2012 with the comment "no Makefiles yet".

llvm-svn: 367567
2019-08-01 14:30:49 +00:00
Nico Weber 5de29a4b0e compiler-rt: Rename .cc file in lib/tsan/tests/{rtl,unit} to .cpp
Like r367463, but for tsan/tests/{rtl,unit}.

llvm-svn: 367566
2019-08-01 14:26:37 +00:00
Nico Weber 5a3bb1a4d6 compiler-rt: Rename .cc file in lib/tsan/rtl to .cpp
Like r367463, but for tsan/rtl.

llvm-svn: 367564
2019-08-01 14:22:42 +00:00
Nico Weber 1265d0f44c compiler-rt: Rename .cc file in lib/msan/tests to .cpp
Like r367463, but for msan/tests.

llvm-svn: 367563
2019-08-01 14:09:37 +00:00
Nico Weber 60c66db476 compiler-rt: Rename .cc file in lib/msan to .cpp
Like r367463, but for msan.

llvm-svn: 367562
2019-08-01 14:08:18 +00:00
Nico Weber ae1fc9baae compiler-rt: Rename .cc file in lib/lsan to .cpp
Like r367463, but for lsan.

llvm-svn: 367561
2019-08-01 14:01:30 +00:00
Nico Weber 9642e337eb compiler-rt: Rename .cc file in lib/{interception/tests,safestack} to .cpp
Like r367463, but for interception/tests and safestack.

llvm-svn: 367560
2019-08-01 13:56:52 +00:00
Nico Weber b47455b573 compiler-rt: Rename .cc file in lib/asan/tests to .cpp
Like r367463, but for asan/tests

llvm-svn: 367559
2019-08-01 13:48:31 +00:00
Nico Weber 217222abea compiler-rt: Rename .cc file in lib/asan to .cpp
Like r367463, but for asan.

llvm-svn: 367558
2019-08-01 13:43:28 +00:00
Nico Weber c58c1c5350 Try to heal bots more after r367551
llvm-svn: 367555
2019-08-01 13:10:43 +00:00
Nico Weber 208ebc9e8b Try to heal bots after r367551
llvm-svn: 367552
2019-08-01 12:42:28 +00:00
Nico Weber a9aa813792 compiler-rt: Rename .cc file in lib/{dfsan,stats,ubsan_minimal} to .cpp
Like r367463, but for dfsan, stats, ubsan_minimal.

llvm-svn: 367551
2019-08-01 12:41:23 +00:00
Nico Weber bc0d697db9 compiler-rt: Rename .cc file in lib/xray/tests/unit to .cpp
Like r367463, but for xray/texts/unit.

llvm-svn: 367550
2019-08-01 12:35:27 +00:00
Nico Weber b301860321 compiler-rt: Rename .cc file in lib/xray to .cpp
Like r367463, but for xray.

llvm-svn: 367546
2019-08-01 12:30:58 +00:00
Nico Weber 45ff4868c4 compiler-rt: Try to appease lint script.
A bot complains:

/b/sanitizer-x86_64-linux-autoconf/build/llvm/projects/compiler-rt/lib/sanitizer_common/tests/malloc_stress_transfer_test.cpp:2:  Streams are highly discouraged.  [readability/streams] [3]
/b/sanitizer-x86_64-linux-autoconf/build/llvm/projects/compiler-rt/lib/sanitizer_common/tests/sanitizer_libc_test.cpp:11:  Streams are highly discouraged.  [readability/streams] [3]
lib/CMakeFiles/SanitizerLintCheck.dir/build.make:57: recipe for target 'lib/CMakeFiles/SanitizerLintCheck' failed

I do not know why this apparently wasn't a problem when the files
had extension .cc.

llvm-svn: 367493
2019-07-31 23:34:07 +00:00
Nico Weber 0e86e7fda3 Attempt to heal bots after r367467
llvm-svn: 367473
2019-07-31 20:12:19 +00:00
Nico Weber c940adeddb compiler-rt: Rename .cc file in lib/sanitizer_common/symbolizer to .cpp
llvm-svn: 367471
2019-07-31 19:41:02 +00:00
Nico Weber f724b460bd Attempt to heal tsan bot after r367463
llvm-svn: 367469
2019-07-31 19:31:36 +00:00
Nico Weber d6d569fc06 compiler-rt: Rename .cc file in lib/sanitizer_common/tests to .cpp
See https://reviews.llvm.org/D58620 for discussion, and for the commands
I ran. In addition I also ran

  for f in $(svn diff | diffstat | grep .cc | cut -f 2 -d ' '); do rg $(basename $f) . ; done

and manually updated references to renamed files found by that.

llvm-svn: 367467
2019-07-31 19:11:14 +00:00
Nico Weber 65492d959b compiler-rt: Rename .cc file in lib/sanitizer_common to .cpp
See https://reviews.llvm.org/D58620 for discussion, and for the commands
I ran. In addition I also ran

  for f in $(svn diff | diffstat | grep .cc | cut -f 2 -d ' '); do rg $f . ; done

and manually updated (many) references to renamed files found by that.

llvm-svn: 367463
2019-07-31 18:51:27 +00:00
Nico Weber b42a1c6967 Attempt to heal bots after r367456
llvm-svn: 367462
2019-07-31 18:43:27 +00:00
Nico Weber 62a0585530 compiler-rt: Rename .cc file in lib/profile to .cpp
See https://reviews.llvm.org/D58620 for discussion.

Note how the comment in the file already said ".cpp" :)

llvm-svn: 367460
2019-07-31 18:21:08 +00:00
Nico Weber ebbce04c14 compiler-rt: Rename .cc files in lib/interception to .cpp.
See https://reviews.llvm.org/D58620 for discussion, and for the commands
I ran. In addition I also ran

  for f in $(svn diff | diffstat | grep .cc | cut -f 2 -d ' '); do rg $f . ; done

and manually updated references to renamed files found by that.

llvm-svn: 367456
2019-07-31 18:01:55 +00:00
Nico Weber 46ba969752 compiler-rt: Rename .cc files in lib/ubsan to .cpp.
See https://reviews.llvm.org/D58620 for discussion, and for the commands
I ran. In addition I also ran

  for f in $(svn diff | diffstat | grep .cc | cut -f 2 -d ' '); do rg $f . ; done

and manually updated references to renamed files found by that.

llvm-svn: 367452
2019-07-31 17:51:05 +00:00
Nico Weber b206c3e3e4 gn build: Add build files for compiler-rt/lib/profile
Differential Revision: https://reviews.llvm.org/D65518

llvm-svn: 367450
2019-07-31 17:15:32 +00:00
Nico Weber e4001bacc2 gn build: Fix redundant object files in builtin lib.
compiler-rt's builtin library has generic implementations of many
functions, and then per-arch optimized implementations of some.

In the CMake build, both filter_builtin_sources() and an explicit loop
at the end of the build file (see D37166) filter out the generic
versions if a per-arch file is present.

The GN build wasn't doing this filtering. Just do the filtering manually
and explicitly, instead of being clever.

While here, also remove files from the mingw/arm build that are
redundantly listed after D39938 / r318139 (both from the CMake and the
GN build).

While here, also fix a target_os -> target_cpu typo.

Differential Revision: https://reviews.llvm.org/D65512

llvm-svn: 367448
2019-07-31 17:08:34 +00:00
Alexander Richardson a4ea27de92 [Sanitizer][ASAN][MSAN] Fix infinite recursion on FreeBSD
Summary:
MSAN was broken on FreeBSD by https://reviews.llvm.org/D55703: after this
change accesses to the key variable call __tls_get_addr, which is
intercepted. The interceptor then calls GetCurrentThread which calls
MsanTSDGet which again calls __tls_get_addr, etc...
Using the default implementation in the SANITIZER_FREEBSD case fixes MSAN
for me.

I then applied the same change to ASAN (introduced in https://reviews.llvm.org/D55596)
but that did not work yet. In the ASAN case, we get infinite recursion
again during initialization, this time because calling pthread_key_create() early on
results in infinite recursion. pthread_key_create() calls sysctlbyname()
which is intercepted but COMMON_INTERCEPTOR_NOTHING_IS_INITIALIZED returns
true, so the interceptor calls internal_sysctlbyname() which then ends up
calling the interceptor again. I fixed this issue by using dlsym() to get
the libc version of sysctlbyname() instead.

This fixes https://llvm.org/PR40761

Reviewers: vitalybuka, krytarowski, devnexen, dim, bsdjhb, #sanitizers, MaskRay

Reviewed By: MaskRay

Subscribers: MaskRay, emaste, kubamracek, jfb, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D65221

llvm-svn: 367442
2019-07-31 16:31:55 +00:00
Dan Liew 62a8728750 [asan_symbolize] More debugging output
When a line fails to match the stackframe regex we now report
it in the log.

rdar://problem/49476995

llvm-svn: 367380
2019-07-31 00:59:10 +00:00
Dan Liew 51b1d858d3 [asan_symbolize] Add `--skip-uuid-validation` option to `ModuleMapPlugIn`.
This option disables the validation of binary UUIDs. This is useful
in environments where the `otool` binary is not available.

rdar://problem/49476995

llvm-svn: 367379
2019-07-31 00:59:10 +00:00
Dan Liew 23fd828d4e [asan_symbolize] Provide better error message when extracting the UUID of a binary fails
rdar://problem/49476995

llvm-svn: 367378
2019-07-31 00:59:09 +00:00
Francis Visoiu Mistrih fb3c53af34 Revert "[compiler-rt] Appending COMPILER_RT_LIBCXX_PATH -isystem include for xray (2)"
This reverts commit r367250. It's failing on green dragon:
http://lab.llvm.org:8080/green/job/clang-stage1-RA/482/console.

llvm-svn: 367265
2019-07-29 22:08:12 +00:00
Puyan Lotfi a9c59b2873 [compiler-rt] Appending COMPILER_RT_LIBCXX_PATH -isystem include for xray (2)
Second attempt.

Haven't found a better way to pass the libcxx include path for building
compiler-rt with libcxx; this seems to be missing only for xray.

Differential Revision: https://reviews.llvm.org/D65307

llvm-svn: 367250
2019-07-29 19:21:58 +00:00
Puyan Lotfi 5bf16ec02b [compiler-rt] Reverting r367239 (xray include path) to fix bot build breakage.
llvm-svn: 367246
2019-07-29 17:50:39 +00:00
Puyan Lotfi ba7b7f1de4 Appending COMPILER_RT_LIBCXX_PATH include path to -isystem for compiler-rt xray
Haven't found a better way to pass the libcxx include path for building
compiler-rt with libcxx; this seems to be missing only for xray.

Differential Revision: https://reviews.llvm.org/D65307

llvm-svn: 367239
2019-07-29 16:46:35 +00:00
Eugene Leviant 45d910de99 Don't initialize interceptor_metadata_map unless SI_POSIX is set
Differential revision: https://reviews.llvm.org/D64794

llvm-svn: 367188
2019-07-28 08:58:44 +00:00
Pirama Arumuga Nainar 6caa8da072 [profile] In Android, do not mkdir() dirs in GCOV_PREFIX
Summary:
In Android, attempting to mkdir() or even stat() top-level directories
like /data causes noisy selinux denials.  During whole-system coverage
instrumentation, this causes a deluge of noisy messages that drown out
legitimate selinux denials, that should be audited and fixed.

To avoid this, skip creating any directory in GCOV_PREFIX (thereby
assuming that it exists).

- Android platform ensures that the GCOV_PREFIX used in Android is
created and read/writable by all processes.
- This only affects the Android platform (by checking against
__ANDROID_API_FUTURE__) and for apps built with Clang coverage, the
runtime will still create any non-existent parent directories for the
coverage files.

Reviewers: srhines, davidxl

Subscribers: krytarowski, #sanitizers, danalbert, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D65245

llvm-svn: 367064
2019-07-25 22:10:56 +00:00
Julian Lettner 758f6da687 [sanitizer_common] Allow customization of StartSymbolizerSubprocess
Summary:
Make SymbolizerProcess::StartSymbolizerSubprocess virtual and protected
to allow subclasses to customize it via "override and call".

Subscribers: kubamracek, #sanitizers, llvm-commits

Tags: #sanitizers, #llvm

Differential Revision: https://reviews.llvm.org/D65252

llvm-svn: 366967
2019-07-25 00:19:02 +00:00
Kostya Kortchinsky 419f1a4185 [scudo][standalone] Optimization pass
Summary:
This introduces a bunch of small optimizations with the purpose of
making the fastpath tighter:
- tag more conditions as `LIKELY`/`UNLIKELY`: as a rule of thumb we
  consider that every operation related to the secondary is unlikely
- attempt to reduce the number of potentially extraneous instructions
- reorganize the `Chunk` header to not straddle a word boundary and
  use more appropriate types

Note that some `LIKELY`/`UNLIKELY` impact might be less obvious as
they are in slow paths (for example in `secondary.cc`), but at this
point I am throwing a pretty wide net, and it's consistant and doesn't
hurt.

This was mosly done for the benfit of Android, but other platforms
benefit from it too. An aarch64 Android benchmark gives:
- before:
```
  BM_youtube/min_time:15.000/repeats:4/manual_time_mean              445244 us       659385 us            4
  BM_youtube/min_time:15.000/repeats:4/manual_time_median            445007 us       658970 us            4
  BM_youtube/min_time:15.000/repeats:4/manual_time_stddev               885 us         1332 us            4
```
- after:
```
  BM_youtube/min_time:15.000/repeats:4/manual_time_mean       415697 us       621925 us            4
  BM_youtube/min_time:15.000/repeats:4/manual_time_median     415913 us       622061 us            4
  BM_youtube/min_time:15.000/repeats:4/manual_time_stddev        990 us         1163 us            4
```

Additional since `-Werror=conversion` is enabled on some platforms we
are built on, enable it upstream to catch things early: a few sign
conversions had slept through and needed additional casting.

Reviewers: hctim, morehouse, eugenis, vitalybuka

Reviewed By: vitalybuka

Subscribers: srhines, mgorny, javed.absar, kristof.beyls, delcypher, #sanitizers, llvm-commits

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D64664

llvm-svn: 366918
2019-07-24 16:36:01 +00:00
Max Moroz 374398afb7 [libFuzzer] Fix __sanitizer_print_memory_profile return type in ExtFunctions.def
Summary:
Looks like a typo, as that function actually returns void and is used
as such in libFuzzer code as well.

Reviewers: kcc, Dor1s

Reviewed By: Dor1s

Subscribers: delcypher, #sanitizers, llvm-commits

Tags: #llvm, #sanitizers

Differential Revision: https://reviews.llvm.org/D65160

llvm-svn: 366834
2019-07-23 18:26:53 +00:00
Rainer Orth 569f92f1e1 [compiler-rt][builtins] Provide __clear_cache for SPARC
While working on https://reviews.llvm.org/D40900, two tests were failing since __clear_cache
aborted.  While libgcc's __clear_cache is just empty, this only happens because
gcc (in gcc/config/sparc/sparc.c (sparc32_initialize_trampoline, sparc64_initialize_trampoline))
emits flush insns directly.

The following patch mimics that.

Tested on sparcv9-sun-solaris2.11.

Differential Revision: https://reviews.llvm.org/D64496

llvm-svn: 366822
2019-07-23 16:33:54 +00:00
Serge Guelton a30a4a35ec Fix asan infinite loop on undefined symbol
Fix llvm#39641

Recommit of r366413

Differential Revision: https://reviews.llvm.org/D63877

> llvm-svn: 366632

llvm-svn: 366638
2019-07-20 17:44:30 +00:00
Serge Guelton 7a3d4c15a7 Revert "Fix asan infinite loop on undefined symbol"
This reverts commit cbd28cd05b.

Buildbot fail: http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/22434/steps/64-bit%20check-asan-dynamic/

llvm-svn: 366634
2019-07-20 13:00:12 +00:00
Serge Guelton cbd28cd05b Fix asan infinite loop on undefined symbol
Fix llvm#39641

Recommit of r366413

Differential Revision: https://reviews.llvm.org/D63877

llvm-svn: 366632
2019-07-20 12:01:18 +00:00
Matthew Voss 407e837540 Revert "Fix asan infinite loop on undefined symbol"
This reverts commit 8e46275488.

This was failing on sanitizer-x86_64-linux and our internal CI.

llvm-svn: 366618
2019-07-19 21:41:07 +00:00
Serge Guelton 8e46275488 Fix asan infinite loop on undefined symbol
Fix llvm#39641

Recommit of r366413

Differential Revision: https://reviews.llvm.org/D63877

llvm-svn: 366588
2019-07-19 15:20:36 +00:00