The MemProf compiler-rt support relies on some of the support only built
when COMPILER_RT_BUILD_SANITIZERS was enabled. This showed up in some
initial bot failures, and I addressed those by making the memprof
runtime build also conditional on COMPILER_RT_BUILD_SANITIZERS
(3ed77ecd0a). However, this resulted in
another inconsistency with how the tests were set up that was hit by
Chromium:
https://bugs.chromium.org/p/chromium/issues/detail?id=1142191
Undo the original bot fix and address this with a more comprehensive fix
that enables memprof to be built even when COMPILER_RT_BUILD_SANITIZERS
is disabled, by also building the necessary pieces under
COMPILER_RT_BUILD_MEMPROF.
Tested by configuring with a similar command as to what was used in the
failing Chromium configure. I reproduced the Chromium failure, as well
as the original bot failure I tried to fix in
3ed77ecd0a, with that fix reverted.
Confirmed it now works.
Differential Revision: https://reviews.llvm.org/D90190
See RFC for background:
http://lists.llvm.org/pipermail/llvm-dev/2020-June/142744.html
Follow on companion to the clang/llvm instrumentation support in D85948
and committed earlier.
This patch adds the compiler-rt runtime support for the memory
profiling.
Note that much of this support was cloned from asan (and then greatly
simplified and renamed). For example the interactions with the
sanitizer_common allocators, error handling, interception, etc.
The bulk of the memory profiling specific code can be found in the
MemInfoBlock, MemInfoBlockCache, and related classes defined and used
in memprof_allocator.cpp.
For now, the memory profile is dumped to text (stderr by default, but
honors the sanitizer_common log_path flag). It is dumped in either a
default verbose format, or an optional terse format.
This patch also adds a set of tests for the core functionality.
Differential Revision: https://reviews.llvm.org/D87120
Following up D81682 and D83903, remove the code for the old value profiling
buckets, which have been replaced with the new, extended buckets and disabled by
default.
Also syncing InstrProfData.inc between compiler-rt and llvm.
Differential Revision: https://reviews.llvm.org/D88838
Add functions exposed via the MSAN interface to enable MSAN within
binaries that perform manual stack switching (e.g. through using fibers
or coroutines).
This functionality is analogous to the fiber APIs available for ASAN and TSAN.
Fixesgoogle/sanitizers#1232
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D86471
Unmapping and remapping is dangerous since another thread could touch
the shadow memory while it is unmapped. But there is really no need to
unmap anyway, since mmap(MAP_FIXED) will happily clobber the existing
mapping with zeroes. This is thread-safe since the mmap() is done under
the same kernel lock as page faults are done.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D85947
Extend the memop value profile buckets to be more flexible (could accommodate a
mix of individual values and ranges) and to cover more value ranges (from 11 to
22 buckets).
Disabled behind a flag (to be enabled separately) and the existing code to be
removed later.
Differential Revision: https://reviews.llvm.org/D81682
This reverts commit 4a539faf74.
There is a __llvm_profile_instrument_range related crash in PGO-instrumented clang:
```
(gdb) bt
llvm::ConstantRange const&, llvm::APInt const&, unsigned int, bool) ()
llvm::ScalarEvolution::getRangeForAffineAR(llvm::SCEV const*, llvm::SCEV
const*, llvm::SCEV const*, unsigned int) ()
```
(The body of __llvm_profile_instrument_range is inlined, so we can only find__llvm_profile_instrument_target in the trace)
```
23│ 0x000055555dba0961 <+65>: nopw %cs:0x0(%rax,%rax,1)
24│ 0x000055555dba096b <+75>: nopl 0x0(%rax,%rax,1)
25│ 0x000055555dba0970 <+80>: mov %rsi,%rbx
26│ 0x000055555dba0973 <+83>: mov 0x8(%rsi),%rsi # %rsi=-1 -> SIGSEGV
27│ 0x000055555dba0977 <+87>: cmp %r15,(%rbx)
28│ 0x000055555dba097a <+90>: je 0x55555dba0a76 <__llvm_profile_instrument_target+342>
```
Extend the memop value profile buckets to be more flexible (could accommodate a
mix of individual values and ranges) and to cover more value ranges (from 11 to
22 buckets).
Disabled behind a flag (to be enabled separately) and the existing code to be
removed later.
Extend the memop value profile buckets to be more flexible (could accommodate a
mix of individual values and ranges) and to cover more value ranges (from 11 to
22 buckets).
Disabled behind a flag (to be enabled separately) and the existing code to be
removed later.
Differential Revision: https://reviews.llvm.org/D81682
Summary: Separate class definition and actual methods implementation. The main
goal is to keep the list of available methods in a concise readable form inside
the class definition.
Reviewers: hctim, metzman
Subscribers: dberris, #sanitizers
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D76651
Note that in C++ the static keyword is implicit for const objects.
In C however it is not, and we get clashes at link time after
including the header from more than one C file:
lib.a(bar.o):(.rodata+0x0): multiple definition of `__tsan_mutex_linker_init'
lib.a(foo.o):(.rodata+0x0): first defined here
lib.a(bar.o):(.rodata+0xc): multiple definition of `__tsan_mutex_not_static'
lib.a(foo.o):(.rodata+0xc): first defined here
<snip>
Indeed both foo.o and bar.o define the clashing symbols:
$ nm foo.o
<snip>
0000000000000000 R __tsan_mutex_linker_init
000000000000000c R __tsan_mutex_not_static
<snip>
Fix it by explicitly making the constants static.
Reviewed-in: https://reviews.llvm.org/D75820
Author: cota (Emilio G. Cota)
Try again with an up-to-date version of D69471 (99317124 was a stale
revision).
---
Revise the coverage mapping format to reduce binary size by:
1. Naming function records and marking them `linkonce_odr`, and
2. Compressing filenames.
This shrinks the size of llc's coverage segment by 82% (334MB -> 62MB)
and speeds up end-to-end single-threaded report generation by 10%. For
reference the compressed name data in llc is 81MB (__llvm_prf_names).
Rationale for changes to the format:
- With the current format, most coverage function records are discarded.
E.g., more than 97% of the records in llc are *duplicate* placeholders
for functions visible-but-not-used in TUs. Placeholders *are* used to
show under-covered functions, but duplicate placeholders waste space.
- We reached general consensus about giving (1) a try at the 2017 code
coverage BoF [1]. The thinking was that using `linkonce_odr` to merge
duplicates is simpler than alternatives like teaching build systems
about a coverage-aware database/module/etc on the side.
- Revising the format is expensive due to the backwards compatibility
requirement, so we might as well compress filenames while we're at it.
This shrinks the encoded filenames in llc by 86% (12MB -> 1.6MB).
See CoverageMappingFormat.rst for the details on what exactly has
changed.
Fixes PR34533 [2], hopefully.
[1] http://lists.llvm.org/pipermail/llvm-dev/2017-October/118428.html
[2] https://bugs.llvm.org/show_bug.cgi?id=34533
Differential Revision: https://reviews.llvm.org/D69471
Revise the coverage mapping format to reduce binary size by:
1. Naming function records and marking them `linkonce_odr`, and
2. Compressing filenames.
This shrinks the size of llc's coverage segment by 82% (334MB -> 62MB)
and speeds up end-to-end single-threaded report generation by 10%. For
reference the compressed name data in llc is 81MB (__llvm_prf_names).
Rationale for changes to the format:
- With the current format, most coverage function records are discarded.
E.g., more than 97% of the records in llc are *duplicate* placeholders
for functions visible-but-not-used in TUs. Placeholders *are* used to
show under-covered functions, but duplicate placeholders waste space.
- We reached general consensus about giving (1) a try at the 2017 code
coverage BoF [1]. The thinking was that using `linkonce_odr` to merge
duplicates is simpler than alternatives like teaching build systems
about a coverage-aware database/module/etc on the side.
- Revising the format is expensive due to the backwards compatibility
requirement, so we might as well compress filenames while we're at it.
This shrinks the encoded filenames in llc by 86% (12MB -> 1.6MB).
See CoverageMappingFormat.rst for the details on what exactly has
changed.
Fixes PR34533 [2], hopefully.
[1] http://lists.llvm.org/pipermail/llvm-dev/2017-October/118428.html
[2] https://bugs.llvm.org/show_bug.cgi?id=34533
Differential Revision: https://reviews.llvm.org/D69471
Summary:
An implementation for `sigaltstack` to make its side effect be visible to MSAN.
```
ninja check-msan
```
Reviewers: vitalybuka, eugenis
Reviewed By: eugenis
Subscribers: dberris, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73816
Patch by Igor Sugak.
Summary:
Some versions of memcpy mark pointer arguments as __nonnull, that triggers UBSan
errors even when the length passed is 0.
Reviewers: manojgupta, metzman
Subscribers: dberris, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D71031
[compiler-rt] FDP: assert that num_bytes_to_consume == 0 when size == 0.
Revise the coverage mapping format to reduce binary size by:
1. Naming function records and marking them `linkonce_odr`, and
2. Compressing filenames.
This shrinks the size of llc's coverage segment by 82% (334MB -> 62MB)
and speeds up end-to-end single-threaded report generation by 10%. For
reference the compressed name data in llc is 81MB (__llvm_prf_names).
Rationale for changes to the format:
- With the current format, most coverage function records are discarded.
E.g., more than 97% of the records in llc are *duplicate* placeholders
for functions visible-but-not-used in TUs. Placeholders *are* used to
show under-covered functions, but duplicate placeholders waste space.
- We reached general consensus about giving (1) a try at the 2017 code
coverage BoF [1]. The thinking was that using `linkonce_odr` to merge
duplicates is simpler than alternatives like teaching build systems
about a coverage-aware database/module/etc on the side.
- Revising the format is expensive due to the backwards compatibility
requirement, so we might as well compress filenames while we're at it.
This shrinks the encoded filenames in llc by 86% (12MB -> 1.6MB).
See CoverageMappingFormat.rst for the details on what exactly has
changed.
Fixes PR34533 [2], hopefully.
[1] http://lists.llvm.org/pipermail/llvm-dev/2017-October/118428.html
[2] https://bugs.llvm.org/show_bug.cgi?id=34533
Differential Revision: https://reviews.llvm.org/D69471
This header fragment is useful on its own for any consumer that wants
to use custom instruction profile runtime with the LLVM instrumentation.
The concrete use case is in Fuchsia's kernel where we want to use
instruction profile instrumentation, but we cannot use the compiler-rt
runtime because it's not designed for use in the kernel environment.
This change allows installing this header as part of compiler-rt.
Differential Revision: https://reviews.llvm.org/D64532
This header fragment is useful on its own for any consumer that wants
to use custom instruction profile runtime with the LLVM instrumentation.
The concrete use case is in Fuchsia's kernel where we want to use
instruction profile instrumentation, but we cannot use the compiler-rt
runtime because it's not designed for use in the kernel environment.
This change allows installing this header as part of compiler-rt.
Differential Revision: https://reviews.llvm.org/D64532
Summary:
Sometimes an allocation stack trace is not very informative. Provide a
way to replace it with a stack trace of the user's choice.
Reviewers: pcc, kcc
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D69208
Summary:
Also slightly cleaned up the comments and changed the header's extension
back to `.h` as per comments on https://reviews.llvm.org/D65812.
New methods added:
* `ConsumeProbability` returns [0.0, 1.0] by consuming an unsigned integer value
from the input data and dividing that value by the integer's max value.
* `ConsumeFloatingPointInRange` returns a floating point value in the given
range. Relies on `ConsumeProbability` method. This method does not have the
limitation of `std::uniform_real_distribution` that requires the given range
to be <= the floating point type's max. If the range is too large, this
implementation will additionally call `ConsumeBool` to decide whether the
result will be in the first or the second half of the range.
* `ConsumeFloatingPoint` returns a floating point value in the range
`[std::numeric_limits<T>::lowest(), std::numeric_limits<T>::min()]`.
Tested on Linux, Mac, Windows.
Reviewers: morehouse
Reviewed By: morehouse
Subscribers: kubamracek, mgorny, dberris, delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D65905
llvm-svn: 368331
Summary:
.hpp makes more sense for this header as it's C++ only, plus it
contains the actual implementation.
Reviewers: Dor1s
Reviewed By: Dor1s
Subscribers: kubamracek, dberris, mgorny, delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D65812
llvm-svn: 368054
Summary:
FuzzedDataProvider is a helper class for writing fuzz targets that fuzz
multple inputs simultaneously. The header is supposed to be used for fuzzing
engine agnostic fuzz targets (i.e. the same target can be used with libFuzzer,
AFL, honggfuzz, and other engines). The common thing though is that fuzz targets
are typically compiled with clang, as it provides all sanitizers as well as
different coverage instrumentation modes. Therefore, making this FDP class a
part of the compiler-rt installation package would make it easier to develop
and distribute fuzz targets across different projects, build systems, etc.
Some context also available in https://github.com/google/oss-fuzz/pull/2547.
This CL does not delete the header from `lib/fuzzer/utils` directory in order to
provide the downstream users some time for a smooth migration to the new
header location.
Reviewers: kcc, morehouse
Reviewed By: morehouse
Subscribers: lebedev.ri, kubamracek, dberris, mgorny, delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D65661
llvm-svn: 367917
Summary:
dfsan_flush() allows to restart tain tracking from scratch in the same process.
The primary purpose right now is to allow more efficient data flow tracing
for DFT fuzzing: https://github.com/google/oss-fuzz/issues/1632
Reviewers: pcc
Reviewed By: pcc
Subscribers: delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D63037
llvm-svn: 363321
Summary:
This allows libFuzzer to unpoison parameter shadow before calling
LLVMFuzzerTestOneInput to eliminate the false positives described
in https://github.com/google/oss-fuzz/issues/2369.
Reviewers: eugenis
Reviewed By: eugenis
Subscribers: llvm-commits, metzman, kcc
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D61751
llvm-svn: 360379
It hasn't seen active development in years, and it hasn't reached a
state where it was useful.
Remove the code until someone is interested in working on it again.
Differential Revision: https://reviews.llvm.org/D59133
llvm-svn: 355862
Teresa Johnson