Summary:
Before this change, XRay would conservatively patch sections of the code
one sled at a time. Upon testing/profiling, this turns out to take an
inordinate amount of time and cycles. For an instrumented clang binary,
the cycles spent both in the patching/unpatching routine constituted 4%
of the cycles -- this didn't count the time spent in the kernel while
performing the mprotect calls in quick succession.
With this change, we're coalescing the number of calls to mprotect from
being linear to the number of instrumentation points, to now being a
lower constant when patching all the sleds through `__xray_patch()` or
`__xray_unpatch()`. In the case of calling `__xray_patch_function()` or
`__xray_unpatch_function()` we're now doing an mprotect call once for
all the sleds for that function (reduction of at least 2x calls to
mprotect).
Reviewers: kpw, eizan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D41153
llvm-svn: 320664
Summary:
The first and only function to start with allows to set the soft or hard RSS
limit at runtime. Add associated tests.
Reviewers: alekseyshl
Reviewed By: alekseyshl
Subscribers: mgorny, #sanitizers, llvm-commits
Differential Revision: https://reviews.llvm.org/D41128
llvm-svn: 320611
Summary: This brings CPU overhead on bzip2 down from 5.5x to 2x.
Reviewers: kcc, alekseyshl
Subscribers: kubamracek, hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D41137
llvm-svn: 320538
This also slightly refactors the code that's checking the directory
presence which allows eliminating one unnecessary variable.
Differential Revision: https://reviews.llvm.org/D40637
llvm-svn: 320446
Summary:
This tests must be linked with -lintl for the gettext(3) features.
Sponsored by <The NetBSD Foundation>
Reviewers: joerg, eugenis, vitalybuka
Reviewed By: vitalybuka
Subscribers: llvm-commits, #sanitizers
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D41013
llvm-svn: 320226
Summary:
This test uses GNU-specific extension to libc: tdestroy() and as-is is not compatible with NetBSD.
Sponsored by <The NetBSD Foundation>
Reviewers: joerg, eugenis, vitalybuka
Reviewed By: vitalybuka
Subscribers: llvm-commits, #sanitizers
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D41011
llvm-svn: 320225
In more recent Linux kernels with 47 bit VMAs the layout of virtual memory
for powerpc64 changed causing the address sanitizer to not work properly. This
patch adds support for 47 bit VMA kernels for powerpc64 and fixes up test
cases.
https://reviews.llvm.org/D40908
There is an associated patch for trunk.
Tested on several 4.x and 3.x kernel releases.
llvm-svn: 320110
Summary:
This change implements the basic mode filtering similar to what we do in
FDR mode. The implementation is slightly simpler in basic-mode filtering
because we have less details to remember, but the idea is the same. At a
high level, we do the following to decide when to filter function call
records:
- We maintain a per-thread "shadow stack" which keeps track of the
XRay instrumented functions we've encountered in a thread's
execution.
- We push an entry onto the stack when we enter an XRay instrumented
function, and note the CPU, TSC, and type of entry (whether we have
payload or not when entering).
- When we encounter an exit event, we determine whether the function
being exited is the same function we've entered recently, was
executing in the same CPU, and the delta of the recent TSC and the
recorded TSC at the top of the stack is less than the equivalent
amount of microseconds we're configured to ignore -- then we un-wind
the record offset an appropriate number of times (so we can
overwrite the records later).
We also support limiting the stack depth of the recorded functions,
so that we don't arbitrarily write deep function call stacks.
Reviewers: eizan, pelikan, kpw, dblaikie
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D40828
llvm-svn: 319762
Summary:
This change allows for registration of multiple logging implementations
through a central mechanism in XRay, mapping an implementation to a
"mode". Modes are strings that are used as keys to determine which
implementation to install through a single API. This mechanism allows
users to choose which implementation to install either from the
environment variable 'XRAY_OPTIONS' with the `xray_mode=` flag, or
programmatically using the `__xray_select_mode(...)` function.
Here, we introduce two API functions for the XRay logging:
__xray_log_register_mode(Mode, Impl): Associates an XRayLogImpl to a
string Mode. We can only have one implementation associated with a given
Mode.
__xray_log_select_mode(Mode): Finds the associated Impl for Mode and
installs it as if by calling `__xray_set_log_impl(...)`.
Along with these changes, we also deprecate the xray_naive_log and
xray_fdr_log flags and encourage users to instead use the xray_mode
flag.
Reviewers: kpw, dblaikie, eizan, pelikan
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D40703
llvm-svn: 319759
Following patch adds support of all memory origins in
CheckForInvalidPointerPair function. For small difference of pointers,
it's directly done in shadow memory (the limit was set to 2048B).
Then we search for origin of first pointer and verify that the second
one has the same origin. If so, we verify that it points either to a same
variable (in case of stack memory or a global variable), or to a same
heap segment.
Committing on behanf of marxin and jakubjelinek.
Reviewers: alekseyshl, kcc
Subscribers: llvm-commits
Differential revision: https://reviews.llvm.org/D40600
llvm-svn: 319668
Summary:
As discussed in https://github.com/google/oss-fuzz/issues/933,
it would be really awesome to be able to use ThinLTO for fuzzing.
However, as @kcc has pointed out, it is currently undefined (untested)
whether the sanitizers actually function properly with LLD and/or LTO.
This patch is inspired by the cfi test, which already do test with LTO
(and/or LLD), since LTO is required for CFI to function.
I started with UBSan, because it's cmakelists / lit.* files appeared
to be the cleanest. This patch adds the infrastructure to easily add
LLD and/or LTO sub-variants of the existing lit test configurations.
Also, this patch adds the LLD flavor, that explicitly does use LLD to link.
The check-ubsan does pass on my machine. And to minimize the [initial]
potential buildbot breakage i have put some restrictions on this flavour.
Please review carefully, i have not worked with lit/sanitizer tests before.
The original attempt, r319525 was reverted in r319526 due
to the failures in compiler-rt standalone builds.
Reviewers: eugenis, vitalybuka
Reviewed By: eugenis
Subscribers: #sanitizers, pcc, kubamracek, mgorny, llvm-commits, mehdi_amini, inglorion, kcc
Differential Revision: https://reviews.llvm.org/D39508
llvm-svn: 319575
This reverts commit r319525.
This change has introduced a problem with the Lit tests build for compiler-rt using Gold: http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/6047/steps/test%20standalone%20compiler-rt/logs/stdio
llvm-lit: /b/sanitizer-x86_64-linux/build/llvm/utils/lit/lit/TestingConfig.py:101: fatal: unable to parse config file '/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/test/profile/Linux/lit.local.cfg', traceback: Traceback (most recent call last):
File "/b/sanitizer-x86_64-linux/build/llvm/utils/lit/lit/TestingConfig.py", line 88, in load_from_path
exec(compile(data, path, 'exec'), cfg_globals, None)
File "/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/test/profile/Linux/lit.local.cfg", line 37, in <module>
if root.host_os not in ['Linux'] or not is_gold_linker_available():
File "/b/sanitizer-x86_64-linux/build/llvm/projects/compiler-rt/test/profile/Linux/lit.local.cfg", line 27, in is_gold_linker_available
stderr = subprocess.PIPE)
File "/usr/lib/python2.7/subprocess.py", line 390, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1024, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
llvm-svn: 319529
Summary:
As discussed in https://github.com/google/oss-fuzz/issues/933,
it would be really awesome to be able to use ThinLTO for fuzzing.
However, as @kcc has pointed out, it is currently undefined (untested)
whether the sanitizers actually function properly with LLD and/or LTO.
This patch is inspired by the cfi test, which already do test with LTO
(and/or LLD), since LTO is required for CFI to function.
I started with UBSan, because it's cmakelists / lit.* files appeared
to be the cleanest. This patch adds the infrastructure to easily add
LLD and/or LTO sub-variants of the existing lit test configurations.
Also, this patch adds the LLD flavor, that explicitly does use LLD to link.
The check-ubsan does pass on my machine. And to minimize the [initial]
potential buildbot breakage i have put some restrictions on this flavour.
Please review carefully, i have not worked with lit/sanitizer tests before.
Reviewers: eugenis, vitalybuka
Reviewed By: eugenis
Subscribers: #sanitizers, pcc, kubamracek, mgorny, llvm-commits, mehdi_amini, inglorion, kcc
Differential Revision: https://reviews.llvm.org/D39508
llvm-svn: 319525
On macOS, we usually don't require launching the target with DYLD_INSERT_LIBRARIES anymore. However, it is still necessary when running a target that is not instrumented (and e.g. dlopen's an instrument library later). In any case, ASan and TSan currently remove themselves from the DYLD_INSERT_LIBRARIES environment variable to avoid passing it onto children. This works well e.g. when instrumenting a shell. A problem arises when the target is a non-instrumented shim (e.g. "xcrun") that either re-execs or launches a child that is supposed to get DYLD_INSERT_LIBRARIES propagated. To support this mode, this patch introduces 'strip_env' flag that can be used to keep DYLD_INSERT_LIBRARIES untouched.
Differential Revision: https://reviews.llvm.org/D39991
llvm-svn: 319365
It's explicitly forbidden to call fclose with NULL, but at least on Darwin, this succeeds and doesn't segfault. To maintain binary compatibility, ASan should survice fclose(NULL) as well.
Differential Revision: https://reviews.llvm.org/D40053
llvm-svn: 319347
Calling getpwnam(NULL) is probably a bug, but at least on Darwin, such a call succeeds without segfaulting. I have some existing code that relies on that. To maintain binary compatibility, ASan should also survive a call to getpwnam with NULL.
Differential Revision: https://reviews.llvm.org/D40052
llvm-svn: 319344
Summary:
Before this patch, XRay's basic (naive mode) logging would be
initialised and installed in an adhoc manner. This patch ports the
implementation of the basic (naive mode) logging implementation to use
the common XRay framework.
We also make the following changes to reduce the variance between the
usage model of basic mode from FDR (flight data recorder) mode:
- Allow programmatic control of the size of the buffers dedicated to
per-thread records. This removes some hard-coded constants and turns
them into runtime-controllable flags and through an Options
structure.
- Default the `xray_naive_log` option to false. For now, the only way
to start basic mode is to set the environment variable, or set the
default at build-time compiler options. Because of this change we've
had to update a couple of tests relying on basic mode being always
on.
- Removed the reliance on a non-trivially destructible per-thread
resource manager. We use a similar trick done in D39526 to use
pthread_key_create() and pthread_setspecific() to ensure that the
per-thread cleanup handling is performed at thread-exit time.
We also radically simplify the code structure for basic mode, to move
most of the implementation in the `__xray` namespace.
Reviewers: pelikan, eizan, kpw
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D40164
llvm-svn: 318734
PDB emission now works well enough that we can rely on it for these
tests to pass.
Differential Revision: https://reviews.llvm.org/D40188
llvm-svn: 318546
The tests are ported as follows:
contiguous_container_crash.cc
use-after-delete.cc
use-after-free.cc
Replace hardwired shadow granularity in CHECK statements with regex.
max_redzone.cc
Bump max_redzone parameter to 32.
memset_test.cc
Bump size parameter of __asan_poison_memory_region to 32.
scariness_score_test.cc
For "far-from-bounds" heap overflow, make sure overflow is more than
one shadow granularity away.
At large shadow granularity, there is not enough redzone between
stack elements to detect far-from-bounds, so fake out that test.
Differential Revision: https://reviews.llvm.org/D39773
llvm-svn: 318470
The mulsc3_test.c was marked as unsupported due to PR32457, the underlying
cause of this PR was fixed in PR28164 so we can remove the unsupported as
it is no longer needed.
Differential Revision: https://reviews.llvm.org/D40076
llvm-svn: 318396
Summary:
This implements an opportunistic check for the RSS limit.
For ASan, this was implemented thanks to a background thread checking the
current RSS vs the set limit every 100ms. This was deemed problematic for Scudo
due to potential Android concerns (Zygote as pointed out by Aleksey) as well as
the general inconvenience of having a permanent background thread.
If a limit (soft or hard) is specified, we will attempt to update the RSS limit
status (exceeded or not) every 100ms. This is done in an opportunistic way: if
we can update it, we do it, if not we return the current status, mostly because
we don't need it to be fully consistent (it's done every 100ms anyway). If the
limit is exceeded `allocate` will act as if OOM for a soft limit, or just die
for a hard limit.
We use the `common_flags()`'s `hard_rss_limit_mb` & `soft_rss_limit_mb` for
configuration of the limits.
Reviewers: alekseyshl
Reviewed By: alekseyshl
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D40038
llvm-svn: 318301
Summary:
This change fixes the XRay trampolines aside from the __xray_CustomEvent
trampoline to align the stack to 16-byte boundaries before calling the
handler. Before this change we've not been explicitly aligning the stack
to 16-byte boundaries, which makes it dangerous when calling handlers
that leave the stack in a state that isn't strictly 16-byte aligned
after calling the handlers.
We add a test that makes sure we can handle these cases appropriately
after the changes, and prevents us from regressing the state moving
forward.
Fixes http://llvm.org/PR35294.
Reviewers: pelikan, pcc
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D40004
llvm-svn: 318261
Allow user to override shadow scale in compiler_rt by passing
-DCOMPILER_RT_ASAN_SHADOW_SCALE=n to CMake. Propagate the override
shadow scale value via a compiler define to compiler-rt and asan
tests. Tests will use the define to partially disable unsupported
tests. Set "-mllvm -asan-mapping-scale=<n>" for compiler_rt tests.
Differential Revision: https://reviews.llvm.org/D39469
llvm-svn: 318038
Multi-config CMake generators need lit to be able to resolve paths of
artifacts from previous build steps at lit time, rather than expect them
to be fully resolved at CMake time as they may contain the build mode.
Differential Revision: https://reviews.llvm.org/D38471
llvm-svn: 318037
If the lookup using RTLD_NEXT failed, the sanitizer runtime library
is later in the library search order than the DSO that we are trying
to intercept, which means that we cannot intercept this function. We
still want the address of the real definition, though, so look it up
using RTLD_DEFAULT.
Differential Revision: https://reviews.llvm.org/D39779
llvm-svn: 317930
Summary:
This change implements the changes required in both clang and
compiler-rt to allow building XRay-instrumented binaries in Darwin. For
now we limit this to x86_64. We also start building the XRay runtime
library in compiler-rt for osx.
A caveat to this is that we don't have the tests set up and running
yet, which we'll do in a set of follow-on changes.
This patch uses the monorepo layout for the coordinated change across
multiple projects.
Reviewers: kubamracek
Subscribers: mgorny, cfe-commits, llvm-commits
Differential Revision: https://reviews.llvm.org/D39114
llvm-svn: 317875
Summary:
The NetBSD specific implementation of cxa_atexit() does not
preserve the 2nd argument if dso is equal to NULL.
Changes:
- Split paths of handling intercepted __cxa_atexit() and atexit(3).
This affects all supported Operating Systems.
- Add a local stack-like structure to hold the __cxa_atexit() context.
atexit(3) is documented in the C standard as calling callback from the
earliest to the oldest entry. This path also fixes potential ABI
problem of passing an argument to a function from the atexit(3)
callback mechanism.
- Add new test to ensure LIFO style of atexit(3) callbacks: atexit3.cc
Proposal to change the behavior of __cxa_atexit() in NetBSD has been rejected.
With the above changes TSan/NetBSD with the current tsan_interceptors.cc
can bootstrap into operation.
Sponsored by <The NetBSD Foundation>
Reviewers: vitalybuka, dvyukov, joerg, kcc, eugenis
Reviewed By: dvyukov
Subscribers: kubamracek, llvm-commits, #sanitizers
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D39619
llvm-svn: 317735
Summary:
The split in D39461 introduced separate C++ flags, but `cxx_flags` needs `-lrt` as well for the standalone build.
Reviewers: alekseyshl
Reviewed By: alekseyshl
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D39497
llvm-svn: 317103
Summary:
Initially, Scudo had a monolithic design where both C and C++ functions were
living in the same library. This was not necessarily ideal, and with the work
on -fsanitize=scudo, it became more apparent that this needed to change.
We are splitting the new/delete interceptor in their own C++ library. This
allows more flexibility, notably with regard to std::bad_alloc when the work is
done. This also allows us to not link new & delete when using pure C.
Additionally, we add the UBSan runtimes with Scudo, in order to be able to have
a -fsanitize=scudo,undefined in Clang (see work in D39334).
The changes in this patch:
- split the cxx specific code in the scudo cmake file into a new library;
(remove the spurious foreach loop, that was not necessary)
- add the UBSan runtimes (both C and C++);
- change the test cmake file to allow for specific C & C++ tests;
- make C tests pure C, rename their extension accordingly.
Reviewers: alekseyshl
Reviewed By: alekseyshl
Subscribers: srhines, mgorny, llvm-commits
Differential Revision: https://reviews.llvm.org/D39461
llvm-svn: 317097
Fails on darwin
Revert "[fuzzer] Script to detect unbalanced allocation in -trace_malloc output"
Needs previous one.
This reverts commit r317034, r317036.
llvm-svn: 317061
Summary:
LSan is functional on PPC64 Linux now, let's enable all tests.
One test required ppc specific changes: use_registers.cc.
Reviewers: eugenis
Subscribers: mgorny, llvm-commits
Differential Revision: https://reviews.llvm.org/D39316
llvm-svn: 316698
Summary:
The 64-bit primary has had random shuffling of chunks for a while, this
implements it for the 32-bit primary. Scudo is currently the only user of
`kRandomShuffleChunks`.
This change consists of a few modifications:
- move the random shuffling functions out of the 64-bit primary to
`sanitizer_common.h`. Alternatively I could move them to
`sanitizer_allocator.h` as they are only used in the allocator, I don't feel
strongly either way;
- small change in the 64-bit primary to make the `rand_state` initialization
`UNLIKELY`;
- addition of a `rand_state` in the 32-bit primary's `SizeClassInfo` and
shuffling of chunks when populating the free list.
- enabling the `random_shuffle.cpp` test on platforms using the 32-bit primary
for Scudo.
Some comments on why the shuffling is done that way. Initially I just
implemented a `Shuffle` function in the `TransferBatch` which was simpler but I
came to realize this wasn't good enough: for chunks of 10000 bytes for example,
with a `CompactSizeClassMap`, a batch holds only 1 chunk, meaning shuffling the
batch has no effect, while a region is usually 1MB, eg: 104 chunks of that size.
So I decided to "stage" the newly gathered chunks in a temporary array that
would be shuffled prior to placing the chunks in batches.
The result is looping twice through n_chunks even if shuffling is not enabled,
but I didn't notice any significant significant performance impact.
Reviewers: alekseyshl
Reviewed By: alekseyshl
Subscribers: srhines, llvm-commits, kubamracek
Differential Revision: https://reviews.llvm.org/D39244
llvm-svn: 316596
ASan allocator stores the requested alignment for new and new[] calls
and on delete and delete[] verifies that alignments do match.
The representable alignments are: default alignment, 8, 16, 32, 64, 128,
256 and 512 bytes. Alignments > 512 are stored as 512, hence two
different alignments > 512 will pass the check (possibly masking the bug),
but limited memory requirements deemed to be a resonable tradeoff for
relaxed conditions.
The feature is controlled by new_delete_type_mismatch flag, the same one
protecting new/delete matching size check.
Differential revision: https://reviews.llvm.org/D38574
Issue: https://github.com/google/sanitizers/issues/799
llvm-svn: 316595
Dean Michael Berris