Commit Graph

114 Commits

Author SHA1 Message Date
Petr Hosek 8908aa4e30 [Fuzzer] Migrate Fuchsia port from launchpad to fdio_spawn
fdio_spawn is the new public API for launching processes in Fuchsia.

Differential Revision: https://reviews.llvm.org/D47649

llvm-svn: 333809
2018-06-02 01:17:10 +00:00
Kostya Serebryany 980e45fe55 [libFuzzer] add collect_data_flow.py that allows to run the data-flow tracer several times on subsets of inputs bytes, to overcome DFSan out-of-label failures
llvm-svn: 333616
2018-05-31 01:27:07 +00:00
Petr Hosek 796bb7d4ce [Fuzzer] Update _zx_port_wait function use in Fuchsia port
Fuchsia's _zx_port_wait no longer takes the count argument. This
change also updates all symbol uses to use the underscored versions.

Differential Revision: https://reviews.llvm.org/D47403

llvm-svn: 333328
2018-05-26 01:02:34 +00:00
George Karpenkov 02c85f1489 [libFuzzer] Run libFuzzer unit tests only on host architecture.
LIT tests are used to test other cross-compiled architectures,
unit tests are only run on the host.
NFC now as currently only a single architecture is supported.

Differential Revision: https://reviews.llvm.org/D47286

llvm-svn: 333241
2018-05-24 23:55:52 +00:00
Kostya Serebryany 4d53b744ca [libFuzzer] DataFlow tracer now tags a subset of the input. A separate script merges traces from the subsets
llvm-svn: 333149
2018-05-24 01:43:48 +00:00
Kostya Serebryany 500ca8713c [libFuzzer] fix two off-by-ones (!!) in the data flow tracer
llvm-svn: 333142
2018-05-23 23:55:54 +00:00
Kostya Serebryany 4925392897 [libFuzzer] change the output format for the DataFlow tracer
llvm-svn: 333122
2018-05-23 20:57:11 +00:00
Max Moroz fe97441c2c [libFuzzer] Don't complain about lack of interesting inputs when -runs=0.
Summary:
The most common usecase for -runs=0 is for generating code coverage
over some corpus. Coverage reports based on sancov are about to be deprecated,
which means some external coverage solution will be used, e.g. Clang source
based code coverage, which does not use any sancov instrumentations and thus
libFuzzer would consider any input to be not interesting in that case.

Reviewers: kcc

Reviewed By: kcc

Subscribers: alex, delcypher, #sanitizers, llvm-commits

Differential Revision: https://reviews.llvm.org/D47271

llvm-svn: 333116
2018-05-23 19:42:30 +00:00
Kostya Serebryany 7f5bf83f53 [libFuzzer] modify -print_corpus_stats to print whether the input reaches the focus function
llvm-svn: 333048
2018-05-23 01:42:53 +00:00
Kostya Serebryany 69c2b71a51 [libFuzzer] reinstate -dump_coverage, which is still in use (reverts r332036)
llvm-svn: 332876
2018-05-21 19:47:00 +00:00
Kostya Serebryany e9c6f06cce [libFuzzer] add an experimental flag -focus_function: libFuzzer will try to focus on inputs that trigger that function
llvm-svn: 332554
2018-05-16 23:26:37 +00:00
Kostya Serebryany 2f7edaeb39 [libFuzzer] deprecate equivalence_server
llvm-svn: 332316
2018-05-15 01:15:47 +00:00
Kostya Serebryany 6a6e690d24 [libFuzzer] refactor the implementation of -print_coverage
llvm-svn: 332073
2018-05-11 01:17:52 +00:00
Kostya Serebryany d80e821646 [libFuzzer] remove the dump_coverage flag, it hasn't been working with the inline sanitizer coverage anyway
llvm-svn: 332036
2018-05-10 20:24:39 +00:00
Kostya Serebryany 59bf1e8b36 [libFuzzer] remove the experimental support for clang coverage instrumentation. This mode has not been used and our experiments with https://github.com/google/fuzzer-test-suite show that this signal is weaker than the SanitizerCoverage
llvm-svn: 332034
2018-05-10 20:12:15 +00:00
Kostya Serebryany f489e2bfef [libFuzzer] Experimental data flow tracer for fuzz targets.
Summary:
Experimental data flow tracer for fuzz targets.
Allows to tell which bytes of the input affect which functions of the fuzz target.

We previously attempted to use DFSan directly in the libFuzzer process,
and that didn't work nicely.
Now we will try to collect the data flow information for the seed corpus
in a separate process (using this tracer), and then use it in the regular libFuzzer runs.

Reviewers: morehouse, pcc, Dor1s

Reviewed By: morehouse, Dor1s

Subscribers: delcypher, #sanitizers, llvm-commits

Differential Revision: https://reviews.llvm.org/D46666

llvm-svn: 332029
2018-05-10 19:59:01 +00:00
Kamil Rytarowski aee7b0f209 wrong usages of sem_open in the libFuzzer
Summary:
Fixed two non-standard usages of sem_open in the libFuzzer library and
one NetBSD-related modification with test script.


  - The return value to indicate error should be SEM_FAILED instead of
    (void *)-1 (please refer to "RETURN VALUE" section in this [[
    http://pubs.opengroup.org/onlinepubs/9699919799/functions/sem_open.html
    | page ]]). Actually, SEM_FAILED != (void *)-1 holds in NetBSD.

  - The SharedMemoryRegion::SemName function should return name
    starting with slash. Because the behaviour of name which does not
    start with slash is unspecified as the [[
    http://pubs.opengroup.org/onlinepubs/9699919799/functions/sem_open.html
    | "DESCRIPTION" section ]] specified:

> If name does not begin with the <slash> character, the effect is implementation-defined.

  - The length of name is limited to 14 in NetBSD, it is suggested to
    reduce the length of equivalence server name in the test script.

Patch by: Yang Zheng

Reviewers: vitalybuka, krytarowski, kcc

Reviewed By: kcc

Subscribers: kcc, #sanitizers, llvm-commits, joerg

Differential Revision: https://reviews.llvm.org/D46622

llvm-svn: 332003
2018-05-10 17:31:06 +00:00
Matt Morehouse 14cf71a3a5 [libFuzzer] Guard symbolization with try-lock.
Summary:
When out-of-memory or timeout occurs, threads can be stopped during
symbolization, thereby causing a deadlock when the OOM/TO handlers
attempt symbolization.  We avoid this deadlock by skipping symbolization
if another thread is symbolizing.

Reviewers: kcc

Reviewed By: kcc

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D46605

llvm-svn: 331825
2018-05-08 23:45:05 +00:00
Matt Morehouse 7764a04af0 [libFuzzer] Don't short-circuit from CrashCallback.
Short-circuiting causes tests to fail on Mac since libFuzzer crashes
rather than exiting with an error code when an unexpected signal
happens.

llvm-svn: 331324
2018-05-02 02:55:28 +00:00
Matt Morehouse 52fd169035 [libFuzzer] Report at most one crash per input.
Summary:
Fixes https://github.com/google/sanitizers/issues/788/, a deadlock
caused by multiple crashes happening at the same time.  Before printing
a crash report, we now test and set an atomic flag.  If the flag was
already set, the crash handler returns immediately.

Reviewers: kcc

Reviewed By: kcc

Subscribers: llvm-commits, kubamracek

Differential Revision: https://reviews.llvm.org/D46277

llvm-svn: 331310
2018-05-01 21:01:53 +00:00
Dan Liew dfd5a90a99 [LibFuzzer] Tweak `MutationDispatcher::Mutate_CopyPart` mutation.
It doesn't make sense to non-deterministically choose between
`CopyPart(..)` and `InsertPart(..)` when it is known that
`InsertPart(..)` will fail.

This upstream's a change from JFS solver's fork of LibFuzzer.

Differential Revision: https://reviews.llvm.org/D45693

llvm-svn: 330687
2018-04-24 06:31:09 +00:00
Matt Morehouse f66221c6ec [libFuzzer] Add feature to not use AFL's deferred forkserver.
A small but substantial minority of libFuzzer-based fuzzers run code that
does not play well with fork in global constructors or LLVMFuzzerInitialize.
This patch allows these fuzzers to use afl_driver by allowing them to
opt-out of using AFL's deferred forkserver which deferres calling fork until
after this code.

Patch By: metzman

Differential Revision: https://reviews.llvm.org/D45744

llvm-svn: 330652
2018-04-23 21:36:21 +00:00
Dan Liew b7a3bc99c9 [LibFuzzer] Report when custom counters are available.
This upstreams a feature from the JFS solver's fork of LibFuzzer.

Differential Revision: https://reviews.llvm.org/D45675

llvm-svn: 330391
2018-04-20 06:46:19 +00:00
Dan Liew 25d0c65ff3 [LibFuzzer] Try to unbreak the `FuzzerMutate.ShuffleBytes1` unit test.
This test is failing on my Linux box. Just increasing the number of
iterations works around this. The divergence is likely due to
our reliance on `std::shuffle()` which is not guaranteed to have
the same behaviour across platforms.

This is a strong argument for us to implement our own shuffle
function to avoid divergence in behaviour across platforms.

Differential Revision: https://reviews.llvm.org/D45767

llvm-svn: 330390
2018-04-20 06:46:14 +00:00
Dan Liew c8879c989e [LibFuzzer] Unbreak the `trace-malloc-unbalanced.test` when using Python 3.
The `unbalanced_allocs.py` script uses Python 2 print statement
and `iteritems()`. Running `2to3` over the script fixes these.

Differential Revision: https://reviews.llvm.org/D45765

llvm-svn: 330389
2018-04-20 06:46:09 +00:00
Petr Hosek dd097445f0 [Fuzzer] Add a missing header in Fuchsia implementation
This is needed for fd_set.

Differential Revision: https://reviews.llvm.org/D45852

llvm-svn: 330383
2018-04-20 00:41:06 +00:00
Petr Hosek 7a31c7ad23 [Fuzzer] Make InterruptHandler non-blocking for Fuchsia
The initial naive approach to simulate SIGINT on Fuchsia was to getchar
and look for ETX. This caused the InterruptHandler thread to lock stdin,
preventing musl's exit() from being able to close the stdio descriptors
and complete. This change uses select() instead.

Patch By: aarongreen

Differential Revision: https://reviews.llvm.org/D45636

llvm-svn: 330328
2018-04-19 14:01:46 +00:00
Petr Hosek 1ba8c8a2cc [Fuzzer] Avoid C++ compiler checks for fuzzer C++ library
When bulding libFuzzer as part of the toolchain, C++ library may not
have been fully built yet so the C++ compiler checks will fail, but we
don't care since we don't need C++ library anyway as we're building
our own.

Differential Revision: https://reviews.llvm.org/D45638

llvm-svn: 330075
2018-04-13 23:05:14 +00:00
Vitaly Buka 9802089e13 libFuzzer, OpenBSD support
Summary:
- Enabling libfuzzer on OpenBSD
- OpenBSD can t support asan, msan ... the tests can t be run.

Patch by David CARLIER

Reviewers: eugenis, phosek, vitalybuka

Reviewed By: vitalybuka

Subscribers: srhines, mgorny, krytarowski, llvm-commits, #sanitizers

Differential Revision: https://reviews.llvm.org/D44877

llvm-svn: 329631
2018-04-09 22:38:26 +00:00
Matt Morehouse fcc97b29c1 [libFuzzer] Print a correct error message when a directory can't be
opened.

Summary:
Currently if the directory cannot be opened for a reason other than
non-existence (e.g. too many open file descriptors) the error message
printed is incredibly confusing.

Patch By: Alex Gaynor

Reviewers: kcc, morehouse

Reviewed By: morehouse

Subscribers: delcypher, llvm-commits, Sanitizers

Differential Revision: https://reviews.llvm.org/D45322

llvm-svn: 329438
2018-04-06 18:15:24 +00:00
Mandeep Singh Grang a9e74a9165 Revert "[compiler-rt] Change std::sort to llvm::sort in response to r327219"
This reverts commit 2ee210e1963e03aacc0f71c50e4994bb5c66586e.

llvm-svn: 327936
2018-03-20 01:17:18 +00:00
Mandeep Singh Grang dc09ebf71b [compiler-rt] Change std::sort to llvm::sort in response to r327219
Summary:
r327219 added wrappers to std::sort which randomly shuffle the container before sorting.
This will help in uncovering non-determinism caused due to undefined sorting
order of objects having the same key.

To make use of that infrastructure we need to invoke llvm::sort instead of std::sort.

Reviewers: kcc, rsmith, RKSimon, eugenis

Reviewed By: RKSimon

Subscribers: efriedma, kubamracek, dberris, #sanitizers, llvm-commits

Differential Revision: https://reviews.llvm.org/D44360

llvm-svn: 327929
2018-03-20 00:44:59 +00:00
Petr Hosek 961235d335 [Fuzzer] Build the shared memory hooks for Fuchsia
This is needed otherwise we'll get undefined references when trying
to use the libFuzzer built for Fuchsia.

Differential Revision: https://reviews.llvm.org/D44590

llvm-svn: 327759
2018-03-16 22:40:55 +00:00
Sylvestre Ledru d9a8b6a745 fix some user facing typos / in the comments
llvm-svn: 327402
2018-03-13 14:35:10 +00:00
Petr Hosek 0b013e041d [Fuzzer] When building for Fuchsia, add an explicit libc++ dependency
libFuzzer dependes on C++ library, when building for Fuchsia, we
need to ensure that libFuzzer is only being built after libc++
has been built, so we add an explicity dependency on it.

Differential Revision: https://reviews.llvm.org/D44340

llvm-svn: 327196
2018-03-10 02:00:18 +00:00
Petr Hosek 2d19f9036e [Fuzzer] Avoid the unnecessary rebuild of the custom libc++
This changes the add_custom_libcxx macro to resemble the
llvm_ExternalProject_Add. The primary motivation is to avoid
unnecessary libFuzzer rebuilds that are being done on every
Ninja/Make invocation. The libc++ should be only rebuilt whenever
the libc++ source itself changes.

Differential Revision: https://reviews.llvm.org/D43213

llvm-svn: 326921
2018-03-07 18:14:09 +00:00
Matt Morehouse ddf352b953 [libFuzzer] Include TEMP_MAX_LEN in Fuzzer::PrintStats.
Reviewers: kcc

Reviewed By: kcc

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D43597

llvm-svn: 325817
2018-02-22 19:00:17 +00:00
Matt Morehouse 36c89b3c77 [libFuzzer] Set -experimental_len_control=1000 as default.
Summary:
Experiments using
https://github.com/google/fuzzer-test-suite/tree/master/engine-comparison
show a significant increase in coverage and reduction in corpus size
with this option enabled.

Addresses https://llvm.org/pr36371.

Reviewers: kcc

Reviewed By: kcc

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D42932

llvm-svn: 325050
2018-02-13 20:52:15 +00:00
Petr Hosek 2f70693e08 [Fuzzer] Use Zircon's public API on Fuchsia
The original libFuzzer Fuchsia port relied on convenience libraries,
but these are not exported as part of Fuchsia sysroot. This change
eliminates the use of these libraries and relies on public API only.

Differential Revision: https://reviews.llvm.org/D42996

llvm-svn: 324454
2018-02-07 08:22:58 +00:00
Jonas Hahnfeld 44ef345c50 [CMake] Remove -stdlib= which is unused when passing -nostdinc++
This avoids the warnings when building with LLVM_ENABLE_LIBCXX
which automatically adds -stdlib=libc++ to CMAKE_CXX_FLAGS.

Differential Revision: https://reviews.llvm.org/D42238

llvm-svn: 323969
2018-02-01 13:57:24 +00:00
Petr Hosek b1eaa56704 [fuzzer] Update and enable libFuzzer on Fuchsia
This change updates the Fuchsia-specific code to use the C++ friendly
duration expressions and flips on the building of
libclang_rt.fuzzer-x86_64.a and similar for Fuchsia. Given that
compiler-rt doesn't build on Fuchsia, test have been run by explicitly
building the library and linking it against
lib/fuzzer/tests/FuzzerUnittest.cpp.

Differential Revision: https://reviews.llvm.org/D42670

llvm-svn: 323828
2018-01-30 22:59:48 +00:00
Matt Morehouse 1d8e5ea250 [libFuzzer] Fix sizeof(ptr) bug.
sizeof(const char *) returns 4 or 8 when what we really want is the size
of the array.

llvm-svn: 323802
2018-01-30 18:25:55 +00:00
Petr Hosek 066e4bf888 Reland "[Fuzzer] Parametrize add_custom_libcxx"
add_custom_libcxx uses the just built compiler and installs the
built libc++, e.g. for testing, neither of which is desirable in
case of Fuzzer where the libc++ should be built using the host
compiler and it's only linked into the libFuzzer and should never
be installed. This change introduces additional arguments to
add_custom_libcxx to allow parametrizing its behavior.

Differential Revision: https://reviews.llvm.org/D42330

llvm-svn: 323054
2018-01-21 01:01:53 +00:00
Petr Hosek 81ac12d1b4 Revert "[Fuzzer] Parametrize add_custom_libcxx"
This reverts commit r323032: failing on the sanitizer-x86_64-linux-autoconf bot.

llvm-svn: 323033
2018-01-20 09:21:00 +00:00
Petr Hosek 94e67be187 [Fuzzer] Parametrize add_custom_libcxx
add_custom_libcxx uses the just built compiler and installs the
built libc++, e.g. for testing, neither of which is desirable in
case of Fuzzer where the libc++ should be built using the host
compiler and it's only linked into the libFuzzer and should never
be installed. This change introduces additional arguments to
add_custom_libcxx to allow parametrizing its behavior.

Differential Revision: https://reviews.llvm.org/D42330

llvm-svn: 323032
2018-01-20 09:03:16 +00:00
Petr Hosek eac2b47b9f Reland "[libFuzzer] Support using libc++"
This is needed in case the users of libFuzzer use libc++ in their
code, which the fuzz target (libFuzzer) will be linked against.
When libc++ source is available, we build a private version of it
and link it against libFuzzer which allows using the same static
library against codebases which use both libc++ and libstdc++.

Differential Revision: https://reviews.llvm.org/D37631

llvm-svn: 322755
2018-01-17 20:39:14 +00:00
Petr Hosek 68bc4bd6e5 Revert "[libFuzzer] Support using libc++"
This reverts commit r322604: test is failing for standalone compiler-rt.

llvm-svn: 322689
2018-01-17 17:24:56 +00:00
Petr Hosek a1b57e694e [libFuzzer] Support using libc++
This is needed in case the users of libFuzzer use libc++ in their
code, which the fuzz target (libFuzzer) will be linked against.
When libc++ source is available, we build a private version of it
and link it against libFuzzer which allows using the same static
library against codebases which use both libc++ and libstdc++.

Differential Revision: https://reviews.llvm.org/D37631

llvm-svn: 322604
2018-01-17 00:42:48 +00:00
Kamil Rytarowski 4463ae4f6d Switch from Bourne shell to simply base shell to build libfuzzer
Summary: It is not necessary launching the build script with bash.

Reviewers: krytarowski

Reviewed By: krytarowski

Subscribers: llvm-commits, #sanitizers

Differential Revision: https://reviews.llvm.org/D42008

llvm-svn: 322422
2018-01-12 23:45:24 +00:00
Kamil Rytarowski e81e944199 lib Fuzzer FreeBSD support
Summary: Patch by David CARLIER

Reviewers: vitalybuka, kcc, dim, emaste, davide, morehouse, george.karpenkov

Reviewed By: morehouse

Subscribers: george.karpenkov, kubamracek, srhines, mgorny, emaste, krytarowski

Differential Revision: https://reviews.llvm.org/D41642

llvm-svn: 322380
2018-01-12 17:15:05 +00:00