Summary:
1. Do not create DFSan labels for the bytes which we do not trace. This is where we run out of labels at the first place.
2. When dumping the traces on the disk, make sure to offset the label identifiers by the number of the first byte in the trace range.
3. For the last label, make sure to write it at the last position of the trace bit string, as that label represents the input size, not any particular byte.
Also fixed the bug with division in python which I've introduced when migrated the scripts to Python3 (`//` is required for integral division).
Otherwise, the scripts are wasting too much time unsuccessfully trying to
collect and process traces from the long inputs. For more context, see
https://github.com/google/oss-fuzz/issues/1632#issuecomment-481761789
Reviewers: kcc
Reviewed By: kcc
Subscribers: delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D60538
llvm-svn: 358311
Summary:
Before this change, MutateWithMask used to assert that Mask should be
of sufficient length (>= Size of the input). However, in real cases we may have
inputs that are longer than the Mask they have inherited from the based inputs.
Reviewers: kcc, morehouse
Reviewed By: kcc
Subscribers: delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D60571
llvm-svn: 358207
Summary:
In case the current corpus input doesn't have bytes going into the
focus function, MutateWithMask is useless and may fail gently, allowing the
default mutation routine happen, rather than crashing on an assertion.
For more context and the initial fix suggestion, see:
https://github.com/google/oss-fuzz/issues/1632#issuecomment-481862879
Reviewers: kcc, morehouse
Reviewed By: kcc
Subscribers: delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D60567
llvm-svn: 358190
Summary:
When using libfuzzer on Windows, in the contents of a crash sample, bytes that can be mistaken for a \n are replaced by a \r\n sequence. As a consequence, crashes are not reproducible. This patch will open files in binary mode to fix this issue. The patch does not affect POSIX systems.
Patch by tuktuk
Reviewers: kcc, vitalybuka
Reviewed By: vitalybuka
Subscribers: dexonsmith, jdoerfert, llvm-commits, #sanitizers
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D60008
llvm-svn: 357807
Summary:
This fixes inconsistent symbol visibility. This shows up as a linker
warning if r336238 (43f633564e338a6dde83d49a48e5bfcbfdce292c) is
reverted.
```
ld: warning: direct access in function 'fuzzer::CleanseCrashInput(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, fuzzer::fuzzer_allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, fuzzer::FuzzingOptions const&)' from file '/Volumes/data/dev/llvm/upstream/master/builds/projects/compiler-rt/lib/fuzzer/tests/libRTFuzzerTest.x86_64.a(FuzzerDriver.cpp.o)' to global weak symbol 'fuzzer::Command::ignoreRemainingArgs()::kIgnoreRemaining' from file 'FuzzerTestObjects.FuzzerUnittest.cpp.x86_64.o' means the weak symbol cannot be overridden
at runtime. This was likely caused by different translation units being compiled with different visibility settings.
```
r336238 just hid the issue rather than fixing the real issue. On macOS
and other platforms we usually compile with `-fvisibility=hidden` but
the unit tests were compiled without this flag.
Reviewers: george.karpenkov, kubamracek, kcc, yln
Subscribers: mgorny, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D58055
llvm-svn: 355143
Summary:
Port libFuzzer's fork mode to Windows.
Implement Windows versions of MkDir, RmDir, and IterateDirRecursive to do this.
Don't print error messages under new normal uses of FileSize (on a non-existent file).
Implement portable way of piping output to /dev/null.
Fix test for Windows and comment fork-sigusr.test on why it won't be ported to Win.
Reviewers: zturner
Reviewed By: zturner
Subscribers: kcc, zturner, jdoerfert, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D58513
llvm-svn: 355019
Summary:
When running the standalone main on a large corpus, I eventually get a
EMFILE error ("Too many open files").
Patch by Paul Chaignon
Reviewers: kcc, vitalybuka
Reviewed By: vitalybuka
Subscribers: lebedev.ri, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D58622
llvm-svn: 354918
According to the logs and local debugging there were two issues:
1) tsan tests listed libc++.a before the source file. That's usually
ok for shared libraries, but the linker will not add symbols from
a static library unless needed at that time. As a result the tests
that rely upon symbols from the library (and not only include the
headers) had undefined references.
To solve this I'm adding a new substitution %link_libcxx_tsan which
expands to libc++.a if available.
2) The target Fuzzer-x86_64-Test linked in SANITIZER_TEST_CXX_LIBRARIES
which defaults to -lstdc++. This resulted in error messages like
hidden symbol '_ZdlPv' is not defined locally
hidden symbol '_Znwm' is not defined locally
when using GNU gold (ld.bfd and lld are fine). Removing the linkage
is fine because we build a custom libc++ for that purpose.
llvm-svn: 354231
This changes add_custom_libcxx to also build libcxxabi and merges
the two into a static and hermetic library.
There are multiple advantages:
1) The resulting libFuzzer doesn't expose C++ internals and looks
like a plain C library.
2) We don't have to manually link in libstdc++ to provide cxxabi.
3) The sanitizer tests cannot interfere with an installed version
of libc++.so in LD_LIBRARY_PATH.
Differential Revision: https://reviews.llvm.org/D58013
llvm-svn: 354212
Add missed value "libcxxabi" and introduce SANITIZER_TEST_CXX for linking
unit tests. This needs to be a full C++ library and cannot be libcxxabi.
Recommit r354132 which I reverted in r354153 because it broke a sanitizer
bot. This was because of the "fixes" for pthread linking, so I've removed
these changes.
Differential Revision: https://reviews.llvm.org/D58012
llvm-svn: 354198
Add missed value "libcxxabi" and introduce SANITIZER_TEST_CXX for linking
unit tests. This needs to be a full C++ library and cannot be libcxxabi.
Differential Revision: https://reviews.llvm.org/D58012
llvm-svn: 354132
The __sanitizer_acquire_crash_state function has int return type, but the
fuzzer's external function definitions give it bool.
Places where __sanitizer_acquire_crash_state is declared:
include/sanitizer_common/sanitizer_interface_defs.h
lib/sanitizer_common/sanitizer_interface_internal.h
lib/sanitizer_common/sanitizer_common.cc
lib/fuzzer/FuzzerExtFunctions.def (this is the only bool)
llvm-svn: 353596
Summary:
Use `_write` instead of the deprecated alias `write` on Windows.
Also, remove comment saying RawPrint is untested on Windows.
Reviewers: vitalybuka
Reviewed By: vitalybuka
Subscribers: vitalybuka
Differential Revision: https://reviews.llvm.org/D57589
llvm-svn: 353108
Summary:
Use RawPrint instead of Printf for instrumentation warning because
Printf doesn't work on Win when instrumentation is being
initialized (since OutputFile is not yet initialized).
Reviewers: kcc
Reviewed By: kcc
Differential Revision: https://reviews.llvm.org/D57531
llvm-svn: 352789
Summary:
Set default `ASAN_OPTIONS` when running libFuzzer tests. This allows us
to remove special casing in code for Darwin where we usually pass
`abort_on_error=0` to override platform defaults for tests.
A previous commit changed the code to make the tests pass:
7764a04af0
Adapted a few tests to use `%env_asan_opts=` instead of directly setting
the environment variable.
rdar://problem/47515276
Reviewers: kcc, george.karpenkov
Differential Revision: https://reviews.llvm.org/D57465
llvm-svn: 352711
Summary:
Use dllexport for all declarations in FuzzerInterface.h Use it for clang
even though clang supports default visibility attribute to prevent a
warning from being thrown when LLVMFuzzerMutate is defined with dllexport.
This makes `FUZZER_INTERFACE_VISIBILITY` (FuzzerInterface.h) consistent with
`ATTRIBUTE_INTERFACE` (FuzzerDefs.h) when using clang on Windows.
Reviewers: vitalybuka, morehouse
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D57305
llvm-svn: 352395
Summary:
Remove this feature as it is unused, buggy, and not worth correcting
since the forkserver makes it difficult.
Reviewers: morehouse, jfb
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D57308
llvm-svn: 352392
Summary:
Enable building libFuzzer with MSVC.
* Don't try to include <endian.h> in FuzzerSHA1.cpp. MSVC
doesn't have this header, and WINDOWS is always little
endian (even on ARM)
Subscribers: srhines, mgorny, javed.absar, kristof.beyls
Differential Revision: https://reviews.llvm.org/D56510
llvm-svn: 351855
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
Summary:
Remove code for handling unstable edges from libFuzzer since
it has not been found useful.
Differential Revision: https://reviews.llvm.org/D56730
llvm-svn: 351262
Summary:
Use alternatename for external functions only when using
MSVC since Clang doesn't support it and MSVC doesn't support
Clang's method (weak aliases).
Reviewers: morehouse
Reviewed By: morehouse
Subscribers: rnk, thakis, mgorny
Differential Revision: https://reviews.llvm.org/D56514
llvm-svn: 351152
Summary:
Replace calls to builtin functions with macros or functions that call the
Windows-equivalents when targeting windows and call the original
builtin functions everywhere else.
This change makes more parts of libFuzzer buildable with MSVC.
Reviewers: vitalybuka
Reviewed By: vitalybuka
Subscribers: mgorny, rnk, thakis
Differential Revision: https://reviews.llvm.org/D56439
llvm-svn: 350766
Include CompilerRTCompile in fuzzer tests explicitly. Otherwise, when
building only libFuzzer, CMake fails due to:
CMake Error at cmake/Modules/AddCompilerRT.cmake:395 (sanitizer_test_compile):
Unknown CMake command "sanitizer_test_compile".
Call Stack (most recent call first):
lib/fuzzer/tests/CMakeLists.txt:53 (generate_compiler_rt_tests)
Differential Revision: https://reviews.llvm.org/D55378
llvm-svn: 348524
This fixes the issue introduced in r345765 which changed the way in
which the embedded libc++ is being built but omitted tests.
Differential Revision: https://reviews.llvm.org/D54058
llvm-svn: 346052
libc++ now supports customizing the ABI namespace directly from the
CMake build so we no longer need to rely on custom CFLAGS.
Differential Revision: https://reviews.llvm.org/D53939
llvm-svn: 345765
Summary:
Silence warning when linking unittest binary by not passing
-lstdc++ to the linker since it is ignored.
Reviewers: morehouse
Reviewed By: morehouse
Subscribers: mgorny
Differential Revision: https://reviews.llvm.org/D53225
llvm-svn: 344480
Without this change, tests in coverage.test and dump_coverage.test are
failing on non-x86_64 platforms.
The diff is copied from sanitizer_common library, an alternative would
be to link it together with libFuzzer.
Differential Revision: https://reviews.llvm.org/D53040
llvm-svn: 344104
This change fixes two aspects of logging on Fuchsia:
1. When logging to stdout, the output file descriptor should not be
closed in ExecuteCommand, as it prevent fdio_spawn_etc from succeeding
in subsequent calls.
2. When logging to a file in anything other than standalone Zircon, the
log file needs to be created in mutable storage. The best indicator
where this is relative to the Fuchsia component will be given by
'-artifact_prefix=...', so save the log file relative to that path.
Patch by: aarongreen
Differential Revision: https://reviews.llvm.org/D52562
llvm-svn: 343607
Summary:
Replace FuzzerExtFunctionsDlsymWin.cpp with FuzzerExtFunctionsWeakAlias.cpp
to get externally defined functions (eg: LLVMFuzzerInitialize,
LLVMFuzzerCustomMutator, etc) working again.
Also enable tests that depended on these functions (on windows)
Reviewers: rnk, morehouse
Reviewed By: rnk, morehouse
Subscribers: rnk, morehouse, mgorny
Differential Revision: https://reviews.llvm.org/D51700
llvm-svn: 342698
This fixes building on a case sensitive filesystem with mingw-w64
headers, where all headers are lowercase, and matches how these
headers are included elsewhere in compiler-rt.
Also include these headers with angle brackets, as they are system
headers.
Differential Revision: https://reviews.llvm.org/D51913
llvm-svn: 341983
Summary:
Enable tests on Windows and make check-fuzzer pass on it. Make
check-fuzzer pass on Windows by fixing libFuzzer, fixing tests, and by
disabling tests on Windows. Most of these are disabled temporarily as
support for the tests and the features they test will be added
incrementally. Other tests will not be enabled since they require
things that are not on Windows (eg: afl_driver tests). Every test
that was explicitly disabled on Windows has a comment explaining why
(unless obvious like merge-posix.test).
The lit.cfg file was modified to support running tests on windows.
fuzzer-dirs.test was fixed by making the Windows implementation print
the same error message as the posix version.
merge-control-file.test was fixed by making the test binary end with
the ".exe" extension (on all platforms).
Patch By: metzman
Reviewers: morehouse
Reviewed By: morehouse
Subscribers: srhines, mgorny
Differential Revision: https://reviews.llvm.org/D51549
llvm-svn: 341385
Summary:
Port libFuzzer to windows-msvc.
This patch allows libFuzzer targets to be built and run on Windows, using -fsanitize=fuzzer and/or fsanitize=fuzzer-no-link. It allows these forms of coverage instrumentation to work on Windows as well.
It does not fix all issues, such as those with -fsanitize-coverage=stack-depth, which is not usable on Windows as of this patch.
It also does not fix any libFuzzer integration tests. Nearly all of them fail to compile, fixing them will come in a later patch, so libFuzzer tests are disabled on Windows until them.
Patch By: metzman
Reviewers: morehouse, rnk
Reviewed By: morehouse, rnk
Subscribers: #sanitizers, delcypher, morehouse, kcc, eraman
Differential Revision: https://reviews.llvm.org/D51022
llvm-svn: 341082
Summary:
This was an experimental feature. After evaluating it with:
1) https://github.com/google/fuzzer-test-suite/tree/master/engine-comparison
2) enabling on real world fuzz targets running at ClusterFuzz and OSS-Fuzz
The following conclusions were made:
1) With fuzz targets that have reached a code coverage plateau, the feature does
not improve libFuzzer's ability to discover new coverage and may actually
negatively impact it.
2) With fuzz targets that have not yet reached a code coverage plateau, the
feature might speed up new units discovery in some cases, but it is quite
rare and hard to confirm with a high level on confidence.
Revert of https://reviews.llvm.org/D48054 and https://reviews.llvm.org/D49621.
Reviewers: metzman, morehouse
Reviewed By: metzman, morehouse
Subscribers: delcypher, #sanitizers, llvm-commits, kcc
Differential Revision: https://reviews.llvm.org/D51455
llvm-svn: 340976
Summary:
Port libFuzzer to windows-msvc.
This patch allows libFuzzer targets to be built and run on Windows, using -fsanitize=fuzzer and/or fsanitize=fuzzer-no-link. It allows these forms of coverage instrumentation to work on Windows as well.
It does not fix all issues, such as those with -fsanitize-coverage=stack-depth, which is not usable on Windows as of this patch.
It also does not fix any libFuzzer integration tests. Nearly all of them fail to compile, fixing them will come in a later patch, so libFuzzer tests are disabled on Windows until them.
Reviewers: morehouse, rnk
Reviewed By: morehouse, rnk
Subscribers: #sanitizers, delcypher, morehouse, kcc, eraman
Differential Revision: https://reviews.llvm.org/D51022
llvm-svn: 340949
Summary:
Port libFuzzer to windows-msvc.
This patch allows libFuzzer targets to be built and run on Windows, using -fsanitize=fuzzer and/or fsanitize=fuzzer-no-link. It allows these forms of coverage instrumentation to work on Windows as well.
It does not fix all issues, such as those with -fsanitize-coverage=stack-depth, which is not usable on Windows as of this patch.
It also does not fix any libFuzzer integration tests. Nearly all of them fail to compile, fixing them will come in a later patch, so libFuzzer tests are disabled on Windows until them.
Patch By: metzman
Reviewers: morehouse, rnk
Reviewed By: morehouse, rnk
Subscribers: morehouse, kcc, eraman
Differential Revision: https://reviews.llvm.org/D51022
llvm-svn: 340860
Now that all Zircon calls have been transitioned to take time as signed
value, we can transition back to ZX_TIME_INFINITE, undoing the change
made in r337802.
Differential Revision: https://reviews.llvm.org/D51266
llvm-svn: 340764
Summary:
Since we're casting from double to size_t during input selection, we
really want a discrete distribution over size_t rather than a piecewise
distribution over doubles.
Reviewers: kcc
Reviewed By: kcc
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D50356
llvm-svn: 339973
Summary:
We only run the 3rd check if 2nd check finds unstable edges.
3rd UpdateUnstableCounters is now merged with ApplyUnstableCounters to only run 1 iteration.
Patch by Kyungtak Woo (@kevinwkt).
Reviewers: Dor1s, metzman, morehouse
Reviewed By: Dor1s, morehouse
Subscribers: delcypher, #sanitizers, llvm-commits, kcc
Differential Revision: https://reviews.llvm.org/D50411
llvm-svn: 339249
Summary:
There may be cases in which a user wants to know which part of their code is unstable.
We use ObservedFuncs and UnstableCounters to print at exit which of the ObservedFunctions
are unstable under the -print_unstable_stats flag.
Patch by Kyungtak Woo (@kevinwkt).
Reviewers: Dor1s, metzman, morehouse
Reviewed By: Dor1s, metzman, morehouse
Subscribers: delcypher, #sanitizers, llvm-commits, kcc
Differential Revision: https://reviews.llvm.org/D50264
llvm-svn: 339081
Summary:
Added functions that calculate stats while fuzz targets are running and give
mutations weight based on how much new coverage they provide, and choose better
performing mutations more often.
Patch by Kodé Williams (@kodewilliams).
Reviewers: Dor1s, metzman, morehouse
Reviewed By: Dor1s, morehouse
Subscribers: delcypher, kcc, llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D49621
llvm-svn: 338776
Summary:
Added a new mode within flag -handle_unstable for new unstable handling algorithm that does the following:
When an edge is shown as unstable, copy to UnstableCounters the value 0.
During ApplyUnstableCounters we copy back the value 0 to ModuleInline8bitCounters if the edge was unstable.
This way we would be ignoring completely features that were collected through non-determinism.
Unstable hits would be counted as if it never hit.
Reviewers: metzman, Dor1s, kcc, morehouse
Reviewed By: metzman, morehouse
Subscribers: delcypher, llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D49684
llvm-svn: 337853
This is a preparation for breaking change when all Zircon calls that
take time as an argument will start using signed valued. We will
transition back to ZX_TIME_INFITINE after all the changes to these
symbols are done and become part of the Fuchsia SDK.
Differential Revision: https://reviews.llvm.org/D49694
llvm-svn: 337802
The path to headers which are installed into libc++ build directory
has changed in r337727 which broke the libFuzzer build.
Differential Revision: https://reviews.llvm.org/D49705
llvm-svn: 337775
Summary:
Created unstable_handle flag that takes 1 or 2, depending on the handling type.
Modified RunOne to accommodate the following heuristic:
Use the first CollectFeatures to count how many features there are.
If no new features, CollectFeatures like before.
If there is new feature, we run CB 2 more times,
Check which edges are unstable per input and we store the least amount of hit counts for each edge.
Apply these hit counts back to inline8bitcounters so that CollectFeatures can work as intended.
Modified UnstableCounters to 8int_t and created a bitset UnstableSet to tell which edges are unstable.
Patch by Kyungtak Woo (@kevinwkt).
Reviewers: Dor1s, metzman, morehouse
Reviewed By: Dor1s, morehouse
Subscribers: delcypher, #sanitizers, llvm-commits, kcc
Differential Revision: https://reviews.llvm.org/D49525
llvm-svn: 337696
Fuchsia doesn't have signals; instead it expects processes to have a
dedicated exception thread that binds to the process' exception port and
waits for exception packets to be delivered. On the other hand,
libFuzzer and sanitizer_common use expect to collect crash information
via libunwind from the same thread that caused the exception.
The long term fix is to improve support for remote unwinding in
libunbwind, plumb this through sanitizer_common and libFuzzer, and
handle the exception exclusively on the exception thread. In the
meantime, this revision has the exception thread "resurrect" the
crashing thread by:
* saving its general purpose register state onto the crashing thread's
stack,
* setting the crashing thread's program counter to an assembly trampoline
with the CFI information needed by libunwind, and
* resuming the crashed thread.
Patch By: aarongreen
Differential Revision: https://reviews.llvm.org/D48509
llvm-svn: 337418
Summary:
Created IterateInline8bitCounters, a single template for visiting Inline8bitCounters (nested for loop)
Made InitializeUnstableCounters and UpdateUnstableCounters both send a lambda to IterateInline8bitCounters.
Patch by Kyungtak Woo (@kevinwkt).
Reviewers: Dor1s, metzman, kcc, morehouse
Reviewed By: metzman, morehouse
Subscribers: delcypher, llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D49453
llvm-svn: 337403
Summary:
Code now exists to track number of mutations that are used in fuzzing in total
and ones that produce new coverage. The stats are currently being dumped to the
command line.
Patch by Kodé Williams (@kodewilliams).
Reviewers: metzman, Dor1s, morehouse, kcc
Reviewed By: Dor1s, morehouse, kcc
Subscribers: delcypher, kubamracek, kcc, morehouse, llvm-commits, #sanitizers, mgorny
Differential Revision: https://reviews.llvm.org/D48054
llvm-svn: 337324
This is a fix for bug 37047.
https://bugs.llvm.org/show_bug.cgi?id=37047
Implemented by basically reversing the logic. Previously all strings
were considered, with some operations excluded. Now strings are excluded
by default, and only strings during the CB considered.
Patch By: pdknsk
Differential Revision: https://reviews.llvm.org/D48800
llvm-svn: 337296
We no longer pass CLANG_DEFAULT_CXX_STDLIB to the runtimes build
as it was causing issues so we can no longer use this variable. We
instead use cxx-headers as a dependency whenever this is available
since both XRay and libFuzzer are built as static libraries so this
is sufficient.
Differential Revision: https://reviews.llvm.org/D49346
llvm-svn: 337199
Summary:
Code now exists to track number of mutations that are used in fuzzing in total
and ones that produce new coverage. The stats are currently being dumped to the
command line.
Patch by Kodé Williams (@kodewilliams).
Reviewers: metzman, Dor1s, morehouse, kcc
Reviewed By: Dor1s, morehouse, kcc
Subscribers: delcypher, kubamracek, kcc, morehouse, llvm-commits, #sanitizers, mgorny
Differential Revision: https://reviews.llvm.org/D48054
llvm-svn: 337194
Summary:
Created a -print_unstable_stats flag.
When -print_unstable_stats=1, we run it 2 more times on interesting inputs poisoning unstable edges in an array.
On program termination, we run PrintUnstableStats() which will print a line with a stability percentage like AFL does.
Patch by Kyungtak Woo (@kevinwkt).
Reviewers: metzman, Dor1s, kcc, morehouse
Reviewed By: metzman, Dor1s, morehouse
Subscribers: delcypher, llvm-commits, #sanitizers, kcc, morehouse, Dor1s
Differential Revision: https://reviews.llvm.org/D49212
llvm-svn: 337187
Summary:
Created a -print_unstable_stats flag.
When -print_unstable_stats=1, we run it 2 more times on interesting inputs poisoning unstable edges in an array.
On program termination, we run PrintUnstableStats() which will print a line with a stability percentage like AFL does.
Patch by Kyungtak Woo (@kevinwkt).
Reviewers: metzman, Dor1s, kcc, morehouse
Reviewed By: metzman, Dor1s, morehouse
Subscribers: delcypher, llvm-commits, #sanitizers, kcc, morehouse, Dor1s
Differential Revision: https://reviews.llvm.org/D49212
llvm-svn: 337175
This reverts rL334510 due to breakage of afl_driver's command line
interface.
Patch By: Jonathan Metzman
Differential Revision: https://reviews.llvm.org/D49141
llvm-svn: 336719
when building with an IDE so that header files show up in the UI.
This massively improves the development workflow in IDEs.
To implement this a new function `compiler_rt_process_sources(...)` has
been added that adds header files to the list of sources when the
generator is an IDE. For non-IDE generators (e.g. Ninja/Makefile) no
changes are made to the list of source files.
The function can be passed a list of headers via the
`ADDITIONAL_HEADERS` argument. For each runtime library a list of
explicit header files has been added and passed via
`ADDITIONAL_HEADERS`. For `tsan` and `sanitizer_common` a list of
headers was already present but it was stale and has been updated
to reflect the current state of the source tree.
The original version of this patch used file globbing (`*.{h,inc,def}`)
to find the headers but the approach was changed due to this being a
CMake anti-pattern (if the list of headers changes CMake won't
automatically re-generate if globbing is used).
The LLVM repo contains a similar function named `llvm_process_sources()`
but we don't use it here for several reasons:
* It depends on the `LLVM_ENABLE_OPTION` cache variable which is
not set in standalone compiler-rt builds.
* We would have to `include(LLVMProcessSources)` which I'd like to
avoid because it would include a bunch of stuff we don't need.
Differential Revision: https://reviews.llvm.org/D48422
llvm-svn: 336663
Code now exists to track number of mutations that are used in fuzzing in
total and ones that produce new coverage. The stats are currently being
dumped to the command line.
Patch By: Kode Williams
Differntial Revision: https://reviews.llvm.org/D48054
llvm-svn: 336597
Summary: This is a minor cosmetic change. When function/path exceed ~1000 characters, the output is truncated before the line-break. I noticed this for NEW_FUNC.
Reviewers: kcc
Reviewed By: kcc
Subscribers: llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D48799
llvm-svn: 336461
Some warnings originating from googletest were causing bots to fail
while bulding unit tests. The sanitizers address this issue by not
using -Werror. We adopt this approach for libFuzzer.
llvm-svn: 335640
On targets that don't link internal libc++ (Fuchsia and Linux) but
use libc++ as their C++ library and libFuzzer is being built using
the just built compiler together with libc++ as part of runtimes, we
need an explicit dependency from libFuzzer object library to libc++ to
make sure the headers are available by the time we start building
libFuzzer.
Differential Revision: https://reviews.llvm.org/D48261
llvm-svn: 334928
Don't hardcode the architecture for Fuzzer tests which breaks when
compiler-rt is being compiled for architectures other than x86_64.
Differential Revision: https://reviews.llvm.org/D48207
llvm-svn: 334852
Handling differently the iterations with the type limit and eventually an error message.
Reviewers: morehouse, kcc
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D47880
llvm-svn: 334510
Summary: Otherwise we print things like [0/1] which is visually confusing.
Reviewers: kcc
Reviewed By: kcc
Subscribers: delcypher, llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D47837
llvm-svn: 334234
On Fuchsia, we use libc++ compiled with ASan for our ASan built
executable which means we cannot use the same libc++ for libFuzzer when
building fuzz targets, instead we'll link a custom internal libc++ into
Fuchsia's build of libFuzzer like we already do on Linux.
Differential Revision: https://reviews.llvm.org/D47835
llvm-svn: 334144
By adding a ctor to create fuzzer_allocator<T> from fuzzer_allocator<U>.
This mimics construcotrs of std::allocator<T>.
Without the constructors, some versions of libstdc++ can't compile
`vector<bool, fuzzer_allocator<bool>>`.
llvm-svn: 334077
Fuchsia's _zx_port_wait no longer takes the count argument. This
change also updates all symbol uses to use the underscored versions.
Differential Revision: https://reviews.llvm.org/D47403
llvm-svn: 333328
LIT tests are used to test other cross-compiled architectures,
unit tests are only run on the host.
NFC now as currently only a single architecture is supported.
Differential Revision: https://reviews.llvm.org/D47286
llvm-svn: 333241
Summary:
The most common usecase for -runs=0 is for generating code coverage
over some corpus. Coverage reports based on sancov are about to be deprecated,
which means some external coverage solution will be used, e.g. Clang source
based code coverage, which does not use any sancov instrumentations and thus
libFuzzer would consider any input to be not interesting in that case.
Reviewers: kcc
Reviewed By: kcc
Subscribers: alex, delcypher, #sanitizers, llvm-commits
Differential Revision: https://reviews.llvm.org/D47271
llvm-svn: 333116
Summary:
Experimental data flow tracer for fuzz targets.
Allows to tell which bytes of the input affect which functions of the fuzz target.
We previously attempted to use DFSan directly in the libFuzzer process,
and that didn't work nicely.
Now we will try to collect the data flow information for the seed corpus
in a separate process (using this tracer), and then use it in the regular libFuzzer runs.
Reviewers: morehouse, pcc, Dor1s
Reviewed By: morehouse, Dor1s
Subscribers: delcypher, #sanitizers, llvm-commits
Differential Revision: https://reviews.llvm.org/D46666
llvm-svn: 332029
Summary:
Fixed two non-standard usages of sem_open in the libFuzzer library and
one NetBSD-related modification with test script.
- The return value to indicate error should be SEM_FAILED instead of
(void *)-1 (please refer to "RETURN VALUE" section in this [[
http://pubs.opengroup.org/onlinepubs/9699919799/functions/sem_open.html
| page ]]). Actually, SEM_FAILED != (void *)-1 holds in NetBSD.
- The SharedMemoryRegion::SemName function should return name
starting with slash. Because the behaviour of name which does not
start with slash is unspecified as the [[
http://pubs.opengroup.org/onlinepubs/9699919799/functions/sem_open.html
| "DESCRIPTION" section ]] specified:
> If name does not begin with the <slash> character, the effect is implementation-defined.
- The length of name is limited to 14 in NetBSD, it is suggested to
reduce the length of equivalence server name in the test script.
Patch by: Yang Zheng
Reviewers: vitalybuka, krytarowski, kcc
Reviewed By: kcc
Subscribers: kcc, #sanitizers, llvm-commits, joerg
Differential Revision: https://reviews.llvm.org/D46622
llvm-svn: 332003
Summary:
When out-of-memory or timeout occurs, threads can be stopped during
symbolization, thereby causing a deadlock when the OOM/TO handlers
attempt symbolization. We avoid this deadlock by skipping symbolization
if another thread is symbolizing.
Reviewers: kcc
Reviewed By: kcc
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D46605
llvm-svn: 331825
Short-circuiting causes tests to fail on Mac since libFuzzer crashes
rather than exiting with an error code when an unexpected signal
happens.
llvm-svn: 331324
Summary:
Fixes https://github.com/google/sanitizers/issues/788/, a deadlock
caused by multiple crashes happening at the same time. Before printing
a crash report, we now test and set an atomic flag. If the flag was
already set, the crash handler returns immediately.
Reviewers: kcc
Reviewed By: kcc
Subscribers: llvm-commits, kubamracek
Differential Revision: https://reviews.llvm.org/D46277
llvm-svn: 331310
It doesn't make sense to non-deterministically choose between
`CopyPart(..)` and `InsertPart(..)` when it is known that
`InsertPart(..)` will fail.
This upstream's a change from JFS solver's fork of LibFuzzer.
Differential Revision: https://reviews.llvm.org/D45693
llvm-svn: 330687
A small but substantial minority of libFuzzer-based fuzzers run code that
does not play well with fork in global constructors or LLVMFuzzerInitialize.
This patch allows these fuzzers to use afl_driver by allowing them to
opt-out of using AFL's deferred forkserver which deferres calling fork until
after this code.
Patch By: metzman
Differential Revision: https://reviews.llvm.org/D45744
llvm-svn: 330652
This test is failing on my Linux box. Just increasing the number of
iterations works around this. The divergence is likely due to
our reliance on `std::shuffle()` which is not guaranteed to have
the same behaviour across platforms.
This is a strong argument for us to implement our own shuffle
function to avoid divergence in behaviour across platforms.
Differential Revision: https://reviews.llvm.org/D45767
llvm-svn: 330390
The initial naive approach to simulate SIGINT on Fuchsia was to getchar
and look for ETX. This caused the InterruptHandler thread to lock stdin,
preventing musl's exit() from being able to close the stdio descriptors
and complete. This change uses select() instead.
Patch By: aarongreen
Differential Revision: https://reviews.llvm.org/D45636
llvm-svn: 330328
When bulding libFuzzer as part of the toolchain, C++ library may not
have been fully built yet so the C++ compiler checks will fail, but we
don't care since we don't need C++ library anyway as we're building
our own.
Differential Revision: https://reviews.llvm.org/D45638
llvm-svn: 330075
Summary:
- Enabling libfuzzer on OpenBSD
- OpenBSD can t support asan, msan ... the tests can t be run.
Patch by David CARLIER
Reviewers: eugenis, phosek, vitalybuka
Reviewed By: vitalybuka
Subscribers: srhines, mgorny, krytarowski, llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D44877
llvm-svn: 329631
opened.
Summary:
Currently if the directory cannot be opened for a reason other than
non-existence (e.g. too many open file descriptors) the error message
printed is incredibly confusing.
Patch By: Alex Gaynor
Reviewers: kcc, morehouse
Reviewed By: morehouse
Subscribers: delcypher, llvm-commits, Sanitizers
Differential Revision: https://reviews.llvm.org/D45322
llvm-svn: 329438
Summary:
r327219 added wrappers to std::sort which randomly shuffle the container before sorting.
This will help in uncovering non-determinism caused due to undefined sorting
order of objects having the same key.
To make use of that infrastructure we need to invoke llvm::sort instead of std::sort.
Reviewers: kcc, rsmith, RKSimon, eugenis
Reviewed By: RKSimon
Subscribers: efriedma, kubamracek, dberris, #sanitizers, llvm-commits
Differential Revision: https://reviews.llvm.org/D44360
llvm-svn: 327929
This is needed otherwise we'll get undefined references when trying
to use the libFuzzer built for Fuchsia.
Differential Revision: https://reviews.llvm.org/D44590
llvm-svn: 327759
libFuzzer dependes on C++ library, when building for Fuchsia, we
need to ensure that libFuzzer is only being built after libc++
has been built, so we add an explicity dependency on it.
Differential Revision: https://reviews.llvm.org/D44340
llvm-svn: 327196
This changes the add_custom_libcxx macro to resemble the
llvm_ExternalProject_Add. The primary motivation is to avoid
unnecessary libFuzzer rebuilds that are being done on every
Ninja/Make invocation. The libc++ should be only rebuilt whenever
the libc++ source itself changes.
Differential Revision: https://reviews.llvm.org/D43213
llvm-svn: 326921
The original libFuzzer Fuchsia port relied on convenience libraries,
but these are not exported as part of Fuchsia sysroot. This change
eliminates the use of these libraries and relies on public API only.
Differential Revision: https://reviews.llvm.org/D42996
llvm-svn: 324454
This avoids the warnings when building with LLVM_ENABLE_LIBCXX
which automatically adds -stdlib=libc++ to CMAKE_CXX_FLAGS.
Differential Revision: https://reviews.llvm.org/D42238
llvm-svn: 323969
This change updates the Fuchsia-specific code to use the C++ friendly
duration expressions and flips on the building of
libclang_rt.fuzzer-x86_64.a and similar for Fuchsia. Given that
compiler-rt doesn't build on Fuchsia, test have been run by explicitly
building the library and linking it against
lib/fuzzer/tests/FuzzerUnittest.cpp.
Differential Revision: https://reviews.llvm.org/D42670
llvm-svn: 323828
add_custom_libcxx uses the just built compiler and installs the
built libc++, e.g. for testing, neither of which is desirable in
case of Fuzzer where the libc++ should be built using the host
compiler and it's only linked into the libFuzzer and should never
be installed. This change introduces additional arguments to
add_custom_libcxx to allow parametrizing its behavior.
Differential Revision: https://reviews.llvm.org/D42330
llvm-svn: 323054
add_custom_libcxx uses the just built compiler and installs the
built libc++, e.g. for testing, neither of which is desirable in
case of Fuzzer where the libc++ should be built using the host
compiler and it's only linked into the libFuzzer and should never
be installed. This change introduces additional arguments to
add_custom_libcxx to allow parametrizing its behavior.
Differential Revision: https://reviews.llvm.org/D42330
llvm-svn: 323032
This is needed in case the users of libFuzzer use libc++ in their
code, which the fuzz target (libFuzzer) will be linked against.
When libc++ source is available, we build a private version of it
and link it against libFuzzer which allows using the same static
library against codebases which use both libc++ and libstdc++.
Differential Revision: https://reviews.llvm.org/D37631
llvm-svn: 322755
This is needed in case the users of libFuzzer use libc++ in their
code, which the fuzz target (libFuzzer) will be linked against.
When libc++ source is available, we build a private version of it
and link it against libFuzzer which allows using the same static
library against codebases which use both libc++ and libstdc++.
Differential Revision: https://reviews.llvm.org/D37631
llvm-svn: 322604
Summary: It is not necessary launching the build script with bash.
Reviewers: krytarowski
Reviewed By: krytarowski
Subscribers: llvm-commits, #sanitizers
Differential Revision: https://reviews.llvm.org/D42008
llvm-svn: 322422
Summary:
Add dummy call of LLVMFuzzerTestOneInput to afl_driver before it starts
executing on actual inputs. Do this so that first time initialization
performed by LLVMFuzzerTestOneInput is not considered code covered by
a particular input.
Patch By: metzman
Reviewers: kcc, morehouse
Reviewed By: kcc
Subscribers: llvm-commits, Sanitizers
Differential Revision: https://reviews.llvm.org/D41193
llvm-svn: 320643
Summary:
This patch adds the initial support for Fuchsia.
- LIBFUZZER_FUCHSIA is added as an OS type in FuzzerDefs.h
- Fuchsia is, by design, not POSIX compliant. However, it does use ELF and
supports common POSIX I/O functions. Thus, FuzzerExtFunctions.h and
FuzzerIO.h are implemented by extending the header guards in
FuzzerExtFunctionsWeak.cpp and FuzzerIOPosix.cpp to include
LIBFUZZER_FUCHSIA.
- The platform-specific portions of FuzzerUtil.h are implemented by
FuzzerUtilFuchsia.cpp, which makes use of exception ports, syscalls, and
the launchpad library.
- The experimental equivalence server is not currently supported, so
FuzzerShmem.h is implemented by stub methods in FuzzerShmemFuchsia.cpp.
Any future implementation will likely involve VMOs.
Tested with ASAN/SanCov on Fuchsia/x86-64 with the canonical toy fuzzer.
Patch By: aarongreen
Reviewers: kcc, morehouse, flowerhack, phosek
Reviewed By: kcc, phosek, Eugene.Zelenko
Subscribers: srhines, mgorny, Eugene.Zelenko
Differential Revision: https://reviews.llvm.org/D40974
llvm-svn: 320210
Summary: With 3 Dictionary objects, each containing space of ~16k DictionaryEntry objects, the MutationDispatcher object is fairly memory heavy. On platforms with a lower default stack size, this can cause panics in FuzzerUnittest as those tests stack-allocate the MutationDispatcher. This may be especially problematic for platforms that do not (yet) have a way to programmatically change their stack size, aside from link-time flags. In general, it seems more prudent to use the heap for an object of this size.
Reviewers: kcc, morehouse
Reviewed By: kcc
Differential Revision: https://reviews.llvm.org/D40926
llvm-svn: 319988
Kostya Serebryany