Previously, when the fuzzing loop replaced an input in the corpus, it didn't update the execution time of the input. Therefore, some schedulers (e.g. Entropic) would adjust weights based on the incorrect execution time.
This patch updates the execution time of the input when replacing it.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D111479
Previously, PrintASCII would print the string "\ta" as "\x09a". However,
in C/C++ those strings are not the same: the trailing 'a' is part of the
escape sequence, which means it's equivalent to "\x9a". This is an
annoying quirk of the standard. (See
https://eel.is/c++draft/lex.ccon#nt:hexadecimal-escape-sequence)
To fix this, output three-digit octal escape sequences instead. Since
octal escapes are limited to max three digits, this avoids the problem
of subsequent characters unintentionally becoming part of the escape
sequence.
Dictionary files still use the non-C-compatible hex escapes, but I
believe we can't change the format since it comes from AFL, and
libfuzzer never writes such files, it only has to read them, so they're
not affected by this change.
Differential revision: https://reviews.llvm.org/D110920
On Fuchsia, killing or exiting a process that has a thread listening to its own process's debugger exception channel can hang. Zircon may kill all the threads, send a synthetic exceptions to debugger, and wait for the debugger to have received them. This means the thread listening to the debug exception channel may be killed even as Zircon is waiting for that thread to drain the exception channel, and the process can become stuck in a half-dead state.
This situation is "weird" as it only arises when a process is trying to debug itself. Unfortunately, this is exactly the scenario for libFuzzer on Fuchsia: FuzzerUtilFuchsia spawns a crash-handling thread that acts like a debugger in order to be able to rewrite the crashed threads stack and resume them into libFuzzer's usual POSIX signal handlers. In practice, approximately 25% of fuzzers appear to hang on exit, after generating output and artifacts. These processes hang around until the platform is torn done, which is typically a ClusterFuzz VM. Thus, real-world impact has been somewhat mitigated. The issue should still be resolved for local users, though.
This change improves the behavior of exit() in libFuzzer by adding an atexit handler which closes an event shared with the crash handling thread. This signals to the crash handler that it should close the exception channel and be joined before the process actually exits.
Reviewed By: charco
Differential Revision: https://reviews.llvm.org/D109258
I found that the initial corpus allocation of fork mode has certain defects.
I designed a new initial corpus allocation strategy based on size grouping.
This method can give more energy to the small seeds in the corpus and
increase the throughput of the test.
Fuzzbench data (glibfuzzer is -fork_corpus_groups=1):
https://www.fuzzbench.com/reports/experimental/2021-08-05-parallel/index.html
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D105084
Extend the existing single-pass algorithm for `Merger::Merge` with an algorithm that gives better results. This new implementation can be used with a new **set_cover_merge=1** flag.
This greedy set cover implementation gives a substantially smaller final corpus (40%-80% less testcases) while preserving the same features/coverage. At the same time, the execution time penalty is not that significant (+50% for ~1M corpus files and far less for smaller corpora). These results were obtained by comparing several targets with varying size corpora.
Change `Merger::CrashResistantMergeInternalStep` to collect all features from each file and not just unique ones. This is needed for the set cover algorithm to work correctly. The implementation of the algorithm in `Merger::SetCoverMerge` uses a bitvector to store features that are covered by a file while performing the pass. Collisions while indexing the bitvector are ignored similarly to the fuzzer.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D105284
Include windows.h with an all lowercase filename; Windows SDK headers
aren't self consistent so they can't be used in an entirely
case sensitive setting, and mingw headers use all lowercase names
for such headers.
This fixes building after 881faf4190.
- Enable extra coverage counters on Windows.
- Update extra_counters.test to run on Windows also.
- Update TableLookupTest.cpp to include the required pragma/declspec for the extra coverage counters.
Patch By: MichaelSquires
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D106676
This commit fixes the CFI directives in the crash trampoline so
libunwind can get a backtrace during a crash.
In order to get a backtrace from a libfuzzer crash in fuchsia, we
resume execution in the crashed thread, forcing it to call the
StaticCrashHandler. We do this by setting a "crash trampoline" that has
all the necessary cfi directives for an unwinder to get full backtrace
for that thread.
Due to a bug in libunwind, it was not possible to restore the RSP
pointer, as it was always set to the call frame address (CFA). The
previous version worked around this issue by setting the CFA to the
value of the stack pointer at the point of the crash.
The bug in libunwind is now fixed[0], so I am correcting the CFI
annotations so that the CFA correctly points to the beginning of the
trampoline's call frame.
[0]: https://reviews.llvm.org/D106626
Reviewed By: mcgrathr
Differential Revision: https://reviews.llvm.org/D106725
This change introduces libMutagen/libclang_rt.mutagen.a as a subset of libFuzzer/libclang_rt.fuzzer.a. This library contains only the fuzzing strategies used by libFuzzer to produce new test inputs from provided inputs, dictionaries, and SanitizerCoverage feedback.
Most of this change is simply moving sections of code to one side or the other of the library boundary. The only meaningful new code is:
* The Mutagen.h interface and its implementation in Mutagen.cpp.
* The following methods in MutagenDispatcher.cpp:
* UseCmp
* UseMemmem
* SetCustomMutator
* SetCustomCrossOver
* LateInitialize (similar to the MutationDispatcher's original constructor)
* Mutate_AddWordFromTORC (uses callbacks instead of accessing TPC directly)
* StartMutationSequence
* MutationSequence
* DictionaryEntrySequence
* RecommendDictionary
* RecommendDictionaryEntry
* FuzzerMutate.cpp (which now justs sets callbacks and handles printing)
* MutagenUnittest.cpp (which adds tests of Mutagen.h)
A note on performance: This change was tested with a 100 passes of test/fuzzer/LargeTest.cpp with 1000 runs per pass, both with and without the change. The running time distribution was qualitatively similar both with and without the change, and the average difference was within 30 microseconds (2.240 ms/run vs 2.212 ms/run, respectively). Both times were much higher than observed with the fully optimized system clang (~0.38 ms/run), most likely due to the combination of CMake "dev mode" settings (e.g. CMAKE_BUILD_TYPE="Debug", LLVM_ENABLE_LTO=OFF, etc.). The difference between the two versions built similarly seems to be "in the noise" and suggests no meaningful performance degradation.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D102447
These have been broken by https://reviews.llvm.org/D104494.
However, `lib/fuzzer/dataflow/` is unused (?) so addressing this is not a priority.
Added TODOs to re-enable them in the future.
Reviewed By: stephan.yichao.zhao
Differential Revision: https://reviews.llvm.org/D104568
cstddef is needed for size_t definition.
(Multiple headers can provide size_t but none of them exists.)
Reviewed By: MaskRay
Differential Revision: https://reviews.llvm.org/D96213
Complete support for fast8:
- amend shadow size and mapping in runtime
- remove fast16 mode and -dfsan-fast-16-labels flag
- remove legacy mode and make fast8 mode the default
- remove dfsan-fast-8-labels flag
- remove functions in dfsan interface only applicable to legacy
- remove legacy-related instrumentation code and tests
- update documentation.
Reviewed By: stephan.yichao.zhao, browneee
Differential Revision: https://reviews.llvm.org/D103745
This change introduces libMutagen/libclang_rt.mutagen.a as a subset of libFuzzer/libclang_rt.fuzzer.a. This library contains only the fuzzing strategies used by libFuzzer to produce new test inputs from provided inputs, dictionaries, and SanitizerCoverage feedback.
Most of this change is simply moving sections of code to one side or the other of the library boundary. The only meaningful new code is:
* The Mutagen.h interface and its implementation in Mutagen.cpp.
* The following methods in MutagenDispatcher.cpp:
* UseCmp
* UseMemmem
* SetCustomMutator
* SetCustomCrossOver
* LateInitialize (similar to the MutationDispatcher's original constructor)
* Mutate_AddWordFromTORC (uses callbacks instead of accessing TPC directly)
* StartMutationSequence
* MutationSequence
* DictionaryEntrySequence
* RecommendDictionary
* RecommendDictionaryEntry
* FuzzerMutate.cpp (which now justs sets callbacks and handles printing)
* MutagenUnittest.cpp (which adds tests of Mutagen.h)
A note on performance: This change was tested with a 100 passes of test/fuzzer/LargeTest.cpp with 1000 runs per pass, both with and without the change. The running time distribution was qualitatively similar both with and without the change, and the average difference was within 30 microseconds (2.240 ms/run vs 2.212 ms/run, respectively). Both times were much higher than observed with the fully optimized system clang (~0.38 ms/run), most likely due to the combination of CMake "dev mode" settings (e.g. CMAKE_BUILD_TYPE="Debug", LLVM_ENABLE_LTO=OFF, etc.). The difference between the two versions built similarly seems to be "in the noise" and suggests no meaningful performance degradation.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D102447
Address sanitizer can detect stack exhaustion via its SEGV handler, which is
executed on a separate stack using the sigaltstack mechanism. When libFuzzer is
used with address sanitizer, it installs its own signal handlers which defer to
those put in place by the sanitizer before performing additional actions. In the
particular case of a stack overflow, the current setup fails because libFuzzer
doesn't preserve the flag for executing the signal handler on a separate stack:
when we run out of stack space, the operating system can't run the SEGV handler,
so address sanitizer never reports the issue. See the included test for an
example.
This commit fixes the issue by making libFuzzer preserve the SA_ONSTACK flag
when installing its signal handlers; the dedicated signal-handler stack set up
by the sanitizer runtime appears to be large enough to support the additional
frames from the fuzzer.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D101824
Currently, the position hint of an entry in the persistent auto
dictionary is fixed to 1. As a consequence, with a 50% chance, the entry
is applied right after the first byte of the input. As the position 1
does not appear to have any particular significance, this is likely a
bug that may have been caused by confusing the constructor parameter
with a success count.
This commit resolves the issue by preserving any existing position hint
or disabling the hint if the original entry didn't have one.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D101686
In the overwrite branch of MutationDispatcher::ApplyDictionaryEntry in
FuzzerMutate.cpp, the index Idx at which W.size() bytes are overwritten
with the word W is chosen uniformly at random in the interval
[0, Size - W.size()). This means that Idx + W.size() will always be
strictly less than Size, i.e., the last byte of the current unit will
never be overwritten.
This is fixed by adding 1 to the exclusive upper bound.
Addresses https://bugs.llvm.org/show_bug.cgi?id=49989.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D101625
In order to integrate libFuzzer with a dynamic symbolic execution tool
Sydr we need to print loaded file paths.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D100303
This change adds a SimpleFastHash64 variant of SimpleFastHash which allows call sites to specify a starting value and get a 64 bit hash in return. This allows a hash to be "resumed" with more data.
A later patch needs this to be able to hash a sequence of module-relative values one at a time, rather than just a region a memory.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D94510
The inlining of this function needs to be disabled as it is part of the
inpsected stack traces. It's string representation will look different
depending on if it was inlined or not which will cause it's string comparison
to fail.
When it was inlined in only one of the two execution stacks,
minimize_two_crashes.test failed on SystemZ. For details see
https://bugs.llvm.org/show_bug.cgi?id=49152.
Reviewers: Ulrich Weigand, Matt Morehouse, Arthur Eubanks
Differential Revision: https://reviews.llvm.org/D97975
Attempting to build a standalone libFuzzer in Fuchsia's default toolchain for the purpose of cross-compiling the unit tests revealed a number of not-quite-proper type conversions. Fuchsia's toolchain include `-std=c++17` and `-Werror`, among others, leading to many errors like `-Wshorten-64-to-32`, `-Wimplicit-float-conversion`, etc.
Most of these have been addressed by simply making the conversion explicit with a `static_cast`. These typically fell into one of two categories: 1) conversions between types where high precision isn't critical, e.g. the "energy" calculations for `InputInfo`, and 2) conversions where the values will never reach the bits being truncated, e.g. `DftTimeInSeconds` is not going to exceed 136 years.
The major exception to this is the number of features: there are several places that treat features as `size_t`, and others as `uint32_t`. This change makes the decision to cap the features at 32 bits. The maximum value of a feature as produced by `TracePC::CollectFeatures` is roughly:
(NumPCsInPCTables + ValueBitMap::kMapSizeInBits + ExtraCountersBegin() - ExtraCountersEnd() + log2(SIZE_MAX)) * 8
It's conceivable for extremely large targets and/or extra counters that this limit could be reached. This shouldn't break fuzzing, but it will cause certain features to collide and lower the fuzzers overall precision. To address this, this change adds a warning to TracePC::PrintModuleInfo about excessive feature size if it is detected, and recommends refactoring the fuzzer into several smaller ones.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D97992
During unit tests, it was observed that crafting an artificially small DSO could cause OOB memory to be accessed. This change fixes that (but again, the affected DSOs are unlikely to ever occur outside unit tests).
Reviewed By: morehouse, charco
Differential Revision: https://reviews.llvm.org/D94507
This change adds additional unit tests for fuzzer::Merger::Parse and fuzzer::Merger::Merge in anticipation of additional changes to the merge control file format to support cross-process fuzzing.
It modifies the parameter handling of Merge slightly in order to make NewFeatures and NewCov consistent with NewFiles; namely, Merge *replaces* the contents of these output parameters rather than accumulating them (thereby fixing a buggy return value).
This is change 1 of (at least) 18 for cross-process fuzzing support.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D94506
This makes `PickValueInArray` work for `std::array<T, s>` (C++11). I've also tested the C++17 `std::array` (with compiler-deduced template parameters)
```
Author:
MarcoFalke <falke.marco@gmail.com>
```
Reviewed By: Dor1s
Differential Revision: https://reviews.llvm.org/D93412
Adds a new option, `handle_winexcept` to try to intercept uncaught
Visual C++ exceptions on Windows. On Linux, such exceptions are handled
implicitly by `std::terminate()` raising `SIBABRT`. This option brings the
Windows behavior in line with Linux.
Unfortunately this exception code is intentionally undocumented, however
has remained stable for the last decade. More information can be found
here: https://devblogs.microsoft.com/oldnewthing/20100730-00/?p=13273
Reviewed By: morehouse, metzman
Differential Revision: https://reviews.llvm.org/D89755
While sanitizers don't use C++ standard library, we could still end
up accidentally including or linking it just by the virtue of using
the C++ compiler. Pass -nostdinc++ and -nostdlib++ to avoid these
accidental dependencies.
Reviewed By: smeenai, vitalybuka
Differential Revision: https://reviews.llvm.org/D88922
As implemented, the `InterruptHandler` thread was spinning trying to
`select()` on a null "stdin", wasting a significant amount of CPU for no
benefit. As Fuchsia does not have a native concept of stdin (or POSIX
signals), this commit simply removes this feature entirely.
Reviewed By: aarongreen
Differential Revision: https://reviews.llvm.org/D89266
-print_full_coverage=1 produces a detailed branch coverage dump when run on a single file.
Uses same infrastructure as -print_coverage flag, but prints all branches (regardless of coverage status) in an easy-to-parse format.
Usage: For internal use with machine learning fuzzing models which require detailed coverage information on seed files to generate mutations.
Differential Revision: https://reviews.llvm.org/D85928
- Fixing VS compiler and other cases settings this time.
Reviewers: dmajor, hans
Reviewed By: hans
Differential Revision: https://reviews.llvm.org/D89759
Revert "Fix compiler-rt build on Windows after D89640"
This reverts commit a7acee89d6.
This reverts commit d09b08919c.
Reason: breaks Linux / x86_64 build.
While sanitizers don't use C++ standard library, we could still end
up accidentally including or linking it just by the virtue of using
the C++ compiler. Pass -nostdinc++ and -nostdlib++ to avoid these
accidental dependencies.
Differential Revision: https://reviews.llvm.org/D88922
Matt Morehouse
gtt1995