Summary: This patch disables implicit builtin knowledge about memcmp-like functions when compiling the program for fuzzing, i.e., when -fsanitize=fuzzer(-no-link) is given. This allows libFuzzer to always intercept memcmp-like functions as it effectively disables optimizing calls to such functions into different forms. This is done by adding a set of flags (-fno-builtin-memcmp and others) in the clang driver. Individual -fno-builtin-* flags previously used in several libFuzzer tests are now removed, as it is now done automatically in the clang driver.
The patch was once reverted in 8ef9e2bf35, as this patch was dependent on a reverted commit f78d9fceea. This reverted commit was recommitted in 831ae45e3d, so relanding this dependent patch too.
Reviewers: morehouse, hctim
Subscribers: cfe-commits, #sanitizers
Tags: #clang, #sanitizers
Differential Revision: https://reviews.llvm.org/D83987
Summary: This patch disables (i) noasan-memcmp64.test on Windows as libFuzzer's interceptors are only supported on Linux for now, and (ii) bcmp.test as on Windows bcmp is not available in strings.h.
Reviewers: morehouse, hctim, kcc
Subscribers: #sanitizers
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D84536
If we define memcmp in an archive, bcmp should be defined as well (many libc
define bcmp/memcmp in one object file). Otherwise if the application calls bcmp
or strcmp which gets optimized to bcmp (SimplifyLibCalls), the undefined
reference may pull in an optimized bcmp/strcmp implementation (libc replacement)
later on the linker command line. If both libFuzzer's memcmp and the optimized
memcmp are strong => there will be a multiple definition error.
Summary: libFuzzer's interceptor support added in 831ae45e3d currently only works on Linux. This patch disables the test cases added as part of that commit on non-Linux platforms.
Reviewers: morehouse, hctim
Subscribers: #sanitizers
Tags: #sanitizers
Differential Revision: https://reviews.llvm.org/D84434
Summary: libFuzzer intercepts certain library functions such as memcmp/strcmp by defining weak hooks. Weak hooks, however, are called only when other runtimes such as ASan is linked. This patch defines libFuzzer's own interceptors, which is linked into the libFuzzer executable when other runtimes are not linked, i.e., when -fsanitize=fuzzer is given, but not others.
The patch once landed but was reverted in 8ef9e2bf35 due to an assertion failure caused by calling an intercepted function, strncmp, while initializing the interceptors in fuzzerInit(). This issue is now fixed by calling libFuzzer's own implementation of library functions (i.e., internal_*) when the fuzzer has not been initialized yet, instead of recursively calling fuzzerInit() again.
Reviewers: kcc, morehouse, hctim
Subscribers: #sanitizers, krytarowski, mgorny, cfe-commits
Tags: #clang, #sanitizers
Differential Revision: https://reviews.llvm.org/D83494
This causes binaries linked with this runtime to crash on startup if
dlsym uses any of the intercepted functions. (For example, that happens
when using tcmalloc as the allocator: dlsym attempts to allocate memory
with malloc, and tcmalloc uses strncmp within its implementation.)
Also revert dependent commit "[libFuzzer] Disable implicit builtin knowledge about memcmp-like functions when -fsanitize=fuzzer-no-link is given."
This reverts commit f78d9fceea and 12d1124c49.
Summary: This patch disables implicit builtin knowledge about memcmp-like functions when compiling the program for fuzzing, i.e., when -fsanitize=fuzzer(-no-link) is given. This allows libFuzzer to always intercept memcmp-like functions as it effectively disables optimizing calls to such functions into different forms. This is done by adding a set of flags (-fno-builtin-memcmp and others) in the clang driver. Individual -fno-builtin-* flags previously used in several libFuzzer tests are now removed, as it is now done automatically in the clang driver.
Reviewers: morehouse, hctim
Subscribers: cfe-commits, #sanitizers
Tags: #clang, #sanitizers
Differential Revision: https://reviews.llvm.org/D83987
Summary: libFuzzer intercepts certain library functions such as memcmp/strcmp by defining weak hooks. Weak hooks, however, are called only when other runtimes such as ASan is linked. This patch defines libFuzzer's own interceptors, which is linked into the libFuzzer executable when other runtimes are not linked, i.e., when -fsanitize=fuzzer is given, but not others.
Reviewers: kcc, morehouse, hctim
Reviewed By: morehouse, hctim
Subscribers: krytarowski, mgorny, cfe-commits, #sanitizers
Tags: #clang, #sanitizers
Differential Revision: https://reviews.llvm.org/D83494
Having the input dumped on failure seems like a better
default: I debugged FileCheck tests for a while without knowing
about this option, which really helps to understand failures.
Remove `-dump-input-on-failure` and the environment variable
FILECHECK_DUMP_INPUT_ON_FAILURE which are now obsolete.
Differential Revision: https://reviews.llvm.org/D81422
Summary:
The number of "inputs have the Data Flow Trace" cannot be greater than
the number of inputs touching the focus function. The existing message is rather
confusing as the same log would mention a greater total number of traces a few
lines above.
Reviewers: kcc, metzman
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D74779
This patch defines `config.apple_platform_min_deployment_target_flag`
in the ASan, LibFuzzer, TSan, and UBSan lit test configs.
rdar://problem/59463146
EXCLUDE_FROM_ALL means something else for add_lit_testsuite as it does
for something like add_executable. Distinguish between the two by
renaming the variable and making it an argument to add_lit_testsuite.
Differential revision: https://reviews.llvm.org/D74168
Summary:
In order to do this `FUZZER_SUPPORTED_OS` had to be pulled out of
`lib/fuzzer/CMakeLists.txt` and into the main config so we can use it
from the `test/fuzzer/CMakeList.txt`. `FUZZER_SUPPORTED_OS` currently
has the same value of `SANITIZER_COMMON_SUPPORTED_OS` which preserves
the existing behaviour but this allows us in the future to adjust the
supported platforms independent of `SANITIZER_COMMON_SUPPORTED_OS`. This
mirrors the other sanitizers.
For non-Apple platforms `FUZZER_SUPPORTED_OS` is not defined and
surprisingly this was the behaviour before this patch because
`SANITIZER_COMMON_SUPPORTED_OS` was actually empty. This appears to
not matter right now because the functions that take an `OS` as an
argument seem to ignore it on non-Apple platforms.
While this change tries to be NFC it is technically not because we
now generate an iossim config whereas previously we didn't. This seems
like the right thing to do because the build system was configured to
compile LibFuzzer for iossim but previously we weren't generating a lit
test config for it. The device/simulator testing configs don't run by
default anyway so this shouldn't break testing.
This change relies on the get_capitalized_apple_platform() function
added in a previous commit.
rdar://problem/58798733
Reviewers: kubamracek, yln
Subscribers: mgorny, #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73243
Summary:
there is an ongoing work on interchangeable custom mutators
(https://github.com/google/clusterfuzz/pull/1333/files#r367706283)
and having some sort of signalling from libFuzzer that it has loaded
a custom mutator would be helpful.
The initial idea was to make the mutator to print something, but given
the anticipated variety of different mutators, it does not seem possible
to make all of them print the same message to signal their execution.
Reviewers: kcc, metzman
Reviewed By: metzman
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D73136
Do not add an lld dependency when this target does not exist. In this
case the system installation of lld is used (or whatever is detected
with -fuse-ld=lld by default).
The main problem here is that `-*-version_min=` was not being passed to
the compiler when building test cases. This can cause problems when
testing on devices running older OSs because Clang would previously
assume the minimum deployment target is the the latest OS in the SDK
which could be much newer than what the device is running.
Previously the generated value looked like this:
`-arch arm64 -isysroot
<path_to_xcode>/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS12.1.sdk`
With this change it now looks like:
`-arch arm64 -stdlib=libc++ -miphoneos-version-min=8.0 -isysroot
<path_to_xcode>/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS12.1.sdk`
This mirrors the setting of config.target_cflags on macOS.
This change is made for ASan, LibFuzzer, TSan, and UBSan.
To implement this a new `get_test_cflags_for_apple_platform()` function
has been added that when given an Apple platform name and architecture
returns a string containing the C compiler flags to use when building
tests. This also calls a new helper function `is_valid_apple_platform()`
that validates Apple platform names.
This is the third attempt at landing the patch.
The first attempt (r359305) had to be reverted (r359327) due to a buildbot
failure. The problem was that calling `get_test_cflags_for_apple_platform()`
can trigger a CMake error if the provided architecture is not supported by the
current CMake configuration. Previously, this could be triggered by passing
`-DCOMPILER_RT_ENABLE_IOS=OFF` to CMake. The root cause is that we were
generating test configurations for a list of architectures without checking if
the relevant Sanitizer actually supported that architecture. We now intersect
the list of architectures for an Apple platform with
`<SANITIZER>_SUPPORTED_ARCH` (where `<SANITIZER>` is a Sanitizer name) to
iterate through the correct list of architectures.
The second attempt (r363633) had to be reverted (r363779) due to a build
failure. The failed build was using a modified Apple toolchain where the iOS
simulator SDK was missing. This exposed a bug in the existing UBSan test
generation code where it was assumed that `COMPILER_RT_ENABLE_IOS` implied that
the toolchain supported both iOS and the iOS simulator. This is not true. This
has been fixed by using the list `SANITIZER_COMMON_SUPPORTED_OS` for the list
of supported Apple platforms for UBSan. For consistency with the other
Sanitizers we also now intersect the list of architectures with
UBSAN_SUPPORTED_ARCH.
rdar://problem/50124489
Differential Revision: https://reviews.llvm.org/D61242
llvm-svn: 373405
Summary:
This change allows to perform corpus merging in two steps. This is useful when
the user wants to address the following two points simultaneously:
1) Get trustworthy incremental stats for the coverage and corpus size changes
when adding new corpus units.
2) Make sure the shorter units will be preferred when two or more units give the
same unique signal (equivalent to the `REDUCE` logic).
This solution was brainstormed together with @kcc, hopefully it looks good to
the other people too. The proposed use case scenario:
1) We have a `fuzz_target` binary and `existing_corpus` directory.
2) We do fuzzing and write new units into the `new_corpus` directory.
3) We want to merge the new corpus into the existing corpus and satisfy the
points mentioned above.
4) We create an empty directory `merged_corpus` and run the first merge step:
`
./fuzz_target -merge=1 -merge_control_file=MCF ./merged_corpus ./existing_corpus
`
this provides the initial stats for `existing_corpus`, e.g. from the output:
`
MERGE-OUTER: 3 new files with 11 new features added; 11 new coverage edges
`
5) We recreate `merged_corpus` directory and run the second merge step:
`
./fuzz_target -merge=1 -merge_control_file=MCF ./merged_corpus ./existing_corpus ./new_corpus
`
this provides the final stats for the merged corpus, e.g. from the output:
`
MERGE-OUTER: 6 new files with 14 new features added; 14 new coverage edges
`
Alternative solutions to this approach are:
A) Store precise coverage information for every unit (not only unique signal).
B) Execute the same two steps without reusing the control file.
Either of these would be suboptimal as it would impose an extra disk or CPU load
respectively, which is bad given the quadratic complexity in the worst case.
Tested on Linux, Mac, Windows.
Reviewers: morehouse, metzman, hctim, kcc
Reviewed By: morehouse
Subscribers: JDevlieghere, delcypher, mgrang, #sanitizers, llvm-commits, kcc
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D66107
llvm-svn: 371620
Summary:
There's no real reason to use clang-cl on Windows, the clang driver
works just as well. This fixes a test which uses the -O0 flag, which was
recently removed from clang-cl to match MSVC, which lacks this flag.
While I'm here, remove the explicit -std=c++11 flag. Previously, this
flag was necessary when the default C++ standard was C++98. Now that the
default is C++14, this is no longer necessary. It's problematic on
Windows, because the Visual C++ standard library relies on C++14
features, and attempting to compile it with C++11 results in errors.
Rather than adding logic to conditionally set the standard to C++11 only
on non-Win, this flag can be removed.
See http://lab.llvm.org:8011/builders/clang-x64-windows-msvc and
https://reviews.llvm.org/D64506.
Reviewers: morehouse, thakis
Subscribers: #sanitizers, llvm-commits
Tags: #sanitizers, #llvm
Differential Revision: https://reviews.llvm.org/D64587
llvm-svn: 365841
These lit configuration files are really Python source code. Using the
.py file extension helps editors and tools use the correct language
mode. LLVM and Clang already use this convention for lit configuration,
this change simply applies it to all of compiler-rt.
Reviewers: vitalybuka, dberris
Differential Revision: https://reviews.llvm.org/D63658
llvm-svn: 364591
This caused Chromium's clang package to stop building, see comment on
https://reviews.llvm.org/D61242 for details.
> Summary:
> The main problem here is that `-*-version_min=` was not being passed to
> the compiler when building test cases. This can cause problems when
> testing on devices running older OSs because Clang would previously
> assume the minimum deployment target is the the latest OS in the SDK
> which could be much newer than what the device is running.
>
> Previously the generated value looked like this:
>
> `-arch arm64 -isysroot
> <path_to_xcode>/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS12.1.sdk`
>
> With this change it now looks like:
>
> `-arch arm64 -stdlib=libc++ -miphoneos-version-min=8.0 -isysroot
> <path_to_xcode>/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS12.1.sdk`
>
> This mirrors the setting of `config.target_cflags` on macOS.
>
> This change is made for ASan, LibFuzzer, TSan, and UBSan.
>
> To implement this a new `get_test_cflags_for_apple_platform()` function
> has been added that when given an Apple platform name and architecture
> returns a string containing the C compiler flags to use when building
> tests. This also calls a new helper function `is_valid_apple_platform()`
> that validates Apple platform names.
>
> This is the second attempt at landing the patch. The first attempt (r359305)
> had to be reverted (r359327) due to a buildbot failure. The problem was
> that calling `get_test_cflags_for_apple_platform()` can trigger a CMake
> error if the provided architecture is not supported by the current
> CMake configuration. Previously, this could be triggered by passing
> `-DCOMPILER_RT_ENABLE_IOS=OFF` to CMake. The root cause is that we were
> generating test configurations for a list of architectures without
> checking if the relevant Sanitizer actually supported that architecture.
> We now intersect the list of architectures for an Apple platform
> with `<SANITIZER>_SUPPORTED_ARCH` (where `<SANITIZER>` is a Sanitizer
> name) to iterate through the correct list of architectures.
>
> rdar://problem/50124489
>
> Reviewers: kubamracek, yln, vsk, juliehockett, phosek
>
> Subscribers: mgorny, javed.absar, kristof.beyls, #sanitizers, llvm-commits
>
> Tags: #llvm, #sanitizers
>
> Differential Revision: https://reviews.llvm.org/D61242
llvm-svn: 363779
Summary:
The main problem here is that `-*-version_min=` was not being passed to
the compiler when building test cases. This can cause problems when
testing on devices running older OSs because Clang would previously
assume the minimum deployment target is the the latest OS in the SDK
which could be much newer than what the device is running.
Previously the generated value looked like this:
`-arch arm64 -isysroot
<path_to_xcode>/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS12.1.sdk`
With this change it now looks like:
`-arch arm64 -stdlib=libc++ -miphoneos-version-min=8.0 -isysroot
<path_to_xcode>/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS12.1.sdk`
This mirrors the setting of `config.target_cflags` on macOS.
This change is made for ASan, LibFuzzer, TSan, and UBSan.
To implement this a new `get_test_cflags_for_apple_platform()` function
has been added that when given an Apple platform name and architecture
returns a string containing the C compiler flags to use when building
tests. This also calls a new helper function `is_valid_apple_platform()`
that validates Apple platform names.
This is the second attempt at landing the patch. The first attempt (r359305)
had to be reverted (r359327) due to a buildbot failure. The problem was
that calling `get_test_cflags_for_apple_platform()` can trigger a CMake
error if the provided architecture is not supported by the current
CMake configuration. Previously, this could be triggered by passing
`-DCOMPILER_RT_ENABLE_IOS=OFF` to CMake. The root cause is that we were
generating test configurations for a list of architectures without
checking if the relevant Sanitizer actually supported that architecture.
We now intersect the list of architectures for an Apple platform
with `<SANITIZER>_SUPPORTED_ARCH` (where `<SANITIZER>` is a Sanitizer
name) to iterate through the correct list of architectures.
rdar://problem/50124489
Reviewers: kubamracek, yln, vsk, juliehockett, phosek
Subscribers: mgorny, javed.absar, kristof.beyls, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D61242
llvm-svn: 363633
Summary:
Some custom mutators may not peform well when size restriction is
enforced by len_control. Because of that, it's safer to disable len_control
by default in such cases, but still allow users to enable it manually.
Bug example: https://bugs.chromium.org/p/chromium/issues/detail?id=919530.
Tested manually with LPM-based and regular fuzz targets.
Reviewers: kcc, vitalybuka, metzman
Reviewed By: kcc, metzman
Subscribers: delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D63334
llvm-svn: 363443
It broke the Windows build:
C:\b\s\w\ir\cache\builder\src\third_party\llvm\compiler-rt\lib\fuzzer\FuzzerDataFlowTrace.cpp(243): error C3861: 'setenv': identifier not found
This also reverts the follow-up r363327.
llvm-svn: 363358
Summary:
This class is useful for writing fuzz target that have multiple inputs.
Current CL imports the existing `FuzzedDataProvider` from Chromium
without any modifications. Feel free to review it thoroughly, if you're
interested, but I'd prefer changing the class in a follow up CL.
The CL also introduces an exhaustive test for the library, as the behavior
of `FuzzedDataProvider` must not change over time.
In follow up CLs I'm planning on changing some implementation details
(I can share a doc with some comments to be addressed). After that, we
will document how `FuzzedDataProvider` should be used.
I have tested this on Linux, Windows and Mac platforms.
Reviewers: morehouse, metzman, kcc
Reviewed By: morehouse
Subscribers: metzman, thakis, rnk, mgorny, ormris, delcypher, #sanitizers, llvm-commits
Tags: #llvm, #sanitizers
Differential Revision: https://reviews.llvm.org/D62733
llvm-svn: 363071
Dokyung Song