This is the original patch in my GNUInstallDirs series, now last to merge as the final piece!
It arose as a new draft of D28234. I initially did the unorthodox thing of pushing to that when I wasn't the original author, but since I ended up
- Using `GNUInstallDirs`, rather than mimicking it, as the original author was hesitant to do but others requested.
- Converting all the packages, not just LLVM, effecting many more projects than LLVM itself.
I figured it was time to make a new revision.
I have used this patch series (and many back-ports) as the basis of https://github.com/NixOS/nixpkgs/pull/111487 for my distro (NixOS), which was merged last spring (2021). It looked like people were generally on board in D28234, but I make note of this here in case extra motivation is useful.
---
As pointed out in the original issue, a central tension is that LLVM already has some partial support for these sorts of things. Variables like `COMPILER_RT_INSTALL_PATH` have already been dealt with. Variables like `LLVM_LIBDIR_SUFFIX` however, will require further work, so that we may use `CMAKE_INSTALL_LIBDIR`.
These remaining items will be addressed in further patches. What is here is now rote and so we should get it out of the way before dealing more intricately with the remainder.
Reviewed By: #libunwind, #libc, #libc_abi, compnerd
Differential Revision: https://reviews.llvm.org/D99484
In C++20 compound assignment to volatile (here `LocalData[I]++`) is
deprecated, so `mutex_test.cpp` fails to compile.
Simply changing it to `LocalData[I] = LocalData[I] + 1` fixes it.
Differential Revision: https://reviews.llvm.org/D117359
Existing code tended to assume that counters had type `uint64_t` and
computed size from the number of counters. Fix this code to directly
compute the counters size in number of bytes where possible. When the
number of counters is needed, use `__llvm_profile_counter_entry_size()`
or `getCounterTypeSize()`. In a later diff these functions will depend
on the profile mode.
Change the meaning of `DataSize` and `CountersSize` to make them more clear.
* `DataSize` (`CountersSize`) - the size of the data (counter) section in bytes.
* `NumData` (`NumCounters`) - the number of data (counter) entries.
Reviewed By: kyulee
Differential Revision: https://reviews.llvm.org/D116179
If `pthread_create` is not available on a platform, we won't be able to check if interceptors work. Use `strcmp` instead.
Reviewed By: yln
Differential Revision: https://reviews.llvm.org/D116989
The test checks that an array of Obj-C literal integers (e.g. `@1`) gets a UBSan
warning when cast to an NSString, however the actual concrete Obj-C class of
literal integers doesn't always need to be __NSCFNumber. Let's relax the test
expectations to allow NSConstantIntegerNumber. Which exact subclass of NSNumber
is used is not actually important for the test (the test is just checking that
the invalid cast warning is thrown).
These tests are now green:
```
Trace.MultiPart
Trace.RestoreAccess
Trace.RestoreMutexLock
TraceAlloc.FinishedThreadReuse
TraceAlloc.FinishedThreadReuse2
TraceAlloc.SingleThread
```
rdar://82107856
D114294 <https://reviews.llvm.org/D114294> broke the Solaris buildbots:
/opt/llvm-buildbot/home/solaris11-amd64/clang-solaris11-amd64/llvm/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cpp:613:29: error: use of undeclared identifier 'NT_GNU_BUILD_ID'
if (nhdr->n_type == NT_GNU_BUILD_ID && nhdr->n_namesz == kGnuNamesz) {
^
Like D107556 <https://reviews.llvm.org/D107556>, it forgot that
`NT_GNU_BUILD_ID` is an unportable GNU extension.
Fixed by making the code conditional on the definition of the macro.
Tested on `amd64-pc-solaris2.11` and `sparcv9-sun-solaris2.11`.
Differential Revision: https://reviews.llvm.org/D117051
indirect branch tracking(IBT) feature aiming to ensure the target address
of an indirect jump/call is not tampered.
When IBT is enabled, each function or target of any indirect jump/call will start
with an 'endbr32/64' instruction otherwise the program will crash during execution.
To build an application with CET enabled. we need to ensure:
1. build the source code with "-fcf-protection=full"
2. all the libraries linked with .o files must be CET enabled too
This patch aims to enable CET for compiler-rt builtins library, we add an option
"COMPILER_RT_ENABLE_CET" whose default value is OFF to enable CET for compiler-rt
in building time and when this option is "ON", "-fcf-protection=full" is added to
BUILTINS_CFLAG and the "endbr32/64" will be placed in the beginning of each assembly
function. We also enabled CET for crtbegin, crtend object files in this patch.
Reviewed by: MaskRay, compnerd, manojgupta, efriedma
Differential Revision: https://reviews.llvm.org/D109811
Signed-off-by: jinge90 <ge.jin@intel.com>
Inspired by LLVM_DEBUG, but using environment variables rather than command line
options.
Code can use ORC_RT_DEBUG(...) (if ORC_RT_DEBUG_TYPE is set), or
ORC_RT_DEBUG_WITH_TYPE(<type>, ...) (if ORC_RT_DEBUG_TYPE is not set. E.g. in
headers).
Debug logging is enabled in the executor by setting the ORC_RT_DEBUG environment
variable. Debug logging can be restricted by type by setting the
ORC_RT_DEBUG_TYPES environment variable to a comma separated list of types,
e.g. ORC_RT_DEBUG_TYPES=macho_platform,sps.
Differential Revision: https://reviews.llvm.org/D116139
When building libcxx, libcxxabi, and libunwind the build environment may
specify any number of sanitizers. For some build feature tests these
sanitizers must be disabled to prevent spurious linking errors. With
-fsanitize= this is straight forward with -fno-sanitize=all. With
-fsanitize-coverage= there is no -fno-sanitize-coverage=all, but there
is the equivalent undocumented but tested -fsanitize-coverage=0.
The current build rules fail to disable 'trace-pc-guard'. By disabling
all sanitize-coverage flags, including 'trace-pc-guard', possible
spurious linker errors are prevented. In particular, this allows libcxx,
libcxxabi, and libunwind to be built with HonggFuzz.
CMAKE_REQUIRED_FLAGS is extra compile flags when running CMake build
configuration steps (like check_cxx_compiler_flag). It does not affect
the compile flags for the actual build of the project (unless of course
these flags change whether or not a given source compiles and links or
not). So libcxx, libcxxabi, and libunwind will still be built with any
specified sanitize-coverage as before. The build configuration steps
(which are mostly checking to see if certain compiler flags are
available) will not try to compile and link "int main() { return 0;}"
(or other specified source) with sanitize-coverage (which can fail to
link at this stage in building, since the final compile flags required
are yet to be determined).
The change to LIBFUZZER_CFLAGS was done to keep it consistent with the
obvious intention of disabling all sanitize-coverage. This appears to
be intentional, preventing the fuzzer driver itself from showing up in
any coverage calculations.
Reviewed By: #libunwind, #libc, #libc_abi, ldionne, phosek
Differential Revision: https://reviews.llvm.org/D116050
Currently we use very common names for macros like ACQUIRE/RELEASE,
which cause conflicts with system headers.
Prefix all macros with SANITIZER_ to avoid conflicts.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D116652
This allows DFSan to find tainted values used to control program behavior.
Reviewed By: morehouse
Differential Revision: https://reviews.llvm.org/D116207
This allows their reuse across projects. The name of the module
is intentionally generic because we would like to move more platform
checks there.
Differential Revision: https://reviews.llvm.org/D115276
This reverts commit 640beb38e7.
That commit caused performance degradtion in Quicksilver test QS:sGPU and a functional test failure in (rocPRIM rocprim.device_segmented_radix_sort).
Reverting until we have a better solution to s_cselect_b64 codegen cleanup
Change-Id: Ibf8e397df94001f248fba609f072088a46abae08
Reviewed By: kzhuravl
Differential Revision: https://reviews.llvm.org/D115960
Change-Id: Id169459ce4dfffa857d5645a0af50b0063ce1105
This will allow linking in the callbacks directly instead of using PLT.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D116182
A signal handler can alter ucontext_t to affect execution after
the signal returns. Check that the contents are initialized.
Restoring unitialized values in registers can't be good.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D116209
ucontext_t can be larger than its static size if it contains
AVX state and YMM/ZMM registers.
Currently a signal handler that tries to access that state
can produce false positives with random origins on stack.
Account for the additional ucontext_t state.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D116208
This is a segmentation fault in INTERCEPTOR function on a special edge
case of strstr libc call. When 'Haystack'(main string to be examined) is
NULL and 'needle'(sub-string to be searched in 'Haystack') is an empty
string then it hits a SEGV while using sanitizers and as a 'string not
found' case otherwise.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D115919
In D116472 we created conditionally defined variables for the tools to
unbreak the legacy build where they are in `llvm/tools`.
The runtimes are not tools, so that flexibility doesn't matter. Still,
it might be nice to define (unconditionally) and use the variable for
the runtimes simply to make the code a bit clearer and document what is
going on.
Also, consistently put project dirs at the beginning, not end of `CMAKE_MODULE_PATH`. This ensures they will properly shadow similarly named stuff that happens to be later on the path.
Reviewed By: mstorsjo, #libunwind, #libc, #libc_abi, ldionne
Differential Revision: https://reviews.llvm.org/D116477
It is likely to become used again, if other projects want their own per-project
install directory variables. `install` is removed from the name since it is not inherently about installing.
Reviewed By: stephenneuendorffer
Differential Revision: https://reviews.llvm.org/D115746
When using debug info for profile correlation, avoid adding duplicate
functions in the synthetic Data section.
Before this patch, n duplicate function entries in the Data section would
cause counter values to be a factor of n larger. I built instrumented
clang with and without debug info correlation and got these summaries.
```
# With Debug Info Correlate
$ llvm-profdata show default.profdata
Instrumentation level: IR entry_first = 0
Total functions: 182530
Maximum function count: 52034
Maximum internal block count: 5763
# Without
$ llvm-profdata show default.profdata
Instrumentation level: IR entry_first = 0
Total functions: 183212
Maximum function count: 52034
Maximum internal block count: 5766
```
The slight difference in counts seem to be mostly from FileSystem and
Map functions and the difference in the number of instrumented functions
seems to come from missing debug info like destructors without source.
Reviewed By: kyulee
Differential Revision: https://reviews.llvm.org/D116051
This patch adds support to read all the PT_NOTE segments in the
executable to find the binary ids. Previously, it was only reading
the first PT_NOTE segment, and this was missing the cases where
binary id is in the following segments. As a result, binary-id.c
and binary-id-padding.c test were failing in the following cases:
1) sanitizer-x86_64-linux bot
https://lab.llvm.org/staging/#/builders/97
2) OpenSuse Tumbleweed
https://github.com/llvm/llvm-project/issues/52695
Differential Revision: https://reviews.llvm.org/D115830
This will allow linking in the callbacks directly instead of using PLT.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D116182
This option is per process anyway. I'd like to add more options, but
having them as parameters of __sanitizer_symbolize_code looks
inconvenient.
Reviewed By: browneee
Differential Revision: https://reviews.llvm.org/D116201
This will allow linking in the callbacks directly instead of using PLT.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D116182
After D116148 the memccpy gets optimized away and the expected
uninitialized memory access does not occur.
Make sure the call does not get optimized away.
The new tsan runtime has 2x more compact shadow.
Adjust shadow ranges accordingly.
Depends on D112603.
Reviewed By: vitalybuka, melver
Differential Revision: https://reviews.llvm.org/D113751
If there are multiple processes, it's hard to understand
what output comes from what process.
VReport prepends pid to the output. Use it.
Depends on D113982.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D113983
Update now after long operations so that we don't use
stale value in subsequent computations.
Depends on D113981.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D113982
Creating threads after a multi-threaded fork is semi-supported,
we don't give particular guarantees, but we try to not fail
on simple cases and we have die_after_fork=0 flag that enables
not dying on creation of threads after a multi-threaded fork.
This flag is used in the wild:
23c052e3e3/SConstruct (L3599)
fork_multithreaded.cpp test started hanging in debug mode
after the recent "tsan: fix deadlock during race reporting" commit,
which added proactive ThreadRegistryLock check in SlotLock.
But the test broke earlier after "tsan: remove quadratic behavior in pthread_join"
commit which made tracking of alive threads based on pthread_t stricter
(CHECK-fail on 2 threads with the same pthread_t, or joining a non-existent thread).
When we start a thread after a multi-threaded fork, the new pthread_t
can actually match one of existing values (for threads that don't exist anymore).
Thread creation started CHECK-failing on this, but the test simply
ignored this CHECK failure in the child thread and "passed".
But after "tsan: fix deadlock during race reporting" the test started hanging dead,
because CHECK failures recursively lock thread registry.
Fix this purging all alive threads from thread registry on fork.
Also the thread registry mutex somehow lost the internal deadlock detector id
and was excluded from deadlock detection. If it would have the id, the CHECK
wouldn't hang because of the nested CHECK failure due to the deadlock.
But then again the test would have silently ignore this error as well
and the bugs wouldn't have been noticed.
Add the deadlock detector id to the thread registry mutex.
Also extend the test to check more cases and detect more bugs.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D116091
If we miss both close of a file descriptor and a subsequent open
if the same file descriptor number, we report false positives
between operations on the old and on the new descriptors.
There are lots of ways to create new file descriptors, but for closing
there is mostly close call. So we try to handle at least it.
However, if the close happens in an ignored library, we miss it
and start reporting false positives.
Handle closing of file descriptors always, even in ignored libraries
(as we do for malloc/free and other critical functions).
But don't imitate memory accesses on close for ignored libraries.
FdClose checks validity of the fd (fd >= 0) itself,
so remove the excessive checks in the callers.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D116095
We used to use u64 as mutex id because it was some
tricky identifier built from address and reuse count.
Now it's just the mutex index in the report (0, 1, 2...),
so use int to represent it.
Depends on D112603.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D113980
These callbacks are used for SSE vector accesses.
In some computational programs these accesses dominate.
Currently we do 2 uninlined 8-byte accesses to handle them.
Inline and optimize them similarly to unaligned accesses.
This reduces the vector access benchmark time from 8 to 3 seconds.
Depends on D112603.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D114594
This also makes the sanitizer_stoptheworld_test cross-platform by using the STL, rather than pthread.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D115204
This attempts to adjust the test to still exercise the expected codepath after D115904. This test is fundementally rather fragile.
Unfortunately, I have not been able to confirm this workaround either does, or does not, work. Attempting check-all with compiler-rt blows through an additional 30GB of disk space so my build config which exceeds my local disk space.
There is a small chance that the slot may be not queued in TraceSwitchPart.
This can happen if the slot has kEpochLast epoch and another thread
in FindSlotAndLock discovered that it's exhausted and removed it from
the slot queue. kEpochLast can happen in 2 cases: (1) if TraceSwitchPart
was called with the slot locked and epoch already at kEpochLast,
or (2) if we've acquired a new slot in SlotLock in the beginning
of the function and the slot was at kEpochLast - 1, so after increment
in SlotAttachAndLock it become kEpochLast.
If this happens we crash on ctx->slot_queue.Remove(thr->slot).
Skip the requeueing if the slot is not queued.
The slot is exhausted, so it must not be ctx->slot_queue.
The existing stress test triggers this with very small probability.
I am not sure how to make this condition more likely to be triggered,
it evaded lots of testing.
Depends on D116040.
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D116041
SlotPairLocker calls SlotLock under ctx->multi_slot_mtx.
SlotLock can invoke global reset DoReset if we are out of slots/epochs.
But DoReset locks ctx->multi_slot_mtx as well, which leads to deadlock.
Resolve the deadlock by removing SlotPairLocker/multi_slot_mtx
and only lock one slot for which we will do RestoreStack.
We need to lock that slot because RestoreStack accesses the slot journal.
But it's unclear why we need to lock the current slot.
Initially I did it just to be on the safer side (but at that time
we dit not lock the second slot, so it was easy just to lock the current slot).
Reviewed By: melver
Differential Revision: https://reviews.llvm.org/D116040
Profile merging is not supported when using debug info profile
correlation because the data section won't be in the binary at runtime.
Change the default profile name in this mode to `default_%p.proflite` so
we don't use profile merging.
Reviewed By: kyulee
Differential Revision: https://reviews.llvm.org/D115979
Previously we would crash in the TSan runtime if the user program passes
a pointer to `malloc_size()` that doesn't point into app memory.
In these cases, `malloc_size()` should return 0.
For ASan, we fixed a similar issue here:
https://reviews.llvm.org/D15008
Radar-Id: rdar://problem/86213149
Differential Revision: https://reviews.llvm.org/D115947
There was a build failure on the `instrprof-debug-info-correlate.c` test
because zlib was missing so we need to require it to run the test.
Differential Revision: https://reviews.llvm.org/D115970
Extend `llvm-profdata` to read in a `.proflite` file and also a debug info file to generate a normal `.profdata` profile. This reduces the binary size by 8.4% when building an instrumented Clang binary without value profiling (164 MB vs 179 MB).
This work is part of the "lightweight instrumentation" RFC: https://groups.google.com/g/llvm-dev/c/r03Z6JoN7d4
This was first landed in https://reviews.llvm.org/D114566 but had to be reverted due to build errors.
Reviewed By: kyulee
Differential Revision: https://reviews.llvm.org/D115915
This reverts commit 3f5f687e2e.
That commit broke building for mingw, where the sanitizers are
built with -nostdinc++, while the added source file includes
the C++ standard library's <algorithm>.
Additionally, the new code fails to build for i386, as it
unconditionally uses the CONTEXT member Rsp.
AARCH64_GET_REG() is used to initialize uptrs, and after D79132
the ptrauth branch of its implementation explicitly casts to uptr.
The non-ptrauth branch returns ucontext->uc_mcontext->__ss.__fp (etc),
which has either type void* or __uint64_t (ref usr/include/mach/arm/_structs.h)
where __uint64_t is a unsigned long long (ref usr/include/arm/_types.h).
uptr is an unsigned long (ref
compiler-rt/lib/sanitizer_common/sanitizer_internal_defs.h). So explicitly
cast to uptr in this branch as well, so that AARCH64_GET_REG() has a
well-defined type.
Then change DUMPREGA64() tu use %lx instead of %llx since that's the right type
for uptr. (Most other places in compiler-rt print uptrs as %p and cast the arg
to (void*), but there are explicit 0x%016 format strings in the surroundings,
so be locally consistent with that.)
No behavior change, in the end it's just 64-bit unsigneds by slightly different
names.
This also makes the sanitizer_stoptheworld_test cross-platform by using the STL, rather than pthread.
Reviewed By: vitalybuka
Differential Revision: https://reviews.llvm.org/D115204
Extend `llvm-profdata` to read in a `.proflite` file and also a debug info file to generate a normal `.profdata` profile. This reduces the binary size by 8.4% when building an instrumented Clang binary without value profiling (164 MB vs 179 MB).
This work is part of the "lightweight instrumentation" RFC: https://groups.google.com/g/llvm-dev/c/r03Z6JoN7d4
Reviewed By: kyulee
Differential Revision: https://reviews.llvm.org/D114566
Alexandre Ganea