UbiquitousOS
/
OpenCloudOS-Kernel
mirror of https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git
11
0
Fork 0
Code Issues Projects Releases Wiki Activity
OpenCloudOS-Kernel/include/asm-generic/atomic-instrumented.h

1834 lines
49 KiB
C
Raw Normal View History

locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
// SPDX-License-Identifier: GPL-2.0
// Generated by scripts/atomic/gen-atomic-instrumented.sh
// DO NOT MODIFY THIS FILE DIRECTLY
locking/atomic, asm-generic, x86: Add comments for atomic instrumentation The comments are factored out from the code changes to make them easier to read. Add them separately to explain some non-obvious aspects. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/cc595efc644bb905407012d82d3eb8bac3368e7a.1517246437.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:07 +08:00
/*
* This file provides wrappers with KASAN instrumentation for atomic operations.
* To use this functionality an arch's atomic.h file needs to define all
* atomic operations with arch_ prefix (e.g. arch_atomic_read()) and include
* this file at the end. This file provides atomic_read() that forwards to
* arch_atomic_read() for actual atomic operation.
* Note: if an arch atomic operation is implemented by means of other atomic
* operations (e.g. atomic_read()/atomic_cmpxchg() loop), then it needs to use
* arch_ variants (i.e. arch_atomic_read()/arch_atomic_cmpxchg()) to avoid
* double instrumentation.
*/
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#ifndef _ASM_GENERIC_ATOMIC_INSTRUMENTED_H
#define _ASM_GENERIC_ATOMIC_INSTRUMENTED_H
#include <linux/build_bug.h>
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
#include <linux/compiler.h>
asm-generic, atomic-instrumented: Use generic instrumented.h This switches atomic-instrumented.h to use the generic instrumentation wrappers provided by instrumented.h. No functional change intended. Suggested-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org>
2020-01-22 00:05:09 +08:00
#include <linux/instrumented.h>
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_read(const atomic_t *v)
{
asm-generic, atomic-instrumented: Use generic instrumented.h This switches atomic-instrumented.h to use the generic instrumentation wrappers provided by instrumented.h. No functional change intended. Suggested-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org>
2020-01-22 00:05:09 +08:00
instrument_atomic_read(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_read(v);
}
#define atomic_read atomic_read
#if defined(arch_atomic_read_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_read_acquire(const atomic_t *v)
{
asm-generic, atomic-instrumented: Use generic instrumented.h This switches atomic-instrumented.h to use the generic instrumentation wrappers provided by instrumented.h. No functional change intended. Suggested-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org>
2020-01-22 00:05:09 +08:00
instrument_atomic_read(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_read_acquire(v);
}
#define atomic_read_acquire atomic_read_acquire
#endif
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline void
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_set(atomic_t *v, int i)
{
asm-generic, atomic-instrumented: Use generic instrumented.h This switches atomic-instrumented.h to use the generic instrumentation wrappers provided by instrumented.h. No functional change intended. Suggested-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org>
2020-01-22 00:05:09 +08:00
instrument_atomic_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
arch_atomic_set(v, i);
}
#define atomic_set atomic_set
#if defined(arch_atomic_set_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline void
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_set_release(atomic_t *v, int i)
{
asm-generic, atomic-instrumented: Use generic instrumented.h This switches atomic-instrumented.h to use the generic instrumentation wrappers provided by instrumented.h. No functional change intended. Suggested-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org>
2020-01-22 00:05:09 +08:00
instrument_atomic_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
arch_atomic_set_release(v, i);
}
#define atomic_set_release atomic_set_release
#endif
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline void
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_add(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
arch_atomic_add(i, v);
}
#define atomic_add atomic_add
#if !defined(arch_atomic_add_return_relaxed) || defined(arch_atomic_add_return)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_add_return(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_add_return(i, v);
}
#define atomic_add_return atomic_add_return
#endif
#if defined(arch_atomic_add_return_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_add_return_acquire(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_add_return_acquire(i, v);
}
#define atomic_add_return_acquire atomic_add_return_acquire
#endif
#if defined(arch_atomic_add_return_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_add_return_release(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_add_return_release(i, v);
}
#define atomic_add_return_release atomic_add_return_release
#endif
#if defined(arch_atomic_add_return_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_add_return_relaxed(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_add_return_relaxed(i, v);
}
#define atomic_add_return_relaxed atomic_add_return_relaxed
#endif
#if !defined(arch_atomic_fetch_add_relaxed) || defined(arch_atomic_fetch_add)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_add(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_add(i, v);
}
#define atomic_fetch_add atomic_fetch_add
#endif
#if defined(arch_atomic_fetch_add_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_add_acquire(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_add_acquire(i, v);
}
#define atomic_fetch_add_acquire atomic_fetch_add_acquire
#endif
#if defined(arch_atomic_fetch_add_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_add_release(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_add_release(i, v);
}
#define atomic_fetch_add_release atomic_fetch_add_release
#endif
#if defined(arch_atomic_fetch_add_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_add_relaxed(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_add_relaxed(i, v);
}
#define atomic_fetch_add_relaxed atomic_fetch_add_relaxed
#endif
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline void
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_sub(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
arch_atomic_sub(i, v);
}
#define atomic_sub atomic_sub
#if !defined(arch_atomic_sub_return_relaxed) || defined(arch_atomic_sub_return)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_sub_return(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_sub_return(i, v);
}
#define atomic_sub_return atomic_sub_return
#endif
#if defined(arch_atomic_sub_return_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_sub_return_acquire(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_sub_return_acquire(i, v);
}
#define atomic_sub_return_acquire atomic_sub_return_acquire
#endif
#if defined(arch_atomic_sub_return_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_sub_return_release(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_sub_return_release(i, v);
}
#define atomic_sub_return_release atomic_sub_return_release
#endif
#if defined(arch_atomic_sub_return_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_sub_return_relaxed(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_sub_return_relaxed(i, v);
}
#define atomic_sub_return_relaxed atomic_sub_return_relaxed
#endif
#if !defined(arch_atomic_fetch_sub_relaxed) || defined(arch_atomic_fetch_sub)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_sub(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_sub(i, v);
}
#define atomic_fetch_sub atomic_fetch_sub
#endif
#if defined(arch_atomic_fetch_sub_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_sub_acquire(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_sub_acquire(i, v);
}
#define atomic_fetch_sub_acquire atomic_fetch_sub_acquire
#endif
#if defined(arch_atomic_fetch_sub_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_sub_release(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_sub_release(i, v);
}
#define atomic_fetch_sub_release atomic_fetch_sub_release
#endif
#if defined(arch_atomic_fetch_sub_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_sub_relaxed(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_sub_relaxed(i, v);
}
#define atomic_fetch_sub_relaxed atomic_fetch_sub_relaxed
#endif
#if defined(arch_atomic_inc)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline void
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_inc(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
arch_atomic_inc(v);
}
#define atomic_inc atomic_inc
#endif
#if defined(arch_atomic_inc_return)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_inc_return(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_inc_return(v);
}
#define atomic_inc_return atomic_inc_return
#endif
#if defined(arch_atomic_inc_return_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_inc_return_acquire(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_inc_return_acquire(v);
}
#define atomic_inc_return_acquire atomic_inc_return_acquire
#endif
#if defined(arch_atomic_inc_return_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_inc_return_release(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_inc_return_release(v);
}
#define atomic_inc_return_release atomic_inc_return_release
#endif
#if defined(arch_atomic_inc_return_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_inc_return_relaxed(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_inc_return_relaxed(v);
}
#define atomic_inc_return_relaxed atomic_inc_return_relaxed
#endif
#if defined(arch_atomic_fetch_inc)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_inc(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_inc(v);
}
#define atomic_fetch_inc atomic_fetch_inc
#endif
#if defined(arch_atomic_fetch_inc_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_inc_acquire(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_inc_acquire(v);
}
#define atomic_fetch_inc_acquire atomic_fetch_inc_acquire
#endif
#if defined(arch_atomic_fetch_inc_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_inc_release(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_inc_release(v);
}
#define atomic_fetch_inc_release atomic_fetch_inc_release
#endif
#if defined(arch_atomic_fetch_inc_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_inc_relaxed(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_inc_relaxed(v);
}
#define atomic_fetch_inc_relaxed atomic_fetch_inc_relaxed
#endif
#if defined(arch_atomic_dec)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline void
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_dec(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
arch_atomic_dec(v);
}
#define atomic_dec atomic_dec
#endif
#if defined(arch_atomic_dec_return)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_dec_return(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_dec_return(v);
}
#define atomic_dec_return atomic_dec_return
#endif
#if defined(arch_atomic_dec_return_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_dec_return_acquire(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_dec_return_acquire(v);
}
#define atomic_dec_return_acquire atomic_dec_return_acquire
#endif
#if defined(arch_atomic_dec_return_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_dec_return_release(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_dec_return_release(v);
}
#define atomic_dec_return_release atomic_dec_return_release
#endif
#if defined(arch_atomic_dec_return_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_dec_return_relaxed(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_dec_return_relaxed(v);
}
#define atomic_dec_return_relaxed atomic_dec_return_relaxed
#endif
#if defined(arch_atomic_fetch_dec)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_dec(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_dec(v);
}
#define atomic_fetch_dec atomic_fetch_dec
#endif
#if defined(arch_atomic_fetch_dec_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_dec_acquire(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_dec_acquire(v);
}
#define atomic_fetch_dec_acquire atomic_fetch_dec_acquire
#endif
#if defined(arch_atomic_fetch_dec_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_dec_release(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_dec_release(v);
}
#define atomic_fetch_dec_release atomic_fetch_dec_release
#endif
#if defined(arch_atomic_fetch_dec_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_dec_relaxed(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_dec_relaxed(v);
}
#define atomic_fetch_dec_relaxed atomic_fetch_dec_relaxed
#endif
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline void
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_and(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
arch_atomic_and(i, v);
}
#define atomic_and atomic_and
#if !defined(arch_atomic_fetch_and_relaxed) || defined(arch_atomic_fetch_and)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_and(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_and(i, v);
}
#define atomic_fetch_and atomic_fetch_and
#endif
#if defined(arch_atomic_fetch_and_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_and_acquire(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_and_acquire(i, v);
}
#define atomic_fetch_and_acquire atomic_fetch_and_acquire
#endif
#if defined(arch_atomic_fetch_and_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_and_release(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_and_release(i, v);
}
#define atomic_fetch_and_release atomic_fetch_and_release
#endif
#if defined(arch_atomic_fetch_and_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_and_relaxed(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_and_relaxed(i, v);
}
#define atomic_fetch_and_relaxed atomic_fetch_and_relaxed
#endif
#if defined(arch_atomic_andnot)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline void
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_andnot(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
arch_atomic_andnot(i, v);
}
#define atomic_andnot atomic_andnot
#endif
#if defined(arch_atomic_fetch_andnot)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_andnot(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_andnot(i, v);
}
#define atomic_fetch_andnot atomic_fetch_andnot
#endif
#if defined(arch_atomic_fetch_andnot_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_andnot_acquire(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_andnot_acquire(i, v);
}
#define atomic_fetch_andnot_acquire atomic_fetch_andnot_acquire
#endif
#if defined(arch_atomic_fetch_andnot_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_andnot_release(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_andnot_release(i, v);
}
#define atomic_fetch_andnot_release atomic_fetch_andnot_release
#endif
#if defined(arch_atomic_fetch_andnot_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_andnot_relaxed(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_andnot_relaxed(i, v);
}
#define atomic_fetch_andnot_relaxed atomic_fetch_andnot_relaxed
#endif
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline void
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_or(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
arch_atomic_or(i, v);
}
#define atomic_or atomic_or
#if !defined(arch_atomic_fetch_or_relaxed) || defined(arch_atomic_fetch_or)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_or(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_or(i, v);
}
#define atomic_fetch_or atomic_fetch_or
#endif
#if defined(arch_atomic_fetch_or_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_or_acquire(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_or_acquire(i, v);
}
#define atomic_fetch_or_acquire atomic_fetch_or_acquire
#endif
#if defined(arch_atomic_fetch_or_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_or_release(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_or_release(i, v);
}
#define atomic_fetch_or_release atomic_fetch_or_release
#endif
#if defined(arch_atomic_fetch_or_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_or_relaxed(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_or_relaxed(i, v);
}
#define atomic_fetch_or_relaxed atomic_fetch_or_relaxed
#endif
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline void
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_xor(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
arch_atomic_xor(i, v);
}
#define atomic_xor atomic_xor
#if !defined(arch_atomic_fetch_xor_relaxed) || defined(arch_atomic_fetch_xor)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_xor(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_xor(i, v);
}
#define atomic_fetch_xor atomic_fetch_xor
#endif
#if defined(arch_atomic_fetch_xor_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_xor_acquire(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_xor_acquire(i, v);
}
#define atomic_fetch_xor_acquire atomic_fetch_xor_acquire
#endif
#if defined(arch_atomic_fetch_xor_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_xor_release(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_xor_release(i, v);
}
#define atomic_fetch_xor_release atomic_fetch_xor_release
#endif
#if defined(arch_atomic_fetch_xor_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_xor_relaxed(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_xor_relaxed(i, v);
}
#define atomic_fetch_xor_relaxed atomic_fetch_xor_relaxed
#endif
#if !defined(arch_atomic_xchg_relaxed) || defined(arch_atomic_xchg)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_xchg(atomic_t *v, int i)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_xchg(v, i);
}
#define atomic_xchg atomic_xchg
#endif
#if defined(arch_atomic_xchg_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_xchg_acquire(atomic_t *v, int i)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_xchg_acquire(v, i);
}
#define atomic_xchg_acquire atomic_xchg_acquire
#endif
#if defined(arch_atomic_xchg_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_xchg_release(atomic_t *v, int i)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_xchg_release(v, i);
}
#define atomic_xchg_release atomic_xchg_release
#endif
#if defined(arch_atomic_xchg_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_xchg_relaxed(atomic_t *v, int i)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_xchg_relaxed(v, i);
}
#define atomic_xchg_relaxed atomic_xchg_relaxed
#endif
#if !defined(arch_atomic_cmpxchg_relaxed) || defined(arch_atomic_cmpxchg)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_cmpxchg(atomic_t *v, int old, int new)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_cmpxchg(v, old, new);
}
#define atomic_cmpxchg atomic_cmpxchg
#endif
#if defined(arch_atomic_cmpxchg_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_cmpxchg_acquire(atomic_t *v, int old, int new)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_cmpxchg_acquire(v, old, new);
}
#define atomic_cmpxchg_acquire atomic_cmpxchg_acquire
#endif
#if defined(arch_atomic_cmpxchg_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_cmpxchg_release(atomic_t *v, int old, int new)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_cmpxchg_release(v, old, new);
}
#define atomic_cmpxchg_release atomic_cmpxchg_release
#endif
#if defined(arch_atomic_cmpxchg_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_cmpxchg_relaxed(atomic_t *v, int old, int new)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_cmpxchg_relaxed(v, old, new);
}
#define atomic_cmpxchg_relaxed atomic_cmpxchg_relaxed
#endif
#if defined(arch_atomic_try_cmpxchg)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_try_cmpxchg(atomic_t *v, int *old, int new)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
instrument_atomic_read_write(old, sizeof(*old));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_try_cmpxchg(v, old, new);
}
#define atomic_try_cmpxchg atomic_try_cmpxchg
#endif
#if defined(arch_atomic_try_cmpxchg_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_try_cmpxchg_acquire(atomic_t *v, int *old, int new)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
instrument_atomic_read_write(old, sizeof(*old));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_try_cmpxchg_acquire(v, old, new);
}
#define atomic_try_cmpxchg_acquire atomic_try_cmpxchg_acquire
#endif
#if defined(arch_atomic_try_cmpxchg_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_try_cmpxchg_release(atomic_t *v, int *old, int new)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
instrument_atomic_read_write(old, sizeof(*old));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_try_cmpxchg_release(v, old, new);
}
#define atomic_try_cmpxchg_release atomic_try_cmpxchg_release
#endif
#if defined(arch_atomic_try_cmpxchg_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_try_cmpxchg_relaxed(atomic_t *v, int *old, int new)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
instrument_atomic_read_write(old, sizeof(*old));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_try_cmpxchg_relaxed(v, old, new);
}
#define atomic_try_cmpxchg_relaxed atomic_try_cmpxchg_relaxed
#endif
#if defined(arch_atomic_sub_and_test)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_sub_and_test(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_sub_and_test(i, v);
}
#define atomic_sub_and_test atomic_sub_and_test
#endif
#if defined(arch_atomic_dec_and_test)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_dec_and_test(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_dec_and_test(v);
}
#define atomic_dec_and_test atomic_dec_and_test
#endif
#if defined(arch_atomic_inc_and_test)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_inc_and_test(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_inc_and_test(v);
}
#define atomic_inc_and_test atomic_inc_and_test
#endif
#if defined(arch_atomic_add_negative)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_add_negative(int i, atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_add_negative(i, v);
}
#define atomic_add_negative atomic_add_negative
#endif
#if defined(arch_atomic_fetch_add_unless)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_fetch_add_unless(atomic_t *v, int a, int u)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_fetch_add_unless(v, a, u);
}
#define atomic_fetch_add_unless atomic_fetch_add_unless
#endif
#if defined(arch_atomic_add_unless)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_add_unless(atomic_t *v, int a, int u)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_add_unless(v, a, u);
}
#define atomic_add_unless atomic_add_unless
#endif
#if defined(arch_atomic_inc_not_zero)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_inc_not_zero(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_inc_not_zero(v);
}
#define atomic_inc_not_zero atomic_inc_not_zero
#endif
#if defined(arch_atomic_inc_unless_negative)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_inc_unless_negative(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_inc_unless_negative(v);
}
#define atomic_inc_unless_negative atomic_inc_unless_negative
#endif
#if defined(arch_atomic_dec_unless_positive)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_dec_unless_positive(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_dec_unless_positive(v);
}
#define atomic_dec_unless_positive atomic_dec_unless_positive
#endif
#if defined(arch_atomic_dec_if_positive)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline int
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic_dec_if_positive(atomic_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic_dec_if_positive(v);
}
#define atomic_dec_if_positive atomic_dec_if_positive
#endif
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_read(const atomic64_t *v)
{
asm-generic, atomic-instrumented: Use generic instrumented.h This switches atomic-instrumented.h to use the generic instrumentation wrappers provided by instrumented.h. No functional change intended. Suggested-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org>
2020-01-22 00:05:09 +08:00
instrument_atomic_read(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_read(v);
}
#define atomic64_read atomic64_read
#if defined(arch_atomic64_read_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_read_acquire(const atomic64_t *v)
{
asm-generic, atomic-instrumented: Use generic instrumented.h This switches atomic-instrumented.h to use the generic instrumentation wrappers provided by instrumented.h. No functional change intended. Suggested-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org>
2020-01-22 00:05:09 +08:00
instrument_atomic_read(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_read_acquire(v);
}
#define atomic64_read_acquire atomic64_read_acquire
#endif
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline void
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_set(atomic64_t *v, s64 i)
{
asm-generic, atomic-instrumented: Use generic instrumented.h This switches atomic-instrumented.h to use the generic instrumentation wrappers provided by instrumented.h. No functional change intended. Suggested-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org>
2020-01-22 00:05:09 +08:00
instrument_atomic_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
arch_atomic64_set(v, i);
}
#define atomic64_set atomic64_set
#if defined(arch_atomic64_set_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline void
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_set_release(atomic64_t *v, s64 i)
{
asm-generic, atomic-instrumented: Use generic instrumented.h This switches atomic-instrumented.h to use the generic instrumentation wrappers provided by instrumented.h. No functional change intended. Suggested-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Paul E. McKenney <paulmck@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org>
2020-01-22 00:05:09 +08:00
instrument_atomic_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
arch_atomic64_set_release(v, i);
}
#define atomic64_set_release atomic64_set_release
#endif
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline void
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_add(s64 i, atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
arch_atomic64_add(i, v);
}
#define atomic64_add atomic64_add
#if !defined(arch_atomic64_add_return_relaxed) || defined(arch_atomic64_add_return)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_add_return(s64 i, atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_add_return(i, v);
}
#define atomic64_add_return atomic64_add_return
#endif
#if defined(arch_atomic64_add_return_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_add_return_acquire(s64 i, atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_add_return_acquire(i, v);
}
#define atomic64_add_return_acquire atomic64_add_return_acquire
#endif
#if defined(arch_atomic64_add_return_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_add_return_release(s64 i, atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_add_return_release(i, v);
}
#define atomic64_add_return_release atomic64_add_return_release
#endif
#if defined(arch_atomic64_add_return_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_add_return_relaxed(s64 i, atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_add_return_relaxed(i, v);
}
#define atomic64_add_return_relaxed atomic64_add_return_relaxed
#endif
#if !defined(arch_atomic64_fetch_add_relaxed) || defined(arch_atomic64_fetch_add)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_add(s64 i, atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_add(i, v);
}
#define atomic64_fetch_add atomic64_fetch_add
#endif
#if defined(arch_atomic64_fetch_add_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_add_acquire(s64 i, atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_add_acquire(i, v);
}
#define atomic64_fetch_add_acquire atomic64_fetch_add_acquire
#endif
#if defined(arch_atomic64_fetch_add_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_add_release(s64 i, atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_add_release(i, v);
}
#define atomic64_fetch_add_release atomic64_fetch_add_release
#endif
#if defined(arch_atomic64_fetch_add_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_add_relaxed(s64 i, atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_add_relaxed(i, v);
}
#define atomic64_fetch_add_relaxed atomic64_fetch_add_relaxed
#endif
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline void
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_sub(s64 i, atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
arch_atomic64_sub(i, v);
}
#define atomic64_sub atomic64_sub
locking/atomic, asm-generic, x86: Add comments for atomic instrumentation The comments are factored out from the code changes to make them easier to read. Add them separately to explain some non-obvious aspects. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/cc595efc644bb905407012d82d3eb8bac3368e7a.1517246437.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:07 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if !defined(arch_atomic64_sub_return_relaxed) || defined(arch_atomic64_sub_return)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_sub_return(s64 i, atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_sub_return(i, v);
}
#define atomic64_sub_return atomic64_sub_return
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_sub_return_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_sub_return_acquire(s64 i, atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_sub_return_acquire(i, v);
}
#define atomic64_sub_return_acquire atomic64_sub_return_acquire
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_sub_return_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_sub_return_release(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_sub_return_release(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_sub_return_release atomic64_sub_return_release
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_sub_return_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_sub_return_relaxed(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_sub_return_relaxed(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_sub_return_relaxed atomic64_sub_return_relaxed
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if !defined(arch_atomic64_fetch_sub_relaxed) || defined(arch_atomic64_fetch_sub)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_sub(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_sub(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_sub atomic64_fetch_sub
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_sub_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_sub_acquire(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_sub_acquire(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_sub_acquire atomic64_fetch_sub_acquire
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_sub_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_sub_release(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_sub_release(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_sub_release atomic64_fetch_sub_release
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_sub_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_sub_relaxed(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_sub_relaxed(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_sub_relaxed atomic64_fetch_sub_relaxed
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_inc)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline void
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_inc(atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
arch_atomic64_inc(v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_inc atomic64_inc
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_inc_return)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_inc_return(atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_inc_return(v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_inc_return atomic64_inc_return
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_inc_return_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_inc_return_acquire(atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_inc_return_acquire(v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_inc_return_acquire atomic64_inc_return_acquire
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
#endif
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_inc_return_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_inc_return_release(atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_inc_return_release(v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_inc_return_release atomic64_inc_return_release
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
#endif
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_inc_return_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_inc_return_relaxed(atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_inc_return_relaxed(v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_inc_return_relaxed atomic64_inc_return_relaxed
atomics/treewide: Make atomic_fetch_add_unless() optional Several architectures these have a near-identical implementation based on atomic_read() and atomic_cmpxchg() which we can instead define in <linux/atomic.h>, so let's do so, using something close to the existing x86 implementation with try_cmpxchg(). Where an architecture provides its own atomic_fetch_add_unless(), it must define a preprocessor symbol for it. The instrumented atomics are updated accordingly. Note that arch/arc's existing atomic_fetch_add_unless() had redundant barriers, as these are already present in its atomic_cmpxchg() implementation. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Geert Uytterhoeven <geert@linux-m68k.org> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Geert Uytterhoeven <geert@linux-m68k.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Palmer Dabbelt <palmer@sifive.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Vineet Gupta <vgupta@synopsys.com> Link: https://lore.kernel.org/lkml/20180621121321.4761-7-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:09 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_inc)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_inc(atomic64_t *v)
atomics: Prepare for atomic64_fetch_add_unless() Currently all architectures must implement atomic_fetch_add_unless(), with common code providing atomic_add_unless(). Architectures must also implement atomic64_add_unless() directly, with no corresponding atomic64_fetch_add_unless(). This divergence is unfortunate, and means that the APIs for atomic_t, atomic64_t, and atomic_long_t differ. In preparation for unifying things, with architectures providing atomic64_fetch_add_unless, this patch adds a generic atomic64_add_unless() which will use atomic64_fetch_add_unless(). The instrumented atomics are updated to take this case into account. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: Albert Ou <albert@sifive.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Matt Turner <mattst88@gmail.com> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Palmer Dabbelt <palmer@sifive.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Richard Henderson <rth@twiddle.net> Cc: Russell King <linux@armlinux.org.uk> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Vineet Gupta <vgupta@synopsys.com> Link: https://lore.kernel.org/lkml/20180621121321.4761-8-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:10 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_inc(v);
atomics: Prepare for atomic64_fetch_add_unless() Currently all architectures must implement atomic_fetch_add_unless(), with common code providing atomic_add_unless(). Architectures must also implement atomic64_add_unless() directly, with no corresponding atomic64_fetch_add_unless(). This divergence is unfortunate, and means that the APIs for atomic_t, atomic64_t, and atomic_long_t differ. In preparation for unifying things, with architectures providing atomic64_fetch_add_unless, this patch adds a generic atomic64_add_unless() which will use atomic64_fetch_add_unless(). The instrumented atomics are updated to take this case into account. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: Albert Ou <albert@sifive.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Matt Turner <mattst88@gmail.com> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Palmer Dabbelt <palmer@sifive.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Richard Henderson <rth@twiddle.net> Cc: Russell King <linux@armlinux.org.uk> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Vineet Gupta <vgupta@synopsys.com> Link: https://lore.kernel.org/lkml/20180621121321.4761-8-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:10 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_inc atomic64_fetch_inc
atomics: Prepare for atomic64_fetch_add_unless() Currently all architectures must implement atomic_fetch_add_unless(), with common code providing atomic_add_unless(). Architectures must also implement atomic64_add_unless() directly, with no corresponding atomic64_fetch_add_unless(). This divergence is unfortunate, and means that the APIs for atomic_t, atomic64_t, and atomic_long_t differ. In preparation for unifying things, with architectures providing atomic64_fetch_add_unless, this patch adds a generic atomic64_add_unless() which will use atomic64_fetch_add_unless(). The instrumented atomics are updated to take this case into account. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: Albert Ou <albert@sifive.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Matt Turner <mattst88@gmail.com> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Palmer Dabbelt <palmer@sifive.com> Cc: Paul Mackerras <paulus@samba.org> Cc: Richard Henderson <rth@twiddle.net> Cc: Russell King <linux@armlinux.org.uk> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Vineet Gupta <vgupta@synopsys.com> Link: https://lore.kernel.org/lkml/20180621121321.4761-8-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:10 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_inc_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_inc_acquire(atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_inc_acquire(v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_inc_acquire atomic64_fetch_inc_acquire
atomics/treewide: Make unconditional inc/dec ops optional Many of the inc/dec ops are mandatory, but for most architectures inc/dec are simply trivial wrappers around their corresponding add/sub ops. Let's make all the inc/dec ops optional, so that we can get rid of these boilerplate wrappers. The instrumented atomics are updated accordingly. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Palmer Dabbelt <palmer@sifive.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/lkml/20180621121321.4761-17-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:19 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_inc_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_inc_release(atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_inc_release(v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_inc_release atomic64_fetch_inc_release
atomics/treewide: Make unconditional inc/dec ops optional Many of the inc/dec ops are mandatory, but for most architectures inc/dec are simply trivial wrappers around their corresponding add/sub ops. Let's make all the inc/dec ops optional, so that we can get rid of these boilerplate wrappers. The instrumented atomics are updated accordingly. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Palmer Dabbelt <palmer@sifive.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/lkml/20180621121321.4761-17-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:19 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_inc_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_inc_relaxed(atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_inc_relaxed(v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_inc_relaxed atomic64_fetch_inc_relaxed
atomics/treewide: Make unconditional inc/dec ops optional Many of the inc/dec ops are mandatory, but for most architectures inc/dec are simply trivial wrappers around their corresponding add/sub ops. Let's make all the inc/dec ops optional, so that we can get rid of these boilerplate wrappers. The instrumented atomics are updated accordingly. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Palmer Dabbelt <palmer@sifive.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/lkml/20180621121321.4761-17-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:19 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_dec)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline void
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_dec(atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
arch_atomic64_dec(v);
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_dec atomic64_dec
atomics/treewide: Make unconditional inc/dec ops optional Many of the inc/dec ops are mandatory, but for most architectures inc/dec are simply trivial wrappers around their corresponding add/sub ops. Let's make all the inc/dec ops optional, so that we can get rid of these boilerplate wrappers. The instrumented atomics are updated accordingly. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Palmer Dabbelt <palmer@sifive.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/lkml/20180621121321.4761-17-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:19 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_dec_return)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_dec_return(atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_dec_return(v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_dec_return atomic64_dec_return
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_dec_return_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_dec_return_acquire(atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_dec_return_acquire(v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_dec_return_acquire atomic64_dec_return_acquire
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_dec_return_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_dec_return_release(atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_dec_return_release(v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_dec_return_release atomic64_dec_return_release
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_dec_return_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_dec_return_relaxed(atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_dec_return_relaxed(v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_dec_return_relaxed atomic64_dec_return_relaxed
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_dec)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_dec(atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_dec(v);
}
#define atomic64_fetch_dec atomic64_fetch_dec
#endif
#if defined(arch_atomic64_fetch_dec_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_dec_acquire(atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_dec_acquire(v);
}
#define atomic64_fetch_dec_acquire atomic64_fetch_dec_acquire
#endif
#if defined(arch_atomic64_fetch_dec_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_dec_release(atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_dec_release(v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_dec_release atomic64_fetch_dec_release
#endif
#if defined(arch_atomic64_fetch_dec_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_dec_relaxed(atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_dec_relaxed(v);
}
#define atomic64_fetch_dec_relaxed atomic64_fetch_dec_relaxed
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline void
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_and(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
arch_atomic64_and(i, v);
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_and atomic64_and
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if !defined(arch_atomic64_fetch_and_relaxed) || defined(arch_atomic64_fetch_and)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_and(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_and(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_and atomic64_fetch_and
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_and_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_and_acquire(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_and_acquire(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_and_acquire atomic64_fetch_and_acquire
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_and_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_and_release(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_and_release(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_and_release atomic64_fetch_and_release
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_and_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_and_relaxed(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_and_relaxed(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_and_relaxed atomic64_fetch_and_relaxed
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_andnot)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline void
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_andnot(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
arch_atomic64_andnot(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_andnot atomic64_andnot
atomics/treewide: Make unconditional inc/dec ops optional Many of the inc/dec ops are mandatory, but for most architectures inc/dec are simply trivial wrappers around their corresponding add/sub ops. Let's make all the inc/dec ops optional, so that we can get rid of these boilerplate wrappers. The instrumented atomics are updated accordingly. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Palmer Dabbelt <palmer@sifive.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/lkml/20180621121321.4761-17-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:19 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_andnot)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_andnot(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_andnot(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_andnot atomic64_fetch_andnot
atomics/treewide: Make unconditional inc/dec ops optional Many of the inc/dec ops are mandatory, but for most architectures inc/dec are simply trivial wrappers around their corresponding add/sub ops. Let's make all the inc/dec ops optional, so that we can get rid of these boilerplate wrappers. The instrumented atomics are updated accordingly. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Palmer Dabbelt <palmer@sifive.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/lkml/20180621121321.4761-17-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:19 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_andnot_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_andnot_acquire(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_andnot_acquire(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_andnot_acquire atomic64_fetch_andnot_acquire
atomics/treewide: Make unconditional inc/dec ops optional Many of the inc/dec ops are mandatory, but for most architectures inc/dec are simply trivial wrappers around their corresponding add/sub ops. Let's make all the inc/dec ops optional, so that we can get rid of these boilerplate wrappers. The instrumented atomics are updated accordingly. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Palmer Dabbelt <palmer@sifive.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/lkml/20180621121321.4761-17-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:19 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_andnot_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_andnot_release(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_andnot_release(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_andnot_release atomic64_fetch_andnot_release
atomics/treewide: Make unconditional inc/dec ops optional Many of the inc/dec ops are mandatory, but for most architectures inc/dec are simply trivial wrappers around their corresponding add/sub ops. Let's make all the inc/dec ops optional, so that we can get rid of these boilerplate wrappers. The instrumented atomics are updated accordingly. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Palmer Dabbelt <palmer@sifive.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/lkml/20180621121321.4761-17-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:19 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_andnot_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_andnot_relaxed(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_andnot_relaxed(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_andnot_relaxed atomic64_fetch_andnot_relaxed
atomics/treewide: Make atomic64_inc_not_zero() optional We define a trivial fallback for atomic_inc_not_zero(), but don't do the same for atomic64_inc_not_zero(), leading most architectures to define the same boilerplate. Let's add a fallback in <linux/atomic.h>, and remove the redundant implementations. Note that atomic64_add_unless() is always defined in <linux/atomic.h>, and promotes its arguments to the requisite types, so we need not do this explicitly. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Palmer Dabbelt <palmer@sifive.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/lkml/20180621121321.4761-6-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:08 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline void
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_or(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
arch_atomic64_or(i, v);
}
#define atomic64_or atomic64_or
#if !defined(arch_atomic64_fetch_or_relaxed) || defined(arch_atomic64_fetch_or)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_or(s64 i, atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_or(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_or atomic64_fetch_or
atomics/treewide: Make conditional inc/dec ops optional The conditional inc/dec ops differ for atomic_t and atomic64_t: - atomic_inc_unless_positive() is optional for atomic_t, and doesn't exist for atomic64_t. - atomic_dec_unless_negative() is optional for atomic_t, and doesn't exist for atomic64_t. - atomic_dec_if_positive is optional for atomic_t, and is mandatory for atomic64_t. Let's make these consistently optional for both. At the same time, let's clean up the existing fallbacks to use atomic_try_cmpxchg(). The instrumented atomics are updated accordingly. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/lkml/20180621121321.4761-18-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:20 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_or_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_or_acquire(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_or_acquire(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_or_acquire atomic64_fetch_or_acquire
atomics/treewide: Make test ops optional Some of the atomics return the result of a test applied after the atomic operation, and almost all architectures implement these as trivial wrappers around the underlying atomic. Specifically: * <atomic>_inc_and_test(v) is (<atomic>_inc_return(v) == 0) * <atomic>_dec_and_test(v) is (<atomic>_dec_return(v) == 0) * <atomic>_sub_and_test(i, v) is (<atomic>_sub_return(i, v) == 0) * <atomic>_add_negative(i, v) is (<atomic>_add_return(i, v) < 0) Rather than have these definitions duplicated in all architectures, with minor inconsistencies in formatting and documentation, let's make these operations optional, with default fallbacks as above. Implementations must now provide a preprocessor symbol. The instrumented atomics are updated accordingly. Both x86 and m68k have custom implementations, which are left as-is, given preprocessor symbols to avoid being overridden. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Geert Uytterhoeven <geert@linux-m68k.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Palmer Dabbelt <palmer@sifive.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/lkml/20180621121321.4761-16-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:18 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_or_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_or_release(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_or_release(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_or_release atomic64_fetch_or_release
atomics/treewide: Make test ops optional Some of the atomics return the result of a test applied after the atomic operation, and almost all architectures implement these as trivial wrappers around the underlying atomic. Specifically: * <atomic>_inc_and_test(v) is (<atomic>_inc_return(v) == 0) * <atomic>_dec_and_test(v) is (<atomic>_dec_return(v) == 0) * <atomic>_sub_and_test(i, v) is (<atomic>_sub_return(i, v) == 0) * <atomic>_add_negative(i, v) is (<atomic>_add_return(i, v) < 0) Rather than have these definitions duplicated in all architectures, with minor inconsistencies in formatting and documentation, let's make these operations optional, with default fallbacks as above. Implementations must now provide a preprocessor symbol. The instrumented atomics are updated accordingly. Both x86 and m68k have custom implementations, which are left as-is, given preprocessor symbols to avoid being overridden. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Geert Uytterhoeven <geert@linux-m68k.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Palmer Dabbelt <palmer@sifive.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/lkml/20180621121321.4761-16-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:18 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_or_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_or_relaxed(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_or_relaxed(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_or_relaxed atomic64_fetch_or_relaxed
atomics/treewide: Make test ops optional Some of the atomics return the result of a test applied after the atomic operation, and almost all architectures implement these as trivial wrappers around the underlying atomic. Specifically: * <atomic>_inc_and_test(v) is (<atomic>_inc_return(v) == 0) * <atomic>_dec_and_test(v) is (<atomic>_dec_return(v) == 0) * <atomic>_sub_and_test(i, v) is (<atomic>_sub_return(i, v) == 0) * <atomic>_add_negative(i, v) is (<atomic>_add_return(i, v) < 0) Rather than have these definitions duplicated in all architectures, with minor inconsistencies in formatting and documentation, let's make these operations optional, with default fallbacks as above. Implementations must now provide a preprocessor symbol. The instrumented atomics are updated accordingly. Both x86 and m68k have custom implementations, which are left as-is, given preprocessor symbols to avoid being overridden. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Geert Uytterhoeven <geert@linux-m68k.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Palmer Dabbelt <palmer@sifive.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/lkml/20180621121321.4761-16-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:18 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline void
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_xor(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
arch_atomic64_xor(i, v);
}
#define atomic64_xor atomic64_xor
#if !defined(arch_atomic64_fetch_xor_relaxed) || defined(arch_atomic64_fetch_xor)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_xor(s64 i, atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_xor(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_xor atomic64_fetch_xor
atomics/treewide: Make test ops optional Some of the atomics return the result of a test applied after the atomic operation, and almost all architectures implement these as trivial wrappers around the underlying atomic. Specifically: * <atomic>_inc_and_test(v) is (<atomic>_inc_return(v) == 0) * <atomic>_dec_and_test(v) is (<atomic>_dec_return(v) == 0) * <atomic>_sub_and_test(i, v) is (<atomic>_sub_return(i, v) == 0) * <atomic>_add_negative(i, v) is (<atomic>_add_return(i, v) < 0) Rather than have these definitions duplicated in all architectures, with minor inconsistencies in formatting and documentation, let's make these operations optional, with default fallbacks as above. Implementations must now provide a preprocessor symbol. The instrumented atomics are updated accordingly. Both x86 and m68k have custom implementations, which are left as-is, given preprocessor symbols to avoid being overridden. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Geert Uytterhoeven <geert@linux-m68k.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Palmer Dabbelt <palmer@sifive.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/lkml/20180621121321.4761-16-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:18 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_xor_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_xor_acquire(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_xor_acquire(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_xor_acquire atomic64_fetch_xor_acquire
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_xor_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_xor_release(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_xor_release(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_xor_release atomic64_fetch_xor_release
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_fetch_xor_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_xor_relaxed(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_xor_relaxed(i, v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_fetch_xor_relaxed atomic64_fetch_xor_relaxed
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if !defined(arch_atomic64_xchg_relaxed) || defined(arch_atomic64_xchg)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_xchg(atomic64_t *v, s64 i)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_xchg(v, i);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_xchg atomic64_xchg
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_xchg_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_xchg_acquire(atomic64_t *v, s64 i)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_xchg_acquire(v, i);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_xchg_acquire atomic64_xchg_acquire
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_xchg_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_xchg_release(atomic64_t *v, s64 i)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_xchg_release(v, i);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_xchg_release atomic64_xchg_release
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_xchg_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_xchg_relaxed(atomic64_t *v, s64 i)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_xchg_relaxed(v, i);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_xchg_relaxed atomic64_xchg_relaxed
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if !defined(arch_atomic64_cmpxchg_relaxed) || defined(arch_atomic64_cmpxchg)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_cmpxchg(v, old, new);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_cmpxchg atomic64_cmpxchg
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_cmpxchg_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_cmpxchg_acquire(atomic64_t *v, s64 old, s64 new)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_cmpxchg_acquire(v, old, new);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_cmpxchg_acquire atomic64_cmpxchg_acquire
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_cmpxchg_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_cmpxchg_release(atomic64_t *v, s64 old, s64 new)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_cmpxchg_release(v, old, new);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_cmpxchg_release atomic64_cmpxchg_release
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_cmpxchg_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_cmpxchg_relaxed(atomic64_t *v, s64 old, s64 new)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_cmpxchg_relaxed(v, old, new);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_cmpxchg_relaxed atomic64_cmpxchg_relaxed
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_try_cmpxchg)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 new)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
instrument_atomic_read_write(old, sizeof(*old));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_try_cmpxchg(v, old, new);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_try_cmpxchg atomic64_try_cmpxchg
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_try_cmpxchg_acquire)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_try_cmpxchg_acquire(atomic64_t *v, s64 *old, s64 new)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
instrument_atomic_read_write(old, sizeof(*old));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_try_cmpxchg_acquire(v, old, new);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_try_cmpxchg_acquire atomic64_try_cmpxchg_acquire
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_try_cmpxchg_release)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_try_cmpxchg_release(atomic64_t *v, s64 *old, s64 new)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
instrument_atomic_read_write(old, sizeof(*old));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_try_cmpxchg_release(v, old, new);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_try_cmpxchg_release atomic64_try_cmpxchg_release
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_try_cmpxchg_relaxed)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_try_cmpxchg_relaxed(atomic64_t *v, s64 *old, s64 new)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
instrument_atomic_read_write(old, sizeof(*old));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_try_cmpxchg_relaxed(v, old, new);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_try_cmpxchg_relaxed atomic64_try_cmpxchg_relaxed
atomics/treewide: Make test ops optional Some of the atomics return the result of a test applied after the atomic operation, and almost all architectures implement these as trivial wrappers around the underlying atomic. Specifically: * <atomic>_inc_and_test(v) is (<atomic>_inc_return(v) == 0) * <atomic>_dec_and_test(v) is (<atomic>_dec_return(v) == 0) * <atomic>_sub_and_test(i, v) is (<atomic>_sub_return(i, v) == 0) * <atomic>_add_negative(i, v) is (<atomic>_add_return(i, v) < 0) Rather than have these definitions duplicated in all architectures, with minor inconsistencies in formatting and documentation, let's make these operations optional, with default fallbacks as above. Implementations must now provide a preprocessor symbol. The instrumented atomics are updated accordingly. Both x86 and m68k have custom implementations, which are left as-is, given preprocessor symbols to avoid being overridden. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Geert Uytterhoeven <geert@linux-m68k.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Palmer Dabbelt <palmer@sifive.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/lkml/20180621121321.4761-16-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:18 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_sub_and_test)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_sub_and_test(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
return arch_atomic64_sub_and_test(i, v);
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_sub_and_test atomic64_sub_and_test
atomics/treewide: Make test ops optional Some of the atomics return the result of a test applied after the atomic operation, and almost all architectures implement these as trivial wrappers around the underlying atomic. Specifically: * <atomic>_inc_and_test(v) is (<atomic>_inc_return(v) == 0) * <atomic>_dec_and_test(v) is (<atomic>_dec_return(v) == 0) * <atomic>_sub_and_test(i, v) is (<atomic>_sub_return(i, v) == 0) * <atomic>_add_negative(i, v) is (<atomic>_add_return(i, v) < 0) Rather than have these definitions duplicated in all architectures, with minor inconsistencies in formatting and documentation, let's make these operations optional, with default fallbacks as above. Implementations must now provide a preprocessor symbol. The instrumented atomics are updated accordingly. Both x86 and m68k have custom implementations, which are left as-is, given preprocessor symbols to avoid being overridden. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Geert Uytterhoeven <geert@linux-m68k.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Palmer Dabbelt <palmer@sifive.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/lkml/20180621121321.4761-16-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:18 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_dec_and_test)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_dec_and_test(atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_dec_and_test(v);
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_dec_and_test atomic64_dec_and_test
atomics/treewide: Make test ops optional Some of the atomics return the result of a test applied after the atomic operation, and almost all architectures implement these as trivial wrappers around the underlying atomic. Specifically: * <atomic>_inc_and_test(v) is (<atomic>_inc_return(v) == 0) * <atomic>_dec_and_test(v) is (<atomic>_dec_return(v) == 0) * <atomic>_sub_and_test(i, v) is (<atomic>_sub_return(i, v) == 0) * <atomic>_add_negative(i, v) is (<atomic>_add_return(i, v) < 0) Rather than have these definitions duplicated in all architectures, with minor inconsistencies in formatting and documentation, let's make these operations optional, with default fallbacks as above. Implementations must now provide a preprocessor symbol. The instrumented atomics are updated accordingly. Both x86 and m68k have custom implementations, which are left as-is, given preprocessor symbols to avoid being overridden. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Geert Uytterhoeven <geert@linux-m68k.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Palmer Dabbelt <palmer@sifive.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/lkml/20180621121321.4761-16-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:18 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_inc_and_test)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_inc_and_test(atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_inc_and_test(v);
}
#define atomic64_inc_and_test atomic64_inc_and_test
#endif
#if defined(arch_atomic64_add_negative)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_add_negative(s64 i, atomic64_t *v)
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
return arch_atomic64_add_negative(i, v);
}
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#define atomic64_add_negative atomic64_add_negative
#endif
#if defined(arch_atomic64_fetch_add_unless)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_fetch_add_unless(atomic64_t *v, s64 a, s64 u)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_fetch_add_unless(v, a, u);
}
#define atomic64_fetch_add_unless atomic64_fetch_add_unless
#endif
#if defined(arch_atomic64_add_unless)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_add_unless(atomic64_t *v, s64 a, s64 u)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_add_unless(v, a, u);
}
#define atomic64_add_unless atomic64_add_unless
#endif
#if defined(arch_atomic64_inc_not_zero)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_inc_not_zero(atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_inc_not_zero(v);
}
#define atomic64_inc_not_zero atomic64_inc_not_zero
#endif
#if defined(arch_atomic64_inc_unless_negative)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_inc_unless_negative(atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_inc_unless_negative(v);
}
#define atomic64_inc_unless_negative atomic64_inc_unless_negative
atomics/treewide: Make test ops optional Some of the atomics return the result of a test applied after the atomic operation, and almost all architectures implement these as trivial wrappers around the underlying atomic. Specifically: * <atomic>_inc_and_test(v) is (<atomic>_inc_return(v) == 0) * <atomic>_dec_and_test(v) is (<atomic>_dec_return(v) == 0) * <atomic>_sub_and_test(i, v) is (<atomic>_sub_return(i, v) == 0) * <atomic>_add_negative(i, v) is (<atomic>_add_return(i, v) < 0) Rather than have these definitions duplicated in all architectures, with minor inconsistencies in formatting and documentation, let's make these operations optional, with default fallbacks as above. Implementations must now provide a preprocessor symbol. The instrumented atomics are updated accordingly. Both x86 and m68k have custom implementations, which are left as-is, given preprocessor symbols to avoid being overridden. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Will Deacon <will.deacon@arm.com> Acked-by: Geert Uytterhoeven <geert@linux-m68k.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Palmer Dabbelt <palmer@sifive.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Link: https://lore.kernel.org/lkml/20180621121321.4761-16-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-06-21 20:13:18 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_atomic64_dec_unless_positive)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline bool
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_dec_unless_positive(atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_dec_unless_positive(v);
}
#define atomic64_dec_unless_positive atomic64_dec_unless_positive
#endif
#if defined(arch_atomic64_dec_if_positive)
asm-generic/atomic: Use __always_inline for pure wrappers Prefer __always_inline for atomic wrappers. When building for size (CC_OPTIMIZE_FOR_SIZE), some compilers appear to be less inclined to inline even relatively small static inline functions that are assumed to be inlinable such as atomic ops. This can cause problems, for example in UACCESS regions. By using __always_inline, we let the real implementation and not the wrapper determine the final inlining preference. For x86 tinyconfig we observe: - vmlinux baseline: 1316204 - vmlinux with patch: 1315988 (-216 bytes) This came up when addressing UACCESS warnings with CC_OPTIMIZE_FOR_SIZE in the KCSAN runtime: http://lkml.kernel.org/r/58708908-84a0-0a81-a836-ad97e33dbb62@infradead.org Reported-by: Randy Dunlap <rdunlap@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2019-11-26 22:04:04 +08:00
static __always_inline s64
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
atomic64_dec_if_positive(atomic64_t *v)
{
locking/atomics: Use read-write instrumentation for atomic RMWs Use instrument_atomic_read_write() for atomic RMW ops. Cc: Will Deacon <will@kernel.org> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: <linux-arch@vger.kernel.org> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Marco Elver <elver@google.com> Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
2020-07-24 15:00:08 +08:00
instrument_atomic_read_write(v, sizeof(*v));
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
return arch_atomic64_dec_if_positive(v);
}
#define atomic64_dec_if_positive atomic64_dec_if_positive
#endif
#if !defined(arch_xchg_relaxed) || defined(arch_xchg)
asm-generic/atomic: Add try_cmpxchg() fallbacks Only x86 provides try_cmpxchg() outside of the atomic_t interfaces, provide generic fallbacks to create this interface from the widely available cmpxchg() function. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/159870621515.1229682.15506193091065001742.stgit@devnote2
2020-08-29 21:03:35 +08:00
#define xchg(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \
arch_xchg(__ai_ptr, __VA_ARGS__); \
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
})
#endif
#if defined(arch_xchg_acquire)
asm-generic/atomic: Add try_cmpxchg() fallbacks Only x86 provides try_cmpxchg() outside of the atomic_t interfaces, provide generic fallbacks to create this interface from the widely available cmpxchg() function. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/159870621515.1229682.15506193091065001742.stgit@devnote2
2020-08-29 21:03:35 +08:00
#define xchg_acquire(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \
arch_xchg_acquire(__ai_ptr, __VA_ARGS__); \
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
})
#endif
#if defined(arch_xchg_release)
asm-generic/atomic: Add try_cmpxchg() fallbacks Only x86 provides try_cmpxchg() outside of the atomic_t interfaces, provide generic fallbacks to create this interface from the widely available cmpxchg() function. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/159870621515.1229682.15506193091065001742.stgit@devnote2
2020-08-29 21:03:35 +08:00
#define xchg_release(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \
arch_xchg_release(__ai_ptr, __VA_ARGS__); \
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
})
#endif
#if defined(arch_xchg_relaxed)
asm-generic/atomic: Add try_cmpxchg() fallbacks Only x86 provides try_cmpxchg() outside of the atomic_t interfaces, provide generic fallbacks to create this interface from the widely available cmpxchg() function. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/159870621515.1229682.15506193091065001742.stgit@devnote2
2020-08-29 21:03:35 +08:00
#define xchg_relaxed(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \
arch_xchg_relaxed(__ai_ptr, __VA_ARGS__); \
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
})
#endif
#if !defined(arch_cmpxchg_relaxed) || defined(arch_cmpxchg)
asm-generic/atomic: Add try_cmpxchg() fallbacks Only x86 provides try_cmpxchg() outside of the atomic_t interfaces, provide generic fallbacks to create this interface from the widely available cmpxchg() function. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/159870621515.1229682.15506193091065001742.stgit@devnote2
2020-08-29 21:03:35 +08:00
#define cmpxchg(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \
arch_cmpxchg(__ai_ptr, __VA_ARGS__); \
locking/atomics: Instrument xchg() While we instrument all of the (non-relaxed) atomic_*() functions and cmpxchg(), we missed xchg(). Let's add instrumentation for xchg(), fixing up x86 to implement arch_xchg(). Signed-off-by: Mark Rutland <mark.rutland@arm.com> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: andy.shevchenko@gmail.com Cc: arnd@arndb.de Cc: aryabinin@virtuozzo.com Cc: catalin.marinas@arm.com Cc: glider@google.com Cc: linux-arm-kernel@lists.infradead.org Cc: parri.andrea@gmail.com Cc: peter@hurleysoftware.com Link: http://lkml.kernel.org/r/20180716113017.3909-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-07-16 19:30:09 +08:00
})
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#endif
#if defined(arch_cmpxchg_acquire)
asm-generic/atomic: Add try_cmpxchg() fallbacks Only x86 provides try_cmpxchg() outside of the atomic_t interfaces, provide generic fallbacks to create this interface from the widely available cmpxchg() function. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/159870621515.1229682.15506193091065001742.stgit@devnote2
2020-08-29 21:03:35 +08:00
#define cmpxchg_acquire(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \
arch_cmpxchg_acquire(__ai_ptr, __VA_ARGS__); \
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
})
#endif
#if defined(arch_cmpxchg_release)
asm-generic/atomic: Add try_cmpxchg() fallbacks Only x86 provides try_cmpxchg() outside of the atomic_t interfaces, provide generic fallbacks to create this interface from the widely available cmpxchg() function. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/159870621515.1229682.15506193091065001742.stgit@devnote2
2020-08-29 21:03:35 +08:00
#define cmpxchg_release(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \
arch_cmpxchg_release(__ai_ptr, __VA_ARGS__); \
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
})
#endif
#if defined(arch_cmpxchg_relaxed)
asm-generic/atomic: Add try_cmpxchg() fallbacks Only x86 provides try_cmpxchg() outside of the atomic_t interfaces, provide generic fallbacks to create this interface from the widely available cmpxchg() function. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/159870621515.1229682.15506193091065001742.stgit@devnote2
2020-08-29 21:03:35 +08:00
#define cmpxchg_relaxed(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \
arch_cmpxchg_relaxed(__ai_ptr, __VA_ARGS__); \
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
})
#endif
locking/atomics: Instrument xchg() While we instrument all of the (non-relaxed) atomic_*() functions and cmpxchg(), we missed xchg(). Let's add instrumentation for xchg(), fixing up x86 to implement arch_xchg(). Signed-off-by: Mark Rutland <mark.rutland@arm.com> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: andy.shevchenko@gmail.com Cc: arnd@arndb.de Cc: aryabinin@virtuozzo.com Cc: catalin.marinas@arm.com Cc: glider@google.com Cc: linux-arm-kernel@lists.infradead.org Cc: parri.andrea@gmail.com Cc: peter@hurleysoftware.com Link: http://lkml.kernel.org/r/20180716113017.3909-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-07-16 19:30:09 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if !defined(arch_cmpxchg64_relaxed) || defined(arch_cmpxchg64)
asm-generic/atomic: Add try_cmpxchg() fallbacks Only x86 provides try_cmpxchg() outside of the atomic_t interfaces, provide generic fallbacks to create this interface from the widely available cmpxchg() function. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/159870621515.1229682.15506193091065001742.stgit@devnote2
2020-08-29 21:03:35 +08:00
#define cmpxchg64(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \
arch_cmpxchg64(__ai_ptr, __VA_ARGS__); \
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
})
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#endif
#if defined(arch_cmpxchg64_acquire)
asm-generic/atomic: Add try_cmpxchg() fallbacks Only x86 provides try_cmpxchg() outside of the atomic_t interfaces, provide generic fallbacks to create this interface from the widely available cmpxchg() function. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/159870621515.1229682.15506193091065001742.stgit@devnote2
2020-08-29 21:03:35 +08:00
#define cmpxchg64_acquire(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \
arch_cmpxchg64_acquire(__ai_ptr, __VA_ARGS__); \
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
})
#endif
#if defined(arch_cmpxchg64_release)
asm-generic/atomic: Add try_cmpxchg() fallbacks Only x86 provides try_cmpxchg() outside of the atomic_t interfaces, provide generic fallbacks to create this interface from the widely available cmpxchg() function. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/159870621515.1229682.15506193091065001742.stgit@devnote2
2020-08-29 21:03:35 +08:00
#define cmpxchg64_release(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \
arch_cmpxchg64_release(__ai_ptr, __VA_ARGS__); \
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
})
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#if defined(arch_cmpxchg64_relaxed)
asm-generic/atomic: Add try_cmpxchg() fallbacks Only x86 provides try_cmpxchg() outside of the atomic_t interfaces, provide generic fallbacks to create this interface from the widely available cmpxchg() function. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/159870621515.1229682.15506193091065001742.stgit@devnote2
2020-08-29 21:03:35 +08:00
#define cmpxchg64_relaxed(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \
arch_cmpxchg64_relaxed(__ai_ptr, __VA_ARGS__); \
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
})
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#endif
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
asm-generic/atomic: Add try_cmpxchg() fallbacks Only x86 provides try_cmpxchg() outside of the atomic_t interfaces, provide generic fallbacks to create this interface from the widely available cmpxchg() function. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/159870621515.1229682.15506193091065001742.stgit@devnote2
2020-08-29 21:03:35 +08:00
#if !defined(arch_try_cmpxchg_relaxed) || defined(arch_try_cmpxchg)
#define try_cmpxchg(ptr, oldp, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
typeof(oldp) __ai_oldp = (oldp); \
instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \
instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \
arch_try_cmpxchg(__ai_ptr, __ai_oldp, __VA_ARGS__); \
})
#endif
#if defined(arch_try_cmpxchg_acquire)
#define try_cmpxchg_acquire(ptr, oldp, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
typeof(oldp) __ai_oldp = (oldp); \
instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \
instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \
arch_try_cmpxchg_acquire(__ai_ptr, __ai_oldp, __VA_ARGS__); \
})
#endif
#if defined(arch_try_cmpxchg_release)
#define try_cmpxchg_release(ptr, oldp, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
typeof(oldp) __ai_oldp = (oldp); \
instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \
instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \
arch_try_cmpxchg_release(__ai_ptr, __ai_oldp, __VA_ARGS__); \
})
#endif
#if defined(arch_try_cmpxchg_relaxed)
#define try_cmpxchg_relaxed(ptr, oldp, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
typeof(oldp) __ai_oldp = (oldp); \
instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \
instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \
arch_try_cmpxchg_relaxed(__ai_ptr, __ai_oldp, __VA_ARGS__); \
})
#endif
#define cmpxchg_local(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \
arch_cmpxchg_local(__ai_ptr, __VA_ARGS__); \
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
})
asm-generic/atomic: Add try_cmpxchg() fallbacks Only x86 provides try_cmpxchg() outside of the atomic_t interfaces, provide generic fallbacks to create this interface from the widely available cmpxchg() function. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/159870621515.1229682.15506193091065001742.stgit@devnote2
2020-08-29 21:03:35 +08:00
#define cmpxchg64_local(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \
arch_cmpxchg64_local(__ai_ptr, __VA_ARGS__); \
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
})
asm-generic/atomic: Add try_cmpxchg() fallbacks Only x86 provides try_cmpxchg() outside of the atomic_t interfaces, provide generic fallbacks to create this interface from the widely available cmpxchg() function. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/159870621515.1229682.15506193091065001742.stgit@devnote2
2020-08-29 21:03:35 +08:00
#define sync_cmpxchg(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \
arch_sync_cmpxchg(__ai_ptr, __VA_ARGS__); \
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
})
asm-generic/atomic: Add try_cmpxchg() fallbacks Only x86 provides try_cmpxchg() outside of the atomic_t interfaces, provide generic fallbacks to create this interface from the widely available cmpxchg() function. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/159870621515.1229682.15506193091065001742.stgit@devnote2
2020-08-29 21:03:35 +08:00
#define cmpxchg_double(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
instrument_atomic_write(__ai_ptr, 2 * sizeof(*__ai_ptr)); \
arch_cmpxchg_double(__ai_ptr, __VA_ARGS__); \
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
})
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
asm-generic/atomic: Add try_cmpxchg() fallbacks Only x86 provides try_cmpxchg() outside of the atomic_t interfaces, provide generic fallbacks to create this interface from the widely available cmpxchg() function. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> Signed-off-by: Ingo Molnar <mingo@kernel.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/159870621515.1229682.15506193091065001742.stgit@devnote2
2020-08-29 21:03:35 +08:00
#define cmpxchg_double_local(ptr, ...) \
({ \
typeof(ptr) __ai_ptr = (ptr); \
instrument_atomic_write(__ai_ptr, 2 * sizeof(*__ai_ptr)); \
arch_cmpxchg_double_local(__ai_ptr, __VA_ARGS__); \
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h The new header allows to wrap per-arch atomic operations and add common functionality to all of them. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-01-30 01:26:04 +08:00
})
locking/atomics: Switch to generated instrumentation As a step towards ensuring the atomic* APIs are consistent, let's switch to wrappers generated by gen-atomic-instrumented.h, using the same table used to generate the fallbacks and atomic-long wrappers. These are checked in rather than generated with Kbuild, since: * This allows inspection of the atomics with git grep and ctags on a pristine tree, which Linus strongly prefers being able to do. * The fallbacks are not affected by machine details or configuration options, so it is not necessary to regenerate them to take these into account. * These are included by files required *very* early in the build process (e.g. for generating bounds.h), and we'd rather not complicate the top-level Kbuild file with dependencies. Generating the atomic headers means that the instrumented wrappers will remain in sync with the rest of the atomic APIs, and we gain all the ordering variants of each atomic without having to manually expanded them all. The KASAN checks are automatically generated based on the function parameters defined in atomics.tbl. Note that try_cmpxchg() now correctly treats 'old' as a parameter that may be written to, and not only read as the hand-written instrumentation assumed. Other than the change to try_cmpxchg(), existing code should not be affected by this patch. The patch introduces instrumentation for all optional atomics (and ordering variants), along with the ifdeffery this requires, enabling other architectures to make use of the instrumented atomics. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: Will Deacon <will.deacon@arm.com> Cc: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com Cc: linuxdrivers@attotech.com Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Boqun Feng <boqun.feng@gmail.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Alexander Potapenko <glider@google.com> Link: http://lkml.kernel.org/r/20180904104830.2975-5-mark.rutland@arm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-09-04 18:48:28 +08:00
#endif /* _ASM_GENERIC_ATOMIC_INSTRUMENTED_H */
locking/atomics: Regenerate the atomics-check SHA1's The include/asm-generic/atomic-instrumented.h checksum got out of sync, so regenerate it. (No change to actual code.) Also make scripts/atomic/gen-atomics.sh executable, to make it easier to use. The auto-generated atomic header signatures are now fine: thule:~/tip> scripts/atomic/check-atomics.sh thule:~/tip> Signed-off-by: Ingo Molnar <mingo@kernel.org> Cc: linux-kernel@vger.kernel.org Cc: Peter Zijlstra <peterz@infradead.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Paul E. McKenney <paulmck@kernel.org> Cc: Will Deacon <will.deacon@arm.com> Signed-off-by: Ingo Molnar <mingo@kernel.org>
2020-11-07 19:54:49 +08:00
// 4bec382e44520f4d8267e42620054db26a659ea3