UbiquitousOS
/
OpenCloudOS-Kernel
mirror of https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git
11
0
Fork 0
Code Issues Projects Releases Wiki Activity
OpenCloudOS-Kernel/arch/powerpc/include/asm/synch.h

57 lines
1.4 KiB
C
Raw Normal View History

[PATCH] powerpc: merge atomic.h, memory.h powerpc: Merge atomic.h and memory.h into powerpc Merged atomic.h into include/powerpc. Moved asm-style HMT_ defines from memory.h into ppc_asm.h, where there were already HMT_defines; moved c-style HMT_ defines to processor.h. Renamed memory.h to synch.h to better reflect its contents. Signed-off-by: Kumar Gala <kumar.gala@freescale.com> Signed-off-by: Becky Bruce <becky.bruce@freescale.com> Signed-off-by: Jon Loeliger <linuxppc@jdl.com> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-09-23 03:20:04 +08:00
#ifndef _ASM_POWERPC_SYNCH_H
#define _ASM_POWERPC_SYNCH_H
[PATCH] powerpc: sanitize header files for user space includes include/asm-ppc/ had #ifdef __KERNEL__ in all header files that are not meant for use by user space, include/asm-powerpc does not have this yet. This patch gets us a lot closer there. There are a few cases where I was not sure, so I left them out. I have verified that no CONFIG_* symbols are used outside of __KERNEL__ any more and that there are no obvious compile errors when including any of the headers in user space libraries. Signed-off-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-12-17 05:43:46 +08:00
#ifdef __KERNEL__
[PATCH] powerpc: merge atomic.h, memory.h powerpc: Merge atomic.h and memory.h into powerpc Merged atomic.h into include/powerpc. Moved asm-style HMT_ defines from memory.h into ppc_asm.h, where there were already HMT_defines; moved c-style HMT_ defines to processor.h. Renamed memory.h to synch.h to better reflect its contents. Signed-off-by: Kumar Gala <kumar.gala@freescale.com> Signed-off-by: Becky Bruce <becky.bruce@freescale.com> Signed-off-by: Jon Loeliger <linuxppc@jdl.com> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-09-23 03:20:04 +08:00
[PATCH] powerpc: use lwsync in atomics, bitops, lock functions eieio is only a store - store ordering. When used to order an unlock operation loads may leak out of the critical region. This is potentially buggy, one example is if a user wants to atomically read a couple of values. We can solve this with an lwsync which orders everything except store - load. I removed the (now unused) EIEIO_ON_SMP macros and the c versions isync_on_smp and eieio_on_smp now we dont use them. I also removed some old comments that were used to identify inline spinlocks in assembly, they dont make sense now our locks are out of line. Another interesting thing was that read_unlock was using an eieio even though the rest of the spinlock code had already been converted to use lwsync. Signed-off-by: Anton Blanchard <anton@samba.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2006-01-13 12:37:17 +08:00
#include <linux/stringify.h>
powerpc: Fixup lwsync at runtime To allow for a single kernel image on e500 v1/v2/mc we need to fixup lwsync at runtime. On e500v1/v2 lwsync causes an illop so we need to patch up the code. We default to 'sync' since that is always safe and if the cpu is capable we will replace 'sync' with 'lwsync'. We introduce CPU_FTR_LWSYNC as a way to determine at runtime if this is needed. This flag could be moved elsewhere since we dont really use it for the normal CPU_FTR purpose. Finally we only store the relative offset in the fixup section to keep it as small as possible rather than using a full fixup_entry. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2008-07-01 23:16:40 +08:00
#include <asm/feature-fixups.h>
[PATCH] powerpc: use lwsync in atomics, bitops, lock functions eieio is only a store - store ordering. When used to order an unlock operation loads may leak out of the critical region. This is potentially buggy, one example is if a user wants to atomically read a couple of values. We can solve this with an lwsync which orders everything except store - load. I removed the (now unused) EIEIO_ON_SMP macros and the c versions isync_on_smp and eieio_on_smp now we dont use them. I also removed some old comments that were used to identify inline spinlocks in assembly, they dont make sense now our locks are out of line. Another interesting thing was that read_unlock was using an eieio even though the rest of the spinlock code had already been converted to use lwsync. Signed-off-by: Anton Blanchard <anton@samba.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2006-01-13 12:37:17 +08:00
powerpc: Optimise smp_wmb Change 2d1b2027626d5151fff8ef7c06ca8e7876a1a510 ("powerpc: Fixup lwsync at runtime") removed __SUBARCH_HAS_LWSYNC, causing smp_wmb to revert back to eieio for all CPUs. This restores the behaviour intorduced in 74f0609526afddd88bef40b651da24f3167b10b2 ("powerpc: Optimise smp_wmb on 64-bit processors"). Signed-off-by: Nick Piggin <npiggin@suse.de> Signed-off-by: Paul Mackerras <paulus@samba.org>
2008-11-12 01:50:48 +08:00
#if defined(__powerpc64__) || defined(CONFIG_PPC_E500MC)
#define __SUBARCH_HAS_LWSYNC
#endif
powerpc: Fixup lwsync at runtime To allow for a single kernel image on e500 v1/v2/mc we need to fixup lwsync at runtime. On e500v1/v2 lwsync causes an illop so we need to patch up the code. We default to 'sync' since that is always safe and if the cpu is capable we will replace 'sync' with 'lwsync'. We introduce CPU_FTR_LWSYNC as a way to determine at runtime if this is needed. This flag could be moved elsewhere since we dont really use it for the normal CPU_FTR purpose. Finally we only store the relative offset in the fixup section to keep it as small as possible rather than using a full fixup_entry. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2008-07-01 23:16:40 +08:00
#ifndef __ASSEMBLY__
extern unsigned int __start___lwsync_fixup, __stop___lwsync_fixup;
extern void do_lwsync_fixups(unsigned long value, void *fixup_start,
void *fixup_end);
static inline void eieio(void)
{
__asm__ __volatile__ ("eieio" : : : "memory");
}
static inline void isync(void)
{
__asm__ __volatile__ ("isync" : : : "memory");
}
#endif /* __ASSEMBLY__ */
[PATCH] powerpc: merge atomic.h, memory.h powerpc: Merge atomic.h and memory.h into powerpc Merged atomic.h into include/powerpc. Moved asm-style HMT_ defines from memory.h into ppc_asm.h, where there were already HMT_defines; moved c-style HMT_ defines to processor.h. Renamed memory.h to synch.h to better reflect its contents. Signed-off-by: Kumar Gala <kumar.gala@freescale.com> Signed-off-by: Becky Bruce <becky.bruce@freescale.com> Signed-off-by: Jon Loeliger <linuxppc@jdl.com> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-09-23 03:20:04 +08:00
powerpc: Fixup lwsync at runtime To allow for a single kernel image on e500 v1/v2/mc we need to fixup lwsync at runtime. On e500v1/v2 lwsync causes an illop so we need to patch up the code. We default to 'sync' since that is always safe and if the cpu is capable we will replace 'sync' with 'lwsync'. We introduce CPU_FTR_LWSYNC as a way to determine at runtime if this is needed. This flag could be moved elsewhere since we dont really use it for the normal CPU_FTR purpose. Finally we only store the relative offset in the fixup section to keep it as small as possible rather than using a full fixup_entry. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2008-07-01 23:16:40 +08:00
#if defined(__powerpc64__)
[PATCH] powerpc: merge atomic.h, memory.h powerpc: Merge atomic.h and memory.h into powerpc Merged atomic.h into include/powerpc. Moved asm-style HMT_ defines from memory.h into ppc_asm.h, where there were already HMT_defines; moved c-style HMT_ defines to processor.h. Renamed memory.h to synch.h to better reflect its contents. Signed-off-by: Kumar Gala <kumar.gala@freescale.com> Signed-off-by: Becky Bruce <becky.bruce@freescale.com> Signed-off-by: Jon Loeliger <linuxppc@jdl.com> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-09-23 03:20:04 +08:00
# define LWSYNC lwsync
powerpc: Fixup lwsync at runtime To allow for a single kernel image on e500 v1/v2/mc we need to fixup lwsync at runtime. On e500v1/v2 lwsync causes an illop so we need to patch up the code. We default to 'sync' since that is always safe and if the cpu is capable we will replace 'sync' with 'lwsync'. We introduce CPU_FTR_LWSYNC as a way to determine at runtime if this is needed. This flag could be moved elsewhere since we dont really use it for the normal CPU_FTR purpose. Finally we only store the relative offset in the fixup section to keep it as small as possible rather than using a full fixup_entry. Signed-off-by: Kumar Gala <galak@kernel.crashing.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
2008-07-01 23:16:40 +08:00
#elif defined(CONFIG_E500)
# define LWSYNC \
START_LWSYNC_SECTION(96); \
sync; \
MAKE_LWSYNC_SECTION_ENTRY(96, __lwsync_fixup);
[PATCH] powerpc: merge atomic.h, memory.h powerpc: Merge atomic.h and memory.h into powerpc Merged atomic.h into include/powerpc. Moved asm-style HMT_ defines from memory.h into ppc_asm.h, where there were already HMT_defines; moved c-style HMT_ defines to processor.h. Renamed memory.h to synch.h to better reflect its contents. Signed-off-by: Kumar Gala <kumar.gala@freescale.com> Signed-off-by: Becky Bruce <becky.bruce@freescale.com> Signed-off-by: Jon Loeliger <linuxppc@jdl.com> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-09-23 03:20:04 +08:00
#else
# define LWSYNC sync
#endif
#ifdef CONFIG_SMP
powerpc: Use lwsync for acquire barrier if CPU supports it Nick Piggin discovered that lwsync barriers around locks were faster than isync on 970. That was a long time ago and I completely dropped the ball in testing his patches across other ppc64 processors. Turns out the idea helps on other chips. Using a microbenchmark that uses a lot of threads to contend on a global pthread mutex (and therefore a global futex), POWER6 improves 8% and POWER7 improves 2%. I checked POWER5 and while I couldn't measure an improvement, there was no regression. This patch uses the lwsync patching code to replace the isyncs with lwsyncs on CPUs that support the instruction. We were marking POWER3 and RS64 as lwsync capable but in reality they treat it as a full sync (ie slow). Remove the CPU_FTR_LWSYNC bit from these CPUs so they continue to use the faster isync method. Signed-off-by: Anton Blanchard <anton@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2010-02-10 09:10:25 +08:00
#define __PPC_ACQUIRE_BARRIER \
START_LWSYNC_SECTION(97); \
isync; \
MAKE_LWSYNC_SECTION_ENTRY(97, __lwsync_fixup);
powerpc: Fix atomic_xxx_return barrier semantics The Documentation/memory-barriers.txt document requires that atomic operations that return a value act as a memory barrier both before and after the actual atomic operation. Our current implementation doesn't guarantee this. More specifically, while a load following the isync can not be issued before stwcx. has completed, that completion doesn't architecturally means that the result of stwcx. is visible to other processors (or any previous stores for that matter) (typically, the other processors L1 caches can still hold the old value). This has caused an actual crash in RCU torture testing on Power 7 This fixes it by changing those atomic ops to use new macros instead of RELEASE/ACQUIRE barriers, called ATOMIC_ENTRY and ATMOIC_EXIT barriers, which are then defined respectively to lwsync and sync. I haven't had a chance to measure the performance impact (or rather what I measured with kernel compiles is in the noise, I yet have to find a more precise benchmark) Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
2011-11-16 01:11:27 +08:00
#define PPC_ACQUIRE_BARRIER "\n" stringify_in_c(__PPC_ACQUIRE_BARRIER)
#define PPC_RELEASE_BARRIER stringify_in_c(LWSYNC) "\n"
powerpc: Make value-returning atomics fully ordered According to memory-barriers.txt: > Any atomic operation that modifies some state in memory and returns > information about the state (old or new) implies an SMP-conditional > general memory barrier (smp_mb()) on each side of the actual > operation ... Which mean these operations should be fully ordered. However on PPC, PPC_ATOMIC_ENTRY_BARRIER is the barrier before the actual operation, which is currently "lwsync" if SMP=y. The leading "lwsync" can not guarantee fully ordered atomics, according to Paul Mckenney: https://lkml.org/lkml/2015/10/14/970 To fix this, we define PPC_ATOMIC_ENTRY_BARRIER as "sync" to guarantee the fully-ordered semantics. This also makes futex atomics fully ordered, which can avoid possible memory ordering problems if userspace code relies on futex system call for fully ordered semantics. Fixes: b97021f85517 ("powerpc: Fix atomic_xxx_return barrier semantics") Cc: stable@vger.kernel.org # 3.2+ Signed-off-by: Boqun Feng <boqun.feng@gmail.com> Reviewed-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2015-11-02 09:30:31 +08:00
#define PPC_ATOMIC_ENTRY_BARRIER "\n" stringify_in_c(sync) "\n"
powerpc: Fix atomic_xxx_return barrier semantics The Documentation/memory-barriers.txt document requires that atomic operations that return a value act as a memory barrier both before and after the actual atomic operation. Our current implementation doesn't guarantee this. More specifically, while a load following the isync can not be issued before stwcx. has completed, that completion doesn't architecturally means that the result of stwcx. is visible to other processors (or any previous stores for that matter) (typically, the other processors L1 caches can still hold the old value). This has caused an actual crash in RCU torture testing on Power 7 This fixes it by changing those atomic ops to use new macros instead of RELEASE/ACQUIRE barriers, called ATOMIC_ENTRY and ATMOIC_EXIT barriers, which are then defined respectively to lwsync and sync. I haven't had a chance to measure the performance impact (or rather what I measured with kernel compiles is in the noise, I yet have to find a more precise benchmark) Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
2011-11-16 01:11:27 +08:00
#define PPC_ATOMIC_EXIT_BARRIER "\n" stringify_in_c(sync) "\n"
[PATCH] powerpc: merge atomic.h, memory.h powerpc: Merge atomic.h and memory.h into powerpc Merged atomic.h into include/powerpc. Moved asm-style HMT_ defines from memory.h into ppc_asm.h, where there were already HMT_defines; moved c-style HMT_ defines to processor.h. Renamed memory.h to synch.h to better reflect its contents. Signed-off-by: Kumar Gala <kumar.gala@freescale.com> Signed-off-by: Becky Bruce <becky.bruce@freescale.com> Signed-off-by: Jon Loeliger <linuxppc@jdl.com> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-09-23 03:20:04 +08:00
#else
powerpc: Rename LWSYNC_ON_SMP to PPC_RELEASE_BARRIER, ISYNC_ON_SMP to PPC_ACQUIRE_BARRIER For performance reasons we are about to change ISYNC_ON_SMP to sometimes be lwsync. Now that the macro name doesn't make sense, change it and LWSYNC_ON_SMP to better explain what the barriers are doing. Signed-off-by: Anton Blanchard <anton@samba.org> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
2010-02-10 09:04:06 +08:00
#define PPC_ACQUIRE_BARRIER
#define PPC_RELEASE_BARRIER
powerpc: Fix atomic_xxx_return barrier semantics The Documentation/memory-barriers.txt document requires that atomic operations that return a value act as a memory barrier both before and after the actual atomic operation. Our current implementation doesn't guarantee this. More specifically, while a load following the isync can not be issued before stwcx. has completed, that completion doesn't architecturally means that the result of stwcx. is visible to other processors (or any previous stores for that matter) (typically, the other processors L1 caches can still hold the old value). This has caused an actual crash in RCU torture testing on Power 7 This fixes it by changing those atomic ops to use new macros instead of RELEASE/ACQUIRE barriers, called ATOMIC_ENTRY and ATMOIC_EXIT barriers, which are then defined respectively to lwsync and sync. I haven't had a chance to measure the performance impact (or rather what I measured with kernel compiles is in the noise, I yet have to find a more precise benchmark) Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
2011-11-16 01:11:27 +08:00
#define PPC_ATOMIC_ENTRY_BARRIER
#define PPC_ATOMIC_EXIT_BARRIER
[PATCH] powerpc: merge atomic.h, memory.h powerpc: Merge atomic.h and memory.h into powerpc Merged atomic.h into include/powerpc. Moved asm-style HMT_ defines from memory.h into ppc_asm.h, where there were already HMT_defines; moved c-style HMT_ defines to processor.h. Renamed memory.h to synch.h to better reflect its contents. Signed-off-by: Kumar Gala <kumar.gala@freescale.com> Signed-off-by: Becky Bruce <becky.bruce@freescale.com> Signed-off-by: Jon Loeliger <linuxppc@jdl.com> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-09-23 03:20:04 +08:00
#endif
[PATCH] powerpc: sanitize header files for user space includes include/asm-ppc/ had #ifdef __KERNEL__ in all header files that are not meant for use by user space, include/asm-powerpc does not have this yet. This patch gets us a lot closer there. There are a few cases where I was not sure, so I left them out. I have verified that no CONFIG_* symbols are used outside of __KERNEL__ any more and that there are no obvious compile errors when including any of the headers in user space libraries. Signed-off-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-12-17 05:43:46 +08:00
#endif /* __KERNEL__ */
[PATCH] powerpc: merge atomic.h, memory.h powerpc: Merge atomic.h and memory.h into powerpc Merged atomic.h into include/powerpc. Moved asm-style HMT_ defines from memory.h into ppc_asm.h, where there were already HMT_defines; moved c-style HMT_ defines to processor.h. Renamed memory.h to synch.h to better reflect its contents. Signed-off-by: Kumar Gala <kumar.gala@freescale.com> Signed-off-by: Becky Bruce <becky.bruce@freescale.com> Signed-off-by: Jon Loeliger <linuxppc@jdl.com> Signed-off-by: Paul Mackerras <paulus@samba.org>
2005-09-23 03:20:04 +08:00
#endif /* _ASM_POWERPC_SYNCH_H */