2011-01-24 15:42:41 +08:00
|
|
|
/*
|
2016-07-08 14:20:49 +08:00
|
|
|
* This file contains idle entry/exit functions for POWER7,
|
|
|
|
|
* POWER8 and POWER9 CPUs.
|
2011-01-24 15:42:41 +08:00
|
|
|
*
|
|
|
|
|
* This program is free software; you can redistribute it and/or
|
|
|
|
|
* modify it under the terms of the GNU General Public License
|
|
|
|
|
* as published by the Free Software Foundation; either version
|
|
|
|
|
* 2 of the License, or (at your option) any later version.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#include <linux/threads.h>
|
|
|
|
|
#include <asm/processor.h>
|
|
|
|
|
#include <asm/page.h>
|
|
|
|
|
#include <asm/cputable.h>
|
|
|
|
|
#include <asm/thread_info.h>
|
|
|
|
|
#include <asm/ppc_asm.h>
|
|
|
|
|
#include <asm/asm-offsets.h>
|
|
|
|
|
#include <asm/ppc-opcode.h>
|
powerpc: Rework lazy-interrupt handling
The current implementation of lazy interrupts handling has some
issues that this tries to address.
We don't do the various workarounds we need to do when re-enabling
interrupts in some cases such as when returning from an interrupt
and thus we may still lose or get delayed decrementer or doorbell
interrupts.
The current scheme also makes it much harder to handle the external
"edge" interrupts provided by some BookE processors when using the
EPR facility (External Proxy) and the Freescale Hypervisor.
Additionally, we tend to keep interrupts hard disabled in a number
of cases, such as decrementer interrupts, external interrupts, or
when a masked decrementer interrupt is pending. This is sub-optimal.
This is an attempt at fixing it all in one go by reworking the way
we do the lazy interrupt disabling from the ground up.
The base idea is to replace the "hard_enabled" field with a
"irq_happened" field in which we store a bit mask of what interrupt
occurred while soft-disabled.
When re-enabling, either via arch_local_irq_restore() or when returning
from an interrupt, we can now decide what to do by testing bits in that
field.
We then implement replaying of the missed interrupts either by
re-using the existing exception frame (in exception exit case) or via
the creation of a new one from an assembly trampoline (in the
arch_local_irq_enable case).
This removes the need to play with the decrementer to try to create
fake interrupts, among others.
In addition, this adds a few refinements:
- We no longer hard disable decrementer interrupts that occur
while soft-disabled. We now simply bump the decrementer back to max
(on BookS) or leave it stopped (on BookE) and continue with hard interrupts
enabled, which means that we'll potentially get better sample quality from
performance monitor interrupts.
- Timer, decrementer and doorbell interrupts now hard-enable
shortly after removing the source of the interrupt, which means
they no longer run entirely hard disabled. Again, this will improve
perf sample quality.
- On Book3E 64-bit, we now make the performance monitor interrupt
act as an NMI like Book3S (the necessary C code for that to work
appear to already be present in the FSL perf code, notably calling
nmi_enter instead of irq_enter). (This also fixes a bug where BookE
perfmon interrupts could clobber r14 ... oops)
- We could make "masked" decrementer interrupts act as NMIs when doing
timer-based perf sampling to improve the sample quality.
Signed-off-by-yet: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
v2:
- Add hard-enable to decrementer, timer and doorbells
- Fix CR clobber in masked irq handling on BookE
- Make embedded perf interrupt act as an NMI
- Add a PACA_HAPPENED_EE_EDGE for use by FSL if they want
to retrigger an interrupt without preventing hard-enable
v3:
- Fix or vs. ori bug on Book3E
- Fix enabling of interrupts for some exceptions on Book3E
v4:
- Fix resend of doorbells on return from interrupt on Book3E
v5:
- Rebased on top of my latest series, which involves some significant
rework of some aspects of the patch.
v6:
- 32-bit compile fix
- more compile fixes with various .config combos
- factor out the asm code to soft-disable interrupts
- remove the C wrapper around preempt_schedule_irq
v7:
- Fix a bug with hard irq state tracking on native power7
2012-03-06 15:27:59 +08:00
|
|
|
#include <asm/hw_irq.h>
|
2012-02-03 08:54:17 +08:00
|
|
|
#include <asm/kvm_book3s_asm.h>
|
2014-02-26 08:08:43 +08:00
|
|
|
#include <asm/opal.h>
|
2014-12-10 02:56:52 +08:00
|
|
|
#include <asm/cpuidle.h>
|
2016-03-01 15:29:20 +08:00
|
|
|
#include <asm/book3s/64/mmu-hash.h>
|
2016-07-08 14:20:49 +08:00
|
|
|
#include <asm/mmu.h>
|
2011-01-24 15:42:41 +08:00
|
|
|
|
|
|
|
|
#undef DEBUG
|
|
|
|
|
|
2014-12-10 02:56:53 +08:00
|
|
|
/*
|
|
|
|
|
* Use unused space in the interrupt stack to save and restore
|
|
|
|
|
* registers for winkle support.
|
|
|
|
|
*/
|
|
|
|
|
#define _SDR1 GPR3
|
|
|
|
|
#define _RPR GPR4
|
|
|
|
|
#define _SPURR GPR5
|
|
|
|
|
#define _PURR GPR6
|
|
|
|
|
#define _TSCR GPR7
|
|
|
|
|
#define _DSCR GPR8
|
|
|
|
|
#define _AMOR GPR9
|
|
|
|
|
#define _WORT GPR10
|
|
|
|
|
#define _WORC GPR11
|
2016-07-08 14:20:49 +08:00
|
|
|
#define _PTCR GPR12
|
|
|
|
|
|
|
|
|
|
#define PSSCR_HV_TEMPLATE PSSCR_ESL | PSSCR_EC | \
|
|
|
|
|
PSSCR_PSLL_MASK | PSSCR_TR_MASK | \
|
|
|
|
|
PSSCR_MTL_MASK
|
2014-12-10 02:56:53 +08:00
|
|
|
|
2014-02-26 08:08:25 +08:00
|
|
|
.text
|
|
|
|
|
|
2016-07-08 14:20:48 +08:00
|
|
|
/*
|
|
|
|
|
* Used by threads before entering deep idle states. Saves SPRs
|
|
|
|
|
* in interrupt stack frame
|
|
|
|
|
*/
|
|
|
|
|
save_sprs_to_stack:
|
|
|
|
|
/*
|
|
|
|
|
* Note all register i.e per-core, per-subcore or per-thread is saved
|
|
|
|
|
* here since any thread in the core might wake up first
|
|
|
|
|
*/
|
2016-07-08 14:20:49 +08:00
|
|
|
BEGIN_FTR_SECTION
|
|
|
|
|
mfspr r3,SPRN_PTCR
|
|
|
|
|
std r3,_PTCR(r1)
|
|
|
|
|
/*
|
|
|
|
|
* Note - SDR1 is dropped in Power ISA v3. Hence not restoring
|
|
|
|
|
* SDR1 here
|
|
|
|
|
*/
|
|
|
|
|
FTR_SECTION_ELSE
|
2016-07-08 14:20:48 +08:00
|
|
|
mfspr r3,SPRN_SDR1
|
|
|
|
|
std r3,_SDR1(r1)
|
2016-07-08 14:20:49 +08:00
|
|
|
ALT_FTR_SECTION_END_IFSET(CPU_FTR_ARCH_300)
|
2016-07-08 14:20:48 +08:00
|
|
|
mfspr r3,SPRN_RPR
|
|
|
|
|
std r3,_RPR(r1)
|
|
|
|
|
mfspr r3,SPRN_SPURR
|
|
|
|
|
std r3,_SPURR(r1)
|
|
|
|
|
mfspr r3,SPRN_PURR
|
|
|
|
|
std r3,_PURR(r1)
|
|
|
|
|
mfspr r3,SPRN_TSCR
|
|
|
|
|
std r3,_TSCR(r1)
|
|
|
|
|
mfspr r3,SPRN_DSCR
|
|
|
|
|
std r3,_DSCR(r1)
|
|
|
|
|
mfspr r3,SPRN_AMOR
|
|
|
|
|
std r3,_AMOR(r1)
|
|
|
|
|
mfspr r3,SPRN_WORT
|
|
|
|
|
std r3,_WORT(r1)
|
|
|
|
|
mfspr r3,SPRN_WORC
|
|
|
|
|
std r3,_WORC(r1)
|
|
|
|
|
|
|
|
|
|
blr
|
|
|
|
|
|
powerpc/powernv: Fix race in updating core_idle_state
core_idle_state is maintained for each core. It uses 0-7 bits to track
whether a thread in the core has entered fastsleep or winkle. 8th bit is
used as a lock bit.
The lock bit is set in these 2 scenarios-
- The thread is first in subcore to wakeup from sleep/winkle.
- If its the last thread in the core about to enter sleep/winkle
While the lock bit is set, if any other thread in the core wakes up, it
loops until the lock bit is cleared before proceeding in the wakeup
path. This helps prevent race conditions w.r.t fastsleep workaround and
prevents threads from switching to process context before core/subcore
resources are restored.
But, in the path to sleep/winkle entry, we currently don't check for
lock-bit. This exposes us to following race when running with subcore
on-
First thread in the subcorea Another thread in the same
waking up core entering sleep/winkle
lwarx r15,0,r14
ori r15,r15,PNV_CORE_IDLE_LOCK_BIT
stwcx. r15,0,r14
[Code to restore subcore state]
lwarx r15,0,r14
[clear thread bit]
stwcx. r15,0,r14
andi. r15,r15,PNV_CORE_IDLE_THREAD_BITS
stw r15,0(r14)
Here, after the thread entering sleep clears its thread bit in
core_idle_state, the value is overwritten by the thread waking up.
In such cases when the core enters fastsleep, code mistakes an idle
thread as running. Because of this, the first thread waking up from
fastsleep which is supposed to resync timebase skips it. So we can
end up having a core with stale timebase value.
This patch fixes the above race by looping on the lock bit even while
entering the idle states.
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
Fixes: 7b54e9f213f76 'powernv/powerpc: Add winkle support for offline cpus'
Cc: stable@vger.kernel.org # 3.19+
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2015-07-07 04:09:23 +08:00
|
|
|
/*
|
|
|
|
|
* Used by threads when the lock bit of core_idle_state is set.
|
|
|
|
|
* Threads will spin in HMT_LOW until the lock bit is cleared.
|
|
|
|
|
* r14 - pointer to core_idle_state
|
|
|
|
|
* r15 - used to load contents of core_idle_state
|
2016-10-21 17:04:17 +08:00
|
|
|
* r9 - used as a temporary variable
|
powerpc/powernv: Fix race in updating core_idle_state
core_idle_state is maintained for each core. It uses 0-7 bits to track
whether a thread in the core has entered fastsleep or winkle. 8th bit is
used as a lock bit.
The lock bit is set in these 2 scenarios-
- The thread is first in subcore to wakeup from sleep/winkle.
- If its the last thread in the core about to enter sleep/winkle
While the lock bit is set, if any other thread in the core wakes up, it
loops until the lock bit is cleared before proceeding in the wakeup
path. This helps prevent race conditions w.r.t fastsleep workaround and
prevents threads from switching to process context before core/subcore
resources are restored.
But, in the path to sleep/winkle entry, we currently don't check for
lock-bit. This exposes us to following race when running with subcore
on-
First thread in the subcorea Another thread in the same
waking up core entering sleep/winkle
lwarx r15,0,r14
ori r15,r15,PNV_CORE_IDLE_LOCK_BIT
stwcx. r15,0,r14
[Code to restore subcore state]
lwarx r15,0,r14
[clear thread bit]
stwcx. r15,0,r14
andi. r15,r15,PNV_CORE_IDLE_THREAD_BITS
stw r15,0(r14)
Here, after the thread entering sleep clears its thread bit in
core_idle_state, the value is overwritten by the thread waking up.
In such cases when the core enters fastsleep, code mistakes an idle
thread as running. Because of this, the first thread waking up from
fastsleep which is supposed to resync timebase skips it. So we can
end up having a core with stale timebase value.
This patch fixes the above race by looping on the lock bit even while
entering the idle states.
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
Fixes: 7b54e9f213f76 'powernv/powerpc: Add winkle support for offline cpus'
Cc: stable@vger.kernel.org # 3.19+
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2015-07-07 04:09:23 +08:00
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
core_idle_lock_held:
|
|
|
|
|
HMT_LOW
|
|
|
|
|
3: lwz r15,0(r14)
|
|
|
|
|
andi. r15,r15,PNV_CORE_IDLE_LOCK_BIT
|
|
|
|
|
bne 3b
|
|
|
|
|
HMT_MEDIUM
|
|
|
|
|
lwarx r15,0,r14
|
2016-10-21 17:04:17 +08:00
|
|
|
andi. r9,r15,PNV_CORE_IDLE_LOCK_BIT
|
|
|
|
|
bne core_idle_lock_held
|
powerpc/powernv: Fix race in updating core_idle_state
core_idle_state is maintained for each core. It uses 0-7 bits to track
whether a thread in the core has entered fastsleep or winkle. 8th bit is
used as a lock bit.
The lock bit is set in these 2 scenarios-
- The thread is first in subcore to wakeup from sleep/winkle.
- If its the last thread in the core about to enter sleep/winkle
While the lock bit is set, if any other thread in the core wakes up, it
loops until the lock bit is cleared before proceeding in the wakeup
path. This helps prevent race conditions w.r.t fastsleep workaround and
prevents threads from switching to process context before core/subcore
resources are restored.
But, in the path to sleep/winkle entry, we currently don't check for
lock-bit. This exposes us to following race when running with subcore
on-
First thread in the subcorea Another thread in the same
waking up core entering sleep/winkle
lwarx r15,0,r14
ori r15,r15,PNV_CORE_IDLE_LOCK_BIT
stwcx. r15,0,r14
[Code to restore subcore state]
lwarx r15,0,r14
[clear thread bit]
stwcx. r15,0,r14
andi. r15,r15,PNV_CORE_IDLE_THREAD_BITS
stw r15,0(r14)
Here, after the thread entering sleep clears its thread bit in
core_idle_state, the value is overwritten by the thread waking up.
In such cases when the core enters fastsleep, code mistakes an idle
thread as running. Because of this, the first thread waking up from
fastsleep which is supposed to resync timebase skips it. So we can
end up having a core with stale timebase value.
This patch fixes the above race by looping on the lock bit even while
entering the idle states.
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
Fixes: 7b54e9f213f76 'powernv/powerpc: Add winkle support for offline cpus'
Cc: stable@vger.kernel.org # 3.19+
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2015-07-07 04:09:23 +08:00
|
|
|
blr
|
|
|
|
|
|
2014-02-26 08:08:25 +08:00
|
|
|
/*
|
|
|
|
|
* Pass requested state in r3:
|
2016-07-08 14:20:49 +08:00
|
|
|
* r3 - PNV_THREAD_NAP/SLEEP/WINKLE in POWER8
|
|
|
|
|
* - Requested STOP state in POWER9
|
2014-05-23 16:15:26 +08:00
|
|
|
*
|
|
|
|
|
* To check IRQ_HAPPENED in r4
|
|
|
|
|
* 0 - don't check
|
|
|
|
|
* 1 - check
|
2016-07-08 14:20:47 +08:00
|
|
|
*
|
|
|
|
|
* Address to 'rfid' to in r5
|
2014-02-26 08:08:25 +08:00
|
|
|
*/
|
2016-07-08 14:20:46 +08:00
|
|
|
_GLOBAL(pnv_powersave_common)
|
2014-02-26 08:08:25 +08:00
|
|
|
/* Use r3 to pass state nap/sleep/winkle */
|
2011-01-24 15:42:41 +08:00
|
|
|
/* NAP is a state loss, we create a regs frame on the
|
|
|
|
|
* stack, fill it up with the state we care about and
|
|
|
|
|
* stick a pointer to it in PACAR1. We really only
|
|
|
|
|
* need to save PC, some CR bits and the NV GPRs,
|
|
|
|
|
* but for now an interrupt frame will do.
|
|
|
|
|
*/
|
|
|
|
|
mflr r0
|
|
|
|
|
std r0,16(r1)
|
|
|
|
|
stdu r1,-INT_FRAME_SIZE(r1)
|
|
|
|
|
std r0,_LINK(r1)
|
|
|
|
|
std r0,_NIP(r1)
|
|
|
|
|
|
|
|
|
|
/* Hard disable interrupts */
|
|
|
|
|
mfmsr r9
|
|
|
|
|
rldicl r9,r9,48,1
|
|
|
|
|
rotldi r9,r9,16
|
|
|
|
|
mtmsrd r9,1 /* hard-disable interrupts */
|
powerpc: Rework lazy-interrupt handling
The current implementation of lazy interrupts handling has some
issues that this tries to address.
We don't do the various workarounds we need to do when re-enabling
interrupts in some cases such as when returning from an interrupt
and thus we may still lose or get delayed decrementer or doorbell
interrupts.
The current scheme also makes it much harder to handle the external
"edge" interrupts provided by some BookE processors when using the
EPR facility (External Proxy) and the Freescale Hypervisor.
Additionally, we tend to keep interrupts hard disabled in a number
of cases, such as decrementer interrupts, external interrupts, or
when a masked decrementer interrupt is pending. This is sub-optimal.
This is an attempt at fixing it all in one go by reworking the way
we do the lazy interrupt disabling from the ground up.
The base idea is to replace the "hard_enabled" field with a
"irq_happened" field in which we store a bit mask of what interrupt
occurred while soft-disabled.
When re-enabling, either via arch_local_irq_restore() or when returning
from an interrupt, we can now decide what to do by testing bits in that
field.
We then implement replaying of the missed interrupts either by
re-using the existing exception frame (in exception exit case) or via
the creation of a new one from an assembly trampoline (in the
arch_local_irq_enable case).
This removes the need to play with the decrementer to try to create
fake interrupts, among others.
In addition, this adds a few refinements:
- We no longer hard disable decrementer interrupts that occur
while soft-disabled. We now simply bump the decrementer back to max
(on BookS) or leave it stopped (on BookE) and continue with hard interrupts
enabled, which means that we'll potentially get better sample quality from
performance monitor interrupts.
- Timer, decrementer and doorbell interrupts now hard-enable
shortly after removing the source of the interrupt, which means
they no longer run entirely hard disabled. Again, this will improve
perf sample quality.
- On Book3E 64-bit, we now make the performance monitor interrupt
act as an NMI like Book3S (the necessary C code for that to work
appear to already be present in the FSL perf code, notably calling
nmi_enter instead of irq_enter). (This also fixes a bug where BookE
perfmon interrupts could clobber r14 ... oops)
- We could make "masked" decrementer interrupts act as NMIs when doing
timer-based perf sampling to improve the sample quality.
Signed-off-by-yet: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
v2:
- Add hard-enable to decrementer, timer and doorbells
- Fix CR clobber in masked irq handling on BookE
- Make embedded perf interrupt act as an NMI
- Add a PACA_HAPPENED_EE_EDGE for use by FSL if they want
to retrigger an interrupt without preventing hard-enable
v3:
- Fix or vs. ori bug on Book3E
- Fix enabling of interrupts for some exceptions on Book3E
v4:
- Fix resend of doorbells on return from interrupt on Book3E
v5:
- Rebased on top of my latest series, which involves some significant
rework of some aspects of the patch.
v6:
- 32-bit compile fix
- more compile fixes with various .config combos
- factor out the asm code to soft-disable interrupts
- remove the C wrapper around preempt_schedule_irq
v7:
- Fix a bug with hard irq state tracking on native power7
2012-03-06 15:27:59 +08:00
|
|
|
|
|
|
|
|
/* Check if something happened while soft-disabled */
|
|
|
|
|
lbz r0,PACAIRQHAPPENED(r13)
|
powerpc/powernv: Don't call generic code on offline cpus
On PowerNV platforms, when a CPU is offline, we put it into nap mode.
It's possible that the CPU wakes up from nap mode while it is still
offline due to a stray IPI. A misdirected device interrupt could also
potentially cause it to wake up. In that circumstance, we need to clear
the interrupt so that the CPU can go back to nap mode.
In the past the clearing of the interrupt was accomplished by briefly
enabling interrupts and allowing the normal interrupt handling code
(do_IRQ() etc.) to handle the interrupt. This has the problem that
this code calls irq_enter() and irq_exit(), which call functions such
as account_system_vtime() which use RCU internally. Use of RCU is not
permitted on offline CPUs and will trigger errors if RCU checking is
enabled.
To avoid calling into any generic code which might use RCU, we adopt
a different method of clearing interrupts on offline CPUs. Since we
are on the PowerNV platform, we know that the system interrupt
controller is a XICS being driven directly (i.e. not via hcalls) by
the kernel. Hence this adds a new icp_native_flush_interrupt()
function to the native-mode XICS driver and arranges to call that
when an offline CPU is woken from nap. This new function reads the
interrupt from the XICS. If it is an IPI, it clears the IPI; if it
is a device interrupt, it prints a warning and disables the source.
Then it does the end-of-interrupt processing for the interrupt.
The other thing that briefly enabling interrupts did was to check and
clear the irq_happened flag in this CPU's PACA. Therefore, after
flushing the interrupt from the XICS, we also clear all bits except
the PACA_IRQ_HARD_DIS (interrupts are hard disabled) bit from the
irq_happened flag. The PACA_IRQ_HARD_DIS flag is set by power7_nap()
and is left set to indicate that interrupts are hard disabled. This
means we then have to ignore that flag in power7_nap(), which is
reasonable since it doesn't indicate that any interrupt event needs
servicing.
Signed-off-by: Paul Mackerras <paulus@samba.org>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2014-09-02 12:23:16 +08:00
|
|
|
andi. r0,r0,~PACA_IRQ_HARD_DIS@l
|
powerpc: Rework lazy-interrupt handling
The current implementation of lazy interrupts handling has some
issues that this tries to address.
We don't do the various workarounds we need to do when re-enabling
interrupts in some cases such as when returning from an interrupt
and thus we may still lose or get delayed decrementer or doorbell
interrupts.
The current scheme also makes it much harder to handle the external
"edge" interrupts provided by some BookE processors when using the
EPR facility (External Proxy) and the Freescale Hypervisor.
Additionally, we tend to keep interrupts hard disabled in a number
of cases, such as decrementer interrupts, external interrupts, or
when a masked decrementer interrupt is pending. This is sub-optimal.
This is an attempt at fixing it all in one go by reworking the way
we do the lazy interrupt disabling from the ground up.
The base idea is to replace the "hard_enabled" field with a
"irq_happened" field in which we store a bit mask of what interrupt
occurred while soft-disabled.
When re-enabling, either via arch_local_irq_restore() or when returning
from an interrupt, we can now decide what to do by testing bits in that
field.
We then implement replaying of the missed interrupts either by
re-using the existing exception frame (in exception exit case) or via
the creation of a new one from an assembly trampoline (in the
arch_local_irq_enable case).
This removes the need to play with the decrementer to try to create
fake interrupts, among others.
In addition, this adds a few refinements:
- We no longer hard disable decrementer interrupts that occur
while soft-disabled. We now simply bump the decrementer back to max
(on BookS) or leave it stopped (on BookE) and continue with hard interrupts
enabled, which means that we'll potentially get better sample quality from
performance monitor interrupts.
- Timer, decrementer and doorbell interrupts now hard-enable
shortly after removing the source of the interrupt, which means
they no longer run entirely hard disabled. Again, this will improve
perf sample quality.
- On Book3E 64-bit, we now make the performance monitor interrupt
act as an NMI like Book3S (the necessary C code for that to work
appear to already be present in the FSL perf code, notably calling
nmi_enter instead of irq_enter). (This also fixes a bug where BookE
perfmon interrupts could clobber r14 ... oops)
- We could make "masked" decrementer interrupts act as NMIs when doing
timer-based perf sampling to improve the sample quality.
Signed-off-by-yet: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
v2:
- Add hard-enable to decrementer, timer and doorbells
- Fix CR clobber in masked irq handling on BookE
- Make embedded perf interrupt act as an NMI
- Add a PACA_HAPPENED_EE_EDGE for use by FSL if they want
to retrigger an interrupt without preventing hard-enable
v3:
- Fix or vs. ori bug on Book3E
- Fix enabling of interrupts for some exceptions on Book3E
v4:
- Fix resend of doorbells on return from interrupt on Book3E
v5:
- Rebased on top of my latest series, which involves some significant
rework of some aspects of the patch.
v6:
- 32-bit compile fix
- more compile fixes with various .config combos
- factor out the asm code to soft-disable interrupts
- remove the C wrapper around preempt_schedule_irq
v7:
- Fix a bug with hard irq state tracking on native power7
2012-03-06 15:27:59 +08:00
|
|
|
beq 1f
|
2014-05-23 16:15:26 +08:00
|
|
|
cmpwi cr0,r4,0
|
|
|
|
|
beq 1f
|
powerpc: Rework lazy-interrupt handling
The current implementation of lazy interrupts handling has some
issues that this tries to address.
We don't do the various workarounds we need to do when re-enabling
interrupts in some cases such as when returning from an interrupt
and thus we may still lose or get delayed decrementer or doorbell
interrupts.
The current scheme also makes it much harder to handle the external
"edge" interrupts provided by some BookE processors when using the
EPR facility (External Proxy) and the Freescale Hypervisor.
Additionally, we tend to keep interrupts hard disabled in a number
of cases, such as decrementer interrupts, external interrupts, or
when a masked decrementer interrupt is pending. This is sub-optimal.
This is an attempt at fixing it all in one go by reworking the way
we do the lazy interrupt disabling from the ground up.
The base idea is to replace the "hard_enabled" field with a
"irq_happened" field in which we store a bit mask of what interrupt
occurred while soft-disabled.
When re-enabling, either via arch_local_irq_restore() or when returning
from an interrupt, we can now decide what to do by testing bits in that
field.
We then implement replaying of the missed interrupts either by
re-using the existing exception frame (in exception exit case) or via
the creation of a new one from an assembly trampoline (in the
arch_local_irq_enable case).
This removes the need to play with the decrementer to try to create
fake interrupts, among others.
In addition, this adds a few refinements:
- We no longer hard disable decrementer interrupts that occur
while soft-disabled. We now simply bump the decrementer back to max
(on BookS) or leave it stopped (on BookE) and continue with hard interrupts
enabled, which means that we'll potentially get better sample quality from
performance monitor interrupts.
- Timer, decrementer and doorbell interrupts now hard-enable
shortly after removing the source of the interrupt, which means
they no longer run entirely hard disabled. Again, this will improve
perf sample quality.
- On Book3E 64-bit, we now make the performance monitor interrupt
act as an NMI like Book3S (the necessary C code for that to work
appear to already be present in the FSL perf code, notably calling
nmi_enter instead of irq_enter). (This also fixes a bug where BookE
perfmon interrupts could clobber r14 ... oops)
- We could make "masked" decrementer interrupts act as NMIs when doing
timer-based perf sampling to improve the sample quality.
Signed-off-by-yet: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
v2:
- Add hard-enable to decrementer, timer and doorbells
- Fix CR clobber in masked irq handling on BookE
- Make embedded perf interrupt act as an NMI
- Add a PACA_HAPPENED_EE_EDGE for use by FSL if they want
to retrigger an interrupt without preventing hard-enable
v3:
- Fix or vs. ori bug on Book3E
- Fix enabling of interrupts for some exceptions on Book3E
v4:
- Fix resend of doorbells on return from interrupt on Book3E
v5:
- Rebased on top of my latest series, which involves some significant
rework of some aspects of the patch.
v6:
- 32-bit compile fix
- more compile fixes with various .config combos
- factor out the asm code to soft-disable interrupts
- remove the C wrapper around preempt_schedule_irq
v7:
- Fix a bug with hard irq state tracking on native power7
2012-03-06 15:27:59 +08:00
|
|
|
addi r1,r1,INT_FRAME_SIZE
|
|
|
|
|
ld r0,16(r1)
|
2015-03-20 07:10:18 +08:00
|
|
|
li r3,0 /* Return 0 (no nap) */
|
powerpc: Rework lazy-interrupt handling
The current implementation of lazy interrupts handling has some
issues that this tries to address.
We don't do the various workarounds we need to do when re-enabling
interrupts in some cases such as when returning from an interrupt
and thus we may still lose or get delayed decrementer or doorbell
interrupts.
The current scheme also makes it much harder to handle the external
"edge" interrupts provided by some BookE processors when using the
EPR facility (External Proxy) and the Freescale Hypervisor.
Additionally, we tend to keep interrupts hard disabled in a number
of cases, such as decrementer interrupts, external interrupts, or
when a masked decrementer interrupt is pending. This is sub-optimal.
This is an attempt at fixing it all in one go by reworking the way
we do the lazy interrupt disabling from the ground up.
The base idea is to replace the "hard_enabled" field with a
"irq_happened" field in which we store a bit mask of what interrupt
occurred while soft-disabled.
When re-enabling, either via arch_local_irq_restore() or when returning
from an interrupt, we can now decide what to do by testing bits in that
field.
We then implement replaying of the missed interrupts either by
re-using the existing exception frame (in exception exit case) or via
the creation of a new one from an assembly trampoline (in the
arch_local_irq_enable case).
This removes the need to play with the decrementer to try to create
fake interrupts, among others.
In addition, this adds a few refinements:
- We no longer hard disable decrementer interrupts that occur
while soft-disabled. We now simply bump the decrementer back to max
(on BookS) or leave it stopped (on BookE) and continue with hard interrupts
enabled, which means that we'll potentially get better sample quality from
performance monitor interrupts.
- Timer, decrementer and doorbell interrupts now hard-enable
shortly after removing the source of the interrupt, which means
they no longer run entirely hard disabled. Again, this will improve
perf sample quality.
- On Book3E 64-bit, we now make the performance monitor interrupt
act as an NMI like Book3S (the necessary C code for that to work
appear to already be present in the FSL perf code, notably calling
nmi_enter instead of irq_enter). (This also fixes a bug where BookE
perfmon interrupts could clobber r14 ... oops)
- We could make "masked" decrementer interrupts act as NMIs when doing
timer-based perf sampling to improve the sample quality.
Signed-off-by-yet: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
v2:
- Add hard-enable to decrementer, timer and doorbells
- Fix CR clobber in masked irq handling on BookE
- Make embedded perf interrupt act as an NMI
- Add a PACA_HAPPENED_EE_EDGE for use by FSL if they want
to retrigger an interrupt without preventing hard-enable
v3:
- Fix or vs. ori bug on Book3E
- Fix enabling of interrupts for some exceptions on Book3E
v4:
- Fix resend of doorbells on return from interrupt on Book3E
v5:
- Rebased on top of my latest series, which involves some significant
rework of some aspects of the patch.
v6:
- 32-bit compile fix
- more compile fixes with various .config combos
- factor out the asm code to soft-disable interrupts
- remove the C wrapper around preempt_schedule_irq
v7:
- Fix a bug with hard irq state tracking on native power7
2012-03-06 15:27:59 +08:00
|
|
|
mtlr r0
|
|
|
|
|
blr
|
|
|
|
|
|
|
|
|
|
1: /* We mark irqs hard disabled as this is the state we'll
|
|
|
|
|
* be in when returning and we need to tell arch_local_irq_restore()
|
|
|
|
|
* about it
|
|
|
|
|
*/
|
|
|
|
|
li r0,PACA_IRQ_HARD_DIS
|
|
|
|
|
stb r0,PACAIRQHAPPENED(r13)
|
|
|
|
|
|
|
|
|
|
/* We haven't lost state ... yet */
|
2011-01-24 15:42:41 +08:00
|
|
|
li r0,0
|
2011-12-06 03:47:26 +08:00
|
|
|
stb r0,PACA_NAPSTATELOST(r13)
|
2011-01-24 15:42:41 +08:00
|
|
|
|
|
|
|
|
/* Continue saving state */
|
|
|
|
|
SAVE_GPR(2, r1)
|
|
|
|
|
SAVE_NVGPRS(r1)
|
2014-02-26 08:08:25 +08:00
|
|
|
mfcr r4
|
|
|
|
|
std r4,_CCR(r1)
|
2011-01-24 15:42:41 +08:00
|
|
|
std r9,_MSR(r1)
|
|
|
|
|
std r1,PACAR1(r13)
|
|
|
|
|
|
powerpc/powernv: Switch off MMU before entering nap/sleep/rvwinkle mode
Currently, when going idle, we set the flag indicating that we are in
nap mode (paca->kvm_hstate.hwthread_state) and then execute the nap
(or sleep or rvwinkle) instruction, all with the MMU on. This is bad
for two reasons: (a) the architecture specifies that those instructions
must be executed with the MMU off, and in fact with only the SF, HV, ME
and possibly RI bits set, and (b) this introduces a race, because as
soon as we set the flag, another thread can switch the MMU to a guest
context. If the race is lost, this thread will typically start looping
on relocation-on ISIs at 0xc...4400.
This fixes it by setting the MSR as required by the architecture before
setting the flag or executing the nap/sleep/rvwinkle instruction.
Cc: stable@vger.kernel.org
[ shreyas@linux.vnet.ibm.com: Edited to handle LE ]
Signed-off-by: Paul Mackerras <paulus@samba.org>
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: linuxppc-dev@lists.ozlabs.org
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2014-12-10 02:56:50 +08:00
|
|
|
/*
|
|
|
|
|
* Go to real mode to do the nap, as required by the architecture.
|
|
|
|
|
* Also, we need to be in real mode before setting hwthread_state,
|
|
|
|
|
* because as soon as we do that, another thread can switch
|
|
|
|
|
* the MMU context to the guest.
|
|
|
|
|
*/
|
2016-07-08 14:20:47 +08:00
|
|
|
LOAD_REG_IMMEDIATE(r7, MSR_IDLE)
|
powerpc/powernv: Switch off MMU before entering nap/sleep/rvwinkle mode
Currently, when going idle, we set the flag indicating that we are in
nap mode (paca->kvm_hstate.hwthread_state) and then execute the nap
(or sleep or rvwinkle) instruction, all with the MMU on. This is bad
for two reasons: (a) the architecture specifies that those instructions
must be executed with the MMU off, and in fact with only the SF, HV, ME
and possibly RI bits set, and (b) this introduces a race, because as
soon as we set the flag, another thread can switch the MMU to a guest
context. If the race is lost, this thread will typically start looping
on relocation-on ISIs at 0xc...4400.
This fixes it by setting the MSR as required by the architecture before
setting the flag or executing the nap/sleep/rvwinkle instruction.
Cc: stable@vger.kernel.org
[ shreyas@linux.vnet.ibm.com: Edited to handle LE ]
Signed-off-by: Paul Mackerras <paulus@samba.org>
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: linuxppc-dev@lists.ozlabs.org
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2014-12-10 02:56:50 +08:00
|
|
|
li r6, MSR_RI
|
|
|
|
|
andc r6, r9, r6
|
|
|
|
|
mtmsrd r6, 1 /* clear RI before setting SRR0/1 */
|
2016-07-08 14:20:47 +08:00
|
|
|
mtspr SPRN_SRR0, r5
|
|
|
|
|
mtspr SPRN_SRR1, r7
|
powerpc/powernv: Switch off MMU before entering nap/sleep/rvwinkle mode
Currently, when going idle, we set the flag indicating that we are in
nap mode (paca->kvm_hstate.hwthread_state) and then execute the nap
(or sleep or rvwinkle) instruction, all with the MMU on. This is bad
for two reasons: (a) the architecture specifies that those instructions
must be executed with the MMU off, and in fact with only the SF, HV, ME
and possibly RI bits set, and (b) this introduces a race, because as
soon as we set the flag, another thread can switch the MMU to a guest
context. If the race is lost, this thread will typically start looping
on relocation-on ISIs at 0xc...4400.
This fixes it by setting the MSR as required by the architecture before
setting the flag or executing the nap/sleep/rvwinkle instruction.
Cc: stable@vger.kernel.org
[ shreyas@linux.vnet.ibm.com: Edited to handle LE ]
Signed-off-by: Paul Mackerras <paulus@samba.org>
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: linuxppc-dev@lists.ozlabs.org
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2014-12-10 02:56:50 +08:00
|
|
|
rfid
|
|
|
|
|
|
2016-07-08 14:20:46 +08:00
|
|
|
.globl pnv_enter_arch207_idle_mode
|
|
|
|
|
pnv_enter_arch207_idle_mode:
|
2016-10-21 17:03:05 +08:00
|
|
|
#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
|
|
|
|
|
/* Tell KVM we're entering idle */
|
|
|
|
|
li r4,KVM_HWTHREAD_IN_IDLE
|
|
|
|
|
/******************************************************/
|
|
|
|
|
/* N O T E W E L L ! ! ! N O T E W E L L */
|
|
|
|
|
/* The following store to HSTATE_HWTHREAD_STATE(r13) */
|
|
|
|
|
/* MUST occur in real mode, i.e. with the MMU off, */
|
|
|
|
|
/* and the MMU must stay off until we clear this flag */
|
|
|
|
|
/* and test HSTATE_HWTHREAD_REQ(r13) in the system */
|
|
|
|
|
/* reset interrupt vector in exceptions-64s.S. */
|
|
|
|
|
/* The reason is that another thread can switch the */
|
|
|
|
|
/* MMU to a guest context whenever this flag is set */
|
|
|
|
|
/* to KVM_HWTHREAD_IN_IDLE, and if the MMU was on, */
|
|
|
|
|
/* that would potentially cause this thread to start */
|
|
|
|
|
/* executing instructions from guest memory in */
|
|
|
|
|
/* hypervisor mode, leading to a host crash or data */
|
|
|
|
|
/* corruption, or worse. */
|
|
|
|
|
/******************************************************/
|
|
|
|
|
stb r4,HSTATE_HWTHREAD_STATE(r13)
|
|
|
|
|
#endif
|
2014-12-10 02:56:52 +08:00
|
|
|
stb r3,PACA_THREAD_IDLE_STATE(r13)
|
2014-12-10 02:56:53 +08:00
|
|
|
cmpwi cr3,r3,PNV_THREAD_SLEEP
|
|
|
|
|
bge cr3,2f
|
2017-01-25 16:36:25 +08:00
|
|
|
IDLE_STATE_ENTER_SEQ_NORET(PPC_NAP)
|
2014-02-26 08:08:25 +08:00
|
|
|
/* No return */
|
2014-12-10 02:56:52 +08:00
|
|
|
2:
|
|
|
|
|
/* Sleep or winkle */
|
|
|
|
|
lbz r7,PACA_THREAD_MASK(r13)
|
|
|
|
|
ld r14,PACA_CORE_IDLE_STATE_PTR(r13)
|
|
|
|
|
lwarx_loop1:
|
|
|
|
|
lwarx r15,0,r14
|
powerpc/powernv: Fix race in updating core_idle_state
core_idle_state is maintained for each core. It uses 0-7 bits to track
whether a thread in the core has entered fastsleep or winkle. 8th bit is
used as a lock bit.
The lock bit is set in these 2 scenarios-
- The thread is first in subcore to wakeup from sleep/winkle.
- If its the last thread in the core about to enter sleep/winkle
While the lock bit is set, if any other thread in the core wakes up, it
loops until the lock bit is cleared before proceeding in the wakeup
path. This helps prevent race conditions w.r.t fastsleep workaround and
prevents threads from switching to process context before core/subcore
resources are restored.
But, in the path to sleep/winkle entry, we currently don't check for
lock-bit. This exposes us to following race when running with subcore
on-
First thread in the subcorea Another thread in the same
waking up core entering sleep/winkle
lwarx r15,0,r14
ori r15,r15,PNV_CORE_IDLE_LOCK_BIT
stwcx. r15,0,r14
[Code to restore subcore state]
lwarx r15,0,r14
[clear thread bit]
stwcx. r15,0,r14
andi. r15,r15,PNV_CORE_IDLE_THREAD_BITS
stw r15,0(r14)
Here, after the thread entering sleep clears its thread bit in
core_idle_state, the value is overwritten by the thread waking up.
In such cases when the core enters fastsleep, code mistakes an idle
thread as running. Because of this, the first thread waking up from
fastsleep which is supposed to resync timebase skips it. So we can
end up having a core with stale timebase value.
This patch fixes the above race by looping on the lock bit even while
entering the idle states.
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
Fixes: 7b54e9f213f76 'powernv/powerpc: Add winkle support for offline cpus'
Cc: stable@vger.kernel.org # 3.19+
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2015-07-07 04:09:23 +08:00
|
|
|
|
|
|
|
|
andi. r9,r15,PNV_CORE_IDLE_LOCK_BIT
|
|
|
|
|
bnel core_idle_lock_held
|
|
|
|
|
|
2014-12-10 02:56:52 +08:00
|
|
|
andc r15,r15,r7 /* Clear thread bit */
|
|
|
|
|
|
|
|
|
|
andi. r15,r15,PNV_CORE_IDLE_THREAD_BITS
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If cr0 = 0, then current thread is the last thread of the core entering
|
|
|
|
|
* sleep. Last thread needs to execute the hardware bug workaround code if
|
|
|
|
|
* required by the platform.
|
|
|
|
|
* Make the workaround call unconditionally here. The below branch call is
|
|
|
|
|
* patched out when the idle states are discovered if the platform does not
|
|
|
|
|
* require it.
|
|
|
|
|
*/
|
|
|
|
|
.global pnv_fastsleep_workaround_at_entry
|
|
|
|
|
pnv_fastsleep_workaround_at_entry:
|
|
|
|
|
beq fastsleep_workaround_at_entry
|
|
|
|
|
|
|
|
|
|
stwcx. r15,0,r14
|
|
|
|
|
bne- lwarx_loop1
|
|
|
|
|
isync
|
|
|
|
|
|
2014-12-10 02:56:53 +08:00
|
|
|
common_enter: /* common code for all the threads entering sleep or winkle */
|
|
|
|
|
bgt cr3,enter_winkle
|
2017-01-25 16:36:25 +08:00
|
|
|
IDLE_STATE_ENTER_SEQ_NORET(PPC_SLEEP)
|
2014-12-10 02:56:52 +08:00
|
|
|
|
|
|
|
|
fastsleep_workaround_at_entry:
|
|
|
|
|
ori r15,r15,PNV_CORE_IDLE_LOCK_BIT
|
|
|
|
|
stwcx. r15,0,r14
|
|
|
|
|
bne- lwarx_loop1
|
|
|
|
|
isync
|
|
|
|
|
|
|
|
|
|
/* Fast sleep workaround */
|
|
|
|
|
li r3,1
|
|
|
|
|
li r4,1
|
2016-07-08 14:37:11 +08:00
|
|
|
bl opal_rm_config_cpu_idle_state
|
2014-12-10 02:56:52 +08:00
|
|
|
|
|
|
|
|
/* Clear Lock bit */
|
|
|
|
|
li r0,0
|
|
|
|
|
lwsync
|
|
|
|
|
stw r0,0(r14)
|
|
|
|
|
b common_enter
|
|
|
|
|
|
2014-12-10 02:56:53 +08:00
|
|
|
enter_winkle:
|
2016-07-08 14:20:48 +08:00
|
|
|
bl save_sprs_to_stack
|
|
|
|
|
|
2017-01-25 16:36:25 +08:00
|
|
|
IDLE_STATE_ENTER_SEQ_NORET(PPC_WINKLE)
|
2012-02-03 08:54:17 +08:00
|
|
|
|
2016-07-08 14:20:49 +08:00
|
|
|
/*
|
|
|
|
|
* r3 - requested stop state
|
|
|
|
|
*/
|
|
|
|
|
power_enter_stop:
|
2016-10-21 17:03:05 +08:00
|
|
|
#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
|
|
|
|
|
/* Tell KVM we're entering idle */
|
|
|
|
|
li r4,KVM_HWTHREAD_IN_IDLE
|
|
|
|
|
/* DO THIS IN REAL MODE! See comment above. */
|
|
|
|
|
stb r4,HSTATE_HWTHREAD_STATE(r13)
|
|
|
|
|
#endif
|
2016-07-08 14:20:49 +08:00
|
|
|
/*
|
|
|
|
|
* Check if the requested state is a deep idle state.
|
|
|
|
|
*/
|
|
|
|
|
LOAD_REG_ADDRBASE(r5,pnv_first_deep_stop_state)
|
|
|
|
|
ld r4,ADDROFF(pnv_first_deep_stop_state)(r5)
|
|
|
|
|
cmpd r3,r4
|
|
|
|
|
bge 2f
|
2017-01-25 16:36:25 +08:00
|
|
|
IDLE_STATE_ENTER_SEQ_NORET(PPC_STOP)
|
2016-07-08 14:20:49 +08:00
|
|
|
2:
|
|
|
|
|
/*
|
|
|
|
|
* Entering deep idle state.
|
|
|
|
|
* Clear thread bit in PACA_CORE_IDLE_STATE, save SPRs to
|
|
|
|
|
* stack and enter stop
|
|
|
|
|
*/
|
|
|
|
|
lbz r7,PACA_THREAD_MASK(r13)
|
|
|
|
|
ld r14,PACA_CORE_IDLE_STATE_PTR(r13)
|
|
|
|
|
|
|
|
|
|
lwarx_loop_stop:
|
|
|
|
|
lwarx r15,0,r14
|
|
|
|
|
andi. r9,r15,PNV_CORE_IDLE_LOCK_BIT
|
|
|
|
|
bnel core_idle_lock_held
|
|
|
|
|
andc r15,r15,r7 /* Clear thread bit */
|
|
|
|
|
|
|
|
|
|
stwcx. r15,0,r14
|
|
|
|
|
bne- lwarx_loop_stop
|
|
|
|
|
isync
|
|
|
|
|
|
|
|
|
|
bl save_sprs_to_stack
|
|
|
|
|
|
2017-01-25 16:36:25 +08:00
|
|
|
IDLE_STATE_ENTER_SEQ_NORET(PPC_STOP)
|
2016-07-08 14:20:49 +08:00
|
|
|
|
2014-02-26 08:08:25 +08:00
|
|
|
_GLOBAL(power7_idle)
|
|
|
|
|
/* Now check if user or arch enabled NAP mode */
|
|
|
|
|
LOAD_REG_ADDRBASE(r3,powersave_nap)
|
|
|
|
|
lwz r4,ADDROFF(powersave_nap)(r3)
|
|
|
|
|
cmpwi 0,r4,0
|
|
|
|
|
beqlr
|
2014-05-23 16:15:26 +08:00
|
|
|
li r3, 1
|
2014-02-26 08:08:25 +08:00
|
|
|
/* fall through */
|
|
|
|
|
|
|
|
|
|
_GLOBAL(power7_nap)
|
2014-05-23 16:15:26 +08:00
|
|
|
mr r4,r3
|
2014-12-10 02:56:52 +08:00
|
|
|
li r3,PNV_THREAD_NAP
|
2016-07-08 14:20:47 +08:00
|
|
|
LOAD_REG_ADDR(r5, pnv_enter_arch207_idle_mode)
|
2016-07-08 14:20:46 +08:00
|
|
|
b pnv_powersave_common
|
2014-02-26 08:08:25 +08:00
|
|
|
/* No return */
|
|
|
|
|
|
|
|
|
|
_GLOBAL(power7_sleep)
|
2014-12-10 02:56:52 +08:00
|
|
|
li r3,PNV_THREAD_SLEEP
|
2014-07-02 11:49:35 +08:00
|
|
|
li r4,1
|
2016-07-08 14:20:47 +08:00
|
|
|
LOAD_REG_ADDR(r5, pnv_enter_arch207_idle_mode)
|
2016-07-08 14:20:46 +08:00
|
|
|
b pnv_powersave_common
|
2014-02-26 08:08:25 +08:00
|
|
|
/* No return */
|
2011-01-24 15:42:41 +08:00
|
|
|
|
2014-12-10 02:56:53 +08:00
|
|
|
_GLOBAL(power7_winkle)
|
2016-07-08 14:20:43 +08:00
|
|
|
li r3,PNV_THREAD_WINKLE
|
2014-12-10 02:56:53 +08:00
|
|
|
li r4,1
|
2016-07-08 14:20:47 +08:00
|
|
|
LOAD_REG_ADDR(r5, pnv_enter_arch207_idle_mode)
|
2016-07-08 14:20:46 +08:00
|
|
|
b pnv_powersave_common
|
2014-12-10 02:56:53 +08:00
|
|
|
/* No return */
|
|
|
|
|
|
2014-07-29 21:10:13 +08:00
|
|
|
#define CHECK_HMI_INTERRUPT \
|
|
|
|
|
mfspr r0,SPRN_SRR1; \
|
|
|
|
|
BEGIN_FTR_SECTION_NESTED(66); \
|
|
|
|
|
rlwinm r0,r0,45-31,0xf; /* extract wake reason field (P8) */ \
|
|
|
|
|
FTR_SECTION_ELSE_NESTED(66); \
|
|
|
|
|
rlwinm r0,r0,45-31,0xe; /* P7 wake reason field is 3 bits */ \
|
|
|
|
|
ALT_FTR_SECTION_END_NESTED_IFSET(CPU_FTR_ARCH_207S, 66); \
|
|
|
|
|
cmpwi r0,0xa; /* Hypervisor maintenance ? */ \
|
|
|
|
|
bne 20f; \
|
|
|
|
|
/* Invoke opal call to handle hmi */ \
|
|
|
|
|
ld r2,PACATOC(r13); \
|
|
|
|
|
ld r1,PACAR1(r13); \
|
|
|
|
|
std r3,ORIG_GPR3(r1); /* Save original r3 */ \
|
KVM: PPC: Book3S HV: Fix TB corruption in guest exit path on HMI interrupt
When a guest is assigned to a core it converts the host Timebase (TB)
into guest TB by adding guest timebase offset before entering into
guest. During guest exit it restores the guest TB to host TB. This means
under certain conditions (Guest migration) host TB and guest TB can differ.
When we get an HMI for TB related issues the opal HMI handler would
try fixing errors and restore the correct host TB value. With no guest
running, we don't have any issues. But with guest running on the core
we run into TB corruption issues.
If we get an HMI while in the guest, the current HMI handler invokes opal
hmi handler before forcing guest to exit. The guest exit path subtracts
the guest TB offset from the current TB value which may have already
been restored with host value by opal hmi handler. This leads to incorrect
host and guest TB values.
With split-core, things become more complex. With split-core, TB also gets
split and each subcore gets its own TB register. When a hmi handler fixes
a TB error and restores the TB value, it affects all the TB values of
sibling subcores on the same core. On TB errors all the thread in the core
gets HMI. With existing code, the individual threads call opal hmi handle
independently which can easily throw TB out of sync if we have guest
running on subcores. Hence we will need to co-ordinate with all the
threads before making opal hmi handler call followed by TB resync.
This patch introduces a sibling subcore state structure (shared by all
threads in the core) in paca which holds information about whether sibling
subcores are in Guest mode or host mode. An array in_guest[] of size
MAX_SUBCORE_PER_CORE=4 is used to maintain the state of each subcore.
The subcore id is used as index into in_guest[] array. Only primary
thread entering/exiting the guest is responsible to set/unset its
designated array element.
On TB error, we get HMI interrupt on every thread on the core. Upon HMI,
this patch will now force guest to vacate the core/subcore. Primary
thread from each subcore will then turn off its respective bit
from the above bitmap during the guest exit path just after the
guest->host partition switch is complete.
All other threads that have just exited the guest OR were already in host
will wait until all other subcores clears their respective bit.
Once all the subcores turn off their respective bit, all threads will
will make call to opal hmi handler.
It is not necessary that opal hmi handler would resync the TB value for
every HMI interrupts. It would do so only for the HMI caused due to
TB errors. For rest, it would not touch TB value. Hence to make things
simpler, primary thread would call TB resync explicitly once for each
core immediately after opal hmi handler instead of subtracting guest
offset from TB. TB resync call will restore the TB with host value.
Thus we can be sure about the TB state.
One of the primary threads exiting the guest will take up the
responsibility of calling TB resync. It will use one of the top bits
(bit 63) from subcore state flags bitmap to make the decision. The first
primary thread (among the subcores) that is able to set the bit will
have to call the TB resync. Rest all other threads will wait until TB
resync is complete. Once TB resync is complete all threads will then
proceed.
Signed-off-by: Mahesh Salgaonkar <mahesh@linux.vnet.ibm.com>
Signed-off-by: Paul Mackerras <paulus@ozlabs.org>
2016-05-15 12:14:26 +08:00
|
|
|
li r3,0; /* NULL argument */ \
|
|
|
|
|
bl hmi_exception_realmode; \
|
|
|
|
|
nop; \
|
2014-07-29 21:10:13 +08:00
|
|
|
ld r3,ORIG_GPR3(r1); /* Restore original r3 */ \
|
|
|
|
|
20: nop;
|
|
|
|
|
|
|
|
|
|
|
2016-07-08 14:20:49 +08:00
|
|
|
/*
|
|
|
|
|
* r3 - requested stop state
|
|
|
|
|
*/
|
|
|
|
|
_GLOBAL(power9_idle_stop)
|
|
|
|
|
LOAD_REG_IMMEDIATE(r4, PSSCR_HV_TEMPLATE)
|
|
|
|
|
or r4,r4,r3
|
|
|
|
|
mtspr SPRN_PSSCR, r4
|
|
|
|
|
li r4, 1
|
|
|
|
|
LOAD_REG_ADDR(r5,power_enter_stop)
|
|
|
|
|
b pnv_powersave_common
|
|
|
|
|
/* No return */
|
2016-07-08 14:20:44 +08:00
|
|
|
/*
|
|
|
|
|
* Called from reset vector. Check whether we have woken up with
|
|
|
|
|
* hypervisor state loss. If yes, restore hypervisor state and return
|
|
|
|
|
* back to reset vector.
|
|
|
|
|
*
|
|
|
|
|
* r13 - Contents of HSPRG0
|
|
|
|
|
* cr3 - set to gt if waking up with partial/complete hypervisor state loss
|
|
|
|
|
*/
|
2016-07-08 14:20:46 +08:00
|
|
|
_GLOBAL(pnv_restore_hyp_resource)
|
2016-07-08 14:20:49 +08:00
|
|
|
BEGIN_FTR_SECTION
|
2016-08-05 21:43:12 +08:00
|
|
|
ld r2,PACATOC(r13);
|
2016-07-08 14:20:49 +08:00
|
|
|
/*
|
|
|
|
|
* POWER ISA 3. Use PSSCR to determine if we
|
|
|
|
|
* are waking up from deep idle state
|
|
|
|
|
*/
|
|
|
|
|
LOAD_REG_ADDRBASE(r5,pnv_first_deep_stop_state)
|
|
|
|
|
ld r4,ADDROFF(pnv_first_deep_stop_state)(r5)
|
|
|
|
|
|
|
|
|
|
mfspr r5,SPRN_PSSCR
|
2016-07-08 14:20:44 +08:00
|
|
|
/*
|
2016-07-08 14:20:49 +08:00
|
|
|
* 0-3 bits correspond to Power-Saving Level Status
|
|
|
|
|
* which indicates the idle state we are waking up from
|
|
|
|
|
*/
|
|
|
|
|
rldicl r5,r5,4,60
|
|
|
|
|
cmpd cr4,r5,r4
|
|
|
|
|
bge cr4,pnv_wakeup_tb_loss
|
|
|
|
|
/*
|
|
|
|
|
* Waking up without hypervisor state loss. Return to
|
|
|
|
|
* reset vector
|
|
|
|
|
*/
|
|
|
|
|
blr
|
|
|
|
|
|
|
|
|
|
END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300)
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* POWER ISA 2.07 or less.
|
2016-07-08 14:20:44 +08:00
|
|
|
* Check if last bit of HSPGR0 is set. This indicates whether we are
|
|
|
|
|
* waking up from winkle.
|
|
|
|
|
*/
|
|
|
|
|
clrldi r5,r13,63
|
|
|
|
|
clrrdi r13,r13,1
|
2016-08-05 21:43:12 +08:00
|
|
|
|
|
|
|
|
/* Now that we are sure r13 is corrected, load TOC */
|
|
|
|
|
ld r2,PACATOC(r13);
|
2016-07-08 14:20:44 +08:00
|
|
|
cmpwi cr4,r5,1
|
|
|
|
|
mtspr SPRN_HSPRG0,r13
|
|
|
|
|
|
|
|
|
|
lbz r0,PACA_THREAD_IDLE_STATE(r13)
|
|
|
|
|
cmpwi cr2,r0,PNV_THREAD_NAP
|
2016-07-08 14:20:46 +08:00
|
|
|
bgt cr2,pnv_wakeup_tb_loss /* Either sleep or Winkle */
|
2016-07-08 14:20:44 +08:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* We fall through here if PACA_THREAD_IDLE_STATE shows we are waking
|
|
|
|
|
* up from nap. At this stage CR3 shouldn't contains 'gt' since that
|
|
|
|
|
* indicates we are waking with hypervisor state loss from nap.
|
|
|
|
|
*/
|
|
|
|
|
bgt cr3,.
|
|
|
|
|
|
|
|
|
|
blr /* Return back to System Reset vector from where
|
2016-07-08 14:20:46 +08:00
|
|
|
pnv_restore_hyp_resource was invoked */
|
2016-07-08 14:20:44 +08:00
|
|
|
|
2016-07-08 14:20:49 +08:00
|
|
|
/*
|
|
|
|
|
* Called if waking up from idle state which can cause either partial or
|
|
|
|
|
* complete hyp state loss.
|
|
|
|
|
* In POWER8, called if waking up from fastsleep or winkle
|
|
|
|
|
* In POWER9, called if waking up from stop state >= pnv_first_deep_stop_state
|
|
|
|
|
*
|
|
|
|
|
* r13 - PACA
|
|
|
|
|
* cr3 - gt if waking up with partial/complete hypervisor state loss
|
2016-09-07 13:16:30 +08:00
|
|
|
* cr4 - gt or eq if waking up from complete hypervisor state loss.
|
2016-07-08 14:20:49 +08:00
|
|
|
*/
|
2016-07-08 14:20:46 +08:00
|
|
|
_GLOBAL(pnv_wakeup_tb_loss)
|
2014-02-26 08:08:43 +08:00
|
|
|
ld r1,PACAR1(r13)
|
2014-12-10 02:56:52 +08:00
|
|
|
/*
|
|
|
|
|
* Before entering any idle state, the NVGPRs are saved in the stack
|
|
|
|
|
* and they are restored before switching to the process context. Hence
|
|
|
|
|
* until they are restored, they are free to be used.
|
|
|
|
|
*
|
2016-07-08 14:20:44 +08:00
|
|
|
* Save SRR1 and LR in NVGPRs as they might be clobbered in
|
2016-07-08 14:37:11 +08:00
|
|
|
* opal_call() (called in CHECK_HMI_INTERRUPT). SRR1 is required
|
2016-07-08 14:20:44 +08:00
|
|
|
* to determine the wakeup reason if we branch to kvm_start_guest. LR
|
|
|
|
|
* is required to return back to reset vector after hypervisor state
|
|
|
|
|
* restore is complete.
|
2014-12-10 02:56:52 +08:00
|
|
|
*/
|
2016-07-08 14:20:44 +08:00
|
|
|
mflr r17
|
2014-12-10 02:56:52 +08:00
|
|
|
mfspr r16,SPRN_SRR1
|
2014-07-29 21:10:13 +08:00
|
|
|
BEGIN_FTR_SECTION
|
|
|
|
|
CHECK_HMI_INTERRUPT
|
|
|
|
|
END_FTR_SECTION_IFSET(CPU_FTR_HVMODE)
|
2014-12-10 02:56:52 +08:00
|
|
|
|
|
|
|
|
lbz r7,PACA_THREAD_MASK(r13)
|
|
|
|
|
ld r14,PACA_CORE_IDLE_STATE_PTR(r13)
|
|
|
|
|
lwarx_loop2:
|
|
|
|
|
lwarx r15,0,r14
|
|
|
|
|
andi. r9,r15,PNV_CORE_IDLE_LOCK_BIT
|
|
|
|
|
/*
|
|
|
|
|
* Lock bit is set in one of the 2 cases-
|
|
|
|
|
* a. In the sleep/winkle enter path, the last thread is executing
|
|
|
|
|
* fastsleep workaround code.
|
|
|
|
|
* b. In the wake up path, another thread is executing fastsleep
|
|
|
|
|
* workaround undo code or resyncing timebase or restoring context
|
|
|
|
|
* In either case loop until the lock bit is cleared.
|
|
|
|
|
*/
|
powerpc/powernv: Fix race in updating core_idle_state
core_idle_state is maintained for each core. It uses 0-7 bits to track
whether a thread in the core has entered fastsleep or winkle. 8th bit is
used as a lock bit.
The lock bit is set in these 2 scenarios-
- The thread is first in subcore to wakeup from sleep/winkle.
- If its the last thread in the core about to enter sleep/winkle
While the lock bit is set, if any other thread in the core wakes up, it
loops until the lock bit is cleared before proceeding in the wakeup
path. This helps prevent race conditions w.r.t fastsleep workaround and
prevents threads from switching to process context before core/subcore
resources are restored.
But, in the path to sleep/winkle entry, we currently don't check for
lock-bit. This exposes us to following race when running with subcore
on-
First thread in the subcorea Another thread in the same
waking up core entering sleep/winkle
lwarx r15,0,r14
ori r15,r15,PNV_CORE_IDLE_LOCK_BIT
stwcx. r15,0,r14
[Code to restore subcore state]
lwarx r15,0,r14
[clear thread bit]
stwcx. r15,0,r14
andi. r15,r15,PNV_CORE_IDLE_THREAD_BITS
stw r15,0(r14)
Here, after the thread entering sleep clears its thread bit in
core_idle_state, the value is overwritten by the thread waking up.
In such cases when the core enters fastsleep, code mistakes an idle
thread as running. Because of this, the first thread waking up from
fastsleep which is supposed to resync timebase skips it. So we can
end up having a core with stale timebase value.
This patch fixes the above race by looping on the lock bit even while
entering the idle states.
Signed-off-by: Shreyas B. Prabhu <shreyas@linux.vnet.ibm.com>
Fixes: 7b54e9f213f76 'powernv/powerpc: Add winkle support for offline cpus'
Cc: stable@vger.kernel.org # 3.19+
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2015-07-07 04:09:23 +08:00
|
|
|
bnel core_idle_lock_held
|
2014-12-10 02:56:52 +08:00
|
|
|
|
|
|
|
|
cmpwi cr2,r15,0
|
2014-12-10 02:56:53 +08:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* At this stage
|
2016-07-08 14:20:49 +08:00
|
|
|
* cr2 - eq if first thread to wakeup in core
|
|
|
|
|
* cr3- gt if waking up with partial/complete hypervisor state loss
|
2016-09-07 13:16:30 +08:00
|
|
|
* cr4 - gt or eq if waking up from complete hypervisor state loss.
|
2014-12-10 02:56:53 +08:00
|
|
|
*/
|
|
|
|
|
|
2014-12-10 02:56:52 +08:00
|
|
|
ori r15,r15,PNV_CORE_IDLE_LOCK_BIT
|
|
|
|
|
stwcx. r15,0,r14
|
|
|
|
|
bne- lwarx_loop2
|
|
|
|
|
isync
|
|
|
|
|
|
2016-07-08 14:20:49 +08:00
|
|
|
BEGIN_FTR_SECTION
|
|
|
|
|
lbz r4,PACA_SUBCORE_SIBLING_MASK(r13)
|
|
|
|
|
and r4,r4,r15
|
|
|
|
|
cmpwi r4,0 /* Check if first in subcore */
|
|
|
|
|
|
|
|
|
|
or r15,r15,r7 /* Set thread bit */
|
|
|
|
|
beq first_thread_in_subcore
|
|
|
|
|
END_FTR_SECTION_IFCLR(CPU_FTR_ARCH_300)
|
|
|
|
|
|
|
|
|
|
or r15,r15,r7 /* Set thread bit */
|
|
|
|
|
beq cr2,first_thread_in_core
|
|
|
|
|
|
|
|
|
|
/* Not first thread in core or subcore to wake up */
|
|
|
|
|
b clear_lock
|
|
|
|
|
|
|
|
|
|
first_thread_in_subcore:
|
2014-12-10 02:56:53 +08:00
|
|
|
/*
|
|
|
|
|
* If waking up from sleep, subcore state is not lost. Hence
|
|
|
|
|
* skip subcore state restore
|
|
|
|
|
*/
|
2016-09-07 13:16:30 +08:00
|
|
|
blt cr4,subcore_state_restored
|
2014-12-10 02:56:53 +08:00
|
|
|
|
|
|
|
|
/* Restore per-subcore state */
|
|
|
|
|
ld r4,_SDR1(r1)
|
|
|
|
|
mtspr SPRN_SDR1,r4
|
2016-07-08 14:20:49 +08:00
|
|
|
|
2014-12-10 02:56:53 +08:00
|
|
|
ld r4,_RPR(r1)
|
|
|
|
|
mtspr SPRN_RPR,r4
|
|
|
|
|
ld r4,_AMOR(r1)
|
|
|
|
|
mtspr SPRN_AMOR,r4
|
|
|
|
|
|
|
|
|
|
subcore_state_restored:
|
|
|
|
|
/*
|
|
|
|
|
* Check if the thread is also the first thread in the core. If not,
|
|
|
|
|
* skip to clear_lock.
|
|
|
|
|
*/
|
|
|
|
|
bne cr2,clear_lock
|
|
|
|
|
|
|
|
|
|
first_thread_in_core:
|
|
|
|
|
|
2014-12-10 02:56:52 +08:00
|
|
|
/*
|
2016-07-08 14:20:49 +08:00
|
|
|
* First thread in the core waking up from any state which can cause
|
|
|
|
|
* partial or complete hypervisor state loss. It needs to
|
2014-12-10 02:56:52 +08:00
|
|
|
* call the fastsleep workaround code if the platform requires it.
|
|
|
|
|
* Call it unconditionally here. The below branch instruction will
|
2016-07-08 14:20:49 +08:00
|
|
|
* be patched out if the platform does not have fastsleep or does not
|
|
|
|
|
* require the workaround. Patching will be performed during the
|
|
|
|
|
* discovery of idle-states.
|
2014-12-10 02:56:52 +08:00
|
|
|
*/
|
|
|
|
|
.global pnv_fastsleep_workaround_at_exit
|
|
|
|
|
pnv_fastsleep_workaround_at_exit:
|
|
|
|
|
b fastsleep_workaround_at_exit
|
|
|
|
|
|
|
|
|
|
timebase_resync:
|
2016-07-08 14:20:49 +08:00
|
|
|
/*
|
|
|
|
|
* Use cr3 which indicates that we are waking up with atleast partial
|
|
|
|
|
* hypervisor state loss to determine if TIMEBASE RESYNC is needed.
|
|
|
|
|
*/
|
2014-12-10 02:56:52 +08:00
|
|
|
ble cr3,clear_lock
|
2014-02-26 08:08:43 +08:00
|
|
|
/* Time base re-sync */
|
2016-07-08 14:37:11 +08:00
|
|
|
bl opal_rm_resync_timebase;
|
2014-12-10 02:56:53 +08:00
|
|
|
/*
|
|
|
|
|
* If waking up from sleep, per core state is not lost, skip to
|
|
|
|
|
* clear_lock.
|
|
|
|
|
*/
|
2016-09-07 13:16:30 +08:00
|
|
|
blt cr4,clear_lock
|
2014-12-10 02:56:53 +08:00
|
|
|
|
2016-07-08 14:20:49 +08:00
|
|
|
/*
|
|
|
|
|
* First thread in the core to wake up and its waking up with
|
|
|
|
|
* complete hypervisor state loss. Restore per core hypervisor
|
|
|
|
|
* state.
|
|
|
|
|
*/
|
|
|
|
|
BEGIN_FTR_SECTION
|
|
|
|
|
ld r4,_PTCR(r1)
|
|
|
|
|
mtspr SPRN_PTCR,r4
|
|
|
|
|
ld r4,_RPR(r1)
|
|
|
|
|
mtspr SPRN_RPR,r4
|
|
|
|
|
END_FTR_SECTION_IFSET(CPU_FTR_ARCH_300)
|
|
|
|
|
|
2014-12-10 02:56:53 +08:00
|
|
|
ld r4,_TSCR(r1)
|
|
|
|
|
mtspr SPRN_TSCR,r4
|
|
|
|
|
ld r4,_WORC(r1)
|
|
|
|
|
mtspr SPRN_WORC,r4
|
|
|
|
|
|
2014-12-10 02:56:52 +08:00
|
|
|
clear_lock:
|
|
|
|
|
andi. r15,r15,PNV_CORE_IDLE_THREAD_BITS
|
|
|
|
|
lwsync
|
|
|
|
|
stw r15,0(r14)
|
|
|
|
|
|
|
|
|
|
common_exit:
|
2014-12-10 02:56:53 +08:00
|
|
|
/*
|
|
|
|
|
* Common to all threads.
|
|
|
|
|
*
|
|
|
|
|
* If waking up from sleep, hypervisor state is not lost. Hence
|
|
|
|
|
* skip hypervisor state restore.
|
|
|
|
|
*/
|
2016-09-07 13:16:30 +08:00
|
|
|
blt cr4,hypervisor_state_restored
|
2014-12-10 02:56:53 +08:00
|
|
|
|
|
|
|
|
/* Waking up from winkle */
|
|
|
|
|
|
2016-07-08 14:20:49 +08:00
|
|
|
BEGIN_MMU_FTR_SECTION
|
|
|
|
|
b no_segments
|
2016-07-27 11:19:01 +08:00
|
|
|
END_MMU_FTR_SECTION_IFSET(MMU_FTR_TYPE_RADIX)
|
2014-12-10 02:56:53 +08:00
|
|
|
/* Restore SLB from PACA */
|
|
|
|
|
ld r8,PACA_SLBSHADOWPTR(r13)
|
|
|
|
|
|
|
|
|
|
.rept SLB_NUM_BOLTED
|
|
|
|
|
li r3, SLBSHADOW_SAVEAREA
|
|
|
|
|
LDX_BE r5, r8, r3
|
|
|
|
|
addi r3, r3, 8
|
|
|
|
|
LDX_BE r6, r8, r3
|
|
|
|
|
andis. r7,r5,SLB_ESID_V@h
|
|
|
|
|
beq 1f
|
|
|
|
|
slbmte r6,r5
|
|
|
|
|
1: addi r8,r8,16
|
|
|
|
|
.endr
|
2016-07-08 14:20:49 +08:00
|
|
|
no_segments:
|
|
|
|
|
|
|
|
|
|
/* Restore per thread state */
|
2014-12-10 02:56:53 +08:00
|
|
|
|
|
|
|
|
ld r4,_SPURR(r1)
|
|
|
|
|
mtspr SPRN_SPURR,r4
|
|
|
|
|
ld r4,_PURR(r1)
|
|
|
|
|
mtspr SPRN_PURR,r4
|
|
|
|
|
ld r4,_DSCR(r1)
|
|
|
|
|
mtspr SPRN_DSCR,r4
|
|
|
|
|
ld r4,_WORT(r1)
|
|
|
|
|
mtspr SPRN_WORT,r4
|
|
|
|
|
|
2016-07-08 14:20:49 +08:00
|
|
|
/* Call cur_cpu_spec->cpu_restore() */
|
|
|
|
|
LOAD_REG_ADDR(r4, cur_cpu_spec)
|
|
|
|
|
ld r4,0(r4)
|
|
|
|
|
ld r12,CPU_SPEC_RESTORE(r4)
|
|
|
|
|
#ifdef PPC64_ELF_ABI_v1
|
|
|
|
|
ld r12,0(r12)
|
|
|
|
|
#endif
|
|
|
|
|
mtctr r12
|
|
|
|
|
bctrl
|
|
|
|
|
|
2014-12-10 02:56:53 +08:00
|
|
|
hypervisor_state_restored:
|
|
|
|
|
|
2014-12-10 02:56:52 +08:00
|
|
|
mtspr SPRN_SRR1,r16
|
2016-07-08 14:20:44 +08:00
|
|
|
mtlr r17
|
|
|
|
|
blr /* Return back to System Reset vector from where
|
2016-07-08 14:20:46 +08:00
|
|
|
pnv_restore_hyp_resource was invoked */
|
2014-02-26 08:08:43 +08:00
|
|
|
|
2014-12-10 02:56:52 +08:00
|
|
|
fastsleep_workaround_at_exit:
|
|
|
|
|
li r3,1
|
|
|
|
|
li r4,0
|
2016-07-08 14:37:11 +08:00
|
|
|
bl opal_rm_config_cpu_idle_state
|
2014-12-10 02:56:52 +08:00
|
|
|
b timebase_resync
|
|
|
|
|
|
powerpc/powernv: Return to cpu offline loop when finished in KVM guest
When a secondary hardware thread has finished running a KVM guest, we
currently put that thread into nap mode using a nap instruction in
the KVM code. This changes the code so that instead of doing a nap
instruction directly, we instead cause the call to power7_nap() that
put the thread into nap mode to return. The reason for doing this is
to avoid having the KVM code having to know what low-power mode to
put the thread into.
In the case of a secondary thread used to run a KVM guest, the thread
will be offline from the point of view of the host kernel, and the
relevant power7_nap() call is the one in pnv_smp_cpu_disable().
In this case we don't want to clear pending IPIs in the offline loop
in that function, since that might cause us to miss the wakeup for
the next time the thread needs to run a guest. To tell whether or
not to clear the interrupt, we use the SRR1 value returned from
power7_nap(), and check if it indicates an external interrupt. We
arrange that the return from power7_nap() when we have finished running
a guest returns 0, so pending interrupts don't get flushed in that
case.
Note that it is important a secondary thread that has finished
executing in the guest, or that didn't have a guest to run, should
not return to power7_nap's caller while the kvm_hstate.hwthread_req
flag in the PACA is non-zero, because the return from power7_nap
will reenable the MMU, and the MMU might still be in guest context.
In this situation we spin at low priority in real mode waiting for
hwthread_req to become zero.
Signed-off-by: Paul Mackerras <paulus@samba.org>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2014-12-03 11:48:40 +08:00
|
|
|
/*
|
|
|
|
|
* R3 here contains the value that will be returned to the caller
|
|
|
|
|
* of power7_nap.
|
|
|
|
|
*/
|
2016-07-08 14:20:46 +08:00
|
|
|
_GLOBAL(pnv_wakeup_loss)
|
2011-01-24 15:42:41 +08:00
|
|
|
ld r1,PACAR1(r13)
|
2014-07-29 21:10:13 +08:00
|
|
|
BEGIN_FTR_SECTION
|
|
|
|
|
CHECK_HMI_INTERRUPT
|
|
|
|
|
END_FTR_SECTION_IFSET(CPU_FTR_HVMODE)
|
2011-01-24 15:42:41 +08:00
|
|
|
REST_NVGPRS(r1)
|
|
|
|
|
REST_GPR(2, r1)
|
powerpc/powernv: Return to cpu offline loop when finished in KVM guest
When a secondary hardware thread has finished running a KVM guest, we
currently put that thread into nap mode using a nap instruction in
the KVM code. This changes the code so that instead of doing a nap
instruction directly, we instead cause the call to power7_nap() that
put the thread into nap mode to return. The reason for doing this is
to avoid having the KVM code having to know what low-power mode to
put the thread into.
In the case of a secondary thread used to run a KVM guest, the thread
will be offline from the point of view of the host kernel, and the
relevant power7_nap() call is the one in pnv_smp_cpu_disable().
In this case we don't want to clear pending IPIs in the offline loop
in that function, since that might cause us to miss the wakeup for
the next time the thread needs to run a guest. To tell whether or
not to clear the interrupt, we use the SRR1 value returned from
power7_nap(), and check if it indicates an external interrupt. We
arrange that the return from power7_nap() when we have finished running
a guest returns 0, so pending interrupts don't get flushed in that
case.
Note that it is important a secondary thread that has finished
executing in the guest, or that didn't have a guest to run, should
not return to power7_nap's caller while the kvm_hstate.hwthread_req
flag in the PACA is non-zero, because the return from power7_nap
will reenable the MMU, and the MMU might still be in guest context.
In this situation we spin at low priority in real mode waiting for
hwthread_req to become zero.
Signed-off-by: Paul Mackerras <paulus@samba.org>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2014-12-03 11:48:40 +08:00
|
|
|
ld r6,_CCR(r1)
|
2011-01-24 15:42:41 +08:00
|
|
|
ld r4,_MSR(r1)
|
|
|
|
|
ld r5,_NIP(r1)
|
|
|
|
|
addi r1,r1,INT_FRAME_SIZE
|
powerpc/powernv: Return to cpu offline loop when finished in KVM guest
When a secondary hardware thread has finished running a KVM guest, we
currently put that thread into nap mode using a nap instruction in
the KVM code. This changes the code so that instead of doing a nap
instruction directly, we instead cause the call to power7_nap() that
put the thread into nap mode to return. The reason for doing this is
to avoid having the KVM code having to know what low-power mode to
put the thread into.
In the case of a secondary thread used to run a KVM guest, the thread
will be offline from the point of view of the host kernel, and the
relevant power7_nap() call is the one in pnv_smp_cpu_disable().
In this case we don't want to clear pending IPIs in the offline loop
in that function, since that might cause us to miss the wakeup for
the next time the thread needs to run a guest. To tell whether or
not to clear the interrupt, we use the SRR1 value returned from
power7_nap(), and check if it indicates an external interrupt. We
arrange that the return from power7_nap() when we have finished running
a guest returns 0, so pending interrupts don't get flushed in that
case.
Note that it is important a secondary thread that has finished
executing in the guest, or that didn't have a guest to run, should
not return to power7_nap's caller while the kvm_hstate.hwthread_req
flag in the PACA is non-zero, because the return from power7_nap
will reenable the MMU, and the MMU might still be in guest context.
In this situation we spin at low priority in real mode waiting for
hwthread_req to become zero.
Signed-off-by: Paul Mackerras <paulus@samba.org>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2014-12-03 11:48:40 +08:00
|
|
|
mtcr r6
|
2011-01-24 15:42:41 +08:00
|
|
|
mtspr SPRN_SRR1,r4
|
|
|
|
|
mtspr SPRN_SRR0,r5
|
|
|
|
|
rfid
|
|
|
|
|
|
powerpc/powernv: Return to cpu offline loop when finished in KVM guest
When a secondary hardware thread has finished running a KVM guest, we
currently put that thread into nap mode using a nap instruction in
the KVM code. This changes the code so that instead of doing a nap
instruction directly, we instead cause the call to power7_nap() that
put the thread into nap mode to return. The reason for doing this is
to avoid having the KVM code having to know what low-power mode to
put the thread into.
In the case of a secondary thread used to run a KVM guest, the thread
will be offline from the point of view of the host kernel, and the
relevant power7_nap() call is the one in pnv_smp_cpu_disable().
In this case we don't want to clear pending IPIs in the offline loop
in that function, since that might cause us to miss the wakeup for
the next time the thread needs to run a guest. To tell whether or
not to clear the interrupt, we use the SRR1 value returned from
power7_nap(), and check if it indicates an external interrupt. We
arrange that the return from power7_nap() when we have finished running
a guest returns 0, so pending interrupts don't get flushed in that
case.
Note that it is important a secondary thread that has finished
executing in the guest, or that didn't have a guest to run, should
not return to power7_nap's caller while the kvm_hstate.hwthread_req
flag in the PACA is non-zero, because the return from power7_nap
will reenable the MMU, and the MMU might still be in guest context.
In this situation we spin at low priority in real mode waiting for
hwthread_req to become zero.
Signed-off-by: Paul Mackerras <paulus@samba.org>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2014-12-03 11:48:40 +08:00
|
|
|
/*
|
|
|
|
|
* R3 here contains the value that will be returned to the caller
|
|
|
|
|
* of power7_nap.
|
|
|
|
|
*/
|
2016-07-08 14:20:46 +08:00
|
|
|
_GLOBAL(pnv_wakeup_noloss)
|
2011-12-06 03:47:26 +08:00
|
|
|
lbz r0,PACA_NAPSTATELOST(r13)
|
|
|
|
|
cmpwi r0,0
|
2016-07-08 14:20:46 +08:00
|
|
|
bne pnv_wakeup_loss
|
2014-07-29 21:10:13 +08:00
|
|
|
BEGIN_FTR_SECTION
|
|
|
|
|
CHECK_HMI_INTERRUPT
|
|
|
|
|
END_FTR_SECTION_IFSET(CPU_FTR_HVMODE)
|
2011-01-24 15:42:41 +08:00
|
|
|
ld r1,PACAR1(r13)
|
powerpc/powernv: Restore non-volatile CRs after nap
Patches 7cba160ad "powernv/cpuidle: Redesign idle states management"
and 77b54e9f2 "powernv/powerpc: Add winkle support for offline cpus"
use non-volatile condition registers (cr2, cr3 and cr4) early in the system
reset interrupt handler (system_reset_pSeries()) before it has been determined
if state loss has occurred. If state loss has not occurred, control returns via
the power7_wakeup_noloss() path which does not restore those condition
registers, leaving them corrupted.
Fix this by restoring the condition registers in the power7_wakeup_noloss()
case.
This is apparent when running a KVM guest on hardware that does not
support winkle or sleep and the guest makes use of secondary threads. In
practice this means Power7 machines, though some early unreleased Power8
machines may also be susceptible.
The secondary CPUs are taken off line before the guest is started and
they call pnv_smp_cpu_kill_self(). This checks support for sleep
states (in this case there is no support) and power7_nap() is called.
When the CPU is woken, power7_nap() returns and because the CPU is
still off line, the main while loop executes again. The sleep states
support test is executed again, but because the tested values cannot
have changed, the compiler has optimized the test away and instead we
rely on the result of the first test, which has been left in cr3
and/or cr4. With the result overwritten, the wrong branch is taken and
power7_winkle() is called on a CPU that does not support it, leading
to it stalling.
Fixes: 7cba160ad789 ("powernv/cpuidle: Redesign idle states management")
Fixes: 77b54e9f213f ("powernv/powerpc: Add winkle support for offline cpus")
[mpe: Massage change log a bit more]
Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2015-05-01 14:50:34 +08:00
|
|
|
ld r6,_CCR(r1)
|
2011-01-24 15:42:41 +08:00
|
|
|
ld r4,_MSR(r1)
|
|
|
|
|
ld r5,_NIP(r1)
|
|
|
|
|
addi r1,r1,INT_FRAME_SIZE
|
powerpc/powernv: Restore non-volatile CRs after nap
Patches 7cba160ad "powernv/cpuidle: Redesign idle states management"
and 77b54e9f2 "powernv/powerpc: Add winkle support for offline cpus"
use non-volatile condition registers (cr2, cr3 and cr4) early in the system
reset interrupt handler (system_reset_pSeries()) before it has been determined
if state loss has occurred. If state loss has not occurred, control returns via
the power7_wakeup_noloss() path which does not restore those condition
registers, leaving them corrupted.
Fix this by restoring the condition registers in the power7_wakeup_noloss()
case.
This is apparent when running a KVM guest on hardware that does not
support winkle or sleep and the guest makes use of secondary threads. In
practice this means Power7 machines, though some early unreleased Power8
machines may also be susceptible.
The secondary CPUs are taken off line before the guest is started and
they call pnv_smp_cpu_kill_self(). This checks support for sleep
states (in this case there is no support) and power7_nap() is called.
When the CPU is woken, power7_nap() returns and because the CPU is
still off line, the main while loop executes again. The sleep states
support test is executed again, but because the tested values cannot
have changed, the compiler has optimized the test away and instead we
rely on the result of the first test, which has been left in cr3
and/or cr4. With the result overwritten, the wrong branch is taken and
power7_winkle() is called on a CPU that does not support it, leading
to it stalling.
Fixes: 7cba160ad789 ("powernv/cpuidle: Redesign idle states management")
Fixes: 77b54e9f213f ("powernv/powerpc: Add winkle support for offline cpus")
[mpe: Massage change log a bit more]
Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
2015-05-01 14:50:34 +08:00
|
|
|
mtcr r6
|
2011-01-24 15:42:41 +08:00
|
|
|
mtspr SPRN_SRR1,r4
|
|
|
|
|
mtspr SPRN_SRR0,r5
|
|
|
|
|
rfid
|