2007-07-18 09:37:07 +08:00
|
|
|
/*
|
2009-02-05 23:57:48 +08:00
|
|
|
* Asm versions of Xen pv-ops, suitable for either direct use or
|
|
|
|
* inlining. The inline versions are the same as the direct-use
|
|
|
|
* versions, with the pre- and post-amble chopped off.
|
|
|
|
*
|
|
|
|
* This code is encoded for size rather than absolute efficiency, with
|
|
|
|
* a view to being able to inline as much as possible.
|
|
|
|
*
|
|
|
|
* We only bother with direct forms (ie, vcpu in pda) of the
|
|
|
|
* operations here; the indirect forms are better handled in C, since
|
|
|
|
* they're generally too large to inline anyway.
|
2007-07-18 09:37:07 +08:00
|
|
|
*/
|
|
|
|
|
|
|
|
#include <asm/thread_info.h>
|
|
|
|
#include <asm/processor-flags.h>
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
#include <asm/segment.h>
|
2012-04-21 03:19:52 +08:00
|
|
|
#include <asm/asm.h>
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
|
|
|
|
#include <xen/interface/xen.h>
|
2007-07-18 09:37:07 +08:00
|
|
|
|
2009-02-03 05:55:42 +08:00
|
|
|
#include "xen-asm.h"
|
2007-07-18 09:37:07 +08:00
|
|
|
|
|
|
|
/*
|
2009-02-05 23:57:48 +08:00
|
|
|
* Force an event check by making a hypercall, but preserve regs
|
|
|
|
* before making the call.
|
2007-07-18 09:37:07 +08:00
|
|
|
*/
|
2009-02-03 05:55:42 +08:00
|
|
|
check_events:
|
|
|
|
push %eax
|
|
|
|
push %ecx
|
|
|
|
push %edx
|
|
|
|
call xen_force_evtchn_callback
|
|
|
|
pop %edx
|
|
|
|
pop %ecx
|
|
|
|
pop %eax
|
2007-07-18 09:37:07 +08:00
|
|
|
ret
|
|
|
|
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
/*
|
2009-02-05 23:57:48 +08:00
|
|
|
* This is run where a normal iret would be run, with the same stack setup:
|
|
|
|
* 8: eflags
|
|
|
|
* 4: cs
|
|
|
|
* esp-> 0: eip
|
|
|
|
*
|
|
|
|
* This attempts to make sure that any pending events are dealt with
|
|
|
|
* on return to usermode, but there is a small window in which an
|
|
|
|
* event can happen just before entering usermode. If the nested
|
|
|
|
* interrupt ends up setting one of the TIF_WORK_MASK pending work
|
|
|
|
* flags, they will not be tested again before returning to
|
|
|
|
* usermode. This means that a process can end up with pending work,
|
|
|
|
* which will be unprocessed until the process enters and leaves the
|
|
|
|
* kernel again, which could be an unbounded amount of time. This
|
|
|
|
* means that a pending signal or reschedule event could be
|
|
|
|
* indefinitely delayed.
|
|
|
|
*
|
|
|
|
* The fix is to notice a nested interrupt in the critical window, and
|
|
|
|
* if one occurs, then fold the nested interrupt into the current
|
|
|
|
* interrupt stack frame, and re-process it iteratively rather than
|
|
|
|
* recursively. This means that it will exit via the normal path, and
|
|
|
|
* all pending work will be dealt with appropriately.
|
|
|
|
*
|
|
|
|
* Because the nested interrupt handler needs to deal with the current
|
|
|
|
* stack state in whatever form its in, we keep things simple by only
|
|
|
|
* using a single register which is pushed/popped on the stack.
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
*/
|
2014-04-11 00:17:09 +08:00
|
|
|
|
|
|
|
.macro POP_FS
|
|
|
|
1:
|
|
|
|
popw %fs
|
|
|
|
.pushsection .fixup, "ax"
|
|
|
|
2: movw $0, (%esp)
|
|
|
|
jmp 1b
|
|
|
|
.popsection
|
|
|
|
_ASM_EXTABLE(1b,2b)
|
|
|
|
.endm
|
|
|
|
|
2008-04-17 23:40:51 +08:00
|
|
|
ENTRY(xen_iret)
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
/* test eflags for special cases */
|
|
|
|
testl $(X86_EFLAGS_VM | XEN_EFLAGS_NMI), 8(%esp)
|
|
|
|
jnz hyper_iret
|
|
|
|
|
|
|
|
push %eax
|
|
|
|
ESP_OFFSET=4 # bytes pushed onto stack
|
|
|
|
|
2014-04-11 00:17:09 +08:00
|
|
|
/* Store vcpu_info pointer for easy access */
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
#ifdef CONFIG_SMP
|
2014-04-11 00:17:09 +08:00
|
|
|
pushw %fs
|
|
|
|
movl $(__KERNEL_PERCPU), %eax
|
|
|
|
movl %eax, %fs
|
|
|
|
movl %fs:xen_vcpu, %eax
|
|
|
|
POP_FS
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
#else
|
2013-01-24 21:11:10 +08:00
|
|
|
movl %ss:xen_vcpu, %eax
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
#endif
|
|
|
|
|
|
|
|
/* check IF state we're restoring */
|
|
|
|
testb $X86_EFLAGS_IF>>8, 8+1+ESP_OFFSET(%esp)
|
|
|
|
|
2009-02-05 23:57:48 +08:00
|
|
|
/*
|
|
|
|
* Maybe enable events. Once this happens we could get a
|
|
|
|
* recursive event, so the critical region starts immediately
|
|
|
|
* afterwards. However, if that happens we don't end up
|
|
|
|
* resuming the code, so we don't have to be worried about
|
|
|
|
* being preempted to another CPU.
|
|
|
|
*/
|
2013-01-24 21:11:10 +08:00
|
|
|
setz %ss:XEN_vcpu_info_mask(%eax)
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
xen_iret_start_crit:
|
|
|
|
|
|
|
|
/* check for unmasked and pending */
|
2013-01-24 21:11:10 +08:00
|
|
|
cmpw $0x0001, %ss:XEN_vcpu_info_pending(%eax)
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
|
2009-02-05 23:57:48 +08:00
|
|
|
/*
|
|
|
|
* If there's something pending, mask events again so we can
|
2011-09-01 19:46:55 +08:00
|
|
|
* jump back into xen_hypervisor_callback. Otherwise do not
|
|
|
|
* touch XEN_vcpu_info_mask.
|
2009-02-05 23:57:48 +08:00
|
|
|
*/
|
2011-09-01 19:46:55 +08:00
|
|
|
jne 1f
|
2013-01-24 21:11:10 +08:00
|
|
|
movb $1, %ss:XEN_vcpu_info_mask(%eax)
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
|
2011-09-01 19:46:55 +08:00
|
|
|
1: popl %eax
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
|
2009-02-05 23:57:48 +08:00
|
|
|
/*
|
|
|
|
* From this point on the registers are restored and the stack
|
|
|
|
* updated, so we don't need to worry about it if we're
|
|
|
|
* preempted
|
|
|
|
*/
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
iret_restore_end:
|
|
|
|
|
2009-02-05 23:57:48 +08:00
|
|
|
/*
|
|
|
|
* Jump to hypervisor_callback after fixing up the stack.
|
|
|
|
* Events are masked, so jumping out of the critical region is
|
|
|
|
* OK.
|
|
|
|
*/
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
je xen_hypervisor_callback
|
|
|
|
|
2008-03-18 07:37:12 +08:00
|
|
|
1: iret
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
xen_iret_end_crit:
|
2012-04-21 03:19:52 +08:00
|
|
|
_ASM_EXTABLE(1b, iret_exc)
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
|
|
|
|
hyper_iret:
|
|
|
|
/* put this out of line since its very rarely used */
|
|
|
|
jmp hypercall_page + __HYPERVISOR_iret * 32
|
|
|
|
|
|
|
|
.globl xen_iret_start_crit, xen_iret_end_crit
|
|
|
|
|
|
|
|
/*
|
2009-02-05 23:57:48 +08:00
|
|
|
* This is called by xen_hypervisor_callback in entry.S when it sees
|
|
|
|
* that the EIP at the time of interrupt was between
|
|
|
|
* xen_iret_start_crit and xen_iret_end_crit. We're passed the EIP in
|
|
|
|
* %eax so we can do a more refined determination of what to do.
|
|
|
|
*
|
|
|
|
* The stack format at this point is:
|
|
|
|
* ----------------
|
|
|
|
* ss : (ss/esp may be present if we came from usermode)
|
|
|
|
* esp :
|
|
|
|
* eflags } outer exception info
|
|
|
|
* cs }
|
|
|
|
* eip }
|
|
|
|
* ---------------- <- edi (copy dest)
|
|
|
|
* eax : outer eax if it hasn't been restored
|
|
|
|
* ----------------
|
|
|
|
* eflags } nested exception info
|
|
|
|
* cs } (no ss/esp because we're nested
|
|
|
|
* eip } from the same ring)
|
|
|
|
* orig_eax }<- esi (copy src)
|
|
|
|
* - - - - - - - -
|
|
|
|
* fs }
|
|
|
|
* es }
|
|
|
|
* ds } SAVE_ALL state
|
|
|
|
* eax }
|
|
|
|
* : :
|
|
|
|
* ebx }<- esp
|
|
|
|
* ----------------
|
|
|
|
*
|
|
|
|
* In order to deliver the nested exception properly, we need to shift
|
|
|
|
* everything from the return addr up to the error code so it sits
|
|
|
|
* just under the outer exception info. This means that when we
|
|
|
|
* handle the exception, we do it in the context of the outer
|
|
|
|
* exception rather than starting a new one.
|
|
|
|
*
|
|
|
|
* The only caveat is that if the outer eax hasn't been restored yet
|
|
|
|
* (ie, it's still on stack), we need to insert its value into the
|
|
|
|
* SAVE_ALL state before going on, since it's usermode state which we
|
|
|
|
* eventually need to restore.
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
*/
|
|
|
|
ENTRY(xen_iret_crit_fixup)
|
|
|
|
/*
|
2009-02-05 23:57:48 +08:00
|
|
|
* Paranoia: Make sure we're really coming from kernel space.
|
|
|
|
* One could imagine a case where userspace jumps into the
|
|
|
|
* critical range address, but just before the CPU delivers a
|
|
|
|
* GP, it decides to deliver an interrupt instead. Unlikely?
|
|
|
|
* Definitely. Easy to avoid? Yes. The Intel documents
|
|
|
|
* explicitly say that the reported EIP for a bad jump is the
|
|
|
|
* jump instruction itself, not the destination, but some
|
|
|
|
* virtual environments get this wrong.
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
*/
|
2008-03-18 07:37:22 +08:00
|
|
|
movl PT_CS(%esp), %ecx
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
andl $SEGMENT_RPL_MASK, %ecx
|
|
|
|
cmpl $USER_RPL, %ecx
|
|
|
|
je 2f
|
|
|
|
|
2008-03-18 07:37:22 +08:00
|
|
|
lea PT_ORIG_EAX(%esp), %esi
|
|
|
|
lea PT_EFLAGS(%esp), %edi
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
|
2009-02-05 23:57:48 +08:00
|
|
|
/*
|
|
|
|
* If eip is before iret_restore_end then stack
|
|
|
|
* hasn't been restored yet.
|
|
|
|
*/
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
cmp $iret_restore_end, %eax
|
|
|
|
jae 1f
|
|
|
|
|
2009-02-05 23:57:48 +08:00
|
|
|
movl 0+4(%edi), %eax /* copy EAX (just above top of frame) */
|
2008-03-18 07:37:22 +08:00
|
|
|
movl %eax, PT_EAX(%esp)
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
|
2009-02-05 23:57:48 +08:00
|
|
|
lea ESP_OFFSET(%edi), %edi /* move dest up over saved regs */
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
|
|
|
|
/* set up the copy */
|
|
|
|
1: std
|
2008-03-18 07:37:22 +08:00
|
|
|
mov $PT_EIP / 4, %ecx /* saved regs up to orig_eax */
|
xen: use iret directly when possible
Most of the time we can simply use the iret instruction to exit the
kernel, rather than having to use the iret hypercall - the only
exception is if we're returning into vm86 mode, or from delivering an
NMI (which we don't support yet).
When running native, iret has the behaviour of testing for a pending
interrupt atomically with re-enabling interrupts. Unfortunately
there's no way to do this with Xen, so there's a window in which we
could get a recursive exception after enabling events but before
actually returning to userspace.
This causes a problem: if the nested interrupt causes one of the
task's TIF_WORK_MASK flags to be set, they will not be checked again
before returning to userspace. This means that pending work may be
left pending indefinitely, until the process enters and leaves the
kernel again. The net effect is that a pending signal or reschedule
event could be delayed for an unbounded amount of time.
To deal with this, the xen event upcall handler checks to see if the
EIP is within the critical section of the iret code, after events
are (potentially) enabled up to the iret itself. If its within this
range, it calls the iret critical section fixup, which adjusts the
stack to deal with any unrestored registers, and then shifts the
stack frame up to replace the previous invocation.
Signed-off-by: Jeremy Fitzhardinge <jeremy@xensource.com>
2007-07-18 09:37:07 +08:00
|
|
|
rep movsl
|
|
|
|
cld
|
|
|
|
|
2009-02-05 23:57:48 +08:00
|
|
|
lea 4(%edi), %esp /* point esp to new frame */
|
2008-03-18 07:37:22 +08:00
|
|
|
2: jmp xen_do_upcall
|
2007-07-18 09:37:07 +08:00
|
|
|
|