2020-08-18 21:57:44 +08:00
|
|
|
/* SPDX-License-Identifier: GPL-2.0 */
|
|
|
|
|
#ifndef _ASM_STATIC_CALL_H
|
|
|
|
|
#define _ASM_STATIC_CALL_H
|
|
|
|
|
|
|
|
|
|
#include <asm/text-patching.h>
|
|
|
|
|
|
|
|
|
|
/*
|
2020-08-18 21:57:45 +08:00
|
|
|
* For CONFIG_HAVE_STATIC_CALL_INLINE, this is a temporary trampoline which
|
|
|
|
|
* uses the current value of the key->func pointer to do an indirect jump to
|
|
|
|
|
* the function. This trampoline is only used during boot, before the call
|
|
|
|
|
* sites get patched by static_call_update(). The name of this trampoline has
|
|
|
|
|
* a magical aspect: objtool uses it to find static call sites so it can create
|
|
|
|
|
* the .static_call_sites section.
|
|
|
|
|
*
|
2020-08-18 21:57:44 +08:00
|
|
|
* For CONFIG_HAVE_STATIC_CALL, this is a permanent trampoline which
|
|
|
|
|
* does a direct jump to the function. The direct jump gets patched by
|
|
|
|
|
* static_call_update().
|
2020-08-18 21:57:45 +08:00
|
|
|
*
|
|
|
|
|
* Having the trampoline in a special section forces GCC to emit a JMP.d32 when
|
|
|
|
|
* it does tail-call optimization on the call; since you cannot compute the
|
|
|
|
|
* relative displacement across sections.
|
2020-08-18 21:57:44 +08:00
|
|
|
*/
|
2020-08-18 21:57:48 +08:00
|
|
|
|
x86,static_call: Use alternative RET encoding
In addition to teaching static_call about the new way to spell 'RET',
there is an added complication in that static_call() is allowed to
rewrite text before it is known which particular spelling is required.
In order to deal with this; have a static_call specific fixup in the
apply_return() 'alternative' patching routine that will rewrite the
static_call trampoline to match the definite sequence.
This in turn creates the problem of uniquely identifying static call
trampolines. Currently trampolines are 8 bytes, the first 5 being the
jmp.d32/ret sequence and the final 3 a byte sequence that spells out
'SCT'.
This sequence is used in __static_call_validate() to ensure it is
patching a trampoline and not a random other jmp.d32. That is,
false-positives shouldn't be plenty, but aren't a big concern.
OTOH the new __static_call_fixup() must not have false-positives, and
'SCT' decodes to the somewhat weird but semi plausible sequence:
push %rbx
rex.XB push %r12
Additionally, there are SLS concerns with immediate jumps. Combined it
seems like a good moment to change the signature to a single 3 byte
trap instruction that is unique to this usage and will not ever get
generated by accident.
As such, change the signature to: '0x0f, 0xb9, 0xcc', which decodes
to:
ud1 %esp, %ecx
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Borislav Petkov <bp@suse.de>
2022-06-15 05:15:39 +08:00
|
|
|
/*
|
|
|
|
|
* The trampoline is 8 bytes and of the general form:
|
|
|
|
|
*
|
|
|
|
|
* jmp.d32 \func
|
|
|
|
|
* ud1 %esp, %ecx
|
|
|
|
|
*
|
|
|
|
|
* That trailing #UD provides both a speculation stop and serves as a unique
|
|
|
|
|
* 3 byte signature identifying static call trampolines. Also see tramp_ud[]
|
|
|
|
|
* and __static_call_fixup().
|
|
|
|
|
*/
|
2020-08-18 21:57:48 +08:00
|
|
|
#define __ARCH_DEFINE_STATIC_CALL_TRAMP(name, insns) \
|
2020-08-18 21:57:45 +08:00
|
|
|
asm(".pushsection .static_call.text, \"ax\" \n" \
|
2020-08-18 21:57:44 +08:00
|
|
|
".align 4 \n" \
|
|
|
|
|
".globl " STATIC_CALL_TRAMP_STR(name) " \n" \
|
|
|
|
|
STATIC_CALL_TRAMP_STR(name) ": \n" \
|
2022-04-19 00:50:22 +08:00
|
|
|
ANNOTATE_NOENDBR \
|
2020-08-18 21:57:48 +08:00
|
|
|
insns " \n" \
|
x86,static_call: Use alternative RET encoding
In addition to teaching static_call about the new way to spell 'RET',
there is an added complication in that static_call() is allowed to
rewrite text before it is known which particular spelling is required.
In order to deal with this; have a static_call specific fixup in the
apply_return() 'alternative' patching routine that will rewrite the
static_call trampoline to match the definite sequence.
This in turn creates the problem of uniquely identifying static call
trampolines. Currently trampolines are 8 bytes, the first 5 being the
jmp.d32/ret sequence and the final 3 a byte sequence that spells out
'SCT'.
This sequence is used in __static_call_validate() to ensure it is
patching a trampoline and not a random other jmp.d32. That is,
false-positives shouldn't be plenty, but aren't a big concern.
OTOH the new __static_call_fixup() must not have false-positives, and
'SCT' decodes to the somewhat weird but semi plausible sequence:
push %rbx
rex.XB push %r12
Additionally, there are SLS concerns with immediate jumps. Combined it
seems like a good moment to change the signature to a single 3 byte
trap instruction that is unique to this usage and will not ever get
generated by accident.
As such, change the signature to: '0x0f, 0xb9, 0xcc', which decodes
to:
ud1 %esp, %ecx
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Borislav Petkov <bp@suse.de>
2022-06-15 05:15:39 +08:00
|
|
|
".byte 0x0f, 0xb9, 0xcc \n" \
|
2020-08-18 21:57:44 +08:00
|
|
|
".type " STATIC_CALL_TRAMP_STR(name) ", @function \n" \
|
|
|
|
|
".size " STATIC_CALL_TRAMP_STR(name) ", . - " STATIC_CALL_TRAMP_STR(name) " \n" \
|
|
|
|
|
".popsection \n")
|
|
|
|
|
|
2020-08-18 21:57:48 +08:00
|
|
|
#define ARCH_DEFINE_STATIC_CALL_TRAMP(name, func) \
|
|
|
|
|
__ARCH_DEFINE_STATIC_CALL_TRAMP(name, ".byte 0xe9; .long " #func " - (. + 4)")
|
|
|
|
|
|
2022-06-28 06:21:17 +08:00
|
|
|
#ifdef CONFIG_RETHUNK
|
x86,static_call: Use alternative RET encoding
In addition to teaching static_call about the new way to spell 'RET',
there is an added complication in that static_call() is allowed to
rewrite text before it is known which particular spelling is required.
In order to deal with this; have a static_call specific fixup in the
apply_return() 'alternative' patching routine that will rewrite the
static_call trampoline to match the definite sequence.
This in turn creates the problem of uniquely identifying static call
trampolines. Currently trampolines are 8 bytes, the first 5 being the
jmp.d32/ret sequence and the final 3 a byte sequence that spells out
'SCT'.
This sequence is used in __static_call_validate() to ensure it is
patching a trampoline and not a random other jmp.d32. That is,
false-positives shouldn't be plenty, but aren't a big concern.
OTOH the new __static_call_fixup() must not have false-positives, and
'SCT' decodes to the somewhat weird but semi plausible sequence:
push %rbx
rex.XB push %r12
Additionally, there are SLS concerns with immediate jumps. Combined it
seems like a good moment to change the signature to a single 3 byte
trap instruction that is unique to this usage and will not ever get
generated by accident.
As such, change the signature to: '0x0f, 0xb9, 0xcc', which decodes
to:
ud1 %esp, %ecx
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Borislav Petkov <bp@suse.de>
2022-06-15 05:15:39 +08:00
|
|
|
#define ARCH_DEFINE_STATIC_CALL_NULL_TRAMP(name) \
|
|
|
|
|
__ARCH_DEFINE_STATIC_CALL_TRAMP(name, "jmp __x86_return_thunk")
|
|
|
|
|
#else
|
2020-08-18 21:57:48 +08:00
|
|
|
#define ARCH_DEFINE_STATIC_CALL_NULL_TRAMP(name) \
|
2021-12-04 21:43:44 +08:00
|
|
|
__ARCH_DEFINE_STATIC_CALL_TRAMP(name, "ret; int3; nop; nop; nop")
|
x86,static_call: Use alternative RET encoding
In addition to teaching static_call about the new way to spell 'RET',
there is an added complication in that static_call() is allowed to
rewrite text before it is known which particular spelling is required.
In order to deal with this; have a static_call specific fixup in the
apply_return() 'alternative' patching routine that will rewrite the
static_call trampoline to match the definite sequence.
This in turn creates the problem of uniquely identifying static call
trampolines. Currently trampolines are 8 bytes, the first 5 being the
jmp.d32/ret sequence and the final 3 a byte sequence that spells out
'SCT'.
This sequence is used in __static_call_validate() to ensure it is
patching a trampoline and not a random other jmp.d32. That is,
false-positives shouldn't be plenty, but aren't a big concern.
OTOH the new __static_call_fixup() must not have false-positives, and
'SCT' decodes to the somewhat weird but semi plausible sequence:
push %rbx
rex.XB push %r12
Additionally, there are SLS concerns with immediate jumps. Combined it
seems like a good moment to change the signature to a single 3 byte
trap instruction that is unique to this usage and will not ever get
generated by accident.
As such, change the signature to: '0x0f, 0xb9, 0xcc', which decodes
to:
ud1 %esp, %ecx
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Borislav Petkov <bp@suse.de>
2022-06-15 05:15:39 +08:00
|
|
|
#endif
|
2020-08-18 21:57:48 +08:00
|
|
|
|
2022-03-14 18:27:35 +08:00
|
|
|
#define ARCH_DEFINE_STATIC_CALL_RET0_TRAMP(name) \
|
|
|
|
|
ARCH_DEFINE_STATIC_CALL_TRAMP(name, __static_call_return0)
|
2021-01-28 07:18:37 +08:00
|
|
|
|
|
|
|
|
#define ARCH_ADD_TRAMP_KEY(name) \
|
|
|
|
|
asm(".pushsection .static_call_tramp_key, \"a\" \n" \
|
|
|
|
|
".long " STATIC_CALL_TRAMP_STR(name) " - . \n" \
|
|
|
|
|
".long " STATIC_CALL_KEY_STR(name) " - . \n" \
|
|
|
|
|
".popsection \n")
|
|
|
|
|
|
x86,static_call: Use alternative RET encoding
In addition to teaching static_call about the new way to spell 'RET',
there is an added complication in that static_call() is allowed to
rewrite text before it is known which particular spelling is required.
In order to deal with this; have a static_call specific fixup in the
apply_return() 'alternative' patching routine that will rewrite the
static_call trampoline to match the definite sequence.
This in turn creates the problem of uniquely identifying static call
trampolines. Currently trampolines are 8 bytes, the first 5 being the
jmp.d32/ret sequence and the final 3 a byte sequence that spells out
'SCT'.
This sequence is used in __static_call_validate() to ensure it is
patching a trampoline and not a random other jmp.d32. That is,
false-positives shouldn't be plenty, but aren't a big concern.
OTOH the new __static_call_fixup() must not have false-positives, and
'SCT' decodes to the somewhat weird but semi plausible sequence:
push %rbx
rex.XB push %r12
Additionally, there are SLS concerns with immediate jumps. Combined it
seems like a good moment to change the signature to a single 3 byte
trap instruction that is unique to this usage and will not ever get
generated by accident.
As such, change the signature to: '0x0f, 0xb9, 0xcc', which decodes
to:
ud1 %esp, %ecx
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Josh Poimboeuf <jpoimboe@kernel.org>
Signed-off-by: Borislav Petkov <bp@suse.de>
2022-06-15 05:15:39 +08:00
|
|
|
extern bool __static_call_fixup(void *tramp, u8 op, void *dest);
|
|
|
|
|
|
2020-08-18 21:57:44 +08:00
|
|
|
#endif /* _ASM_STATIC_CALL_H */
|