2010-02-19 08:13:04 +08:00
|
|
|
/*
|
2015-10-21 16:54:38 +08:00
|
|
|
* Copyright (C) 2015 Imagination Technologies
|
|
|
|
* Author: Alex Smith <alex.smith@imgtec.com>
|
2010-02-19 08:13:04 +08:00
|
|
|
*
|
2015-10-21 16:54:38 +08:00
|
|
|
* This program is free software; you can redistribute it and/or modify it
|
|
|
|
* under the terms of the GNU General Public License as published by the
|
|
|
|
* Free Software Foundation; either version 2 of the License, or (at your
|
|
|
|
* option) any later version.
|
2010-02-19 08:13:04 +08:00
|
|
|
*/
|
|
|
|
|
|
|
|
#include <linux/binfmts.h>
|
|
|
|
#include <linux/elf.h>
|
2015-10-21 16:54:38 +08:00
|
|
|
#include <linux/err.h>
|
|
|
|
#include <linux/init.h>
|
MIPS: VDSO: Add implementations of gettimeofday() and clock_gettime()
Add user-mode implementations of gettimeofday() and clock_gettime() to
the VDSO. This is currently usable with 2 clocksources: the CP0 count
register, which is accessible to user-mode via RDHWR on R2 and later
cores, or the MIPS Global Interrupt Controller (GIC) timer, which
provides a "user-mode visible" section containing a mirror of its
counter registers. This section must be mapped into user memory, which
is done below the VDSO data page.
When a supported clocksource is not in use, the VDSO functions will
return -ENOSYS, which causes libc to fall back on the standard syscall
path.
When support for neither of these clocksources is compiled into the
kernel at all, the VDSO still provides clock_gettime(), as the coarse
realtime/monotonic clocks can still be implemented. However,
gettimeofday() is not provided in this case as nothing can be done
without a suitable clocksource. This causes the symbol lookup to fail
in libc and it will then always use the standard syscall path.
This patch includes a workaround for a bug in QEMU which results in
RDHWR on the CP0 count register always returning a constant (incorrect)
value. A fix for this has been submitted, and the workaround can be
removed after the fix has been in stable releases for a reasonable
amount of time.
A simple performance test which calls gettimeofday() 1000 times in a
loop and calculates the average execution time gives the following
results on a Malta + I6400 (running at 20MHz):
- Syscall: ~31000 ns
- VDSO (GIC): ~15000 ns
- VDSO (CP0): ~9500 ns
[markos.chandras@imgtec.com:
- Minor code re-arrangements in order for mappings to be made
in the order they appear to the process' address space.
- Move do_{monotonic, realtime} outside of the MIPS_CLOCK_VSYSCALL ifdef
- Use gic_get_usm_range so we can do the GIC mapping in the
arch/mips/kernel/vdso instead of the GIC irqchip driver]
Signed-off-by: Alex Smith <alex.smith@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/11338/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2015-10-21 16:57:44 +08:00
|
|
|
#include <linux/ioport.h>
|
|
|
|
#include <linux/irqchip/mips-gic.h>
|
2015-10-21 16:54:38 +08:00
|
|
|
#include <linux/mm.h>
|
|
|
|
#include <linux/sched.h>
|
|
|
|
#include <linux/slab.h>
|
MIPS: VDSO: Add implementations of gettimeofday() and clock_gettime()
Add user-mode implementations of gettimeofday() and clock_gettime() to
the VDSO. This is currently usable with 2 clocksources: the CP0 count
register, which is accessible to user-mode via RDHWR on R2 and later
cores, or the MIPS Global Interrupt Controller (GIC) timer, which
provides a "user-mode visible" section containing a mirror of its
counter registers. This section must be mapped into user memory, which
is done below the VDSO data page.
When a supported clocksource is not in use, the VDSO functions will
return -ENOSYS, which causes libc to fall back on the standard syscall
path.
When support for neither of these clocksources is compiled into the
kernel at all, the VDSO still provides clock_gettime(), as the coarse
realtime/monotonic clocks can still be implemented. However,
gettimeofday() is not provided in this case as nothing can be done
without a suitable clocksource. This causes the symbol lookup to fail
in libc and it will then always use the standard syscall path.
This patch includes a workaround for a bug in QEMU which results in
RDHWR on the CP0 count register always returning a constant (incorrect)
value. A fix for this has been submitted, and the workaround can be
removed after the fix has been in stable releases for a reasonable
amount of time.
A simple performance test which calls gettimeofday() 1000 times in a
loop and calculates the average execution time gives the following
results on a Malta + I6400 (running at 20MHz):
- Syscall: ~31000 ns
- VDSO (GIC): ~15000 ns
- VDSO (CP0): ~9500 ns
[markos.chandras@imgtec.com:
- Minor code re-arrangements in order for mappings to be made
in the order they appear to the process' address space.
- Move do_{monotonic, realtime} outside of the MIPS_CLOCK_VSYSCALL ifdef
- Use gic_get_usm_range so we can do the GIC mapping in the
arch/mips/kernel/vdso instead of the GIC irqchip driver]
Signed-off-by: Alex Smith <alex.smith@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/11338/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2015-10-21 16:57:44 +08:00
|
|
|
#include <linux/timekeeper_internal.h>
|
2010-02-19 08:13:04 +08:00
|
|
|
|
2015-10-21 16:54:38 +08:00
|
|
|
#include <asm/abi.h>
|
2010-02-19 08:13:04 +08:00
|
|
|
#include <asm/vdso.h>
|
2015-10-21 16:54:38 +08:00
|
|
|
|
|
|
|
/* Kernel-provided data used by the VDSO. */
|
|
|
|
static union mips_vdso_data vdso_data __page_aligned_data;
|
2010-02-19 08:13:04 +08:00
|
|
|
|
|
|
|
/*
|
MIPS: VDSO: Add implementations of gettimeofday() and clock_gettime()
Add user-mode implementations of gettimeofday() and clock_gettime() to
the VDSO. This is currently usable with 2 clocksources: the CP0 count
register, which is accessible to user-mode via RDHWR on R2 and later
cores, or the MIPS Global Interrupt Controller (GIC) timer, which
provides a "user-mode visible" section containing a mirror of its
counter registers. This section must be mapped into user memory, which
is done below the VDSO data page.
When a supported clocksource is not in use, the VDSO functions will
return -ENOSYS, which causes libc to fall back on the standard syscall
path.
When support for neither of these clocksources is compiled into the
kernel at all, the VDSO still provides clock_gettime(), as the coarse
realtime/monotonic clocks can still be implemented. However,
gettimeofday() is not provided in this case as nothing can be done
without a suitable clocksource. This causes the symbol lookup to fail
in libc and it will then always use the standard syscall path.
This patch includes a workaround for a bug in QEMU which results in
RDHWR on the CP0 count register always returning a constant (incorrect)
value. A fix for this has been submitted, and the workaround can be
removed after the fix has been in stable releases for a reasonable
amount of time.
A simple performance test which calls gettimeofday() 1000 times in a
loop and calculates the average execution time gives the following
results on a Malta + I6400 (running at 20MHz):
- Syscall: ~31000 ns
- VDSO (GIC): ~15000 ns
- VDSO (CP0): ~9500 ns
[markos.chandras@imgtec.com:
- Minor code re-arrangements in order for mappings to be made
in the order they appear to the process' address space.
- Move do_{monotonic, realtime} outside of the MIPS_CLOCK_VSYSCALL ifdef
- Use gic_get_usm_range so we can do the GIC mapping in the
arch/mips/kernel/vdso instead of the GIC irqchip driver]
Signed-off-by: Alex Smith <alex.smith@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/11338/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2015-10-21 16:57:44 +08:00
|
|
|
* Mapping for the VDSO data/GIC pages. The real pages are mapped manually, as
|
2015-10-21 16:54:38 +08:00
|
|
|
* what we map and where within the area they are mapped is determined at
|
|
|
|
* runtime.
|
2010-02-19 08:13:04 +08:00
|
|
|
*/
|
2015-10-21 16:54:38 +08:00
|
|
|
static struct page *no_pages[] = { NULL };
|
|
|
|
static struct vm_special_mapping vdso_vvar_mapping = {
|
|
|
|
.name = "[vvar]",
|
|
|
|
.pages = no_pages,
|
|
|
|
};
|
2010-02-19 08:13:04 +08:00
|
|
|
|
2015-10-21 16:54:38 +08:00
|
|
|
static void __init init_vdso_image(struct mips_vdso_image *image)
|
2010-02-19 08:13:04 +08:00
|
|
|
{
|
2015-10-21 16:54:38 +08:00
|
|
|
unsigned long num_pages, i;
|
2016-09-07 20:37:01 +08:00
|
|
|
unsigned long data_pfn;
|
2015-10-21 16:54:38 +08:00
|
|
|
|
|
|
|
BUG_ON(!PAGE_ALIGNED(image->data));
|
|
|
|
BUG_ON(!PAGE_ALIGNED(image->size));
|
|
|
|
|
|
|
|
num_pages = image->size / PAGE_SIZE;
|
|
|
|
|
2016-09-07 20:37:01 +08:00
|
|
|
data_pfn = __phys_to_pfn(__pa_symbol(image->data));
|
|
|
|
for (i = 0; i < num_pages; i++)
|
|
|
|
image->mapping.pages[i] = pfn_to_page(data_pfn + i);
|
2010-02-19 08:13:04 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static int __init init_vdso(void)
|
|
|
|
{
|
2015-10-21 16:54:38 +08:00
|
|
|
init_vdso_image(&vdso_image);
|
|
|
|
|
|
|
|
#ifdef CONFIG_MIPS32_O32
|
|
|
|
init_vdso_image(&vdso_image_o32);
|
2010-02-19 08:13:04 +08:00
|
|
|
#endif
|
|
|
|
|
2015-10-21 16:54:38 +08:00
|
|
|
#ifdef CONFIG_MIPS32_N32
|
|
|
|
init_vdso_image(&vdso_image_n32);
|
|
|
|
#endif
|
2010-02-19 08:13:04 +08:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
2010-06-17 06:00:28 +08:00
|
|
|
subsys_initcall(init_vdso);
|
2010-02-19 08:13:04 +08:00
|
|
|
|
MIPS: VDSO: Add implementations of gettimeofday() and clock_gettime()
Add user-mode implementations of gettimeofday() and clock_gettime() to
the VDSO. This is currently usable with 2 clocksources: the CP0 count
register, which is accessible to user-mode via RDHWR on R2 and later
cores, or the MIPS Global Interrupt Controller (GIC) timer, which
provides a "user-mode visible" section containing a mirror of its
counter registers. This section must be mapped into user memory, which
is done below the VDSO data page.
When a supported clocksource is not in use, the VDSO functions will
return -ENOSYS, which causes libc to fall back on the standard syscall
path.
When support for neither of these clocksources is compiled into the
kernel at all, the VDSO still provides clock_gettime(), as the coarse
realtime/monotonic clocks can still be implemented. However,
gettimeofday() is not provided in this case as nothing can be done
without a suitable clocksource. This causes the symbol lookup to fail
in libc and it will then always use the standard syscall path.
This patch includes a workaround for a bug in QEMU which results in
RDHWR on the CP0 count register always returning a constant (incorrect)
value. A fix for this has been submitted, and the workaround can be
removed after the fix has been in stable releases for a reasonable
amount of time.
A simple performance test which calls gettimeofday() 1000 times in a
loop and calculates the average execution time gives the following
results on a Malta + I6400 (running at 20MHz):
- Syscall: ~31000 ns
- VDSO (GIC): ~15000 ns
- VDSO (CP0): ~9500 ns
[markos.chandras@imgtec.com:
- Minor code re-arrangements in order for mappings to be made
in the order they appear to the process' address space.
- Move do_{monotonic, realtime} outside of the MIPS_CLOCK_VSYSCALL ifdef
- Use gic_get_usm_range so we can do the GIC mapping in the
arch/mips/kernel/vdso instead of the GIC irqchip driver]
Signed-off-by: Alex Smith <alex.smith@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/11338/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2015-10-21 16:57:44 +08:00
|
|
|
void update_vsyscall(struct timekeeper *tk)
|
|
|
|
{
|
|
|
|
vdso_data_write_begin(&vdso_data);
|
|
|
|
|
|
|
|
vdso_data.xtime_sec = tk->xtime_sec;
|
|
|
|
vdso_data.xtime_nsec = tk->tkr_mono.xtime_nsec;
|
|
|
|
vdso_data.wall_to_mono_sec = tk->wall_to_monotonic.tv_sec;
|
|
|
|
vdso_data.wall_to_mono_nsec = tk->wall_to_monotonic.tv_nsec;
|
|
|
|
vdso_data.cs_shift = tk->tkr_mono.shift;
|
|
|
|
|
|
|
|
vdso_data.clock_mode = tk->tkr_mono.clock->archdata.vdso_clock_mode;
|
|
|
|
if (vdso_data.clock_mode != VDSO_CLOCK_NONE) {
|
|
|
|
vdso_data.cs_mult = tk->tkr_mono.mult;
|
|
|
|
vdso_data.cs_cycle_last = tk->tkr_mono.cycle_last;
|
|
|
|
vdso_data.cs_mask = tk->tkr_mono.mask;
|
|
|
|
}
|
|
|
|
|
|
|
|
vdso_data_write_end(&vdso_data);
|
|
|
|
}
|
|
|
|
|
|
|
|
void update_vsyscall_tz(void)
|
|
|
|
{
|
|
|
|
if (vdso_data.clock_mode != VDSO_CLOCK_NONE) {
|
|
|
|
vdso_data.tz_minuteswest = sys_tz.tz_minuteswest;
|
|
|
|
vdso_data.tz_dsttime = sys_tz.tz_dsttime;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2010-02-19 08:13:04 +08:00
|
|
|
int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)
|
|
|
|
{
|
2015-10-21 16:54:38 +08:00
|
|
|
struct mips_vdso_image *image = current->thread.abi->vdso;
|
2010-02-19 08:13:04 +08:00
|
|
|
struct mm_struct *mm = current->mm;
|
MIPS: VDSO: Add implementations of gettimeofday() and clock_gettime()
Add user-mode implementations of gettimeofday() and clock_gettime() to
the VDSO. This is currently usable with 2 clocksources: the CP0 count
register, which is accessible to user-mode via RDHWR on R2 and later
cores, or the MIPS Global Interrupt Controller (GIC) timer, which
provides a "user-mode visible" section containing a mirror of its
counter registers. This section must be mapped into user memory, which
is done below the VDSO data page.
When a supported clocksource is not in use, the VDSO functions will
return -ENOSYS, which causes libc to fall back on the standard syscall
path.
When support for neither of these clocksources is compiled into the
kernel at all, the VDSO still provides clock_gettime(), as the coarse
realtime/monotonic clocks can still be implemented. However,
gettimeofday() is not provided in this case as nothing can be done
without a suitable clocksource. This causes the symbol lookup to fail
in libc and it will then always use the standard syscall path.
This patch includes a workaround for a bug in QEMU which results in
RDHWR on the CP0 count register always returning a constant (incorrect)
value. A fix for this has been submitted, and the workaround can be
removed after the fix has been in stable releases for a reasonable
amount of time.
A simple performance test which calls gettimeofday() 1000 times in a
loop and calculates the average execution time gives the following
results on a Malta + I6400 (running at 20MHz):
- Syscall: ~31000 ns
- VDSO (GIC): ~15000 ns
- VDSO (CP0): ~9500 ns
[markos.chandras@imgtec.com:
- Minor code re-arrangements in order for mappings to be made
in the order they appear to the process' address space.
- Move do_{monotonic, realtime} outside of the MIPS_CLOCK_VSYSCALL ifdef
- Use gic_get_usm_range so we can do the GIC mapping in the
arch/mips/kernel/vdso instead of the GIC irqchip driver]
Signed-off-by: Alex Smith <alex.smith@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/11338/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2015-10-21 16:57:44 +08:00
|
|
|
unsigned long gic_size, vvar_size, size, base, data_addr, vdso_addr;
|
2015-10-21 16:54:38 +08:00
|
|
|
struct vm_area_struct *vma;
|
MIPS: VDSO: Add implementations of gettimeofday() and clock_gettime()
Add user-mode implementations of gettimeofday() and clock_gettime() to
the VDSO. This is currently usable with 2 clocksources: the CP0 count
register, which is accessible to user-mode via RDHWR on R2 and later
cores, or the MIPS Global Interrupt Controller (GIC) timer, which
provides a "user-mode visible" section containing a mirror of its
counter registers. This section must be mapped into user memory, which
is done below the VDSO data page.
When a supported clocksource is not in use, the VDSO functions will
return -ENOSYS, which causes libc to fall back on the standard syscall
path.
When support for neither of these clocksources is compiled into the
kernel at all, the VDSO still provides clock_gettime(), as the coarse
realtime/monotonic clocks can still be implemented. However,
gettimeofday() is not provided in this case as nothing can be done
without a suitable clocksource. This causes the symbol lookup to fail
in libc and it will then always use the standard syscall path.
This patch includes a workaround for a bug in QEMU which results in
RDHWR on the CP0 count register always returning a constant (incorrect)
value. A fix for this has been submitted, and the workaround can be
removed after the fix has been in stable releases for a reasonable
amount of time.
A simple performance test which calls gettimeofday() 1000 times in a
loop and calculates the average execution time gives the following
results on a Malta + I6400 (running at 20MHz):
- Syscall: ~31000 ns
- VDSO (GIC): ~15000 ns
- VDSO (CP0): ~9500 ns
[markos.chandras@imgtec.com:
- Minor code re-arrangements in order for mappings to be made
in the order they appear to the process' address space.
- Move do_{monotonic, realtime} outside of the MIPS_CLOCK_VSYSCALL ifdef
- Use gic_get_usm_range so we can do the GIC mapping in the
arch/mips/kernel/vdso instead of the GIC irqchip driver]
Signed-off-by: Alex Smith <alex.smith@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/11338/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2015-10-21 16:57:44 +08:00
|
|
|
struct resource gic_res;
|
2015-10-21 16:54:38 +08:00
|
|
|
int ret;
|
2010-02-19 08:13:04 +08:00
|
|
|
|
2016-05-24 07:25:54 +08:00
|
|
|
if (down_write_killable(&mm->mmap_sem))
|
|
|
|
return -EINTR;
|
2010-02-19 08:13:04 +08:00
|
|
|
|
MIPS: Use per-mm page to execute branch delay slot instructions
In some cases the kernel needs to execute an instruction from the delay
slot of an emulated branch instruction. These cases include:
- Emulated floating point branch instructions (bc1[ft]l?) for systems
which don't include an FPU, or upon which the kernel is run with the
"nofpu" parameter.
- MIPSr6 systems running binaries targeting older revisions of the
architecture, which may include branch instructions whose encodings
are no longer valid in MIPSr6.
Executing instructions from such delay slots is done by writing the
instruction to memory followed by a trap, as part of an "emuframe", and
executing it. This avoids the requirement of an emulator for the entire
MIPS instruction set. Prior to this patch such emuframes are written to
the user stack and executed from there.
This patch moves FP branch delay emuframes off of the user stack and
into a per-mm page. Allocating a page per-mm leaves userland with access
to only what it had access to previously, and compared to other
solutions is relatively simple.
When a thread requires a delay slot emulation, it is allocated a frame.
A thread may only have one frame allocated at any one time, since it may
only ever be executing one instruction at any one time. In order to
ensure that we can free up allocated frame later, its index is recorded
in struct thread_struct. In the typical case, after executing the delay
slot instruction we'll execute a break instruction with the BRK_MEMU
code. This traps back to the kernel & leads to a call to do_dsemulret
which frees the allocated frame & moves the user PC back to the
instruction that would have executed following the emulated branch.
In some cases the delay slot instruction may be invalid, such as a
branch, or may trigger an exception. In these cases the BRK_MEMU break
instruction will not be hit. In order to ensure that frames are freed
this patch introduces dsemul_thread_cleanup() and calls it to free any
allocated frame upon thread exit. If the instruction generated an
exception & leads to a signal being delivered to the thread, or indeed
if a signal simply happens to be delivered to the thread whilst it is
executing from the struct emuframe, then we need to take care to exit
the frame appropriately. This is done by either rolling back the user PC
to the branch or advancing it to the continuation PC prior to signal
delivery, using dsemul_thread_rollback(). If this were not done then a
sigreturn would return to the struct emuframe, and if that frame had
meanwhile been used in response to an emulated branch instruction within
the signal handler then we would execute the wrong user code.
Whilst a user could theoretically place something like a compact branch
to self in a delay slot and cause their thread to become stuck in an
infinite loop with the frame never being deallocated, this would:
- Only affect the users single process.
- Be architecturally invalid since there would be a branch in the
delay slot, which is forbidden.
- Be extremely unlikely to happen by mistake, and provide a program
with no more ability to harm the system than a simple infinite loop
would.
If a thread requires a delay slot emulation & no frame is available to
it (ie. the process has enough other threads that all frames are
currently in use) then the thread joins a waitqueue. It will sleep until
a frame is freed by another thread in the process.
Since we now know whether a thread has an allocated frame due to our
tracking of its index, the cookie field of struct emuframe is removed as
we can be more certain whether we have a valid frame. Since a thread may
only ever have a single frame at any given time, the epc field of struct
emuframe is also removed & the PC to continue from is instead stored in
struct thread_struct. Together these changes simplify & shrink struct
emuframe somewhat, allowing twice as many frames to fit into the page
allocated for them.
The primary benefit of this patch is that we are now free to mark the
user stack non-executable where that is possible.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: Maciej Rozycki <maciej.rozycki@imgtec.com>
Cc: Faraz Shahbazker <faraz.shahbazker@imgtec.com>
Cc: Raghu Gandham <raghu.gandham@imgtec.com>
Cc: Matthew Fortune <matthew.fortune@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/13764/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2016-07-08 18:06:19 +08:00
|
|
|
/* Map delay slot emulation page */
|
|
|
|
base = mmap_region(NULL, STACK_TOP, PAGE_SIZE,
|
|
|
|
VM_READ|VM_WRITE|VM_EXEC|
|
|
|
|
VM_MAYREAD|VM_MAYWRITE|VM_MAYEXEC,
|
2017-02-25 06:58:22 +08:00
|
|
|
0, NULL);
|
MIPS: Use per-mm page to execute branch delay slot instructions
In some cases the kernel needs to execute an instruction from the delay
slot of an emulated branch instruction. These cases include:
- Emulated floating point branch instructions (bc1[ft]l?) for systems
which don't include an FPU, or upon which the kernel is run with the
"nofpu" parameter.
- MIPSr6 systems running binaries targeting older revisions of the
architecture, which may include branch instructions whose encodings
are no longer valid in MIPSr6.
Executing instructions from such delay slots is done by writing the
instruction to memory followed by a trap, as part of an "emuframe", and
executing it. This avoids the requirement of an emulator for the entire
MIPS instruction set. Prior to this patch such emuframes are written to
the user stack and executed from there.
This patch moves FP branch delay emuframes off of the user stack and
into a per-mm page. Allocating a page per-mm leaves userland with access
to only what it had access to previously, and compared to other
solutions is relatively simple.
When a thread requires a delay slot emulation, it is allocated a frame.
A thread may only have one frame allocated at any one time, since it may
only ever be executing one instruction at any one time. In order to
ensure that we can free up allocated frame later, its index is recorded
in struct thread_struct. In the typical case, after executing the delay
slot instruction we'll execute a break instruction with the BRK_MEMU
code. This traps back to the kernel & leads to a call to do_dsemulret
which frees the allocated frame & moves the user PC back to the
instruction that would have executed following the emulated branch.
In some cases the delay slot instruction may be invalid, such as a
branch, or may trigger an exception. In these cases the BRK_MEMU break
instruction will not be hit. In order to ensure that frames are freed
this patch introduces dsemul_thread_cleanup() and calls it to free any
allocated frame upon thread exit. If the instruction generated an
exception & leads to a signal being delivered to the thread, or indeed
if a signal simply happens to be delivered to the thread whilst it is
executing from the struct emuframe, then we need to take care to exit
the frame appropriately. This is done by either rolling back the user PC
to the branch or advancing it to the continuation PC prior to signal
delivery, using dsemul_thread_rollback(). If this were not done then a
sigreturn would return to the struct emuframe, and if that frame had
meanwhile been used in response to an emulated branch instruction within
the signal handler then we would execute the wrong user code.
Whilst a user could theoretically place something like a compact branch
to self in a delay slot and cause their thread to become stuck in an
infinite loop with the frame never being deallocated, this would:
- Only affect the users single process.
- Be architecturally invalid since there would be a branch in the
delay slot, which is forbidden.
- Be extremely unlikely to happen by mistake, and provide a program
with no more ability to harm the system than a simple infinite loop
would.
If a thread requires a delay slot emulation & no frame is available to
it (ie. the process has enough other threads that all frames are
currently in use) then the thread joins a waitqueue. It will sleep until
a frame is freed by another thread in the process.
Since we now know whether a thread has an allocated frame due to our
tracking of its index, the cookie field of struct emuframe is removed as
we can be more certain whether we have a valid frame. Since a thread may
only ever have a single frame at any given time, the epc field of struct
emuframe is also removed & the PC to continue from is instead stored in
struct thread_struct. Together these changes simplify & shrink struct
emuframe somewhat, allowing twice as many frames to fit into the page
allocated for them.
The primary benefit of this patch is that we are now free to mark the
user stack non-executable where that is possible.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: Maciej Rozycki <maciej.rozycki@imgtec.com>
Cc: Faraz Shahbazker <faraz.shahbazker@imgtec.com>
Cc: Raghu Gandham <raghu.gandham@imgtec.com>
Cc: Matthew Fortune <matthew.fortune@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/13764/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2016-07-08 18:06:19 +08:00
|
|
|
if (IS_ERR_VALUE(base)) {
|
|
|
|
ret = base;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
MIPS: VDSO: Add implementations of gettimeofday() and clock_gettime()
Add user-mode implementations of gettimeofday() and clock_gettime() to
the VDSO. This is currently usable with 2 clocksources: the CP0 count
register, which is accessible to user-mode via RDHWR on R2 and later
cores, or the MIPS Global Interrupt Controller (GIC) timer, which
provides a "user-mode visible" section containing a mirror of its
counter registers. This section must be mapped into user memory, which
is done below the VDSO data page.
When a supported clocksource is not in use, the VDSO functions will
return -ENOSYS, which causes libc to fall back on the standard syscall
path.
When support for neither of these clocksources is compiled into the
kernel at all, the VDSO still provides clock_gettime(), as the coarse
realtime/monotonic clocks can still be implemented. However,
gettimeofday() is not provided in this case as nothing can be done
without a suitable clocksource. This causes the symbol lookup to fail
in libc and it will then always use the standard syscall path.
This patch includes a workaround for a bug in QEMU which results in
RDHWR on the CP0 count register always returning a constant (incorrect)
value. A fix for this has been submitted, and the workaround can be
removed after the fix has been in stable releases for a reasonable
amount of time.
A simple performance test which calls gettimeofday() 1000 times in a
loop and calculates the average execution time gives the following
results on a Malta + I6400 (running at 20MHz):
- Syscall: ~31000 ns
- VDSO (GIC): ~15000 ns
- VDSO (CP0): ~9500 ns
[markos.chandras@imgtec.com:
- Minor code re-arrangements in order for mappings to be made
in the order they appear to the process' address space.
- Move do_{monotonic, realtime} outside of the MIPS_CLOCK_VSYSCALL ifdef
- Use gic_get_usm_range so we can do the GIC mapping in the
arch/mips/kernel/vdso instead of the GIC irqchip driver]
Signed-off-by: Alex Smith <alex.smith@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/11338/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2015-10-21 16:57:44 +08:00
|
|
|
/*
|
|
|
|
* Determine total area size. This includes the VDSO data itself, the
|
|
|
|
* data page, and the GIC user page if present. Always create a mapping
|
|
|
|
* for the GIC user area if the GIC is present regardless of whether it
|
|
|
|
* is the current clocksource, in case it comes into use later on. We
|
|
|
|
* only map a page even though the total area is 64K, as we only need
|
|
|
|
* the counter registers at the start.
|
|
|
|
*/
|
|
|
|
gic_size = gic_present ? PAGE_SIZE : 0;
|
|
|
|
vvar_size = gic_size + PAGE_SIZE;
|
|
|
|
size = vvar_size + image->size;
|
|
|
|
|
|
|
|
base = get_unmapped_area(NULL, 0, size, 0, 0);
|
2015-10-21 16:54:38 +08:00
|
|
|
if (IS_ERR_VALUE(base)) {
|
|
|
|
ret = base;
|
|
|
|
goto out;
|
2010-02-19 08:13:04 +08:00
|
|
|
}
|
|
|
|
|
MIPS: VDSO: Add implementations of gettimeofday() and clock_gettime()
Add user-mode implementations of gettimeofday() and clock_gettime() to
the VDSO. This is currently usable with 2 clocksources: the CP0 count
register, which is accessible to user-mode via RDHWR on R2 and later
cores, or the MIPS Global Interrupt Controller (GIC) timer, which
provides a "user-mode visible" section containing a mirror of its
counter registers. This section must be mapped into user memory, which
is done below the VDSO data page.
When a supported clocksource is not in use, the VDSO functions will
return -ENOSYS, which causes libc to fall back on the standard syscall
path.
When support for neither of these clocksources is compiled into the
kernel at all, the VDSO still provides clock_gettime(), as the coarse
realtime/monotonic clocks can still be implemented. However,
gettimeofday() is not provided in this case as nothing can be done
without a suitable clocksource. This causes the symbol lookup to fail
in libc and it will then always use the standard syscall path.
This patch includes a workaround for a bug in QEMU which results in
RDHWR on the CP0 count register always returning a constant (incorrect)
value. A fix for this has been submitted, and the workaround can be
removed after the fix has been in stable releases for a reasonable
amount of time.
A simple performance test which calls gettimeofday() 1000 times in a
loop and calculates the average execution time gives the following
results on a Malta + I6400 (running at 20MHz):
- Syscall: ~31000 ns
- VDSO (GIC): ~15000 ns
- VDSO (CP0): ~9500 ns
[markos.chandras@imgtec.com:
- Minor code re-arrangements in order for mappings to be made
in the order they appear to the process' address space.
- Move do_{monotonic, realtime} outside of the MIPS_CLOCK_VSYSCALL ifdef
- Use gic_get_usm_range so we can do the GIC mapping in the
arch/mips/kernel/vdso instead of the GIC irqchip driver]
Signed-off-by: Alex Smith <alex.smith@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/11338/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2015-10-21 16:57:44 +08:00
|
|
|
data_addr = base + gic_size;
|
|
|
|
vdso_addr = data_addr + PAGE_SIZE;
|
2015-10-21 16:54:38 +08:00
|
|
|
|
MIPS: VDSO: Add implementations of gettimeofday() and clock_gettime()
Add user-mode implementations of gettimeofday() and clock_gettime() to
the VDSO. This is currently usable with 2 clocksources: the CP0 count
register, which is accessible to user-mode via RDHWR on R2 and later
cores, or the MIPS Global Interrupt Controller (GIC) timer, which
provides a "user-mode visible" section containing a mirror of its
counter registers. This section must be mapped into user memory, which
is done below the VDSO data page.
When a supported clocksource is not in use, the VDSO functions will
return -ENOSYS, which causes libc to fall back on the standard syscall
path.
When support for neither of these clocksources is compiled into the
kernel at all, the VDSO still provides clock_gettime(), as the coarse
realtime/monotonic clocks can still be implemented. However,
gettimeofday() is not provided in this case as nothing can be done
without a suitable clocksource. This causes the symbol lookup to fail
in libc and it will then always use the standard syscall path.
This patch includes a workaround for a bug in QEMU which results in
RDHWR on the CP0 count register always returning a constant (incorrect)
value. A fix for this has been submitted, and the workaround can be
removed after the fix has been in stable releases for a reasonable
amount of time.
A simple performance test which calls gettimeofday() 1000 times in a
loop and calculates the average execution time gives the following
results on a Malta + I6400 (running at 20MHz):
- Syscall: ~31000 ns
- VDSO (GIC): ~15000 ns
- VDSO (CP0): ~9500 ns
[markos.chandras@imgtec.com:
- Minor code re-arrangements in order for mappings to be made
in the order they appear to the process' address space.
- Move do_{monotonic, realtime} outside of the MIPS_CLOCK_VSYSCALL ifdef
- Use gic_get_usm_range so we can do the GIC mapping in the
arch/mips/kernel/vdso instead of the GIC irqchip driver]
Signed-off-by: Alex Smith <alex.smith@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/11338/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2015-10-21 16:57:44 +08:00
|
|
|
vma = _install_special_mapping(mm, base, vvar_size,
|
2015-10-21 16:54:38 +08:00
|
|
|
VM_READ | VM_MAYREAD,
|
|
|
|
&vdso_vvar_mapping);
|
|
|
|
if (IS_ERR(vma)) {
|
|
|
|
ret = PTR_ERR(vma);
|
|
|
|
goto out;
|
|
|
|
}
|
2010-02-19 08:13:04 +08:00
|
|
|
|
MIPS: VDSO: Add implementations of gettimeofday() and clock_gettime()
Add user-mode implementations of gettimeofday() and clock_gettime() to
the VDSO. This is currently usable with 2 clocksources: the CP0 count
register, which is accessible to user-mode via RDHWR on R2 and later
cores, or the MIPS Global Interrupt Controller (GIC) timer, which
provides a "user-mode visible" section containing a mirror of its
counter registers. This section must be mapped into user memory, which
is done below the VDSO data page.
When a supported clocksource is not in use, the VDSO functions will
return -ENOSYS, which causes libc to fall back on the standard syscall
path.
When support for neither of these clocksources is compiled into the
kernel at all, the VDSO still provides clock_gettime(), as the coarse
realtime/monotonic clocks can still be implemented. However,
gettimeofday() is not provided in this case as nothing can be done
without a suitable clocksource. This causes the symbol lookup to fail
in libc and it will then always use the standard syscall path.
This patch includes a workaround for a bug in QEMU which results in
RDHWR on the CP0 count register always returning a constant (incorrect)
value. A fix for this has been submitted, and the workaround can be
removed after the fix has been in stable releases for a reasonable
amount of time.
A simple performance test which calls gettimeofday() 1000 times in a
loop and calculates the average execution time gives the following
results on a Malta + I6400 (running at 20MHz):
- Syscall: ~31000 ns
- VDSO (GIC): ~15000 ns
- VDSO (CP0): ~9500 ns
[markos.chandras@imgtec.com:
- Minor code re-arrangements in order for mappings to be made
in the order they appear to the process' address space.
- Move do_{monotonic, realtime} outside of the MIPS_CLOCK_VSYSCALL ifdef
- Use gic_get_usm_range so we can do the GIC mapping in the
arch/mips/kernel/vdso instead of the GIC irqchip driver]
Signed-off-by: Alex Smith <alex.smith@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/11338/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2015-10-21 16:57:44 +08:00
|
|
|
/* Map GIC user page. */
|
|
|
|
if (gic_size) {
|
|
|
|
ret = gic_get_usm_range(&gic_res);
|
|
|
|
if (ret)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
ret = io_remap_pfn_range(vma, base,
|
|
|
|
gic_res.start >> PAGE_SHIFT,
|
|
|
|
gic_size,
|
|
|
|
pgprot_noncached(PAGE_READONLY));
|
|
|
|
if (ret)
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
2015-10-21 16:54:38 +08:00
|
|
|
/* Map data page. */
|
MIPS: VDSO: Add implementations of gettimeofday() and clock_gettime()
Add user-mode implementations of gettimeofday() and clock_gettime() to
the VDSO. This is currently usable with 2 clocksources: the CP0 count
register, which is accessible to user-mode via RDHWR on R2 and later
cores, or the MIPS Global Interrupt Controller (GIC) timer, which
provides a "user-mode visible" section containing a mirror of its
counter registers. This section must be mapped into user memory, which
is done below the VDSO data page.
When a supported clocksource is not in use, the VDSO functions will
return -ENOSYS, which causes libc to fall back on the standard syscall
path.
When support for neither of these clocksources is compiled into the
kernel at all, the VDSO still provides clock_gettime(), as the coarse
realtime/monotonic clocks can still be implemented. However,
gettimeofday() is not provided in this case as nothing can be done
without a suitable clocksource. This causes the symbol lookup to fail
in libc and it will then always use the standard syscall path.
This patch includes a workaround for a bug in QEMU which results in
RDHWR on the CP0 count register always returning a constant (incorrect)
value. A fix for this has been submitted, and the workaround can be
removed after the fix has been in stable releases for a reasonable
amount of time.
A simple performance test which calls gettimeofday() 1000 times in a
loop and calculates the average execution time gives the following
results on a Malta + I6400 (running at 20MHz):
- Syscall: ~31000 ns
- VDSO (GIC): ~15000 ns
- VDSO (CP0): ~9500 ns
[markos.chandras@imgtec.com:
- Minor code re-arrangements in order for mappings to be made
in the order they appear to the process' address space.
- Move do_{monotonic, realtime} outside of the MIPS_CLOCK_VSYSCALL ifdef
- Use gic_get_usm_range so we can do the GIC mapping in the
arch/mips/kernel/vdso instead of the GIC irqchip driver]
Signed-off-by: Alex Smith <alex.smith@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/11338/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2015-10-21 16:57:44 +08:00
|
|
|
ret = remap_pfn_range(vma, data_addr,
|
2015-10-21 16:54:38 +08:00
|
|
|
virt_to_phys(&vdso_data) >> PAGE_SHIFT,
|
|
|
|
PAGE_SIZE, PAGE_READONLY);
|
2010-02-19 08:13:04 +08:00
|
|
|
if (ret)
|
2015-10-21 16:54:38 +08:00
|
|
|
goto out;
|
|
|
|
|
|
|
|
/* Map VDSO image. */
|
|
|
|
vma = _install_special_mapping(mm, vdso_addr, image->size,
|
|
|
|
VM_READ | VM_EXEC |
|
|
|
|
VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC,
|
|
|
|
&image->mapping);
|
|
|
|
if (IS_ERR(vma)) {
|
|
|
|
ret = PTR_ERR(vma);
|
|
|
|
goto out;
|
|
|
|
}
|
2010-02-19 08:13:04 +08:00
|
|
|
|
2015-10-21 16:54:38 +08:00
|
|
|
mm->context.vdso = (void *)vdso_addr;
|
|
|
|
ret = 0;
|
2010-02-19 08:13:04 +08:00
|
|
|
|
2015-10-21 16:54:38 +08:00
|
|
|
out:
|
2010-02-19 08:13:04 +08:00
|
|
|
up_write(&mm->mmap_sem);
|
|
|
|
return ret;
|
|
|
|
}
|