ARM: 8331/1: VDSO initialization, mapping, and synchronization
Initialize the VDSO page list at boot, install the VDSO mapping at
exec time, and update the data page during timer ticks. This code is
not built if CONFIG_VDSO is not enabled.
Account for the VDSO length when randomizing the offset from the
stack. The [vdso] and [vvar] pages are placed immediately following
the sigpage with separate _install_special_mapping calls.
We want to "penalize" systems lacking the arch timer as little
as possible. Previous versions of this code installed the VDSO
unconditionally and unmodified, making it a measurably slower way for
glibc to invoke the real syscalls on such systems. E.g. calling
gettimeofday via glibc goes from ~560ns to ~630ns on i.MX6Q.
If we can indicate to glibc that the time-related APIs in the VDSO are
not accelerated, glibc can continue to invoke the syscalls directly
instead of dispatching through the VDSO only to fall back to the slow
path.
Thus, if the architected timer is unusable for whatever reason, patch
the VDSO at boot time so that symbol lookups for gettimeofday and
clock_gettime return NULL. (This is similar to what powerpc does and
borrows code from there.) This allows glibc to perform the syscall
directly instead of passing control to the VDSO, which minimizes the
penalty. In my measurements the time taken for a gettimeofday call
via glibc goes from ~560ns to ~580ns (again on i.MX6Q), and this is
solely due to adding a test and branch to glibc's gettimeofday syscall
wrapper.
An alternative to patching the VDSO at boot would be to not install
the VDSO at all when the arch timer isn't usable. Another alternative
is to include a separate "dummy" vdso.so without gettimeofday and
clock_gettime, which would be selected at boot time. Either of these
would get cumbersome if the VDSO were to gain support for an API such
as getcpu which is unrelated to arch timer support.
Signed-off-by: Nathan Lynch <nathan_lynch@mentor.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-03-26 02:15:08 +08:00
|
|
|
/*
|
|
|
|
* Adapted from arm64 version.
|
|
|
|
*
|
|
|
|
* Copyright (C) 2012 ARM Limited
|
|
|
|
* Copyright (C) 2015 Mentor Graphics Corporation.
|
|
|
|
*
|
|
|
|
* This program is free software; you can redistribute it and/or modify
|
|
|
|
* it under the terms of the GNU General Public License version 2 as
|
|
|
|
* published by the Free Software Foundation.
|
|
|
|
*
|
|
|
|
* This program is distributed in the hope that it will be useful,
|
|
|
|
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
* GNU General Public License for more details.
|
|
|
|
*
|
|
|
|
* You should have received a copy of the GNU General Public License
|
|
|
|
* along with this program. If not, see <http://www.gnu.org/licenses/>.
|
|
|
|
*/
|
|
|
|
|
2016-08-15 16:20:21 +08:00
|
|
|
#include <linux/cache.h>
|
ARM: 8331/1: VDSO initialization, mapping, and synchronization
Initialize the VDSO page list at boot, install the VDSO mapping at
exec time, and update the data page during timer ticks. This code is
not built if CONFIG_VDSO is not enabled.
Account for the VDSO length when randomizing the offset from the
stack. The [vdso] and [vvar] pages are placed immediately following
the sigpage with separate _install_special_mapping calls.
We want to "penalize" systems lacking the arch timer as little
as possible. Previous versions of this code installed the VDSO
unconditionally and unmodified, making it a measurably slower way for
glibc to invoke the real syscalls on such systems. E.g. calling
gettimeofday via glibc goes from ~560ns to ~630ns on i.MX6Q.
If we can indicate to glibc that the time-related APIs in the VDSO are
not accelerated, glibc can continue to invoke the syscalls directly
instead of dispatching through the VDSO only to fall back to the slow
path.
Thus, if the architected timer is unusable for whatever reason, patch
the VDSO at boot time so that symbol lookups for gettimeofday and
clock_gettime return NULL. (This is similar to what powerpc does and
borrows code from there.) This allows glibc to perform the syscall
directly instead of passing control to the VDSO, which minimizes the
penalty. In my measurements the time taken for a gettimeofday call
via glibc goes from ~560ns to ~580ns (again on i.MX6Q), and this is
solely due to adding a test and branch to glibc's gettimeofday syscall
wrapper.
An alternative to patching the VDSO at boot would be to not install
the VDSO at all when the arch timer isn't usable. Another alternative
is to include a separate "dummy" vdso.so without gettimeofday and
clock_gettime, which would be selected at boot time. Either of these
would get cumbersome if the VDSO were to gain support for an API such
as getcpu which is unrelated to arch timer support.
Signed-off-by: Nathan Lynch <nathan_lynch@mentor.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-03-26 02:15:08 +08:00
|
|
|
#include <linux/elf.h>
|
|
|
|
#include <linux/err.h>
|
|
|
|
#include <linux/kernel.h>
|
|
|
|
#include <linux/mm.h>
|
|
|
|
#include <linux/of.h>
|
|
|
|
#include <linux/printk.h>
|
|
|
|
#include <linux/slab.h>
|
|
|
|
#include <linux/timekeeper_internal.h>
|
|
|
|
#include <linux/vmalloc.h>
|
|
|
|
#include <asm/arch_timer.h>
|
|
|
|
#include <asm/barrier.h>
|
|
|
|
#include <asm/cacheflush.h>
|
|
|
|
#include <asm/page.h>
|
|
|
|
#include <asm/vdso.h>
|
|
|
|
#include <asm/vdso_datapage.h>
|
|
|
|
#include <clocksource/arm_arch_timer.h>
|
|
|
|
|
|
|
|
#define MAX_SYMNAME 64
|
|
|
|
|
|
|
|
static struct page **vdso_text_pagelist;
|
|
|
|
|
|
|
|
/* Total number of pages needed for the data and text portions of the VDSO. */
|
2016-08-15 16:20:21 +08:00
|
|
|
unsigned int vdso_total_pages __ro_after_init;
|
ARM: 8331/1: VDSO initialization, mapping, and synchronization
Initialize the VDSO page list at boot, install the VDSO mapping at
exec time, and update the data page during timer ticks. This code is
not built if CONFIG_VDSO is not enabled.
Account for the VDSO length when randomizing the offset from the
stack. The [vdso] and [vvar] pages are placed immediately following
the sigpage with separate _install_special_mapping calls.
We want to "penalize" systems lacking the arch timer as little
as possible. Previous versions of this code installed the VDSO
unconditionally and unmodified, making it a measurably slower way for
glibc to invoke the real syscalls on such systems. E.g. calling
gettimeofday via glibc goes from ~560ns to ~630ns on i.MX6Q.
If we can indicate to glibc that the time-related APIs in the VDSO are
not accelerated, glibc can continue to invoke the syscalls directly
instead of dispatching through the VDSO only to fall back to the slow
path.
Thus, if the architected timer is unusable for whatever reason, patch
the VDSO at boot time so that symbol lookups for gettimeofday and
clock_gettime return NULL. (This is similar to what powerpc does and
borrows code from there.) This allows glibc to perform the syscall
directly instead of passing control to the VDSO, which minimizes the
penalty. In my measurements the time taken for a gettimeofday call
via glibc goes from ~560ns to ~580ns (again on i.MX6Q), and this is
solely due to adding a test and branch to glibc's gettimeofday syscall
wrapper.
An alternative to patching the VDSO at boot would be to not install
the VDSO at all when the arch timer isn't usable. Another alternative
is to include a separate "dummy" vdso.so without gettimeofday and
clock_gettime, which would be selected at boot time. Either of these
would get cumbersome if the VDSO were to gain support for an API such
as getcpu which is unrelated to arch timer support.
Signed-off-by: Nathan Lynch <nathan_lynch@mentor.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-03-26 02:15:08 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The VDSO data page.
|
|
|
|
*/
|
|
|
|
static union vdso_data_store vdso_data_store __page_aligned_data;
|
|
|
|
static struct vdso_data *vdso_data = &vdso_data_store.data;
|
|
|
|
|
2016-08-15 16:20:21 +08:00
|
|
|
static struct page *vdso_data_page __ro_after_init;
|
|
|
|
static const struct vm_special_mapping vdso_data_mapping = {
|
ARM: 8331/1: VDSO initialization, mapping, and synchronization
Initialize the VDSO page list at boot, install the VDSO mapping at
exec time, and update the data page during timer ticks. This code is
not built if CONFIG_VDSO is not enabled.
Account for the VDSO length when randomizing the offset from the
stack. The [vdso] and [vvar] pages are placed immediately following
the sigpage with separate _install_special_mapping calls.
We want to "penalize" systems lacking the arch timer as little
as possible. Previous versions of this code installed the VDSO
unconditionally and unmodified, making it a measurably slower way for
glibc to invoke the real syscalls on such systems. E.g. calling
gettimeofday via glibc goes from ~560ns to ~630ns on i.MX6Q.
If we can indicate to glibc that the time-related APIs in the VDSO are
not accelerated, glibc can continue to invoke the syscalls directly
instead of dispatching through the VDSO only to fall back to the slow
path.
Thus, if the architected timer is unusable for whatever reason, patch
the VDSO at boot time so that symbol lookups for gettimeofday and
clock_gettime return NULL. (This is similar to what powerpc does and
borrows code from there.) This allows glibc to perform the syscall
directly instead of passing control to the VDSO, which minimizes the
penalty. In my measurements the time taken for a gettimeofday call
via glibc goes from ~560ns to ~580ns (again on i.MX6Q), and this is
solely due to adding a test and branch to glibc's gettimeofday syscall
wrapper.
An alternative to patching the VDSO at boot would be to not install
the VDSO at all when the arch timer isn't usable. Another alternative
is to include a separate "dummy" vdso.so without gettimeofday and
clock_gettime, which would be selected at boot time. Either of these
would get cumbersome if the VDSO were to gain support for an API such
as getcpu which is unrelated to arch timer support.
Signed-off-by: Nathan Lynch <nathan_lynch@mentor.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-03-26 02:15:08 +08:00
|
|
|
.name = "[vvar]",
|
|
|
|
.pages = &vdso_data_page,
|
|
|
|
};
|
|
|
|
|
2017-06-20 00:32:42 +08:00
|
|
|
static int vdso_mremap(const struct vm_special_mapping *sm,
|
|
|
|
struct vm_area_struct *new_vma)
|
|
|
|
{
|
|
|
|
unsigned long new_size = new_vma->vm_end - new_vma->vm_start;
|
|
|
|
unsigned long vdso_size;
|
|
|
|
|
|
|
|
/* without VVAR page */
|
|
|
|
vdso_size = (vdso_total_pages - 1) << PAGE_SHIFT;
|
|
|
|
|
|
|
|
if (vdso_size != new_size)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
current->mm->context.vdso = new_vma->vm_start;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2016-08-15 16:20:21 +08:00
|
|
|
static struct vm_special_mapping vdso_text_mapping __ro_after_init = {
|
ARM: 8331/1: VDSO initialization, mapping, and synchronization
Initialize the VDSO page list at boot, install the VDSO mapping at
exec time, and update the data page during timer ticks. This code is
not built if CONFIG_VDSO is not enabled.
Account for the VDSO length when randomizing the offset from the
stack. The [vdso] and [vvar] pages are placed immediately following
the sigpage with separate _install_special_mapping calls.
We want to "penalize" systems lacking the arch timer as little
as possible. Previous versions of this code installed the VDSO
unconditionally and unmodified, making it a measurably slower way for
glibc to invoke the real syscalls on such systems. E.g. calling
gettimeofday via glibc goes from ~560ns to ~630ns on i.MX6Q.
If we can indicate to glibc that the time-related APIs in the VDSO are
not accelerated, glibc can continue to invoke the syscalls directly
instead of dispatching through the VDSO only to fall back to the slow
path.
Thus, if the architected timer is unusable for whatever reason, patch
the VDSO at boot time so that symbol lookups for gettimeofday and
clock_gettime return NULL. (This is similar to what powerpc does and
borrows code from there.) This allows glibc to perform the syscall
directly instead of passing control to the VDSO, which minimizes the
penalty. In my measurements the time taken for a gettimeofday call
via glibc goes from ~560ns to ~580ns (again on i.MX6Q), and this is
solely due to adding a test and branch to glibc's gettimeofday syscall
wrapper.
An alternative to patching the VDSO at boot would be to not install
the VDSO at all when the arch timer isn't usable. Another alternative
is to include a separate "dummy" vdso.so without gettimeofday and
clock_gettime, which would be selected at boot time. Either of these
would get cumbersome if the VDSO were to gain support for an API such
as getcpu which is unrelated to arch timer support.
Signed-off-by: Nathan Lynch <nathan_lynch@mentor.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-03-26 02:15:08 +08:00
|
|
|
.name = "[vdso]",
|
2017-06-20 00:32:42 +08:00
|
|
|
.mremap = vdso_mremap,
|
ARM: 8331/1: VDSO initialization, mapping, and synchronization
Initialize the VDSO page list at boot, install the VDSO mapping at
exec time, and update the data page during timer ticks. This code is
not built if CONFIG_VDSO is not enabled.
Account for the VDSO length when randomizing the offset from the
stack. The [vdso] and [vvar] pages are placed immediately following
the sigpage with separate _install_special_mapping calls.
We want to "penalize" systems lacking the arch timer as little
as possible. Previous versions of this code installed the VDSO
unconditionally and unmodified, making it a measurably slower way for
glibc to invoke the real syscalls on such systems. E.g. calling
gettimeofday via glibc goes from ~560ns to ~630ns on i.MX6Q.
If we can indicate to glibc that the time-related APIs in the VDSO are
not accelerated, glibc can continue to invoke the syscalls directly
instead of dispatching through the VDSO only to fall back to the slow
path.
Thus, if the architected timer is unusable for whatever reason, patch
the VDSO at boot time so that symbol lookups for gettimeofday and
clock_gettime return NULL. (This is similar to what powerpc does and
borrows code from there.) This allows glibc to perform the syscall
directly instead of passing control to the VDSO, which minimizes the
penalty. In my measurements the time taken for a gettimeofday call
via glibc goes from ~560ns to ~580ns (again on i.MX6Q), and this is
solely due to adding a test and branch to glibc's gettimeofday syscall
wrapper.
An alternative to patching the VDSO at boot would be to not install
the VDSO at all when the arch timer isn't usable. Another alternative
is to include a separate "dummy" vdso.so without gettimeofday and
clock_gettime, which would be selected at boot time. Either of these
would get cumbersome if the VDSO were to gain support for an API such
as getcpu which is unrelated to arch timer support.
Signed-off-by: Nathan Lynch <nathan_lynch@mentor.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-03-26 02:15:08 +08:00
|
|
|
};
|
|
|
|
|
|
|
|
struct elfinfo {
|
|
|
|
Elf32_Ehdr *hdr; /* ptr to ELF */
|
|
|
|
Elf32_Sym *dynsym; /* ptr to .dynsym section */
|
|
|
|
unsigned long dynsymsize; /* size of .dynsym section */
|
|
|
|
char *dynstr; /* ptr to .dynstr section */
|
|
|
|
};
|
|
|
|
|
|
|
|
/* Cached result of boot-time check for whether the arch timer exists,
|
|
|
|
* and if so, whether the virtual counter is useable.
|
|
|
|
*/
|
2016-08-15 16:20:21 +08:00
|
|
|
static bool cntvct_ok __ro_after_init;
|
ARM: 8331/1: VDSO initialization, mapping, and synchronization
Initialize the VDSO page list at boot, install the VDSO mapping at
exec time, and update the data page during timer ticks. This code is
not built if CONFIG_VDSO is not enabled.
Account for the VDSO length when randomizing the offset from the
stack. The [vdso] and [vvar] pages are placed immediately following
the sigpage with separate _install_special_mapping calls.
We want to "penalize" systems lacking the arch timer as little
as possible. Previous versions of this code installed the VDSO
unconditionally and unmodified, making it a measurably slower way for
glibc to invoke the real syscalls on such systems. E.g. calling
gettimeofday via glibc goes from ~560ns to ~630ns on i.MX6Q.
If we can indicate to glibc that the time-related APIs in the VDSO are
not accelerated, glibc can continue to invoke the syscalls directly
instead of dispatching through the VDSO only to fall back to the slow
path.
Thus, if the architected timer is unusable for whatever reason, patch
the VDSO at boot time so that symbol lookups for gettimeofday and
clock_gettime return NULL. (This is similar to what powerpc does and
borrows code from there.) This allows glibc to perform the syscall
directly instead of passing control to the VDSO, which minimizes the
penalty. In my measurements the time taken for a gettimeofday call
via glibc goes from ~560ns to ~580ns (again on i.MX6Q), and this is
solely due to adding a test and branch to glibc's gettimeofday syscall
wrapper.
An alternative to patching the VDSO at boot would be to not install
the VDSO at all when the arch timer isn't usable. Another alternative
is to include a separate "dummy" vdso.so without gettimeofday and
clock_gettime, which would be selected at boot time. Either of these
would get cumbersome if the VDSO were to gain support for an API such
as getcpu which is unrelated to arch timer support.
Signed-off-by: Nathan Lynch <nathan_lynch@mentor.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-03-26 02:15:08 +08:00
|
|
|
|
|
|
|
static bool __init cntvct_functional(void)
|
|
|
|
{
|
|
|
|
struct device_node *np;
|
|
|
|
bool ret = false;
|
|
|
|
|
|
|
|
if (!IS_ENABLED(CONFIG_ARM_ARCH_TIMER))
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
/* The arm_arch_timer core should export
|
|
|
|
* arch_timer_use_virtual or similar so we don't have to do
|
|
|
|
* this.
|
|
|
|
*/
|
|
|
|
np = of_find_compatible_node(NULL, NULL, "arm,armv7-timer");
|
|
|
|
if (!np)
|
|
|
|
goto out_put;
|
|
|
|
|
|
|
|
if (of_property_read_bool(np, "arm,cpu-registers-not-fw-configured"))
|
|
|
|
goto out_put;
|
|
|
|
|
|
|
|
ret = true;
|
|
|
|
|
|
|
|
out_put:
|
|
|
|
of_node_put(np);
|
|
|
|
out:
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void * __init find_section(Elf32_Ehdr *ehdr, const char *name,
|
|
|
|
unsigned long *size)
|
|
|
|
{
|
|
|
|
Elf32_Shdr *sechdrs;
|
|
|
|
unsigned int i;
|
|
|
|
char *secnames;
|
|
|
|
|
|
|
|
/* Grab section headers and strings so we can tell who is who */
|
|
|
|
sechdrs = (void *)ehdr + ehdr->e_shoff;
|
|
|
|
secnames = (void *)ehdr + sechdrs[ehdr->e_shstrndx].sh_offset;
|
|
|
|
|
|
|
|
/* Find the section they want */
|
|
|
|
for (i = 1; i < ehdr->e_shnum; i++) {
|
|
|
|
if (strcmp(secnames + sechdrs[i].sh_name, name) == 0) {
|
|
|
|
if (size)
|
|
|
|
*size = sechdrs[i].sh_size;
|
|
|
|
return (void *)ehdr + sechdrs[i].sh_offset;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (size)
|
|
|
|
*size = 0;
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
static Elf32_Sym * __init find_symbol(struct elfinfo *lib, const char *symname)
|
|
|
|
{
|
|
|
|
unsigned int i;
|
|
|
|
|
|
|
|
for (i = 0; i < (lib->dynsymsize / sizeof(Elf32_Sym)); i++) {
|
|
|
|
char name[MAX_SYMNAME], *c;
|
|
|
|
|
|
|
|
if (lib->dynsym[i].st_name == 0)
|
|
|
|
continue;
|
|
|
|
strlcpy(name, lib->dynstr + lib->dynsym[i].st_name,
|
|
|
|
MAX_SYMNAME);
|
|
|
|
c = strchr(name, '@');
|
|
|
|
if (c)
|
|
|
|
*c = 0;
|
|
|
|
if (strcmp(symname, name) == 0)
|
|
|
|
return &lib->dynsym[i];
|
|
|
|
}
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void __init vdso_nullpatch_one(struct elfinfo *lib, const char *symname)
|
|
|
|
{
|
|
|
|
Elf32_Sym *sym;
|
|
|
|
|
|
|
|
sym = find_symbol(lib, symname);
|
|
|
|
if (!sym)
|
|
|
|
return;
|
|
|
|
|
|
|
|
sym->st_name = 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void __init patch_vdso(void *ehdr)
|
|
|
|
{
|
|
|
|
struct elfinfo einfo;
|
|
|
|
|
|
|
|
einfo = (struct elfinfo) {
|
|
|
|
.hdr = ehdr,
|
|
|
|
};
|
|
|
|
|
|
|
|
einfo.dynsym = find_section(einfo.hdr, ".dynsym", &einfo.dynsymsize);
|
|
|
|
einfo.dynstr = find_section(einfo.hdr, ".dynstr", NULL);
|
|
|
|
|
|
|
|
/* If the virtual counter is absent or non-functional we don't
|
|
|
|
* want programs to incur the slight additional overhead of
|
|
|
|
* dispatching through the VDSO only to fall back to syscalls.
|
|
|
|
*/
|
|
|
|
if (!cntvct_ok) {
|
|
|
|
vdso_nullpatch_one(&einfo, "__vdso_gettimeofday");
|
|
|
|
vdso_nullpatch_one(&einfo, "__vdso_clock_gettime");
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static int __init vdso_init(void)
|
|
|
|
{
|
|
|
|
unsigned int text_pages;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
if (memcmp(&vdso_start, "\177ELF", 4)) {
|
|
|
|
pr_err("VDSO is not a valid ELF object!\n");
|
|
|
|
return -ENOEXEC;
|
|
|
|
}
|
|
|
|
|
|
|
|
text_pages = (&vdso_end - &vdso_start) >> PAGE_SHIFT;
|
|
|
|
pr_debug("vdso: %i text pages at base %p\n", text_pages, &vdso_start);
|
|
|
|
|
|
|
|
/* Allocate the VDSO text pagelist */
|
|
|
|
vdso_text_pagelist = kcalloc(text_pages, sizeof(struct page *),
|
|
|
|
GFP_KERNEL);
|
|
|
|
if (vdso_text_pagelist == NULL)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
/* Grab the VDSO data page. */
|
|
|
|
vdso_data_page = virt_to_page(vdso_data);
|
|
|
|
|
|
|
|
/* Grab the VDSO text pages. */
|
|
|
|
for (i = 0; i < text_pages; i++) {
|
|
|
|
struct page *page;
|
|
|
|
|
|
|
|
page = virt_to_page(&vdso_start + i * PAGE_SIZE);
|
|
|
|
vdso_text_pagelist[i] = page;
|
|
|
|
}
|
|
|
|
|
|
|
|
vdso_text_mapping.pages = vdso_text_pagelist;
|
|
|
|
|
|
|
|
vdso_total_pages = 1; /* for the data/vvar page */
|
|
|
|
vdso_total_pages += text_pages;
|
|
|
|
|
|
|
|
cntvct_ok = cntvct_functional();
|
|
|
|
|
|
|
|
patch_vdso(&vdso_start);
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
arch_initcall(vdso_init);
|
|
|
|
|
|
|
|
static int install_vvar(struct mm_struct *mm, unsigned long addr)
|
|
|
|
{
|
|
|
|
struct vm_area_struct *vma;
|
|
|
|
|
|
|
|
vma = _install_special_mapping(mm, addr, PAGE_SIZE,
|
|
|
|
VM_READ | VM_MAYREAD,
|
|
|
|
&vdso_data_mapping);
|
|
|
|
|
2015-12-09 00:30:25 +08:00
|
|
|
return PTR_ERR_OR_ZERO(vma);
|
ARM: 8331/1: VDSO initialization, mapping, and synchronization
Initialize the VDSO page list at boot, install the VDSO mapping at
exec time, and update the data page during timer ticks. This code is
not built if CONFIG_VDSO is not enabled.
Account for the VDSO length when randomizing the offset from the
stack. The [vdso] and [vvar] pages are placed immediately following
the sigpage with separate _install_special_mapping calls.
We want to "penalize" systems lacking the arch timer as little
as possible. Previous versions of this code installed the VDSO
unconditionally and unmodified, making it a measurably slower way for
glibc to invoke the real syscalls on such systems. E.g. calling
gettimeofday via glibc goes from ~560ns to ~630ns on i.MX6Q.
If we can indicate to glibc that the time-related APIs in the VDSO are
not accelerated, glibc can continue to invoke the syscalls directly
instead of dispatching through the VDSO only to fall back to the slow
path.
Thus, if the architected timer is unusable for whatever reason, patch
the VDSO at boot time so that symbol lookups for gettimeofday and
clock_gettime return NULL. (This is similar to what powerpc does and
borrows code from there.) This allows glibc to perform the syscall
directly instead of passing control to the VDSO, which minimizes the
penalty. In my measurements the time taken for a gettimeofday call
via glibc goes from ~560ns to ~580ns (again on i.MX6Q), and this is
solely due to adding a test and branch to glibc's gettimeofday syscall
wrapper.
An alternative to patching the VDSO at boot would be to not install
the VDSO at all when the arch timer isn't usable. Another alternative
is to include a separate "dummy" vdso.so without gettimeofday and
clock_gettime, which would be selected at boot time. Either of these
would get cumbersome if the VDSO were to gain support for an API such
as getcpu which is unrelated to arch timer support.
Signed-off-by: Nathan Lynch <nathan_lynch@mentor.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-03-26 02:15:08 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/* assumes mmap_sem is write-locked */
|
|
|
|
void arm_install_vdso(struct mm_struct *mm, unsigned long addr)
|
|
|
|
{
|
|
|
|
struct vm_area_struct *vma;
|
|
|
|
unsigned long len;
|
|
|
|
|
|
|
|
mm->context.vdso = 0;
|
|
|
|
|
|
|
|
if (vdso_text_pagelist == NULL)
|
|
|
|
return;
|
|
|
|
|
|
|
|
if (install_vvar(mm, addr))
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* Account for vvar page. */
|
|
|
|
addr += PAGE_SIZE;
|
|
|
|
len = (vdso_total_pages - 1) << PAGE_SHIFT;
|
|
|
|
|
|
|
|
vma = _install_special_mapping(mm, addr, len,
|
|
|
|
VM_READ | VM_EXEC | VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC,
|
|
|
|
&vdso_text_mapping);
|
|
|
|
|
|
|
|
if (!IS_ERR(vma))
|
|
|
|
mm->context.vdso = addr;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void vdso_write_begin(struct vdso_data *vdata)
|
|
|
|
{
|
|
|
|
++vdso_data->seq_count;
|
|
|
|
smp_wmb(); /* Pairs with smp_rmb in vdso_read_retry */
|
|
|
|
}
|
|
|
|
|
|
|
|
static void vdso_write_end(struct vdso_data *vdata)
|
|
|
|
{
|
|
|
|
smp_wmb(); /* Pairs with smp_rmb in vdso_read_begin */
|
|
|
|
++vdso_data->seq_count;
|
|
|
|
}
|
|
|
|
|
|
|
|
static bool tk_is_cntvct(const struct timekeeper *tk)
|
|
|
|
{
|
|
|
|
if (!IS_ENABLED(CONFIG_ARM_ARCH_TIMER))
|
|
|
|
return false;
|
|
|
|
|
2016-09-22 16:35:18 +08:00
|
|
|
if (!tk->tkr_mono.clock->archdata.vdso_direct)
|
ARM: 8331/1: VDSO initialization, mapping, and synchronization
Initialize the VDSO page list at boot, install the VDSO mapping at
exec time, and update the data page during timer ticks. This code is
not built if CONFIG_VDSO is not enabled.
Account for the VDSO length when randomizing the offset from the
stack. The [vdso] and [vvar] pages are placed immediately following
the sigpage with separate _install_special_mapping calls.
We want to "penalize" systems lacking the arch timer as little
as possible. Previous versions of this code installed the VDSO
unconditionally and unmodified, making it a measurably slower way for
glibc to invoke the real syscalls on such systems. E.g. calling
gettimeofday via glibc goes from ~560ns to ~630ns on i.MX6Q.
If we can indicate to glibc that the time-related APIs in the VDSO are
not accelerated, glibc can continue to invoke the syscalls directly
instead of dispatching through the VDSO only to fall back to the slow
path.
Thus, if the architected timer is unusable for whatever reason, patch
the VDSO at boot time so that symbol lookups for gettimeofday and
clock_gettime return NULL. (This is similar to what powerpc does and
borrows code from there.) This allows glibc to perform the syscall
directly instead of passing control to the VDSO, which minimizes the
penalty. In my measurements the time taken for a gettimeofday call
via glibc goes from ~560ns to ~580ns (again on i.MX6Q), and this is
solely due to adding a test and branch to glibc's gettimeofday syscall
wrapper.
An alternative to patching the VDSO at boot would be to not install
the VDSO at all when the arch timer isn't usable. Another alternative
is to include a separate "dummy" vdso.so without gettimeofday and
clock_gettime, which would be selected at boot time. Either of these
would get cumbersome if the VDSO were to gain support for an API such
as getcpu which is unrelated to arch timer support.
Signed-off-by: Nathan Lynch <nathan_lynch@mentor.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-03-26 02:15:08 +08:00
|
|
|
return false;
|
|
|
|
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* update_vsyscall - update the vdso data page
|
|
|
|
*
|
|
|
|
* Increment the sequence counter, making it odd, indicating to
|
|
|
|
* userspace that an update is in progress. Update the fields used
|
|
|
|
* for coarse clocks and, if the architected system timer is in use,
|
|
|
|
* the fields used for high precision clocks. Increment the sequence
|
|
|
|
* counter again, making it even, indicating to userspace that the
|
|
|
|
* update is finished.
|
|
|
|
*
|
|
|
|
* Userspace is expected to sample seq_count before reading any other
|
|
|
|
* fields from the data page. If seq_count is odd, userspace is
|
|
|
|
* expected to wait until it becomes even. After copying data from
|
|
|
|
* the page, userspace must sample seq_count again; if it has changed
|
|
|
|
* from its previous value, userspace must retry the whole sequence.
|
|
|
|
*
|
|
|
|
* Calls to update_vsyscall are serialized by the timekeeping core.
|
|
|
|
*/
|
|
|
|
void update_vsyscall(struct timekeeper *tk)
|
|
|
|
{
|
|
|
|
struct timespec64 *wtm = &tk->wall_to_monotonic;
|
|
|
|
|
|
|
|
if (!cntvct_ok) {
|
|
|
|
/* The entry points have been zeroed, so there is no
|
|
|
|
* point in updating the data page.
|
|
|
|
*/
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
vdso_write_begin(vdso_data);
|
|
|
|
|
|
|
|
vdso_data->tk_is_cntvct = tk_is_cntvct(tk);
|
2015-08-11 00:36:06 +08:00
|
|
|
vdso_data->xtime_coarse_sec = tk->xtime_sec;
|
|
|
|
vdso_data->xtime_coarse_nsec = (u32)(tk->tkr_mono.xtime_nsec >>
|
|
|
|
tk->tkr_mono.shift);
|
ARM: 8331/1: VDSO initialization, mapping, and synchronization
Initialize the VDSO page list at boot, install the VDSO mapping at
exec time, and update the data page during timer ticks. This code is
not built if CONFIG_VDSO is not enabled.
Account for the VDSO length when randomizing the offset from the
stack. The [vdso] and [vvar] pages are placed immediately following
the sigpage with separate _install_special_mapping calls.
We want to "penalize" systems lacking the arch timer as little
as possible. Previous versions of this code installed the VDSO
unconditionally and unmodified, making it a measurably slower way for
glibc to invoke the real syscalls on such systems. E.g. calling
gettimeofday via glibc goes from ~560ns to ~630ns on i.MX6Q.
If we can indicate to glibc that the time-related APIs in the VDSO are
not accelerated, glibc can continue to invoke the syscalls directly
instead of dispatching through the VDSO only to fall back to the slow
path.
Thus, if the architected timer is unusable for whatever reason, patch
the VDSO at boot time so that symbol lookups for gettimeofday and
clock_gettime return NULL. (This is similar to what powerpc does and
borrows code from there.) This allows glibc to perform the syscall
directly instead of passing control to the VDSO, which minimizes the
penalty. In my measurements the time taken for a gettimeofday call
via glibc goes from ~560ns to ~580ns (again on i.MX6Q), and this is
solely due to adding a test and branch to glibc's gettimeofday syscall
wrapper.
An alternative to patching the VDSO at boot would be to not install
the VDSO at all when the arch timer isn't usable. Another alternative
is to include a separate "dummy" vdso.so without gettimeofday and
clock_gettime, which would be selected at boot time. Either of these
would get cumbersome if the VDSO were to gain support for an API such
as getcpu which is unrelated to arch timer support.
Signed-off-by: Nathan Lynch <nathan_lynch@mentor.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-03-26 02:15:08 +08:00
|
|
|
vdso_data->wtm_clock_sec = wtm->tv_sec;
|
|
|
|
vdso_data->wtm_clock_nsec = wtm->tv_nsec;
|
|
|
|
|
|
|
|
if (vdso_data->tk_is_cntvct) {
|
Merge branch 'for-linus' of git://ftp.arm.linux.org.uk/~rmk/linux-arm
Pull ARM updates from Russell King:
"Included in this update are both some long term fixes and some new
features.
Fixes:
- An integer overflow in the calculation of ELF_ET_DYN_BASE.
- Avoiding OOMs for high-order IOMMU allocations
- SMP requires the data cache to be enabled for synchronisation
primitives to work, so prevent the CPU_DCACHE_DISABLE option being
visible on SMP builds.
- A bug going back 10+ years in the noMMU ARM94* CPU support code,
where it corrupts registers. Found by folk getting Linux running
on their cameras.
- Versatile Express needs an errata workaround enabled for CPU
hot-unplug to work.
Features:
- Clean up module linker by handling out of range relocations
separately from relocation cases we don't handle.
- Fix a long term bug in the pci_mmap_page_range() code, which we
hope won't impact userspace (we hope there's no users of the
existing broken interface.)
- Don't map DMA coherent allocations when we don't have a MMU.
- Drop experimental status for SMP_ON_UP.
- Warn when DT doesn't specify ePAPR mandatory cache properties.
- Add documentation concerning how we find the start of physical
memory for AUTO_ZRELADDR kernels, detailing why we have chosen the
mask and the implications of changing it.
- Updates from Ard Biesheuvel to address some issues with large
kernels (such as allyesconfig) failing to link.
- Allow hibernation to work on modern (ARMv7) CPUs - this appears to
have never worked in the past on these CPUs.
- Enable IRQ_SHOW_LEVEL, which changes the /proc/interrupts output
format (hopefully without userspace breaking... let's hope that if
it causes someone a problem, they tell us.)
- Fix tegra-ahb DT offsets.
- Rework ARM errata 643719 code (and ARMv7 flush_cache_louis()/
flush_dcache_all()) code to be more efficient, and enable this
errata workaround by default for ARMv7+SMP CPUs. This complements
the Versatile Express fix above.
- Rework ARMv7 context code for errata 430973, so that only Cortex A8
CPUs are impacted by the branch target buffer flush when this
errata is enabled. Also update the help text to indicate that all
r1p* A8 CPUs are impacted.
- Switch ARM to the generic show_mem() implementation, it conveys all
the information which we were already reporting.
- Prevent slow timer sources being used for udelay() - timers running
at less than 1MHz are not useful for this, and can cause udelay()
to return immediately, without any wait. Using such a slow timer
is silly.
- VDSO support for 32-bit ARM, mainly for gettimeofday() using the
ARM architected timer.
- Perf support for Scorpion performance monitoring units"
vdso semantic conflict fixed up as per linux-next.
* 'for-linus' of git://ftp.arm.linux.org.uk/~rmk/linux-arm: (52 commits)
ARM: update errata 430973 documentation to cover Cortex A8 r1p*
ARM: ensure delay timer has sufficient accuracy for delays
ARM: switch to use the generic show_mem() implementation
ARM: proc-v7: avoid errata 430973 workaround for non-Cortex A8 CPUs
ARM: enable ARM errata 643719 workaround by default
ARM: cache-v7: optimise test for Cortex A9 r0pX devices
ARM: cache-v7: optimise branches in v7_flush_cache_louis
ARM: cache-v7: consolidate initialisation of cache level index
ARM: cache-v7: shift CLIDR to extract appropriate field before masking
ARM: cache-v7: use movw/movt instructions
ARM: allow 16-bit instructions in ALT_UP()
ARM: proc-arm94*.S: fix setup function
ARM: vexpress: fix CPU hotplug with CT9x4 tile.
ARM: 8276/1: Make CPU_DCACHE_DISABLE depend on !SMP
ARM: 8335/1: Documentation: DT bindings: Tegra AHB: document the legacy base address
ARM: 8334/1: amba: tegra-ahb: detect and correct bogus base address
ARM: 8333/1: amba: tegra-ahb: fix register offsets in the macros
ARM: 8339/1: Enable CONFIG_GENERIC_IRQ_SHOW_LEVEL
ARM: 8338/1: kexec: Relax SMP validation to improve DT compatibility
ARM: 8337/1: mm: Do not invoke OOM for higher order IOMMU DMA allocations
...
2015-04-15 12:03:26 +08:00
|
|
|
vdso_data->cs_cycle_last = tk->tkr_mono.cycle_last;
|
ARM: 8331/1: VDSO initialization, mapping, and synchronization
Initialize the VDSO page list at boot, install the VDSO mapping at
exec time, and update the data page during timer ticks. This code is
not built if CONFIG_VDSO is not enabled.
Account for the VDSO length when randomizing the offset from the
stack. The [vdso] and [vvar] pages are placed immediately following
the sigpage with separate _install_special_mapping calls.
We want to "penalize" systems lacking the arch timer as little
as possible. Previous versions of this code installed the VDSO
unconditionally and unmodified, making it a measurably slower way for
glibc to invoke the real syscalls on such systems. E.g. calling
gettimeofday via glibc goes from ~560ns to ~630ns on i.MX6Q.
If we can indicate to glibc that the time-related APIs in the VDSO are
not accelerated, glibc can continue to invoke the syscalls directly
instead of dispatching through the VDSO only to fall back to the slow
path.
Thus, if the architected timer is unusable for whatever reason, patch
the VDSO at boot time so that symbol lookups for gettimeofday and
clock_gettime return NULL. (This is similar to what powerpc does and
borrows code from there.) This allows glibc to perform the syscall
directly instead of passing control to the VDSO, which minimizes the
penalty. In my measurements the time taken for a gettimeofday call
via glibc goes from ~560ns to ~580ns (again on i.MX6Q), and this is
solely due to adding a test and branch to glibc's gettimeofday syscall
wrapper.
An alternative to patching the VDSO at boot would be to not install
the VDSO at all when the arch timer isn't usable. Another alternative
is to include a separate "dummy" vdso.so without gettimeofday and
clock_gettime, which would be selected at boot time. Either of these
would get cumbersome if the VDSO were to gain support for an API such
as getcpu which is unrelated to arch timer support.
Signed-off-by: Nathan Lynch <nathan_lynch@mentor.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-03-26 02:15:08 +08:00
|
|
|
vdso_data->xtime_clock_sec = tk->xtime_sec;
|
Merge branch 'for-linus' of git://ftp.arm.linux.org.uk/~rmk/linux-arm
Pull ARM updates from Russell King:
"Included in this update are both some long term fixes and some new
features.
Fixes:
- An integer overflow in the calculation of ELF_ET_DYN_BASE.
- Avoiding OOMs for high-order IOMMU allocations
- SMP requires the data cache to be enabled for synchronisation
primitives to work, so prevent the CPU_DCACHE_DISABLE option being
visible on SMP builds.
- A bug going back 10+ years in the noMMU ARM94* CPU support code,
where it corrupts registers. Found by folk getting Linux running
on their cameras.
- Versatile Express needs an errata workaround enabled for CPU
hot-unplug to work.
Features:
- Clean up module linker by handling out of range relocations
separately from relocation cases we don't handle.
- Fix a long term bug in the pci_mmap_page_range() code, which we
hope won't impact userspace (we hope there's no users of the
existing broken interface.)
- Don't map DMA coherent allocations when we don't have a MMU.
- Drop experimental status for SMP_ON_UP.
- Warn when DT doesn't specify ePAPR mandatory cache properties.
- Add documentation concerning how we find the start of physical
memory for AUTO_ZRELADDR kernels, detailing why we have chosen the
mask and the implications of changing it.
- Updates from Ard Biesheuvel to address some issues with large
kernels (such as allyesconfig) failing to link.
- Allow hibernation to work on modern (ARMv7) CPUs - this appears to
have never worked in the past on these CPUs.
- Enable IRQ_SHOW_LEVEL, which changes the /proc/interrupts output
format (hopefully without userspace breaking... let's hope that if
it causes someone a problem, they tell us.)
- Fix tegra-ahb DT offsets.
- Rework ARM errata 643719 code (and ARMv7 flush_cache_louis()/
flush_dcache_all()) code to be more efficient, and enable this
errata workaround by default for ARMv7+SMP CPUs. This complements
the Versatile Express fix above.
- Rework ARMv7 context code for errata 430973, so that only Cortex A8
CPUs are impacted by the branch target buffer flush when this
errata is enabled. Also update the help text to indicate that all
r1p* A8 CPUs are impacted.
- Switch ARM to the generic show_mem() implementation, it conveys all
the information which we were already reporting.
- Prevent slow timer sources being used for udelay() - timers running
at less than 1MHz are not useful for this, and can cause udelay()
to return immediately, without any wait. Using such a slow timer
is silly.
- VDSO support for 32-bit ARM, mainly for gettimeofday() using the
ARM architected timer.
- Perf support for Scorpion performance monitoring units"
vdso semantic conflict fixed up as per linux-next.
* 'for-linus' of git://ftp.arm.linux.org.uk/~rmk/linux-arm: (52 commits)
ARM: update errata 430973 documentation to cover Cortex A8 r1p*
ARM: ensure delay timer has sufficient accuracy for delays
ARM: switch to use the generic show_mem() implementation
ARM: proc-v7: avoid errata 430973 workaround for non-Cortex A8 CPUs
ARM: enable ARM errata 643719 workaround by default
ARM: cache-v7: optimise test for Cortex A9 r0pX devices
ARM: cache-v7: optimise branches in v7_flush_cache_louis
ARM: cache-v7: consolidate initialisation of cache level index
ARM: cache-v7: shift CLIDR to extract appropriate field before masking
ARM: cache-v7: use movw/movt instructions
ARM: allow 16-bit instructions in ALT_UP()
ARM: proc-arm94*.S: fix setup function
ARM: vexpress: fix CPU hotplug with CT9x4 tile.
ARM: 8276/1: Make CPU_DCACHE_DISABLE depend on !SMP
ARM: 8335/1: Documentation: DT bindings: Tegra AHB: document the legacy base address
ARM: 8334/1: amba: tegra-ahb: detect and correct bogus base address
ARM: 8333/1: amba: tegra-ahb: fix register offsets in the macros
ARM: 8339/1: Enable CONFIG_GENERIC_IRQ_SHOW_LEVEL
ARM: 8338/1: kexec: Relax SMP validation to improve DT compatibility
ARM: 8337/1: mm: Do not invoke OOM for higher order IOMMU DMA allocations
...
2015-04-15 12:03:26 +08:00
|
|
|
vdso_data->xtime_clock_snsec = tk->tkr_mono.xtime_nsec;
|
|
|
|
vdso_data->cs_mult = tk->tkr_mono.mult;
|
|
|
|
vdso_data->cs_shift = tk->tkr_mono.shift;
|
|
|
|
vdso_data->cs_mask = tk->tkr_mono.mask;
|
ARM: 8331/1: VDSO initialization, mapping, and synchronization
Initialize the VDSO page list at boot, install the VDSO mapping at
exec time, and update the data page during timer ticks. This code is
not built if CONFIG_VDSO is not enabled.
Account for the VDSO length when randomizing the offset from the
stack. The [vdso] and [vvar] pages are placed immediately following
the sigpage with separate _install_special_mapping calls.
We want to "penalize" systems lacking the arch timer as little
as possible. Previous versions of this code installed the VDSO
unconditionally and unmodified, making it a measurably slower way for
glibc to invoke the real syscalls on such systems. E.g. calling
gettimeofday via glibc goes from ~560ns to ~630ns on i.MX6Q.
If we can indicate to glibc that the time-related APIs in the VDSO are
not accelerated, glibc can continue to invoke the syscalls directly
instead of dispatching through the VDSO only to fall back to the slow
path.
Thus, if the architected timer is unusable for whatever reason, patch
the VDSO at boot time so that symbol lookups for gettimeofday and
clock_gettime return NULL. (This is similar to what powerpc does and
borrows code from there.) This allows glibc to perform the syscall
directly instead of passing control to the VDSO, which minimizes the
penalty. In my measurements the time taken for a gettimeofday call
via glibc goes from ~560ns to ~580ns (again on i.MX6Q), and this is
solely due to adding a test and branch to glibc's gettimeofday syscall
wrapper.
An alternative to patching the VDSO at boot would be to not install
the VDSO at all when the arch timer isn't usable. Another alternative
is to include a separate "dummy" vdso.so without gettimeofday and
clock_gettime, which would be selected at boot time. Either of these
would get cumbersome if the VDSO were to gain support for an API such
as getcpu which is unrelated to arch timer support.
Signed-off-by: Nathan Lynch <nathan_lynch@mentor.com>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
2015-03-26 02:15:08 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
vdso_write_end(vdso_data);
|
|
|
|
|
|
|
|
flush_dcache_page(virt_to_page(vdso_data));
|
|
|
|
}
|
|
|
|
|
|
|
|
void update_vsyscall_tz(void)
|
|
|
|
{
|
|
|
|
vdso_data->tz_minuteswest = sys_tz.tz_minuteswest;
|
|
|
|
vdso_data->tz_dsttime = sys_tz.tz_dsttime;
|
|
|
|
flush_dcache_page(virt_to_page(vdso_data));
|
|
|
|
}
|