include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 16:04:11 +08:00
|
|
|
#include <linux/gfp.h>
|
2009-04-11 02:33:10 +08:00
|
|
|
#include <linux/initrd.h>
|
2009-03-04 17:46:40 +08:00
|
|
|
#include <linux/ioport.h>
|
2009-03-03 19:15:06 +08:00
|
|
|
#include <linux/swap.h>
|
2010-08-26 04:39:17 +08:00
|
|
|
#include <linux/memblock.h>
|
2018-06-14 06:48:28 +08:00
|
|
|
#include <linux/swapfile.h>
|
|
|
|
#include <linux/swapops.h>
|
2019-04-24 00:58:11 +08:00
|
|
|
#include <linux/kmemleak.h>
|
2019-04-27 07:22:46 +08:00
|
|
|
#include <linux/sched/task.h>
|
2009-03-04 17:46:40 +08:00
|
|
|
|
2017-05-09 06:58:11 +08:00
|
|
|
#include <asm/set_memory.h>
|
2017-01-27 17:27:10 +08:00
|
|
|
#include <asm/e820/api.h>
|
2009-03-05 20:55:08 +08:00
|
|
|
#include <asm/init.h>
|
2009-03-03 19:15:06 +08:00
|
|
|
#include <asm/page.h>
|
2009-03-04 17:46:40 +08:00
|
|
|
#include <asm/page_types.h>
|
2009-03-03 19:15:06 +08:00
|
|
|
#include <asm/sections.h>
|
2009-05-06 20:06:47 +08:00
|
|
|
#include <asm/setup.h>
|
2009-03-05 20:55:05 +08:00
|
|
|
#include <asm/tlbflush.h>
|
2009-04-28 21:00:50 +08:00
|
|
|
#include <asm/tlb.h>
|
2009-07-01 22:24:23 +08:00
|
|
|
#include <asm/proto.h>
|
2011-11-01 21:58:22 +08:00
|
|
|
#include <asm/dma.h> /* for MAX_DMA_PFN */
|
2012-12-21 15:44:31 +08:00
|
|
|
#include <asm/microcode.h>
|
x86/mm: Implement ASLR for kernel memory regions
Randomizes the virtual address space of kernel memory regions for
x86_64. This first patch adds the infrastructure and does not randomize
any region. The following patches will randomize the physical memory
mapping, vmalloc and vmemmap regions.
This security feature mitigates exploits relying on predictable kernel
addresses. These addresses can be used to disclose the kernel modules
base addresses or corrupt specific structures to elevate privileges
bypassing the current implementation of KASLR. This feature can be
enabled with the CONFIG_RANDOMIZE_MEMORY option.
The order of each memory region is not changed. The feature looks at the
available space for the regions based on different configuration options
and randomizes the base and space between each. The size of the physical
memory mapping is the available physical memory. No performance impact
was detected while testing the feature.
Entropy is generated using the KASLR early boot functions now shared in
the lib directory (originally written by Kees Cook). Randomization is
done on PGD & PUD page table levels to increase possible addresses. The
physical memory mapping code was adapted to support PUD level virtual
addresses. This implementation on the best configuration provides 30,000
possible virtual addresses in average for each memory region. An
additional low memory page is used to ensure each CPU can start with a
PGD aligned virtual address (for realmode).
x86/dump_pagetable was updated to correctly display each region.
Updated documentation on x86_64 memory layout accordingly.
Performance data, after all patches in the series:
Kernbench shows almost no difference (-+ less than 1%):
Before:
Average Optimal load -j 12 Run (std deviation): Elapsed Time 102.63 (1.2695)
User Time 1034.89 (1.18115) System Time 87.056 (0.456416) Percent CPU 1092.9
(13.892) Context Switches 199805 (3455.33) Sleeps 97907.8 (900.636)
After:
Average Optimal load -j 12 Run (std deviation): Elapsed Time 102.489 (1.10636)
User Time 1034.86 (1.36053) System Time 87.764 (0.49345) Percent CPU 1095
(12.7715) Context Switches 199036 (4298.1) Sleeps 97681.6 (1031.11)
Hackbench shows 0% difference on average (hackbench 90 repeated 10 times):
attemp,before,after 1,0.076,0.069 2,0.072,0.069 3,0.066,0.066 4,0.066,0.068
5,0.066,0.067 6,0.066,0.069 7,0.067,0.066 8,0.063,0.067 9,0.067,0.065
10,0.068,0.071 average,0.0677,0.0677
Signed-off-by: Thomas Garnier <thgarnie@google.com>
Signed-off-by: Kees Cook <keescook@chromium.org>
Cc: Alexander Kuleshov <kuleshovmail@gmail.com>
Cc: Alexander Popov <alpopov@ptsecurity.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
Cc: Baoquan He <bhe@redhat.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Borislav Petkov <bp@suse.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Christian Borntraeger <borntraeger@de.ibm.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Dave Young <dyoung@redhat.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Jan Beulich <JBeulich@suse.com>
Cc: Joerg Roedel <jroedel@suse.de>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Lv Zheng <lv.zheng@intel.com>
Cc: Mark Salter <msalter@redhat.com>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Matt Fleming <matt@codeblueprint.co.uk>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephen Smalley <sds@tycho.nsa.gov>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Toshi Kani <toshi.kani@hpe.com>
Cc: Xiao Guangrong <guangrong.xiao@linux.intel.com>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: kernel-hardening@lists.openwall.com
Cc: linux-doc@vger.kernel.org
Link: http://lkml.kernel.org/r/1466556426-32664-6-git-send-email-keescook@chromium.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2016-06-22 08:47:02 +08:00
|
|
|
#include <asm/kaslr.h>
|
2017-07-28 18:23:12 +08:00
|
|
|
#include <asm/hypervisor.h>
|
2017-09-11 08:48:27 +08:00
|
|
|
#include <asm/cpufeature.h>
|
2017-12-04 22:07:36 +08:00
|
|
|
#include <asm/pti.h>
|
2019-04-27 07:22:46 +08:00
|
|
|
#include <asm/text-patching.h>
|
2020-06-06 20:26:29 +08:00
|
|
|
#include <asm/memtype.h>
|
2023-01-09 23:09:22 +08:00
|
|
|
#include <asm/paravirt.h>
|
2009-04-28 21:00:50 +08:00
|
|
|
|
2014-07-31 23:40:59 +08:00
|
|
|
/*
|
|
|
|
* We need to define the tracepoints somewhere, and tlb.c
|
2021-03-18 22:28:01 +08:00
|
|
|
* is only compiled when SMP=y.
|
2014-07-31 23:40:59 +08:00
|
|
|
*/
|
|
|
|
#include <trace/events/tlb.h>
|
|
|
|
|
2012-11-17 11:39:01 +08:00
|
|
|
#include "mm_internal.h"
|
|
|
|
|
2014-11-03 21:01:47 +08:00
|
|
|
/*
|
|
|
|
* Tables translating between page_cache_type_t and pte encoding.
|
2015-03-05 15:58:44 +08:00
|
|
|
*
|
2015-07-23 02:06:11 +08:00
|
|
|
* The default values are defined statically as minimal supported mode;
|
|
|
|
* WC and WT fall back to UC-. pat_init() updates these values to support
|
|
|
|
* more cache modes, WC and WT, when it is safe to do so. See pat_init()
|
|
|
|
* for the details. Note, __early_ioremap() used during early boot-time
|
|
|
|
* takes pgprot_t (pte encoding) and does not use these tables.
|
2015-03-05 15:58:44 +08:00
|
|
|
*
|
|
|
|
* Index into __cachemode2pte_tbl[] is the cachemode.
|
|
|
|
*
|
|
|
|
* Index into __pte2cachemode_tbl[] are the caching attribute bits of the pte
|
|
|
|
* (_PAGE_PWT, _PAGE_PCD, _PAGE_PAT) at index bit positions 0, 1, 2.
|
2014-11-03 21:01:47 +08:00
|
|
|
*/
|
2020-04-08 23:27:45 +08:00
|
|
|
static uint16_t __cachemode2pte_tbl[_PAGE_CACHE_MODE_NUM] = {
|
2015-03-05 15:58:44 +08:00
|
|
|
[_PAGE_CACHE_MODE_WB ] = 0 | 0 ,
|
2015-06-05 00:55:10 +08:00
|
|
|
[_PAGE_CACHE_MODE_WC ] = 0 | _PAGE_PCD,
|
2015-03-05 15:58:44 +08:00
|
|
|
[_PAGE_CACHE_MODE_UC_MINUS] = 0 | _PAGE_PCD,
|
|
|
|
[_PAGE_CACHE_MODE_UC ] = _PAGE_PWT | _PAGE_PCD,
|
|
|
|
[_PAGE_CACHE_MODE_WT ] = 0 | _PAGE_PCD,
|
|
|
|
[_PAGE_CACHE_MODE_WP ] = 0 | _PAGE_PCD,
|
2014-11-03 21:01:47 +08:00
|
|
|
};
|
2015-03-05 15:58:44 +08:00
|
|
|
|
2020-04-08 23:27:45 +08:00
|
|
|
unsigned long cachemode2protval(enum page_cache_mode pcm)
|
|
|
|
{
|
|
|
|
if (likely(pcm == 0))
|
|
|
|
return 0;
|
|
|
|
return __cachemode2pte_tbl[pcm];
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(cachemode2protval);
|
2015-03-05 15:58:44 +08:00
|
|
|
|
2020-04-08 23:27:43 +08:00
|
|
|
static uint8_t __pte2cachemode_tbl[8] = {
|
2015-03-05 15:58:44 +08:00
|
|
|
[__pte2cm_idx( 0 | 0 | 0 )] = _PAGE_CACHE_MODE_WB,
|
2015-06-05 00:55:10 +08:00
|
|
|
[__pte2cm_idx(_PAGE_PWT | 0 | 0 )] = _PAGE_CACHE_MODE_UC_MINUS,
|
2015-03-05 15:58:44 +08:00
|
|
|
[__pte2cm_idx( 0 | _PAGE_PCD | 0 )] = _PAGE_CACHE_MODE_UC_MINUS,
|
|
|
|
[__pte2cm_idx(_PAGE_PWT | _PAGE_PCD | 0 )] = _PAGE_CACHE_MODE_UC,
|
|
|
|
[__pte2cm_idx( 0 | 0 | _PAGE_PAT)] = _PAGE_CACHE_MODE_WB,
|
2015-06-05 00:55:10 +08:00
|
|
|
[__pte2cm_idx(_PAGE_PWT | 0 | _PAGE_PAT)] = _PAGE_CACHE_MODE_UC_MINUS,
|
2015-03-05 15:58:44 +08:00
|
|
|
[__pte2cm_idx(0 | _PAGE_PCD | _PAGE_PAT)] = _PAGE_CACHE_MODE_UC_MINUS,
|
2014-11-03 21:01:47 +08:00
|
|
|
[__pte2cm_idx(_PAGE_PWT | _PAGE_PCD | _PAGE_PAT)] = _PAGE_CACHE_MODE_UC,
|
|
|
|
};
|
|
|
|
|
2022-07-08 21:14:56 +08:00
|
|
|
/*
|
|
|
|
* Check that the write-protect PAT entry is set for write-protect.
|
|
|
|
* To do this without making assumptions how PAT has been set up (Xen has
|
|
|
|
* another layout than the kernel), translate the _PAGE_CACHE_MODE_WP cache
|
|
|
|
* mode via the __cachemode2pte_tbl[] into protection bits (those protection
|
|
|
|
* bits will select a cache mode of WP or better), and then translate the
|
|
|
|
* protection bits back into the cache mode using __pte2cm_idx() and the
|
|
|
|
* __pte2cachemode_tbl[] array. This will return the really used cache mode.
|
|
|
|
*/
|
2020-04-08 23:27:42 +08:00
|
|
|
bool x86_has_pat_wp(void)
|
|
|
|
{
|
2022-07-08 21:14:56 +08:00
|
|
|
uint16_t prot = __cachemode2pte_tbl[_PAGE_CACHE_MODE_WP];
|
|
|
|
|
|
|
|
return __pte2cachemode_tbl[__pte2cm_idx(prot)] == _PAGE_CACHE_MODE_WP;
|
2020-04-08 23:27:42 +08:00
|
|
|
}
|
|
|
|
|
2020-04-08 23:27:43 +08:00
|
|
|
enum page_cache_mode pgprot2cachemode(pgprot_t pgprot)
|
|
|
|
{
|
|
|
|
unsigned long masked;
|
|
|
|
|
|
|
|
masked = pgprot_val(pgprot) & _PAGE_CACHE_MASK;
|
|
|
|
if (likely(masked == 0))
|
|
|
|
return 0;
|
|
|
|
return __pte2cachemode_tbl[__pte2cm_idx(masked)];
|
|
|
|
}
|
2014-11-03 21:01:47 +08:00
|
|
|
|
2012-11-17 11:39:07 +08:00
|
|
|
static unsigned long __initdata pgt_buf_start;
|
|
|
|
static unsigned long __initdata pgt_buf_end;
|
|
|
|
static unsigned long __initdata pgt_buf_top;
|
2009-03-05 20:55:05 +08:00
|
|
|
|
2012-11-17 11:39:02 +08:00
|
|
|
static unsigned long min_pfn_mapped;
|
|
|
|
|
2013-01-25 04:19:42 +08:00
|
|
|
static bool __initdata can_use_brk_pgt = true;
|
|
|
|
|
2012-11-17 11:39:05 +08:00
|
|
|
/*
|
|
|
|
* Pages returned are already directly mapped.
|
|
|
|
*
|
|
|
|
* Changing that is likely to break Xen, see commit:
|
|
|
|
*
|
|
|
|
* 279b706 x86,xen: introduce x86_init.mapping.pagetable_reserve
|
|
|
|
*
|
|
|
|
* for detailed information.
|
|
|
|
*/
|
2012-11-17 11:39:04 +08:00
|
|
|
__ref void *alloc_low_pages(unsigned int num)
|
2012-11-17 11:39:01 +08:00
|
|
|
{
|
|
|
|
unsigned long pfn;
|
2012-11-17 11:39:04 +08:00
|
|
|
int i;
|
2012-11-17 11:39:01 +08:00
|
|
|
|
|
|
|
if (after_bootmem) {
|
2012-11-17 11:39:04 +08:00
|
|
|
unsigned int order;
|
2012-11-17 11:39:01 +08:00
|
|
|
|
2012-11-17 11:39:04 +08:00
|
|
|
order = get_order((unsigned long)num << PAGE_SHIFT);
|
2017-11-16 09:35:54 +08:00
|
|
|
return (void *)__get_free_pages(GFP_ATOMIC | __GFP_ZERO, order);
|
2012-11-17 11:39:01 +08:00
|
|
|
}
|
|
|
|
|
2013-01-25 04:19:42 +08:00
|
|
|
if ((pgt_buf_end + num) > pgt_buf_top || !can_use_brk_pgt) {
|
2018-08-20 23:24:20 +08:00
|
|
|
unsigned long ret = 0;
|
|
|
|
|
|
|
|
if (min_pfn_mapped < max_pfn_mapped) {
|
2021-09-03 06:00:26 +08:00
|
|
|
ret = memblock_phys_alloc_range(
|
|
|
|
PAGE_SIZE * num, PAGE_SIZE,
|
2018-08-20 23:24:20 +08:00
|
|
|
min_pfn_mapped << PAGE_SHIFT,
|
2021-09-03 06:00:26 +08:00
|
|
|
max_pfn_mapped << PAGE_SHIFT);
|
2018-08-20 23:24:20 +08:00
|
|
|
}
|
2021-09-03 06:00:26 +08:00
|
|
|
if (!ret && can_use_brk_pgt)
|
2018-08-20 23:24:20 +08:00
|
|
|
ret = __pa(extend_brk(PAGE_SIZE * num, PAGE_SIZE));
|
|
|
|
|
2012-11-17 11:39:01 +08:00
|
|
|
if (!ret)
|
2013-11-13 07:08:28 +08:00
|
|
|
panic("alloc_low_pages: can not alloc memory");
|
2018-08-20 23:24:20 +08:00
|
|
|
|
2012-11-17 11:39:01 +08:00
|
|
|
pfn = ret >> PAGE_SHIFT;
|
2012-11-17 11:39:04 +08:00
|
|
|
} else {
|
|
|
|
pfn = pgt_buf_end;
|
|
|
|
pgt_buf_end += num;
|
|
|
|
}
|
|
|
|
|
|
|
|
for (i = 0; i < num; i++) {
|
|
|
|
void *adr;
|
|
|
|
|
|
|
|
adr = __va((pfn + i) << PAGE_SHIFT);
|
|
|
|
clear_page(adr);
|
|
|
|
}
|
2012-11-17 11:39:01 +08:00
|
|
|
|
2012-11-17 11:39:04 +08:00
|
|
|
return __va(pfn << PAGE_SHIFT);
|
2012-11-17 11:39:01 +08:00
|
|
|
}
|
|
|
|
|
2016-08-10 01:11:05 +08:00
|
|
|
/*
|
2020-12-16 04:56:41 +08:00
|
|
|
* By default need to be able to allocate page tables below PGD firstly for
|
|
|
|
* the 0-ISA_END_ADDRESS range and secondly for the initial PMD_SIZE mapping.
|
|
|
|
* With KASLR memory randomization, depending on the machine e820 memory and the
|
|
|
|
* PUD alignment, twice that many pages may be needed when KASLR memory
|
2016-08-10 01:11:05 +08:00
|
|
|
* randomization is enabled.
|
|
|
|
*/
|
2020-12-16 04:56:41 +08:00
|
|
|
|
|
|
|
#ifndef CONFIG_X86_5LEVEL
|
|
|
|
#define INIT_PGD_PAGE_TABLES 3
|
|
|
|
#else
|
|
|
|
#define INIT_PGD_PAGE_TABLES 4
|
|
|
|
#endif
|
|
|
|
|
2016-08-10 01:11:05 +08:00
|
|
|
#ifndef CONFIG_RANDOMIZE_MEMORY
|
2020-12-16 04:56:41 +08:00
|
|
|
#define INIT_PGD_PAGE_COUNT (2 * INIT_PGD_PAGE_TABLES)
|
2016-08-10 01:11:05 +08:00
|
|
|
#else
|
2020-12-16 04:56:41 +08:00
|
|
|
#define INIT_PGD_PAGE_COUNT (4 * INIT_PGD_PAGE_TABLES)
|
2016-08-10 01:11:05 +08:00
|
|
|
#endif
|
2020-12-16 04:56:41 +08:00
|
|
|
|
2016-08-10 01:11:05 +08:00
|
|
|
#define INIT_PGT_BUF_SIZE (INIT_PGD_PAGE_COUNT * PAGE_SIZE)
|
2012-11-17 11:38:58 +08:00
|
|
|
RESERVE_BRK(early_pgt_alloc, INIT_PGT_BUF_SIZE);
|
|
|
|
void __init early_alloc_pgt_buf(void)
|
|
|
|
{
|
|
|
|
unsigned long tables = INIT_PGT_BUF_SIZE;
|
|
|
|
phys_addr_t base;
|
|
|
|
|
|
|
|
base = __pa(extend_brk(tables, PAGE_SIZE));
|
|
|
|
|
|
|
|
pgt_buf_start = base >> PAGE_SHIFT;
|
|
|
|
pgt_buf_end = pgt_buf_start;
|
|
|
|
pgt_buf_top = pgt_buf_start + (tables >> PAGE_SHIFT);
|
|
|
|
}
|
|
|
|
|
2009-03-05 20:55:05 +08:00
|
|
|
int after_bootmem;
|
|
|
|
|
2015-03-05 15:18:23 +08:00
|
|
|
early_param_on_off("gbpages", "nogbpages", direct_gbpages, CONFIG_X86_DIRECT_GBPAGES);
|
2012-11-17 11:39:08 +08:00
|
|
|
|
2012-10-25 03:24:44 +08:00
|
|
|
struct map_range {
|
|
|
|
unsigned long start;
|
|
|
|
unsigned long end;
|
|
|
|
unsigned page_size_mask;
|
|
|
|
};
|
|
|
|
|
2012-11-17 11:38:38 +08:00
|
|
|
static int page_size_mask;
|
2009-03-05 20:55:05 +08:00
|
|
|
|
2020-04-21 17:20:39 +08:00
|
|
|
/*
|
|
|
|
* Save some of cr4 feature set we're using (e.g. Pentium 4MB
|
|
|
|
* enable and PPro Global page enable), so that any CPU's that boot
|
|
|
|
* up after us can get the correct flags. Invoked on the boot CPU.
|
|
|
|
*/
|
|
|
|
static inline void cr4_set_bits_and_update_boot(unsigned long mask)
|
|
|
|
{
|
|
|
|
mmu_cr4_features |= mask;
|
|
|
|
if (trampoline_cr4_features)
|
|
|
|
*trampoline_cr4_features = mmu_cr4_features;
|
|
|
|
cr4_set_bits(mask);
|
|
|
|
}
|
|
|
|
|
2012-11-17 11:38:41 +08:00
|
|
|
static void __init probe_page_size_mask(void)
|
2012-11-17 11:38:38 +08:00
|
|
|
{
|
|
|
|
/*
|
2017-11-16 09:36:02 +08:00
|
|
|
* For pagealloc debugging, identity mapping will use small pages.
|
2012-11-17 11:38:38 +08:00
|
|
|
* This will simplify cpa(), which otherwise needs to support splitting
|
|
|
|
* large pages into small in interrupt context, etc.
|
|
|
|
*/
|
2017-11-16 09:36:02 +08:00
|
|
|
if (boot_cpu_has(X86_FEATURE_PSE) && !debug_pagealloc_enabled())
|
2012-11-17 11:38:38 +08:00
|
|
|
page_size_mask |= 1 << PG_LEVEL_2M;
|
2017-06-12 15:21:30 +08:00
|
|
|
else
|
|
|
|
direct_gbpages = 0;
|
2012-11-17 11:38:38 +08:00
|
|
|
|
|
|
|
/* Enable PSE if available */
|
2016-03-29 23:42:03 +08:00
|
|
|
if (boot_cpu_has(X86_FEATURE_PSE))
|
2014-10-25 06:58:07 +08:00
|
|
|
cr4_set_bits_and_update_boot(X86_CR4_PSE);
|
2012-11-17 11:38:38 +08:00
|
|
|
|
|
|
|
/* Enable PGE if available */
|
2017-12-04 22:07:34 +08:00
|
|
|
__supported_pte_mask &= ~_PAGE_GLOBAL;
|
2016-03-29 23:42:02 +08:00
|
|
|
if (boot_cpu_has(X86_FEATURE_PGE)) {
|
2014-10-25 06:58:07 +08:00
|
|
|
cr4_set_bits_and_update_boot(X86_CR4_PGE);
|
x86/pti: Never implicitly clear _PAGE_GLOBAL for kernel image
Summary:
In current kernels, with PTI enabled, no pages are marked Global. This
potentially increases TLB misses. But, the mechanism by which the Global
bit is set and cleared is rather haphazard. This patch makes the process
more explicit. In the end, it leaves us with Global entries in the page
tables for the areas truly shared by userspace and kernel and increases
TLB hit rates.
The place this patch really shines in on systems without PCIDs. In this
case, we are using an lseek microbenchmark[1] to see how a reasonably
non-trivial syscall behaves. Higher is better:
No Global pages (baseline): 6077741 lseeks/sec
88 Global Pages (this set): 7528609 lseeks/sec (+23.9%)
On a modern Skylake desktop with PCIDs, the benefits are tangible, but not
huge for a kernel compile (lower is better):
No Global pages (baseline): 186.951 seconds time elapsed ( +- 0.35% )
28 Global pages (this set): 185.756 seconds time elapsed ( +- 0.09% )
-1.195 seconds (-0.64%)
I also re-checked everything using the lseek1 test[1]:
No Global pages (baseline): 15783951 lseeks/sec
28 Global pages (this set): 16054688 lseeks/sec
+270737 lseeks/sec (+1.71%)
The effect is more visible, but still modest.
Details:
The kernel page tables are inherited from head_64.S which rudely marks
them as _PAGE_GLOBAL. For PTI, we have been relying on the grace of
$DEITY and some insane behavior in pageattr.c to clear _PAGE_GLOBAL.
This patch tries to do better.
First, stop filtering out "unsupported" bits from being cleared in the
pageattr code. It's fine to filter out *setting* these bits but it
is insane to keep us from clearing them.
Then, *explicitly* go clear _PAGE_GLOBAL from the kernel identity map.
Do not rely on pageattr to do it magically.
After this patch, we can see that "GLB" shows up in each copy of the
page tables, that we have the same number of global entries in each
and that they are the *same* entries.
/sys/kernel/debug/page_tables/current_kernel:11
/sys/kernel/debug/page_tables/current_user:11
/sys/kernel/debug/page_tables/kernel:11
9caae8ad6a1fb53aca2407ec037f612d current_kernel.GLB
9caae8ad6a1fb53aca2407ec037f612d current_user.GLB
9caae8ad6a1fb53aca2407ec037f612d kernel.GLB
A quick visual audit also shows that all the entries make sense.
0xfffffe0000000000 is the cpu_entry_area and 0xffffffff81c00000
is the entry/exit text:
0xfffffe0000000000-0xfffffe0000002000 8K ro GLB NX pte
0xfffffe0000002000-0xfffffe0000003000 4K RW GLB NX pte
0xfffffe0000003000-0xfffffe0000006000 12K ro GLB NX pte
0xfffffe0000006000-0xfffffe0000007000 4K ro GLB x pte
0xfffffe0000007000-0xfffffe000000d000 24K RW GLB NX pte
0xfffffe000002d000-0xfffffe000002e000 4K ro GLB NX pte
0xfffffe000002e000-0xfffffe000002f000 4K RW GLB NX pte
0xfffffe000002f000-0xfffffe0000032000 12K ro GLB NX pte
0xfffffe0000032000-0xfffffe0000033000 4K ro GLB x pte
0xfffffe0000033000-0xfffffe0000039000 24K RW GLB NX pte
0xffffffff81c00000-0xffffffff81e00000 2M ro PSE GLB x pmd
[1.] https://github.com/antonblanchard/will-it-scale/blob/master/tests/lseek1.c
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Arjan van de Ven <arjan@linux.intel.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Kees Cook <keescook@google.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Nadav Amit <namit@vmware.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-mm@kvack.org
Link: http://lkml.kernel.org/r/20180406205517.C80FBE05@viggo.jf.intel.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-04-07 04:55:17 +08:00
|
|
|
__supported_pte_mask |= _PAGE_GLOBAL;
|
2017-12-04 22:07:34 +08:00
|
|
|
}
|
2015-03-05 15:25:01 +08:00
|
|
|
|
2018-04-07 04:55:06 +08:00
|
|
|
/* By the default is everything supported: */
|
|
|
|
__default_kernel_pte_mask = __supported_pte_mask;
|
|
|
|
/* Except when with PTI where the kernel is mostly non-Global: */
|
|
|
|
if (cpu_feature_enabled(X86_FEATURE_PTI))
|
|
|
|
__default_kernel_pte_mask &= ~_PAGE_GLOBAL;
|
|
|
|
|
2015-03-05 15:25:01 +08:00
|
|
|
/* Enable 1 GB linear kernel mappings if available: */
|
2016-03-29 23:41:58 +08:00
|
|
|
if (direct_gbpages && boot_cpu_has(X86_FEATURE_GBPAGES)) {
|
2015-03-05 15:25:01 +08:00
|
|
|
printk(KERN_INFO "Using GB pages for direct mapping\n");
|
|
|
|
page_size_mask |= 1 << PG_LEVEL_1G;
|
|
|
|
} else {
|
|
|
|
direct_gbpages = 0;
|
|
|
|
}
|
2012-11-17 11:38:38 +08:00
|
|
|
}
|
2011-04-14 22:49:41 +08:00
|
|
|
|
2017-09-11 08:48:27 +08:00
|
|
|
static void setup_pcid(void)
|
|
|
|
{
|
2017-12-04 22:08:01 +08:00
|
|
|
if (!IS_ENABLED(CONFIG_X86_64))
|
|
|
|
return;
|
|
|
|
|
|
|
|
if (!boot_cpu_has(X86_FEATURE_PCID))
|
|
|
|
return;
|
|
|
|
|
|
|
|
if (boot_cpu_has(X86_FEATURE_PGE)) {
|
|
|
|
/*
|
|
|
|
* This can't be cr4_set_bits_and_update_boot() -- the
|
|
|
|
* trampoline code can't handle CR4.PCIDE and it wouldn't
|
|
|
|
* do any good anyway. Despite the name,
|
|
|
|
* cr4_set_bits_and_update_boot() doesn't actually cause
|
|
|
|
* the bits in question to remain set all the way through
|
|
|
|
* the secondary boot asm.
|
|
|
|
*
|
|
|
|
* Instead, we brute-force it and set CR4.PCIDE manually in
|
|
|
|
* start_secondary().
|
|
|
|
*/
|
|
|
|
cr4_set_bits(X86_CR4_PCIDE);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* INVPCID's single-context modes (2/3) only work if we set
|
|
|
|
* X86_CR4_PCIDE, *and* we INVPCID support. It's unusable
|
|
|
|
* on systems that have X86_CR4_PCIDE clear, or that have
|
|
|
|
* no INVPCID support at all.
|
|
|
|
*/
|
|
|
|
if (boot_cpu_has(X86_FEATURE_INVPCID))
|
|
|
|
setup_force_cpu_cap(X86_FEATURE_INVPCID_SINGLE);
|
|
|
|
} else {
|
|
|
|
/*
|
|
|
|
* flush_tlb_all(), as currently implemented, won't work if
|
|
|
|
* PCID is on but PGE is not. Since that combination
|
|
|
|
* doesn't exist on real hardware, there's no reason to try
|
|
|
|
* to fully support it, but it's polite to avoid corrupting
|
|
|
|
* data if we're on an improperly configured VM.
|
|
|
|
*/
|
|
|
|
setup_clear_cpu_cap(X86_FEATURE_PCID);
|
2017-09-11 08:48:27 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2009-03-05 20:55:05 +08:00
|
|
|
#ifdef CONFIG_X86_32
|
|
|
|
#define NR_RANGE_MR 3
|
|
|
|
#else /* CONFIG_X86_64 */
|
|
|
|
#define NR_RANGE_MR 5
|
|
|
|
#endif
|
|
|
|
|
2009-03-12 20:40:06 +08:00
|
|
|
static int __meminit save_mr(struct map_range *mr, int nr_range,
|
|
|
|
unsigned long start_pfn, unsigned long end_pfn,
|
|
|
|
unsigned long page_size_mask)
|
2009-03-05 20:55:05 +08:00
|
|
|
{
|
|
|
|
if (start_pfn < end_pfn) {
|
|
|
|
if (nr_range >= NR_RANGE_MR)
|
|
|
|
panic("run out of range for init_memory_mapping\n");
|
|
|
|
mr[nr_range].start = start_pfn<<PAGE_SHIFT;
|
|
|
|
mr[nr_range].end = end_pfn<<PAGE_SHIFT;
|
|
|
|
mr[nr_range].page_size_mask = page_size_mask;
|
|
|
|
nr_range++;
|
|
|
|
}
|
|
|
|
|
|
|
|
return nr_range;
|
|
|
|
}
|
|
|
|
|
2012-11-17 11:38:55 +08:00
|
|
|
/*
|
|
|
|
* adjust the page_size_mask for small range to go with
|
|
|
|
* big page size instead small one if nearby are ram too.
|
|
|
|
*/
|
2016-08-03 05:03:33 +08:00
|
|
|
static void __ref adjust_range_page_size_mask(struct map_range *mr,
|
2012-11-17 11:38:55 +08:00
|
|
|
int nr_range)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < nr_range; i++) {
|
|
|
|
if ((page_size_mask & (1<<PG_LEVEL_2M)) &&
|
|
|
|
!(mr[i].page_size_mask & (1<<PG_LEVEL_2M))) {
|
|
|
|
unsigned long start = round_down(mr[i].start, PMD_SIZE);
|
|
|
|
unsigned long end = round_up(mr[i].end, PMD_SIZE);
|
|
|
|
|
|
|
|
#ifdef CONFIG_X86_32
|
|
|
|
if ((end >> PAGE_SHIFT) > max_low_pfn)
|
|
|
|
continue;
|
|
|
|
#endif
|
|
|
|
|
|
|
|
if (memblock_is_region_memory(start, end - start))
|
|
|
|
mr[i].page_size_mask |= 1<<PG_LEVEL_2M;
|
|
|
|
}
|
|
|
|
if ((page_size_mask & (1<<PG_LEVEL_1G)) &&
|
|
|
|
!(mr[i].page_size_mask & (1<<PG_LEVEL_1G))) {
|
|
|
|
unsigned long start = round_down(mr[i].start, PUD_SIZE);
|
|
|
|
unsigned long end = round_up(mr[i].end, PUD_SIZE);
|
|
|
|
|
|
|
|
if (memblock_is_region_memory(start, end - start))
|
|
|
|
mr[i].page_size_mask |= 1<<PG_LEVEL_1G;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-02-11 05:20:30 +08:00
|
|
|
static const char *page_size_string(struct map_range *mr)
|
|
|
|
{
|
|
|
|
static const char str_1g[] = "1G";
|
|
|
|
static const char str_2m[] = "2M";
|
|
|
|
static const char str_4m[] = "4M";
|
|
|
|
static const char str_4k[] = "4k";
|
|
|
|
|
|
|
|
if (mr->page_size_mask & (1<<PG_LEVEL_1G))
|
|
|
|
return str_1g;
|
|
|
|
/*
|
|
|
|
* 32-bit without PAE has a 4M large page size.
|
|
|
|
* PG_LEVEL_2M is misnamed, but we can at least
|
|
|
|
* print out the right size in the string.
|
|
|
|
*/
|
|
|
|
if (IS_ENABLED(CONFIG_X86_32) &&
|
|
|
|
!IS_ENABLED(CONFIG_X86_PAE) &&
|
|
|
|
mr->page_size_mask & (1<<PG_LEVEL_2M))
|
|
|
|
return str_4m;
|
|
|
|
|
|
|
|
if (mr->page_size_mask & (1<<PG_LEVEL_2M))
|
|
|
|
return str_2m;
|
|
|
|
|
|
|
|
return str_4k;
|
|
|
|
}
|
|
|
|
|
2012-11-17 11:38:39 +08:00
|
|
|
static int __meminit split_mem_range(struct map_range *mr, int nr_range,
|
|
|
|
unsigned long start,
|
|
|
|
unsigned long end)
|
2009-03-05 20:55:05 +08:00
|
|
|
{
|
2012-11-17 11:39:15 +08:00
|
|
|
unsigned long start_pfn, end_pfn, limit_pfn;
|
2012-11-17 11:39:14 +08:00
|
|
|
unsigned long pfn;
|
2012-11-17 11:38:39 +08:00
|
|
|
int i;
|
2009-03-05 20:55:05 +08:00
|
|
|
|
2012-11-17 11:39:15 +08:00
|
|
|
limit_pfn = PFN_DOWN(end);
|
|
|
|
|
2009-03-05 20:55:05 +08:00
|
|
|
/* head if not big page alignment ? */
|
2012-11-17 11:39:14 +08:00
|
|
|
pfn = start_pfn = PFN_DOWN(start);
|
2009-03-05 20:55:05 +08:00
|
|
|
#ifdef CONFIG_X86_32
|
|
|
|
/*
|
|
|
|
* Don't use a large page for the first 2/4MB of memory
|
|
|
|
* because there are often fixed size MTRRs in there
|
|
|
|
* and overlapping MTRRs into large pages can cause
|
|
|
|
* slowdowns.
|
|
|
|
*/
|
2012-11-17 11:39:14 +08:00
|
|
|
if (pfn == 0)
|
2012-11-17 11:39:13 +08:00
|
|
|
end_pfn = PFN_DOWN(PMD_SIZE);
|
2009-03-05 20:55:05 +08:00
|
|
|
else
|
2012-11-17 11:39:14 +08:00
|
|
|
end_pfn = round_up(pfn, PFN_DOWN(PMD_SIZE));
|
2009-03-05 20:55:05 +08:00
|
|
|
#else /* CONFIG_X86_64 */
|
2012-11-17 11:39:14 +08:00
|
|
|
end_pfn = round_up(pfn, PFN_DOWN(PMD_SIZE));
|
2009-03-05 20:55:05 +08:00
|
|
|
#endif
|
2012-11-17 11:39:15 +08:00
|
|
|
if (end_pfn > limit_pfn)
|
|
|
|
end_pfn = limit_pfn;
|
2009-03-05 20:55:05 +08:00
|
|
|
if (start_pfn < end_pfn) {
|
|
|
|
nr_range = save_mr(mr, nr_range, start_pfn, end_pfn, 0);
|
2012-11-17 11:39:14 +08:00
|
|
|
pfn = end_pfn;
|
2009-03-05 20:55:05 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/* big page (2M) range */
|
2012-11-17 11:39:14 +08:00
|
|
|
start_pfn = round_up(pfn, PFN_DOWN(PMD_SIZE));
|
2009-03-05 20:55:05 +08:00
|
|
|
#ifdef CONFIG_X86_32
|
2012-11-17 11:39:15 +08:00
|
|
|
end_pfn = round_down(limit_pfn, PFN_DOWN(PMD_SIZE));
|
2009-03-05 20:55:05 +08:00
|
|
|
#else /* CONFIG_X86_64 */
|
2012-11-17 11:39:14 +08:00
|
|
|
end_pfn = round_up(pfn, PFN_DOWN(PUD_SIZE));
|
2012-11-17 11:39:15 +08:00
|
|
|
if (end_pfn > round_down(limit_pfn, PFN_DOWN(PMD_SIZE)))
|
|
|
|
end_pfn = round_down(limit_pfn, PFN_DOWN(PMD_SIZE));
|
2009-03-05 20:55:05 +08:00
|
|
|
#endif
|
|
|
|
|
|
|
|
if (start_pfn < end_pfn) {
|
|
|
|
nr_range = save_mr(mr, nr_range, start_pfn, end_pfn,
|
|
|
|
page_size_mask & (1<<PG_LEVEL_2M));
|
2012-11-17 11:39:14 +08:00
|
|
|
pfn = end_pfn;
|
2009-03-05 20:55:05 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
#ifdef CONFIG_X86_64
|
|
|
|
/* big page (1G) range */
|
2012-11-17 11:39:14 +08:00
|
|
|
start_pfn = round_up(pfn, PFN_DOWN(PUD_SIZE));
|
2012-11-17 11:39:15 +08:00
|
|
|
end_pfn = round_down(limit_pfn, PFN_DOWN(PUD_SIZE));
|
2009-03-05 20:55:05 +08:00
|
|
|
if (start_pfn < end_pfn) {
|
|
|
|
nr_range = save_mr(mr, nr_range, start_pfn, end_pfn,
|
|
|
|
page_size_mask &
|
|
|
|
((1<<PG_LEVEL_2M)|(1<<PG_LEVEL_1G)));
|
2012-11-17 11:39:14 +08:00
|
|
|
pfn = end_pfn;
|
2009-03-05 20:55:05 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/* tail is not big page (1G) alignment */
|
2012-11-17 11:39:14 +08:00
|
|
|
start_pfn = round_up(pfn, PFN_DOWN(PMD_SIZE));
|
2012-11-17 11:39:15 +08:00
|
|
|
end_pfn = round_down(limit_pfn, PFN_DOWN(PMD_SIZE));
|
2009-03-05 20:55:05 +08:00
|
|
|
if (start_pfn < end_pfn) {
|
|
|
|
nr_range = save_mr(mr, nr_range, start_pfn, end_pfn,
|
|
|
|
page_size_mask & (1<<PG_LEVEL_2M));
|
2012-11-17 11:39:14 +08:00
|
|
|
pfn = end_pfn;
|
2009-03-05 20:55:05 +08:00
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
|
|
|
/* tail is not big page (2M) alignment */
|
2012-11-17 11:39:14 +08:00
|
|
|
start_pfn = pfn;
|
2012-11-17 11:39:15 +08:00
|
|
|
end_pfn = limit_pfn;
|
2009-03-05 20:55:05 +08:00
|
|
|
nr_range = save_mr(mr, nr_range, start_pfn, end_pfn, 0);
|
|
|
|
|
x86: Fix adjust_range_size_mask calling position
Commit
8d57470d x86, mm: setup page table in top-down
causes a kernel panic while setting mem=2G.
[mem 0x00000000-0x000fffff] page 4k
[mem 0x7fe00000-0x7fffffff] page 1G
[mem 0x7c000000-0x7fdfffff] page 1G
[mem 0x00100000-0x001fffff] page 4k
[mem 0x00200000-0x7bffffff] page 2M
for last entry is not what we want, we should have
[mem 0x00200000-0x3fffffff] page 2M
[mem 0x40000000-0x7bffffff] page 1G
Actually we merge the continuous ranges with same page size too early.
in this case, before merging we have
[mem 0x00200000-0x3fffffff] page 2M
[mem 0x40000000-0x7bffffff] page 2M
after merging them, will get
[mem 0x00200000-0x7bffffff] page 2M
even we can use 1G page to map
[mem 0x40000000-0x7bffffff]
that will cause problem, because we already map
[mem 0x7fe00000-0x7fffffff] page 1G
[mem 0x7c000000-0x7fdfffff] page 1G
with 1G page, aka [0x40000000-0x7fffffff] is mapped with 1G page already.
During phys_pud_init() for [0x40000000-0x7bffffff], it will not
reuse existing that pud page, and allocate new one then try to use
2M page to map it instead, as page_size_mask does not include
PG_LEVEL_1G. At end will have [7c000000-0x7fffffff] not mapped, loop
in phys_pmd_init stop mapping at 0x7bffffff.
That is right behavoir, it maps exact range with exact page size that
we ask, and we should explicitly call it to map [7c000000-0x7fffffff]
before or after mapping 0x40000000-0x7bffffff.
Anyway we need to make sure ranges' page_size_mask correct and consistent
after split_mem_range for each range.
Fix that by calling adjust_range_size_mask before merging range
with same page size.
-v2: update change log.
-v3: add more explanation why [7c000000-0x7fffffff] is not mapped, and
it causes panic.
Bisected-by: "Xie, ChanglongX" <changlongx.xie@intel.com>
Bisected-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Reported-and-tested-by: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1370015587-20835-1-git-send-email-yinghai@kernel.org
Cc: <stable@vger.kernel.org> v3.9
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-05-31 23:53:07 +08:00
|
|
|
if (!after_bootmem)
|
|
|
|
adjust_range_page_size_mask(mr, nr_range);
|
|
|
|
|
2009-03-05 20:55:05 +08:00
|
|
|
/* try to merge same page size and continuous */
|
|
|
|
for (i = 0; nr_range > 1 && i < nr_range - 1; i++) {
|
|
|
|
unsigned long old_start;
|
|
|
|
if (mr[i].end != mr[i+1].start ||
|
|
|
|
mr[i].page_size_mask != mr[i+1].page_size_mask)
|
|
|
|
continue;
|
|
|
|
/* move it */
|
|
|
|
old_start = mr[i].start;
|
|
|
|
memmove(&mr[i], &mr[i+1],
|
|
|
|
(nr_range - 1 - i) * sizeof(struct map_range));
|
|
|
|
mr[i--].start = old_start;
|
|
|
|
nr_range--;
|
|
|
|
}
|
|
|
|
|
|
|
|
for (i = 0; i < nr_range; i++)
|
2015-09-18 04:27:57 +08:00
|
|
|
pr_debug(" [mem %#010lx-%#010lx] page %s\n",
|
2012-05-30 06:06:29 +08:00
|
|
|
mr[i].start, mr[i].end - 1,
|
2015-02-11 05:20:30 +08:00
|
|
|
page_size_string(&mr[i]));
|
2009-03-05 20:55:05 +08:00
|
|
|
|
2012-11-17 11:38:39 +08:00
|
|
|
return nr_range;
|
|
|
|
}
|
|
|
|
|
2017-01-29 00:29:08 +08:00
|
|
|
struct range pfn_mapped[E820_MAX_ENTRIES];
|
2013-01-25 04:20:05 +08:00
|
|
|
int nr_pfn_mapped;
|
2012-11-17 11:38:52 +08:00
|
|
|
|
|
|
|
static void add_pfn_range_mapped(unsigned long start_pfn, unsigned long end_pfn)
|
|
|
|
{
|
2017-01-29 00:29:08 +08:00
|
|
|
nr_pfn_mapped = add_range_with_merge(pfn_mapped, E820_MAX_ENTRIES,
|
2012-11-17 11:38:52 +08:00
|
|
|
nr_pfn_mapped, start_pfn, end_pfn);
|
2017-01-29 00:29:08 +08:00
|
|
|
nr_pfn_mapped = clean_sort_range(pfn_mapped, E820_MAX_ENTRIES);
|
2012-11-17 11:38:52 +08:00
|
|
|
|
|
|
|
max_pfn_mapped = max(max_pfn_mapped, end_pfn);
|
|
|
|
|
|
|
|
if (start_pfn < (1UL<<(32-PAGE_SHIFT)))
|
|
|
|
max_low_pfn_mapped = max(max_low_pfn_mapped,
|
|
|
|
min(end_pfn, 1UL<<(32-PAGE_SHIFT)));
|
|
|
|
}
|
|
|
|
|
|
|
|
bool pfn_range_is_mapped(unsigned long start_pfn, unsigned long end_pfn)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < nr_pfn_mapped; i++)
|
|
|
|
if ((start_pfn >= pfn_mapped[i].start) &&
|
|
|
|
(end_pfn <= pfn_mapped[i].end))
|
|
|
|
return true;
|
|
|
|
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2012-11-17 11:38:39 +08:00
|
|
|
/*
|
|
|
|
* Setup the direct mapping of the physical memory at PAGE_OFFSET.
|
|
|
|
* This runs before bootmem is initialized and gets pages directly from
|
|
|
|
* the physical memory. To access them they are temporarily mapped.
|
|
|
|
*/
|
2016-08-03 05:03:33 +08:00
|
|
|
unsigned long __ref init_memory_mapping(unsigned long start,
|
2020-04-11 05:33:24 +08:00
|
|
|
unsigned long end, pgprot_t prot)
|
2012-11-17 11:38:39 +08:00
|
|
|
{
|
|
|
|
struct map_range mr[NR_RANGE_MR];
|
|
|
|
unsigned long ret = 0;
|
|
|
|
int nr_range, i;
|
|
|
|
|
2015-09-18 04:27:57 +08:00
|
|
|
pr_debug("init_memory_mapping: [mem %#010lx-%#010lx]\n",
|
2012-11-17 11:38:39 +08:00
|
|
|
start, end - 1);
|
|
|
|
|
|
|
|
memset(mr, 0, sizeof(mr));
|
|
|
|
nr_range = split_mem_range(mr, 0, start, end);
|
|
|
|
|
2009-03-05 20:55:05 +08:00
|
|
|
for (i = 0; i < nr_range; i++)
|
|
|
|
ret = kernel_physical_mapping_init(mr[i].start, mr[i].end,
|
2020-04-11 05:33:24 +08:00
|
|
|
mr[i].page_size_mask,
|
|
|
|
prot);
|
2009-03-05 20:55:05 +08:00
|
|
|
|
2012-11-17 11:38:52 +08:00
|
|
|
add_pfn_range_mapped(start >> PAGE_SHIFT, ret >> PAGE_SHIFT);
|
|
|
|
|
2012-11-17 11:38:44 +08:00
|
|
|
return ret >> PAGE_SHIFT;
|
|
|
|
}
|
|
|
|
|
2012-11-17 11:38:52 +08:00
|
|
|
/*
|
2013-05-09 23:57:42 +08:00
|
|
|
* We need to iterate through the E820 memory map and create direct mappings
|
2017-01-29 00:09:33 +08:00
|
|
|
* for only E820_TYPE_RAM and E820_KERN_RESERVED regions. We cannot simply
|
2013-05-09 23:57:42 +08:00
|
|
|
* create direct mappings for all pfns from [0 to max_low_pfn) and
|
|
|
|
* [4GB to max_pfn) because of possible memory holes in high addresses
|
|
|
|
* that cannot be marked as UC by fixed/variable range MTRRs.
|
|
|
|
* Depending on the alignment of E820 ranges, this may possibly result
|
|
|
|
* in using smaller size (i.e. 4K instead of 2M or 1G) page tables.
|
|
|
|
*
|
|
|
|
* init_mem_mapping() calls init_range_memory_mapping() with big range.
|
|
|
|
* That range would have hole in the middle or ends, and only ram parts
|
|
|
|
* will be mapped in init_range_memory_mapping().
|
2012-11-17 11:38:52 +08:00
|
|
|
*/
|
2012-11-17 11:38:58 +08:00
|
|
|
static unsigned long __init init_range_memory_mapping(
|
2012-11-17 11:39:18 +08:00
|
|
|
unsigned long r_start,
|
|
|
|
unsigned long r_end)
|
2012-11-17 11:38:52 +08:00
|
|
|
{
|
|
|
|
unsigned long start_pfn, end_pfn;
|
2012-11-17 11:38:58 +08:00
|
|
|
unsigned long mapped_ram_size = 0;
|
2012-11-17 11:38:52 +08:00
|
|
|
int i;
|
|
|
|
|
|
|
|
for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, NULL) {
|
2012-11-17 11:39:18 +08:00
|
|
|
u64 start = clamp_val(PFN_PHYS(start_pfn), r_start, r_end);
|
|
|
|
u64 end = clamp_val(PFN_PHYS(end_pfn), r_start, r_end);
|
|
|
|
if (start >= end)
|
2012-11-17 11:38:52 +08:00
|
|
|
continue;
|
|
|
|
|
2013-01-25 04:19:42 +08:00
|
|
|
/*
|
|
|
|
* if it is overlapping with brk pgt, we need to
|
|
|
|
* alloc pgt buf from memblock instead.
|
|
|
|
*/
|
|
|
|
can_use_brk_pgt = max(start, (u64)pgt_buf_end<<PAGE_SHIFT) >=
|
|
|
|
min(end, (u64)pgt_buf_top<<PAGE_SHIFT);
|
2020-04-11 05:33:24 +08:00
|
|
|
init_memory_mapping(start, end, PAGE_KERNEL);
|
2012-11-17 11:38:58 +08:00
|
|
|
mapped_ram_size += end - start;
|
2013-01-25 04:19:42 +08:00
|
|
|
can_use_brk_pgt = true;
|
2012-11-17 11:38:52 +08:00
|
|
|
}
|
2012-11-17 11:38:58 +08:00
|
|
|
|
|
|
|
return mapped_ram_size;
|
2012-11-17 11:38:52 +08:00
|
|
|
}
|
|
|
|
|
2013-09-07 10:07:09 +08:00
|
|
|
static unsigned long __init get_new_step_size(unsigned long step_size)
|
|
|
|
{
|
|
|
|
/*
|
2014-12-20 00:10:54 +08:00
|
|
|
* Initial mapped size is PMD_SIZE (2M).
|
2013-09-07 10:07:09 +08:00
|
|
|
* We can not set step_size to be PUD_SIZE (1G) yet.
|
|
|
|
* In worse case, when we cross the 1G boundary, and
|
|
|
|
* PG_LEVEL_2M is not set, we will need 1+1+512 pages (2M + 8k)
|
2014-12-20 00:10:54 +08:00
|
|
|
* to map 1G range with PTE. Hence we use one less than the
|
|
|
|
* difference of page table level shifts.
|
2013-09-07 10:07:09 +08:00
|
|
|
*
|
2014-12-20 00:10:54 +08:00
|
|
|
* Don't need to worry about overflow in the top-down case, on 32bit,
|
|
|
|
* when step_size is 0, round_down() returns 0 for start, and that
|
|
|
|
* turns it into 0x100000000ULL.
|
|
|
|
* In the bottom-up case, round_up(x, 0) returns 0 though too, which
|
|
|
|
* needs to be taken into consideration by the code below.
|
2013-09-07 10:07:09 +08:00
|
|
|
*/
|
2014-12-20 00:10:54 +08:00
|
|
|
return step_size << (PMD_SHIFT - PAGE_SHIFT - 1);
|
2013-09-07 10:07:09 +08:00
|
|
|
}
|
|
|
|
|
2013-11-13 07:08:02 +08:00
|
|
|
/**
|
|
|
|
* memory_map_top_down - Map [map_start, map_end) top down
|
|
|
|
* @map_start: start address of the target memory range
|
|
|
|
* @map_end: end address of the target memory range
|
|
|
|
*
|
|
|
|
* This function will setup direct mapping for memory range
|
|
|
|
* [map_start, map_end) in top-down. That said, the page tables
|
|
|
|
* will be allocated at the end of the memory, and we map the
|
|
|
|
* memory in top-down.
|
|
|
|
*/
|
|
|
|
static void __init memory_map_top_down(unsigned long map_start,
|
|
|
|
unsigned long map_end)
|
2012-11-17 11:38:44 +08:00
|
|
|
{
|
2020-09-28 18:00:04 +08:00
|
|
|
unsigned long real_end, last_start;
|
2012-11-17 11:38:58 +08:00
|
|
|
unsigned long step_size;
|
|
|
|
unsigned long addr;
|
|
|
|
unsigned long mapped_ram_size = 0;
|
2012-11-17 11:38:45 +08:00
|
|
|
|
2021-09-03 06:00:26 +08:00
|
|
|
/*
|
|
|
|
* Systems that have many reserved areas near top of the memory,
|
|
|
|
* e.g. QEMU with less than 1G RAM and EFI enabled, or Xen, will
|
|
|
|
* require lots of 4K mappings which may exhaust pgt_buf.
|
|
|
|
* Start with top-most PMD_SIZE range aligned at PMD_SIZE to ensure
|
|
|
|
* there is enough mapped memory that can be allocated from
|
|
|
|
* memblock.
|
|
|
|
*/
|
|
|
|
addr = memblock_phys_alloc_range(PMD_SIZE, PMD_SIZE, map_start,
|
|
|
|
map_end);
|
2021-11-06 04:43:19 +08:00
|
|
|
memblock_phys_free(addr, PMD_SIZE);
|
2012-11-17 11:38:58 +08:00
|
|
|
real_end = addr + PMD_SIZE;
|
|
|
|
|
|
|
|
/* step_size need to be small so pgt_buf from BRK could cover it */
|
|
|
|
step_size = PMD_SIZE;
|
|
|
|
max_pfn_mapped = 0; /* will get exact value next */
|
|
|
|
min_pfn_mapped = real_end >> PAGE_SHIFT;
|
2020-09-28 18:00:04 +08:00
|
|
|
last_start = real_end;
|
2013-05-09 23:57:42 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* We start from the top (end of memory) and go to the bottom.
|
|
|
|
* The memblock_find_in_range() gets us a block of RAM from the
|
|
|
|
* end of RAM in [min_pfn_mapped, max_pfn_mapped) used as new pages
|
|
|
|
* for page table.
|
|
|
|
*/
|
2013-11-13 07:08:02 +08:00
|
|
|
while (last_start > map_start) {
|
2020-09-28 18:00:04 +08:00
|
|
|
unsigned long start;
|
|
|
|
|
2012-11-17 11:38:58 +08:00
|
|
|
if (last_start > step_size) {
|
|
|
|
start = round_down(last_start - 1, step_size);
|
2013-11-13 07:08:02 +08:00
|
|
|
if (start < map_start)
|
|
|
|
start = map_start;
|
2012-11-17 11:38:58 +08:00
|
|
|
} else
|
2013-11-13 07:08:02 +08:00
|
|
|
start = map_start;
|
2014-12-20 00:10:54 +08:00
|
|
|
mapped_ram_size += init_range_memory_mapping(start,
|
2012-11-17 11:38:58 +08:00
|
|
|
last_start);
|
|
|
|
last_start = start;
|
|
|
|
min_pfn_mapped = last_start >> PAGE_SHIFT;
|
2014-12-20 00:10:54 +08:00
|
|
|
if (mapped_ram_size >= step_size)
|
2013-09-07 10:07:09 +08:00
|
|
|
step_size = get_new_step_size(step_size);
|
2012-11-17 11:38:58 +08:00
|
|
|
}
|
|
|
|
|
2013-11-13 07:08:02 +08:00
|
|
|
if (real_end < map_end)
|
|
|
|
init_range_memory_mapping(real_end, map_end);
|
|
|
|
}
|
|
|
|
|
x86/mem-hotplug: support initialize page tables in bottom-up
The Linux kernel cannot migrate pages used by the kernel. As a result,
kernel pages cannot be hot-removed. So we cannot allocate hotpluggable
memory for the kernel.
In a memory hotplug system, any numa node the kernel resides in should be
unhotpluggable. And for a modern server, each node could have at least
16GB memory. So memory around the kernel image is highly likely
unhotpluggable.
ACPI SRAT (System Resource Affinity Table) contains the memory hotplug
info. But before SRAT is parsed, memblock has already started to allocate
memory for the kernel. So we need to prevent memblock from doing this.
So direct memory mapping page tables setup is the case.
init_mem_mapping() is called before SRAT is parsed. To prevent page
tables being allocated within hotpluggable memory, we will use bottom-up
direction to allocate page tables from the end of kernel image to the
higher memory.
Note:
As for allocating page tables in lower memory, TJ said:
: This is an optional behavior which is triggered by a very specific kernel
: boot param, which I suspect is gonna need to stick around to support
: memory hotplug in the current setup unless we add another layer of address
: translation to support memory hotplug.
As for page tables may occupy too much lower memory if using 4K mapping
(CONFIG_DEBUG_PAGEALLOC and CONFIG_KMEMCHECK both disable using >4k
pages), TJ said:
: But as I said in the same paragraph, parsing SRAT earlier doesn't solve
: the problem in itself either. Ignoring the option if 4k mapping is
: required and memory consumption would be prohibitive should work, no?
: Something like that would be necessary if we're gonna worry about cases
: like this no matter how we implement it, but, frankly, I'm not sure this
: is something worth worrying about.
Signed-off-by: Tang Chen <tangchen@cn.fujitsu.com>
Signed-off-by: Zhang Yanfei <zhangyanfei@cn.fujitsu.com>
Acked-by: Tejun Heo <tj@kernel.org>
Acked-by: Toshi Kani <toshi.kani@hp.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Wanpeng Li <liwanp@linux.vnet.ibm.com>
Cc: Thomas Renninger <trenn@suse.de>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Jiang Liu <jiang.liu@huawei.com>
Cc: Wen Congyang <wency@cn.fujitsu.com>
Cc: Lai Jiangshan <laijs@cn.fujitsu.com>
Cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Cc: Taku Izumi <izumi.taku@jp.fujitsu.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Michal Nazarewicz <mina86@mina86.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-11-13 07:08:05 +08:00
|
|
|
/**
|
|
|
|
* memory_map_bottom_up - Map [map_start, map_end) bottom up
|
|
|
|
* @map_start: start address of the target memory range
|
|
|
|
* @map_end: end address of the target memory range
|
|
|
|
*
|
|
|
|
* This function will setup direct mapping for memory range
|
|
|
|
* [map_start, map_end) in bottom-up. Since we have limited the
|
|
|
|
* bottom-up allocation above the kernel, the page tables will
|
|
|
|
* be allocated just above the kernel and we map the memory
|
|
|
|
* in [map_start, map_end) in bottom-up.
|
|
|
|
*/
|
|
|
|
static void __init memory_map_bottom_up(unsigned long map_start,
|
|
|
|
unsigned long map_end)
|
|
|
|
{
|
2014-12-20 00:10:54 +08:00
|
|
|
unsigned long next, start;
|
x86/mem-hotplug: support initialize page tables in bottom-up
The Linux kernel cannot migrate pages used by the kernel. As a result,
kernel pages cannot be hot-removed. So we cannot allocate hotpluggable
memory for the kernel.
In a memory hotplug system, any numa node the kernel resides in should be
unhotpluggable. And for a modern server, each node could have at least
16GB memory. So memory around the kernel image is highly likely
unhotpluggable.
ACPI SRAT (System Resource Affinity Table) contains the memory hotplug
info. But before SRAT is parsed, memblock has already started to allocate
memory for the kernel. So we need to prevent memblock from doing this.
So direct memory mapping page tables setup is the case.
init_mem_mapping() is called before SRAT is parsed. To prevent page
tables being allocated within hotpluggable memory, we will use bottom-up
direction to allocate page tables from the end of kernel image to the
higher memory.
Note:
As for allocating page tables in lower memory, TJ said:
: This is an optional behavior which is triggered by a very specific kernel
: boot param, which I suspect is gonna need to stick around to support
: memory hotplug in the current setup unless we add another layer of address
: translation to support memory hotplug.
As for page tables may occupy too much lower memory if using 4K mapping
(CONFIG_DEBUG_PAGEALLOC and CONFIG_KMEMCHECK both disable using >4k
pages), TJ said:
: But as I said in the same paragraph, parsing SRAT earlier doesn't solve
: the problem in itself either. Ignoring the option if 4k mapping is
: required and memory consumption would be prohibitive should work, no?
: Something like that would be necessary if we're gonna worry about cases
: like this no matter how we implement it, but, frankly, I'm not sure this
: is something worth worrying about.
Signed-off-by: Tang Chen <tangchen@cn.fujitsu.com>
Signed-off-by: Zhang Yanfei <zhangyanfei@cn.fujitsu.com>
Acked-by: Tejun Heo <tj@kernel.org>
Acked-by: Toshi Kani <toshi.kani@hp.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Wanpeng Li <liwanp@linux.vnet.ibm.com>
Cc: Thomas Renninger <trenn@suse.de>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Jiang Liu <jiang.liu@huawei.com>
Cc: Wen Congyang <wency@cn.fujitsu.com>
Cc: Lai Jiangshan <laijs@cn.fujitsu.com>
Cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Cc: Taku Izumi <izumi.taku@jp.fujitsu.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Michal Nazarewicz <mina86@mina86.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-11-13 07:08:05 +08:00
|
|
|
unsigned long mapped_ram_size = 0;
|
|
|
|
/* step_size need to be small so pgt_buf from BRK could cover it */
|
|
|
|
unsigned long step_size = PMD_SIZE;
|
|
|
|
|
|
|
|
start = map_start;
|
|
|
|
min_pfn_mapped = start >> PAGE_SHIFT;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We start from the bottom (@map_start) and go to the top (@map_end).
|
|
|
|
* The memblock_find_in_range() gets us a block of RAM from the
|
|
|
|
* end of RAM in [min_pfn_mapped, max_pfn_mapped) used as new pages
|
|
|
|
* for page table.
|
|
|
|
*/
|
|
|
|
while (start < map_end) {
|
2014-12-20 00:10:54 +08:00
|
|
|
if (step_size && map_end - start > step_size) {
|
x86/mem-hotplug: support initialize page tables in bottom-up
The Linux kernel cannot migrate pages used by the kernel. As a result,
kernel pages cannot be hot-removed. So we cannot allocate hotpluggable
memory for the kernel.
In a memory hotplug system, any numa node the kernel resides in should be
unhotpluggable. And for a modern server, each node could have at least
16GB memory. So memory around the kernel image is highly likely
unhotpluggable.
ACPI SRAT (System Resource Affinity Table) contains the memory hotplug
info. But before SRAT is parsed, memblock has already started to allocate
memory for the kernel. So we need to prevent memblock from doing this.
So direct memory mapping page tables setup is the case.
init_mem_mapping() is called before SRAT is parsed. To prevent page
tables being allocated within hotpluggable memory, we will use bottom-up
direction to allocate page tables from the end of kernel image to the
higher memory.
Note:
As for allocating page tables in lower memory, TJ said:
: This is an optional behavior which is triggered by a very specific kernel
: boot param, which I suspect is gonna need to stick around to support
: memory hotplug in the current setup unless we add another layer of address
: translation to support memory hotplug.
As for page tables may occupy too much lower memory if using 4K mapping
(CONFIG_DEBUG_PAGEALLOC and CONFIG_KMEMCHECK both disable using >4k
pages), TJ said:
: But as I said in the same paragraph, parsing SRAT earlier doesn't solve
: the problem in itself either. Ignoring the option if 4k mapping is
: required and memory consumption would be prohibitive should work, no?
: Something like that would be necessary if we're gonna worry about cases
: like this no matter how we implement it, but, frankly, I'm not sure this
: is something worth worrying about.
Signed-off-by: Tang Chen <tangchen@cn.fujitsu.com>
Signed-off-by: Zhang Yanfei <zhangyanfei@cn.fujitsu.com>
Acked-by: Tejun Heo <tj@kernel.org>
Acked-by: Toshi Kani <toshi.kani@hp.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Wanpeng Li <liwanp@linux.vnet.ibm.com>
Cc: Thomas Renninger <trenn@suse.de>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Jiang Liu <jiang.liu@huawei.com>
Cc: Wen Congyang <wency@cn.fujitsu.com>
Cc: Lai Jiangshan <laijs@cn.fujitsu.com>
Cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Cc: Taku Izumi <izumi.taku@jp.fujitsu.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Michal Nazarewicz <mina86@mina86.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-11-13 07:08:05 +08:00
|
|
|
next = round_up(start + 1, step_size);
|
|
|
|
if (next > map_end)
|
|
|
|
next = map_end;
|
2014-12-20 00:10:54 +08:00
|
|
|
} else {
|
x86/mem-hotplug: support initialize page tables in bottom-up
The Linux kernel cannot migrate pages used by the kernel. As a result,
kernel pages cannot be hot-removed. So we cannot allocate hotpluggable
memory for the kernel.
In a memory hotplug system, any numa node the kernel resides in should be
unhotpluggable. And for a modern server, each node could have at least
16GB memory. So memory around the kernel image is highly likely
unhotpluggable.
ACPI SRAT (System Resource Affinity Table) contains the memory hotplug
info. But before SRAT is parsed, memblock has already started to allocate
memory for the kernel. So we need to prevent memblock from doing this.
So direct memory mapping page tables setup is the case.
init_mem_mapping() is called before SRAT is parsed. To prevent page
tables being allocated within hotpluggable memory, we will use bottom-up
direction to allocate page tables from the end of kernel image to the
higher memory.
Note:
As for allocating page tables in lower memory, TJ said:
: This is an optional behavior which is triggered by a very specific kernel
: boot param, which I suspect is gonna need to stick around to support
: memory hotplug in the current setup unless we add another layer of address
: translation to support memory hotplug.
As for page tables may occupy too much lower memory if using 4K mapping
(CONFIG_DEBUG_PAGEALLOC and CONFIG_KMEMCHECK both disable using >4k
pages), TJ said:
: But as I said in the same paragraph, parsing SRAT earlier doesn't solve
: the problem in itself either. Ignoring the option if 4k mapping is
: required and memory consumption would be prohibitive should work, no?
: Something like that would be necessary if we're gonna worry about cases
: like this no matter how we implement it, but, frankly, I'm not sure this
: is something worth worrying about.
Signed-off-by: Tang Chen <tangchen@cn.fujitsu.com>
Signed-off-by: Zhang Yanfei <zhangyanfei@cn.fujitsu.com>
Acked-by: Tejun Heo <tj@kernel.org>
Acked-by: Toshi Kani <toshi.kani@hp.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Wanpeng Li <liwanp@linux.vnet.ibm.com>
Cc: Thomas Renninger <trenn@suse.de>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Jiang Liu <jiang.liu@huawei.com>
Cc: Wen Congyang <wency@cn.fujitsu.com>
Cc: Lai Jiangshan <laijs@cn.fujitsu.com>
Cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Cc: Taku Izumi <izumi.taku@jp.fujitsu.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Michal Nazarewicz <mina86@mina86.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-11-13 07:08:05 +08:00
|
|
|
next = map_end;
|
2014-12-20 00:10:54 +08:00
|
|
|
}
|
x86/mem-hotplug: support initialize page tables in bottom-up
The Linux kernel cannot migrate pages used by the kernel. As a result,
kernel pages cannot be hot-removed. So we cannot allocate hotpluggable
memory for the kernel.
In a memory hotplug system, any numa node the kernel resides in should be
unhotpluggable. And for a modern server, each node could have at least
16GB memory. So memory around the kernel image is highly likely
unhotpluggable.
ACPI SRAT (System Resource Affinity Table) contains the memory hotplug
info. But before SRAT is parsed, memblock has already started to allocate
memory for the kernel. So we need to prevent memblock from doing this.
So direct memory mapping page tables setup is the case.
init_mem_mapping() is called before SRAT is parsed. To prevent page
tables being allocated within hotpluggable memory, we will use bottom-up
direction to allocate page tables from the end of kernel image to the
higher memory.
Note:
As for allocating page tables in lower memory, TJ said:
: This is an optional behavior which is triggered by a very specific kernel
: boot param, which I suspect is gonna need to stick around to support
: memory hotplug in the current setup unless we add another layer of address
: translation to support memory hotplug.
As for page tables may occupy too much lower memory if using 4K mapping
(CONFIG_DEBUG_PAGEALLOC and CONFIG_KMEMCHECK both disable using >4k
pages), TJ said:
: But as I said in the same paragraph, parsing SRAT earlier doesn't solve
: the problem in itself either. Ignoring the option if 4k mapping is
: required and memory consumption would be prohibitive should work, no?
: Something like that would be necessary if we're gonna worry about cases
: like this no matter how we implement it, but, frankly, I'm not sure this
: is something worth worrying about.
Signed-off-by: Tang Chen <tangchen@cn.fujitsu.com>
Signed-off-by: Zhang Yanfei <zhangyanfei@cn.fujitsu.com>
Acked-by: Tejun Heo <tj@kernel.org>
Acked-by: Toshi Kani <toshi.kani@hp.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Wanpeng Li <liwanp@linux.vnet.ibm.com>
Cc: Thomas Renninger <trenn@suse.de>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Jiang Liu <jiang.liu@huawei.com>
Cc: Wen Congyang <wency@cn.fujitsu.com>
Cc: Lai Jiangshan <laijs@cn.fujitsu.com>
Cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Cc: Taku Izumi <izumi.taku@jp.fujitsu.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Michal Nazarewicz <mina86@mina86.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-11-13 07:08:05 +08:00
|
|
|
|
2014-12-20 00:10:54 +08:00
|
|
|
mapped_ram_size += init_range_memory_mapping(start, next);
|
x86/mem-hotplug: support initialize page tables in bottom-up
The Linux kernel cannot migrate pages used by the kernel. As a result,
kernel pages cannot be hot-removed. So we cannot allocate hotpluggable
memory for the kernel.
In a memory hotplug system, any numa node the kernel resides in should be
unhotpluggable. And for a modern server, each node could have at least
16GB memory. So memory around the kernel image is highly likely
unhotpluggable.
ACPI SRAT (System Resource Affinity Table) contains the memory hotplug
info. But before SRAT is parsed, memblock has already started to allocate
memory for the kernel. So we need to prevent memblock from doing this.
So direct memory mapping page tables setup is the case.
init_mem_mapping() is called before SRAT is parsed. To prevent page
tables being allocated within hotpluggable memory, we will use bottom-up
direction to allocate page tables from the end of kernel image to the
higher memory.
Note:
As for allocating page tables in lower memory, TJ said:
: This is an optional behavior which is triggered by a very specific kernel
: boot param, which I suspect is gonna need to stick around to support
: memory hotplug in the current setup unless we add another layer of address
: translation to support memory hotplug.
As for page tables may occupy too much lower memory if using 4K mapping
(CONFIG_DEBUG_PAGEALLOC and CONFIG_KMEMCHECK both disable using >4k
pages), TJ said:
: But as I said in the same paragraph, parsing SRAT earlier doesn't solve
: the problem in itself either. Ignoring the option if 4k mapping is
: required and memory consumption would be prohibitive should work, no?
: Something like that would be necessary if we're gonna worry about cases
: like this no matter how we implement it, but, frankly, I'm not sure this
: is something worth worrying about.
Signed-off-by: Tang Chen <tangchen@cn.fujitsu.com>
Signed-off-by: Zhang Yanfei <zhangyanfei@cn.fujitsu.com>
Acked-by: Tejun Heo <tj@kernel.org>
Acked-by: Toshi Kani <toshi.kani@hp.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Wanpeng Li <liwanp@linux.vnet.ibm.com>
Cc: Thomas Renninger <trenn@suse.de>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Jiang Liu <jiang.liu@huawei.com>
Cc: Wen Congyang <wency@cn.fujitsu.com>
Cc: Lai Jiangshan <laijs@cn.fujitsu.com>
Cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Cc: Taku Izumi <izumi.taku@jp.fujitsu.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Michal Nazarewicz <mina86@mina86.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-11-13 07:08:05 +08:00
|
|
|
start = next;
|
|
|
|
|
2014-12-20 00:10:54 +08:00
|
|
|
if (mapped_ram_size >= step_size)
|
x86/mem-hotplug: support initialize page tables in bottom-up
The Linux kernel cannot migrate pages used by the kernel. As a result,
kernel pages cannot be hot-removed. So we cannot allocate hotpluggable
memory for the kernel.
In a memory hotplug system, any numa node the kernel resides in should be
unhotpluggable. And for a modern server, each node could have at least
16GB memory. So memory around the kernel image is highly likely
unhotpluggable.
ACPI SRAT (System Resource Affinity Table) contains the memory hotplug
info. But before SRAT is parsed, memblock has already started to allocate
memory for the kernel. So we need to prevent memblock from doing this.
So direct memory mapping page tables setup is the case.
init_mem_mapping() is called before SRAT is parsed. To prevent page
tables being allocated within hotpluggable memory, we will use bottom-up
direction to allocate page tables from the end of kernel image to the
higher memory.
Note:
As for allocating page tables in lower memory, TJ said:
: This is an optional behavior which is triggered by a very specific kernel
: boot param, which I suspect is gonna need to stick around to support
: memory hotplug in the current setup unless we add another layer of address
: translation to support memory hotplug.
As for page tables may occupy too much lower memory if using 4K mapping
(CONFIG_DEBUG_PAGEALLOC and CONFIG_KMEMCHECK both disable using >4k
pages), TJ said:
: But as I said in the same paragraph, parsing SRAT earlier doesn't solve
: the problem in itself either. Ignoring the option if 4k mapping is
: required and memory consumption would be prohibitive should work, no?
: Something like that would be necessary if we're gonna worry about cases
: like this no matter how we implement it, but, frankly, I'm not sure this
: is something worth worrying about.
Signed-off-by: Tang Chen <tangchen@cn.fujitsu.com>
Signed-off-by: Zhang Yanfei <zhangyanfei@cn.fujitsu.com>
Acked-by: Tejun Heo <tj@kernel.org>
Acked-by: Toshi Kani <toshi.kani@hp.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Wanpeng Li <liwanp@linux.vnet.ibm.com>
Cc: Thomas Renninger <trenn@suse.de>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Jiang Liu <jiang.liu@huawei.com>
Cc: Wen Congyang <wency@cn.fujitsu.com>
Cc: Lai Jiangshan <laijs@cn.fujitsu.com>
Cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Cc: Taku Izumi <izumi.taku@jp.fujitsu.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Michal Nazarewicz <mina86@mina86.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-11-13 07:08:05 +08:00
|
|
|
step_size = get_new_step_size(step_size);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2020-06-09 12:33:01 +08:00
|
|
|
/*
|
|
|
|
* The real mode trampoline, which is required for bootstrapping CPUs
|
|
|
|
* occupies only a small area under the low 1MB. See reserve_real_mode()
|
|
|
|
* for details.
|
|
|
|
*
|
|
|
|
* If KASLR is disabled the first PGD entry of the direct mapping is copied
|
|
|
|
* to map the real mode trampoline.
|
|
|
|
*
|
|
|
|
* If KASLR is enabled, copy only the PUD which covers the low 1MB
|
|
|
|
* area. This limits the randomization granularity to 1GB for both 4-level
|
|
|
|
* and 5-level paging.
|
|
|
|
*/
|
|
|
|
static void __init init_trampoline(void)
|
|
|
|
{
|
|
|
|
#ifdef CONFIG_X86_64
|
2021-12-02 23:32:23 +08:00
|
|
|
/*
|
|
|
|
* The code below will alias kernel page-tables in the user-range of the
|
|
|
|
* address space, including the Global bit. So global TLB entries will
|
|
|
|
* be created when using the trampoline page-table.
|
|
|
|
*/
|
2020-06-09 12:33:01 +08:00
|
|
|
if (!kaslr_memory_enabled())
|
|
|
|
trampoline_pgd_entry = init_top_pgt[pgd_index(__PAGE_OFFSET)];
|
|
|
|
else
|
|
|
|
init_trampoline_kaslr();
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
2013-11-13 07:08:02 +08:00
|
|
|
void __init init_mem_mapping(void)
|
|
|
|
{
|
|
|
|
unsigned long end;
|
|
|
|
|
2017-12-04 22:07:36 +08:00
|
|
|
pti_check_boottime_disable();
|
2013-11-13 07:08:02 +08:00
|
|
|
probe_page_size_mask();
|
2017-09-11 08:48:27 +08:00
|
|
|
setup_pcid();
|
2013-11-13 07:08:02 +08:00
|
|
|
|
|
|
|
#ifdef CONFIG_X86_64
|
|
|
|
end = max_pfn << PAGE_SHIFT;
|
|
|
|
#else
|
|
|
|
end = max_low_pfn << PAGE_SHIFT;
|
|
|
|
#endif
|
|
|
|
|
|
|
|
/* the ISA range is always mapped regardless of memory holes */
|
2020-04-11 05:33:24 +08:00
|
|
|
init_memory_mapping(0, ISA_END_ADDRESS, PAGE_KERNEL);
|
2013-11-13 07:08:02 +08:00
|
|
|
|
2016-06-22 08:47:01 +08:00
|
|
|
/* Init the trampoline, possibly with KASLR memory offset */
|
|
|
|
init_trampoline();
|
|
|
|
|
x86/mem-hotplug: support initialize page tables in bottom-up
The Linux kernel cannot migrate pages used by the kernel. As a result,
kernel pages cannot be hot-removed. So we cannot allocate hotpluggable
memory for the kernel.
In a memory hotplug system, any numa node the kernel resides in should be
unhotpluggable. And for a modern server, each node could have at least
16GB memory. So memory around the kernel image is highly likely
unhotpluggable.
ACPI SRAT (System Resource Affinity Table) contains the memory hotplug
info. But before SRAT is parsed, memblock has already started to allocate
memory for the kernel. So we need to prevent memblock from doing this.
So direct memory mapping page tables setup is the case.
init_mem_mapping() is called before SRAT is parsed. To prevent page
tables being allocated within hotpluggable memory, we will use bottom-up
direction to allocate page tables from the end of kernel image to the
higher memory.
Note:
As for allocating page tables in lower memory, TJ said:
: This is an optional behavior which is triggered by a very specific kernel
: boot param, which I suspect is gonna need to stick around to support
: memory hotplug in the current setup unless we add another layer of address
: translation to support memory hotplug.
As for page tables may occupy too much lower memory if using 4K mapping
(CONFIG_DEBUG_PAGEALLOC and CONFIG_KMEMCHECK both disable using >4k
pages), TJ said:
: But as I said in the same paragraph, parsing SRAT earlier doesn't solve
: the problem in itself either. Ignoring the option if 4k mapping is
: required and memory consumption would be prohibitive should work, no?
: Something like that would be necessary if we're gonna worry about cases
: like this no matter how we implement it, but, frankly, I'm not sure this
: is something worth worrying about.
Signed-off-by: Tang Chen <tangchen@cn.fujitsu.com>
Signed-off-by: Zhang Yanfei <zhangyanfei@cn.fujitsu.com>
Acked-by: Tejun Heo <tj@kernel.org>
Acked-by: Toshi Kani <toshi.kani@hp.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Wanpeng Li <liwanp@linux.vnet.ibm.com>
Cc: Thomas Renninger <trenn@suse.de>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: Jiang Liu <jiang.liu@huawei.com>
Cc: Wen Congyang <wency@cn.fujitsu.com>
Cc: Lai Jiangshan <laijs@cn.fujitsu.com>
Cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Cc: Taku Izumi <izumi.taku@jp.fujitsu.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Michal Nazarewicz <mina86@mina86.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kamezawa Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-11-13 07:08:05 +08:00
|
|
|
/*
|
|
|
|
* If the allocation is in bottom-up direction, we setup direct mapping
|
|
|
|
* in bottom-up, otherwise we setup direct mapping in top-down.
|
|
|
|
*/
|
|
|
|
if (memblock_bottom_up()) {
|
|
|
|
unsigned long kernel_end = __pa_symbol(_end);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* we need two separate calls here. This is because we want to
|
|
|
|
* allocate page tables above the kernel. So we first map
|
|
|
|
* [kernel_end, end) to make memory above the kernel be mapped
|
|
|
|
* as soon as possible. And then use page tables allocated above
|
|
|
|
* the kernel to map [ISA_END_ADDRESS, kernel_end).
|
|
|
|
*/
|
|
|
|
memory_map_bottom_up(kernel_end, end);
|
|
|
|
memory_map_bottom_up(ISA_END_ADDRESS, kernel_end);
|
|
|
|
} else {
|
|
|
|
memory_map_top_down(ISA_END_ADDRESS, end);
|
|
|
|
}
|
2012-11-17 11:38:58 +08:00
|
|
|
|
2012-11-17 11:38:57 +08:00
|
|
|
#ifdef CONFIG_X86_64
|
|
|
|
if (max_pfn > max_low_pfn) {
|
2021-03-22 05:28:53 +08:00
|
|
|
/* can we preserve max_low_pfn ?*/
|
2012-11-17 11:38:57 +08:00
|
|
|
max_low_pfn = max_pfn;
|
|
|
|
}
|
2012-11-17 11:39:06 +08:00
|
|
|
#else
|
|
|
|
early_ioremap_page_table_range_init();
|
x86, 64bit: Use a #PF handler to materialize early mappings on demand
Linear mode (CR0.PG = 0) is mutually exclusive with 64-bit mode; all
64-bit code has to use page tables. This makes it awkward before we
have first set up properly all-covering page tables to access objects
that are outside the static kernel range.
So far we have dealt with that simply by mapping a fixed amount of
low memory, but that fails in at least two upcoming use cases:
1. We will support load and run kernel, struct boot_params, ramdisk,
command line, etc. above the 4 GiB mark.
2. need to access ramdisk early to get microcode to update that as
early possible.
We could use early_iomap to access them too, but it will make code to
messy and hard to be unified with 32 bit.
Hence, set up a #PF table and use a fixed number of buffers to set up
page tables on demand. If the buffers fill up then we simply flush
them and start over. These buffers are all in __initdata, so it does
not increase RAM usage at runtime.
Thus, with the help of the #PF handler, we can set the final kernel
mapping from blank, and switch to init_level4_pgt later.
During the switchover in head_64.S, before #PF handler is available,
we use three pages to handle kernel crossing 1G, 512G boundaries with
sharing page by playing games with page aliasing: the same page is
mapped twice in the higher-level tables with appropriate wraparound.
The kernel region itself will be properly mapped; other mappings may
be spurious.
early_make_pgtable is using kernel high mapping address to access pages
to set page table.
-v4: Add phys_base offset to make kexec happy, and add
init_mapping_kernel() - Yinghai
-v5: fix compiling with xen, and add back ident level3 and level2 for xen
also move back init_level4_pgt from BSS to DATA again.
because we have to clear it anyway. - Yinghai
-v6: switch to init_level4_pgt in init_mem_mapping. - Yinghai
-v7: remove not needed clear_page for init_level4_page
it is with fill 512,8,0 already in head_64.S - Yinghai
-v8: we need to keep that handler alive until init_mem_mapping and don't
let early_trap_init to trash that early #PF handler.
So split early_trap_pf_init out and move it down. - Yinghai
-v9: switchover only cover kernel space instead of 1G so could avoid
touch possible mem holes. - Yinghai
-v11: change far jmp back to far return to initial_code, that is needed
to fix failure that is reported by Konrad on AMD systems. - Yinghai
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Link: http://lkml.kernel.org/r/1359058816-7615-12-git-send-email-yinghai@kernel.org
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
2013-01-25 04:19:52 +08:00
|
|
|
#endif
|
|
|
|
|
2012-11-17 11:39:06 +08:00
|
|
|
load_cr3(swapper_pg_dir);
|
|
|
|
__flush_tlb_all();
|
|
|
|
|
2017-11-09 21:27:35 +08:00
|
|
|
x86_init.hyper.init_mem_mapping();
|
2017-07-28 18:23:12 +08:00
|
|
|
|
2012-11-17 11:38:44 +08:00
|
|
|
early_memtest(0, max_pfn_mapped << PAGE_SHIFT);
|
2012-11-17 11:38:41 +08:00
|
|
|
}
|
2009-03-03 19:15:06 +08:00
|
|
|
|
2019-04-27 07:22:46 +08:00
|
|
|
/*
|
|
|
|
* Initialize an mm_struct to be used during poking and a pointer to be used
|
|
|
|
* during patching.
|
|
|
|
*/
|
|
|
|
void __init poking_init(void)
|
|
|
|
{
|
|
|
|
spinlock_t *ptl;
|
|
|
|
pte_t *ptep;
|
|
|
|
|
2022-10-26 03:38:21 +08:00
|
|
|
poking_mm = mm_alloc();
|
2019-04-27 07:22:46 +08:00
|
|
|
BUG_ON(!poking_mm);
|
|
|
|
|
2023-01-09 23:09:22 +08:00
|
|
|
/* Xen PV guests need the PGD to be pinned. */
|
|
|
|
paravirt_arch_dup_mmap(NULL, poking_mm);
|
|
|
|
|
2019-04-27 07:22:46 +08:00
|
|
|
/*
|
|
|
|
* Randomize the poking address, but make sure that the following page
|
|
|
|
* will be mapped at the same PMD. We need 2 pages, so find space for 3,
|
|
|
|
* and adjust the address if the PMD ends after the first one.
|
|
|
|
*/
|
|
|
|
poking_addr = TASK_UNMAPPED_BASE;
|
|
|
|
if (IS_ENABLED(CONFIG_RANDOMIZE_BASE))
|
|
|
|
poking_addr += (kaslr_get_random_long("Poking") & PAGE_MASK) %
|
|
|
|
(TASK_SIZE - TASK_UNMAPPED_BASE - 3 * PAGE_SIZE);
|
|
|
|
|
|
|
|
if (((poking_addr + PAGE_SIZE) & ~PMD_MASK) == 0)
|
|
|
|
poking_addr += PAGE_SIZE;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We need to trigger the allocation of the page-tables that will be
|
|
|
|
* needed for poking now. Later, poking may be performed in an atomic
|
|
|
|
* section, which might cause allocation to fail.
|
|
|
|
*/
|
|
|
|
ptep = get_locked_pte(poking_mm, poking_addr, &ptl);
|
|
|
|
BUG_ON(!ptep);
|
|
|
|
pte_unmap_unlock(ptep, ptl);
|
|
|
|
}
|
|
|
|
|
2009-03-04 17:46:40 +08:00
|
|
|
/*
|
|
|
|
* devmem_is_allowed() checks to see if /dev/mem access to a certain address
|
|
|
|
* is valid. The argument is a physical page number.
|
|
|
|
*
|
2017-04-06 00:39:08 +08:00
|
|
|
* On x86, access has to be given to the first megabyte of RAM because that
|
|
|
|
* area traditionally contains BIOS code and data regions used by X, dosemu,
|
|
|
|
* and similar apps. Since they map the entire memory range, the whole range
|
|
|
|
* must be allowed (for mapping), but any areas that would otherwise be
|
|
|
|
* disallowed are flagged as being "zero filled" instead of rejected.
|
|
|
|
* Access has to be given to non-kernel-ram areas as well, these contain the
|
|
|
|
* PCI mmio resources as well as potential bios/acpi data regions.
|
2009-03-04 17:46:40 +08:00
|
|
|
*/
|
|
|
|
int devmem_is_allowed(unsigned long pagenr)
|
|
|
|
{
|
2018-06-15 06:26:24 +08:00
|
|
|
if (region_intersects(PFN_PHYS(pagenr), PAGE_SIZE,
|
|
|
|
IORESOURCE_SYSTEM_RAM, IORES_DESC_NONE)
|
|
|
|
!= REGION_DISJOINT) {
|
2017-04-06 00:39:08 +08:00
|
|
|
/*
|
|
|
|
* For disallowed memory regions in the low 1MB range,
|
|
|
|
* request that the page be shown as all zeros.
|
|
|
|
*/
|
|
|
|
if (pagenr < 256)
|
|
|
|
return 2;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This must follow RAM test, since System RAM is considered a
|
2022-07-07 19:54:42 +08:00
|
|
|
* restricted resource under CONFIG_STRICT_DEVMEM.
|
2017-04-06 00:39:08 +08:00
|
|
|
*/
|
|
|
|
if (iomem_is_exclusive(pagenr << PAGE_SHIFT)) {
|
|
|
|
/* Low 1MB bypasses iomem restrictions. */
|
|
|
|
if (pagenr < 256)
|
|
|
|
return 1;
|
|
|
|
|
2009-03-04 17:46:40 +08:00
|
|
|
return 0;
|
2017-04-06 00:39:08 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
return 1;
|
2009-03-04 17:46:40 +08:00
|
|
|
}
|
|
|
|
|
2018-12-28 16:36:03 +08:00
|
|
|
void free_init_pages(const char *what, unsigned long begin, unsigned long end)
|
2009-03-03 19:15:06 +08:00
|
|
|
{
|
x86: Make sure free_init_pages() frees pages on page boundary
When CONFIG_NO_BOOTMEM=y, it could use memory more effiently, or
in a more compact fashion.
Example:
Allocated new RAMDISK: 00ec2000 - 0248ce57
Move RAMDISK from 000000002ea04000 - 000000002ffcee56 to 00ec2000 - 0248ce56
The new RAMDISK's end is not page aligned.
Last page could be shared with other users.
When free_init_pages are called for initrd or .init, the page
could be freed and we could corrupt other data.
code segment in free_init_pages():
| for (; addr < end; addr += PAGE_SIZE) {
| ClearPageReserved(virt_to_page(addr));
| init_page_count(virt_to_page(addr));
| memset((void *)(addr & ~(PAGE_SIZE-1)),
| POISON_FREE_INITMEM, PAGE_SIZE);
| free_page(addr);
| totalram_pages++;
| }
last half page could be used as one whole free page.
So page align the boundaries.
-v2: make the original initramdisk to be aligned, according to
Johannes, otherwise we have the chance to lose one page.
we still need to keep initrd_end not aligned, otherwise it could
confuse decompressor.
-v3: change to WARN_ON instead, suggested by Johannes.
-v4: use PAGE_ALIGN, suggested by Johannes.
We may fix that macro name later to PAGE_ALIGN_UP, and PAGE_ALIGN_DOWN
Add comments about assuming ramdisk start is aligned
in relocate_initrd(), change to re get ramdisk_image instead of save it
to make diff smaller. Add warning for wrong range, suggested by Johannes.
-v6: remove one WARN()
We need to align beginning in free_init_pages()
do not copy more than ramdisk_size, noticed by Johannes
Reported-by: Stanislaw Gruszka <sgruszka@redhat.com>
Tested-by: Stanislaw Gruszka <sgruszka@redhat.com>
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: David Miller <davem@davemloft.net>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
LKML-Reference: <1269830604-26214-3-git-send-email-yinghai@kernel.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2010-03-29 10:42:55 +08:00
|
|
|
unsigned long begin_aligned, end_aligned;
|
2009-03-03 19:15:06 +08:00
|
|
|
|
x86: Make sure free_init_pages() frees pages on page boundary
When CONFIG_NO_BOOTMEM=y, it could use memory more effiently, or
in a more compact fashion.
Example:
Allocated new RAMDISK: 00ec2000 - 0248ce57
Move RAMDISK from 000000002ea04000 - 000000002ffcee56 to 00ec2000 - 0248ce56
The new RAMDISK's end is not page aligned.
Last page could be shared with other users.
When free_init_pages are called for initrd or .init, the page
could be freed and we could corrupt other data.
code segment in free_init_pages():
| for (; addr < end; addr += PAGE_SIZE) {
| ClearPageReserved(virt_to_page(addr));
| init_page_count(virt_to_page(addr));
| memset((void *)(addr & ~(PAGE_SIZE-1)),
| POISON_FREE_INITMEM, PAGE_SIZE);
| free_page(addr);
| totalram_pages++;
| }
last half page could be used as one whole free page.
So page align the boundaries.
-v2: make the original initramdisk to be aligned, according to
Johannes, otherwise we have the chance to lose one page.
we still need to keep initrd_end not aligned, otherwise it could
confuse decompressor.
-v3: change to WARN_ON instead, suggested by Johannes.
-v4: use PAGE_ALIGN, suggested by Johannes.
We may fix that macro name later to PAGE_ALIGN_UP, and PAGE_ALIGN_DOWN
Add comments about assuming ramdisk start is aligned
in relocate_initrd(), change to re get ramdisk_image instead of save it
to make diff smaller. Add warning for wrong range, suggested by Johannes.
-v6: remove one WARN()
We need to align beginning in free_init_pages()
do not copy more than ramdisk_size, noticed by Johannes
Reported-by: Stanislaw Gruszka <sgruszka@redhat.com>
Tested-by: Stanislaw Gruszka <sgruszka@redhat.com>
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: David Miller <davem@davemloft.net>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
LKML-Reference: <1269830604-26214-3-git-send-email-yinghai@kernel.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2010-03-29 10:42:55 +08:00
|
|
|
/* Make sure boundaries are page aligned */
|
|
|
|
begin_aligned = PAGE_ALIGN(begin);
|
|
|
|
end_aligned = end & PAGE_MASK;
|
|
|
|
|
|
|
|
if (WARN_ON(begin_aligned != begin || end_aligned != end)) {
|
|
|
|
begin = begin_aligned;
|
|
|
|
end = end_aligned;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (begin >= end)
|
2009-03-03 19:15:06 +08:00
|
|
|
return;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If debugging page accesses then do not free this memory but
|
|
|
|
* mark them not present - any buggy init-section access will
|
|
|
|
* create a kernel page fault:
|
|
|
|
*/
|
2016-03-16 05:57:39 +08:00
|
|
|
if (debug_pagealloc_enabled()) {
|
|
|
|
pr_info("debug: unmapping init [mem %#010lx-%#010lx]\n",
|
|
|
|
begin, end - 1);
|
2019-04-24 00:58:11 +08:00
|
|
|
/*
|
|
|
|
* Inform kmemleak about the hole in the memory since the
|
|
|
|
* corresponding pages will be unmapped.
|
|
|
|
*/
|
|
|
|
kmemleak_free_part((void *)begin, end - begin);
|
2016-03-16 05:57:39 +08:00
|
|
|
set_memory_np(begin, (end - begin) >> PAGE_SHIFT);
|
|
|
|
} else {
|
|
|
|
/*
|
|
|
|
* We just marked the kernel text read only above, now that
|
|
|
|
* we are going to free part of that, we need to make that
|
|
|
|
* writeable and non-executable first.
|
|
|
|
*/
|
|
|
|
set_memory_nx(begin, (end - begin) >> PAGE_SHIFT);
|
|
|
|
set_memory_rw(begin, (end - begin) >> PAGE_SHIFT);
|
2009-03-03 19:15:06 +08:00
|
|
|
|
2016-03-16 05:57:39 +08:00
|
|
|
free_reserved_area((void *)begin, (void *)end,
|
|
|
|
POISON_FREE_INITMEM, what);
|
|
|
|
}
|
2009-03-03 19:15:06 +08:00
|
|
|
}
|
|
|
|
|
2018-08-03 06:58:29 +08:00
|
|
|
/*
|
|
|
|
* begin/end can be in the direct map or the "high kernel mapping"
|
|
|
|
* used for the kernel image only. free_init_pages() will do the
|
|
|
|
* right thing for either kind of address.
|
|
|
|
*/
|
2019-10-30 05:13:49 +08:00
|
|
|
void free_kernel_image_pages(const char *what, void *begin, void *end)
|
2018-08-03 06:58:29 +08:00
|
|
|
{
|
x86/mm/init: Remove freed kernel image areas from alias mapping
The kernel image is mapped into two places in the virtual address space
(addresses without KASLR, of course):
1. The kernel direct map (0xffff880000000000)
2. The "high kernel map" (0xffffffff81000000)
We actually execute out of #2. If we get the address of a kernel symbol,
it points to #2, but almost all physical-to-virtual translations point to
Parts of the "high kernel map" alias are mapped in the userspace page
tables with the Global bit for performance reasons. The parts that we map
to userspace do not (er, should not) have secrets. When PTI is enabled then
the global bit is usually not set in the high mapping and just used to
compensate for poor performance on systems which lack PCID.
This is fine, except that some areas in the kernel image that are adjacent
to the non-secret-containing areas are unused holes. We free these holes
back into the normal page allocator and reuse them as normal kernel memory.
The memory will, of course, get *used* via the normal map, but the alias
mapping is kept.
This otherwise unused alias mapping of the holes will, by default keep the
Global bit, be mapped out to userspace, and be vulnerable to Meltdown.
Remove the alias mapping of these pages entirely. This is likely to
fracture the 2M page mapping the kernel image near these areas, but this
should affect a minority of the area.
The pageattr code changes *all* aliases mapping the physical pages that it
operates on (by default). We only want to modify a single alias, so we
need to tweak its behavior.
This unmapping behavior is currently dependent on PTI being in place.
Going forward, we should at least consider doing this for all
configurations. Having an extra read-write alias for memory is not exactly
ideal for debugging things like random memory corruption and this does
undercut features like DEBUG_PAGEALLOC or future work like eXclusive Page
Frame Ownership (XPFO).
Before this patch:
current_kernel:---[ High Kernel Mapping ]---
current_kernel-0xffffffff80000000-0xffffffff81000000 16M pmd
current_kernel-0xffffffff81000000-0xffffffff81e00000 14M ro PSE GLB x pmd
current_kernel-0xffffffff81e00000-0xffffffff81e11000 68K ro GLB x pte
current_kernel-0xffffffff81e11000-0xffffffff82000000 1980K RW NX pte
current_kernel-0xffffffff82000000-0xffffffff82600000 6M ro PSE GLB NX pmd
current_kernel-0xffffffff82600000-0xffffffff82c00000 6M RW PSE NX pmd
current_kernel-0xffffffff82c00000-0xffffffff82e00000 2M RW NX pte
current_kernel-0xffffffff82e00000-0xffffffff83200000 4M RW PSE NX pmd
current_kernel-0xffffffff83200000-0xffffffffa0000000 462M pmd
current_user:---[ High Kernel Mapping ]---
current_user-0xffffffff80000000-0xffffffff81000000 16M pmd
current_user-0xffffffff81000000-0xffffffff81e00000 14M ro PSE GLB x pmd
current_user-0xffffffff81e00000-0xffffffff81e11000 68K ro GLB x pte
current_user-0xffffffff81e11000-0xffffffff82000000 1980K RW NX pte
current_user-0xffffffff82000000-0xffffffff82600000 6M ro PSE GLB NX pmd
current_user-0xffffffff82600000-0xffffffffa0000000 474M pmd
After this patch:
current_kernel:---[ High Kernel Mapping ]---
current_kernel-0xffffffff80000000-0xffffffff81000000 16M pmd
current_kernel-0xffffffff81000000-0xffffffff81e00000 14M ro PSE GLB x pmd
current_kernel-0xffffffff81e00000-0xffffffff81e11000 68K ro GLB x pte
current_kernel-0xffffffff81e11000-0xffffffff82000000 1980K pte
current_kernel-0xffffffff82000000-0xffffffff82400000 4M ro PSE GLB NX pmd
current_kernel-0xffffffff82400000-0xffffffff82488000 544K ro NX pte
current_kernel-0xffffffff82488000-0xffffffff82600000 1504K pte
current_kernel-0xffffffff82600000-0xffffffff82c00000 6M RW PSE NX pmd
current_kernel-0xffffffff82c00000-0xffffffff82c0d000 52K RW NX pte
current_kernel-0xffffffff82c0d000-0xffffffff82dc0000 1740K pte
current_user:---[ High Kernel Mapping ]---
current_user-0xffffffff80000000-0xffffffff81000000 16M pmd
current_user-0xffffffff81000000-0xffffffff81e00000 14M ro PSE GLB x pmd
current_user-0xffffffff81e00000-0xffffffff81e11000 68K ro GLB x pte
current_user-0xffffffff81e11000-0xffffffff82000000 1980K pte
current_user-0xffffffff82000000-0xffffffff82400000 4M ro PSE GLB NX pmd
current_user-0xffffffff82400000-0xffffffff82488000 544K ro NX pte
current_user-0xffffffff82488000-0xffffffff82600000 1504K pte
current_user-0xffffffff82600000-0xffffffffa0000000 474M pmd
[ tglx: Do not unmap on 32bit as there is only one mapping ]
Fixes: 0f561fce4d69 ("x86/pti: Enable global pages for shared areas")
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Kees Cook <keescook@google.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Joerg Roedel <jroedel@suse.de>
Link: https://lkml.kernel.org/r/20180802225831.5F6A2BFC@viggo.jf.intel.com
2018-08-03 06:58:31 +08:00
|
|
|
unsigned long begin_ul = (unsigned long)begin;
|
|
|
|
unsigned long end_ul = (unsigned long)end;
|
|
|
|
unsigned long len_pages = (end_ul - begin_ul) >> PAGE_SHIFT;
|
|
|
|
|
2019-10-30 05:13:49 +08:00
|
|
|
free_init_pages(what, begin_ul, end_ul);
|
x86/mm/init: Remove freed kernel image areas from alias mapping
The kernel image is mapped into two places in the virtual address space
(addresses without KASLR, of course):
1. The kernel direct map (0xffff880000000000)
2. The "high kernel map" (0xffffffff81000000)
We actually execute out of #2. If we get the address of a kernel symbol,
it points to #2, but almost all physical-to-virtual translations point to
Parts of the "high kernel map" alias are mapped in the userspace page
tables with the Global bit for performance reasons. The parts that we map
to userspace do not (er, should not) have secrets. When PTI is enabled then
the global bit is usually not set in the high mapping and just used to
compensate for poor performance on systems which lack PCID.
This is fine, except that some areas in the kernel image that are adjacent
to the non-secret-containing areas are unused holes. We free these holes
back into the normal page allocator and reuse them as normal kernel memory.
The memory will, of course, get *used* via the normal map, but the alias
mapping is kept.
This otherwise unused alias mapping of the holes will, by default keep the
Global bit, be mapped out to userspace, and be vulnerable to Meltdown.
Remove the alias mapping of these pages entirely. This is likely to
fracture the 2M page mapping the kernel image near these areas, but this
should affect a minority of the area.
The pageattr code changes *all* aliases mapping the physical pages that it
operates on (by default). We only want to modify a single alias, so we
need to tweak its behavior.
This unmapping behavior is currently dependent on PTI being in place.
Going forward, we should at least consider doing this for all
configurations. Having an extra read-write alias for memory is not exactly
ideal for debugging things like random memory corruption and this does
undercut features like DEBUG_PAGEALLOC or future work like eXclusive Page
Frame Ownership (XPFO).
Before this patch:
current_kernel:---[ High Kernel Mapping ]---
current_kernel-0xffffffff80000000-0xffffffff81000000 16M pmd
current_kernel-0xffffffff81000000-0xffffffff81e00000 14M ro PSE GLB x pmd
current_kernel-0xffffffff81e00000-0xffffffff81e11000 68K ro GLB x pte
current_kernel-0xffffffff81e11000-0xffffffff82000000 1980K RW NX pte
current_kernel-0xffffffff82000000-0xffffffff82600000 6M ro PSE GLB NX pmd
current_kernel-0xffffffff82600000-0xffffffff82c00000 6M RW PSE NX pmd
current_kernel-0xffffffff82c00000-0xffffffff82e00000 2M RW NX pte
current_kernel-0xffffffff82e00000-0xffffffff83200000 4M RW PSE NX pmd
current_kernel-0xffffffff83200000-0xffffffffa0000000 462M pmd
current_user:---[ High Kernel Mapping ]---
current_user-0xffffffff80000000-0xffffffff81000000 16M pmd
current_user-0xffffffff81000000-0xffffffff81e00000 14M ro PSE GLB x pmd
current_user-0xffffffff81e00000-0xffffffff81e11000 68K ro GLB x pte
current_user-0xffffffff81e11000-0xffffffff82000000 1980K RW NX pte
current_user-0xffffffff82000000-0xffffffff82600000 6M ro PSE GLB NX pmd
current_user-0xffffffff82600000-0xffffffffa0000000 474M pmd
After this patch:
current_kernel:---[ High Kernel Mapping ]---
current_kernel-0xffffffff80000000-0xffffffff81000000 16M pmd
current_kernel-0xffffffff81000000-0xffffffff81e00000 14M ro PSE GLB x pmd
current_kernel-0xffffffff81e00000-0xffffffff81e11000 68K ro GLB x pte
current_kernel-0xffffffff81e11000-0xffffffff82000000 1980K pte
current_kernel-0xffffffff82000000-0xffffffff82400000 4M ro PSE GLB NX pmd
current_kernel-0xffffffff82400000-0xffffffff82488000 544K ro NX pte
current_kernel-0xffffffff82488000-0xffffffff82600000 1504K pte
current_kernel-0xffffffff82600000-0xffffffff82c00000 6M RW PSE NX pmd
current_kernel-0xffffffff82c00000-0xffffffff82c0d000 52K RW NX pte
current_kernel-0xffffffff82c0d000-0xffffffff82dc0000 1740K pte
current_user:---[ High Kernel Mapping ]---
current_user-0xffffffff80000000-0xffffffff81000000 16M pmd
current_user-0xffffffff81000000-0xffffffff81e00000 14M ro PSE GLB x pmd
current_user-0xffffffff81e00000-0xffffffff81e11000 68K ro GLB x pte
current_user-0xffffffff81e11000-0xffffffff82000000 1980K pte
current_user-0xffffffff82000000-0xffffffff82400000 4M ro PSE GLB NX pmd
current_user-0xffffffff82400000-0xffffffff82488000 544K ro NX pte
current_user-0xffffffff82488000-0xffffffff82600000 1504K pte
current_user-0xffffffff82600000-0xffffffffa0000000 474M pmd
[ tglx: Do not unmap on 32bit as there is only one mapping ]
Fixes: 0f561fce4d69 ("x86/pti: Enable global pages for shared areas")
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Kees Cook <keescook@google.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Joerg Roedel <jroedel@suse.de>
Link: https://lkml.kernel.org/r/20180802225831.5F6A2BFC@viggo.jf.intel.com
2018-08-03 06:58:31 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* PTI maps some of the kernel into userspace. For performance,
|
|
|
|
* this includes some kernel areas that do not contain secrets.
|
|
|
|
* Those areas might be adjacent to the parts of the kernel image
|
|
|
|
* being freed, which may contain secrets. Remove the "high kernel
|
|
|
|
* image mapping" for these freed areas, ensuring they are not even
|
|
|
|
* potentially vulnerable to Meltdown regardless of the specific
|
|
|
|
* optimizations PTI is currently using.
|
|
|
|
*
|
|
|
|
* The "noalias" prevents unmapping the direct map alias which is
|
|
|
|
* needed to access the freed pages.
|
|
|
|
*
|
|
|
|
* This is only valid for 64bit kernels. 32bit has only one mapping
|
|
|
|
* which can't be treated in this way for obvious reasons.
|
|
|
|
*/
|
|
|
|
if (IS_ENABLED(CONFIG_X86_64) && cpu_feature_enabled(X86_FEATURE_PTI))
|
|
|
|
set_memory_np_noalias(begin_ul, len_pages);
|
2018-08-03 06:58:29 +08:00
|
|
|
}
|
|
|
|
|
x86/e820: Use much less memory for e820/e820_saved, save up to 120k
The maximum size of e820 map array for EFI systems is defined as
E820_X_MAX (E820MAX + 3 * MAX_NUMNODES).
In x86_64 defconfig, this ends up with E820_X_MAX = 320, e820 and e820_saved
are 6404 bytes each.
With larger configs, for example Fedora kernels, E820_X_MAX = 3200, e820
and e820_saved are 64004 bytes each. Most of this space is wasted.
Typical machines have some 20-30 e820 areas at most.
After previous patch, e820 and e820_saved are pointers to e280 maps.
Change them to initially point to maps which are __initdata.
At the very end of kernel init, just before __init[data] sections are freed
in free_initmem(), allocate smaller blocks, copy maps there,
and change pointers.
The late switch makes sure that all functions which can be used to change
e820 maps are no longer accessible (they are all __init functions).
Run-tested.
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
Acked-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Yinghai Lu <yinghai@kernel.org>
Cc: linux-kernel@vger.kernel.org
Link: http://lkml.kernel.org/r/20160918182125.21000-1-dvlasenk@redhat.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2016-09-19 02:21:25 +08:00
|
|
|
void __ref free_initmem(void)
|
2009-03-03 19:15:06 +08:00
|
|
|
{
|
2017-01-29 05:52:16 +08:00
|
|
|
e820__reallocate_tables();
|
2016-09-18 05:39:26 +08:00
|
|
|
|
2018-09-14 21:45:58 +08:00
|
|
|
mem_encrypt_free_decrypted_mem();
|
|
|
|
|
2019-10-30 05:13:49 +08:00
|
|
|
free_kernel_image_pages("unused kernel image (initmem)",
|
|
|
|
&__init_begin, &__init_end);
|
2009-03-03 19:15:06 +08:00
|
|
|
}
|
2009-03-04 17:13:40 +08:00
|
|
|
|
|
|
|
#ifdef CONFIG_BLK_DEV_INITRD
|
2012-06-18 18:30:20 +08:00
|
|
|
void __init free_initrd_mem(unsigned long start, unsigned long end)
|
2009-03-04 17:13:40 +08:00
|
|
|
{
|
x86: Make sure free_init_pages() frees pages on page boundary
When CONFIG_NO_BOOTMEM=y, it could use memory more effiently, or
in a more compact fashion.
Example:
Allocated new RAMDISK: 00ec2000 - 0248ce57
Move RAMDISK from 000000002ea04000 - 000000002ffcee56 to 00ec2000 - 0248ce56
The new RAMDISK's end is not page aligned.
Last page could be shared with other users.
When free_init_pages are called for initrd or .init, the page
could be freed and we could corrupt other data.
code segment in free_init_pages():
| for (; addr < end; addr += PAGE_SIZE) {
| ClearPageReserved(virt_to_page(addr));
| init_page_count(virt_to_page(addr));
| memset((void *)(addr & ~(PAGE_SIZE-1)),
| POISON_FREE_INITMEM, PAGE_SIZE);
| free_page(addr);
| totalram_pages++;
| }
last half page could be used as one whole free page.
So page align the boundaries.
-v2: make the original initramdisk to be aligned, according to
Johannes, otherwise we have the chance to lose one page.
we still need to keep initrd_end not aligned, otherwise it could
confuse decompressor.
-v3: change to WARN_ON instead, suggested by Johannes.
-v4: use PAGE_ALIGN, suggested by Johannes.
We may fix that macro name later to PAGE_ALIGN_UP, and PAGE_ALIGN_DOWN
Add comments about assuming ramdisk start is aligned
in relocate_initrd(), change to re get ramdisk_image instead of save it
to make diff smaller. Add warning for wrong range, suggested by Johannes.
-v6: remove one WARN()
We need to align beginning in free_init_pages()
do not copy more than ramdisk_size, noticed by Johannes
Reported-by: Stanislaw Gruszka <sgruszka@redhat.com>
Tested-by: Stanislaw Gruszka <sgruszka@redhat.com>
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: David Miller <davem@davemloft.net>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
LKML-Reference: <1269830604-26214-3-git-send-email-yinghai@kernel.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2010-03-29 10:42:55 +08:00
|
|
|
/*
|
|
|
|
* end could be not aligned, and We can not align that,
|
2021-03-18 22:28:01 +08:00
|
|
|
* decompressor could be confused by aligned initrd_end
|
x86: Make sure free_init_pages() frees pages on page boundary
When CONFIG_NO_BOOTMEM=y, it could use memory more effiently, or
in a more compact fashion.
Example:
Allocated new RAMDISK: 00ec2000 - 0248ce57
Move RAMDISK from 000000002ea04000 - 000000002ffcee56 to 00ec2000 - 0248ce56
The new RAMDISK's end is not page aligned.
Last page could be shared with other users.
When free_init_pages are called for initrd or .init, the page
could be freed and we could corrupt other data.
code segment in free_init_pages():
| for (; addr < end; addr += PAGE_SIZE) {
| ClearPageReserved(virt_to_page(addr));
| init_page_count(virt_to_page(addr));
| memset((void *)(addr & ~(PAGE_SIZE-1)),
| POISON_FREE_INITMEM, PAGE_SIZE);
| free_page(addr);
| totalram_pages++;
| }
last half page could be used as one whole free page.
So page align the boundaries.
-v2: make the original initramdisk to be aligned, according to
Johannes, otherwise we have the chance to lose one page.
we still need to keep initrd_end not aligned, otherwise it could
confuse decompressor.
-v3: change to WARN_ON instead, suggested by Johannes.
-v4: use PAGE_ALIGN, suggested by Johannes.
We may fix that macro name later to PAGE_ALIGN_UP, and PAGE_ALIGN_DOWN
Add comments about assuming ramdisk start is aligned
in relocate_initrd(), change to re get ramdisk_image instead of save it
to make diff smaller. Add warning for wrong range, suggested by Johannes.
-v6: remove one WARN()
We need to align beginning in free_init_pages()
do not copy more than ramdisk_size, noticed by Johannes
Reported-by: Stanislaw Gruszka <sgruszka@redhat.com>
Tested-by: Stanislaw Gruszka <sgruszka@redhat.com>
Signed-off-by: Yinghai Lu <yinghai@kernel.org>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: David Miller <davem@davemloft.net>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
LKML-Reference: <1269830604-26214-3-git-send-email-yinghai@kernel.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2010-03-29 10:42:55 +08:00
|
|
|
* We already reserve the end partial page before in
|
|
|
|
* - i386_start_kernel()
|
|
|
|
* - x86_64_start_kernel()
|
|
|
|
* - relocate_initrd()
|
|
|
|
* So here We can do PAGE_ALIGN() safely to get partial page to be freed
|
|
|
|
*/
|
2013-07-04 06:02:58 +08:00
|
|
|
free_init_pages("initrd", start, PAGE_ALIGN(end));
|
2009-03-04 17:13:40 +08:00
|
|
|
}
|
|
|
|
#endif
|
2011-11-01 21:58:22 +08:00
|
|
|
|
2017-01-28 19:45:40 +08:00
|
|
|
/*
|
|
|
|
* Calculate the precise size of the DMA zone (first 16 MB of RAM),
|
|
|
|
* and pass it to the MM layer - to help it set zone watermarks more
|
|
|
|
* accurately.
|
|
|
|
*
|
|
|
|
* Done on 64-bit systems only for the time being, although 32-bit systems
|
|
|
|
* might benefit from this as well.
|
|
|
|
*/
|
|
|
|
void __init memblock_find_dma_reserve(void)
|
|
|
|
{
|
|
|
|
#ifdef CONFIG_X86_64
|
|
|
|
u64 nr_pages = 0, nr_free_pages = 0;
|
|
|
|
unsigned long start_pfn, end_pfn;
|
|
|
|
phys_addr_t start_addr, end_addr;
|
|
|
|
int i;
|
|
|
|
u64 u;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Iterate over all memory ranges (free and reserved ones alike),
|
|
|
|
* to calculate the total number of pages in the first 16 MB of RAM:
|
|
|
|
*/
|
|
|
|
nr_pages = 0;
|
|
|
|
for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, NULL) {
|
|
|
|
start_pfn = min(start_pfn, MAX_DMA_PFN);
|
|
|
|
end_pfn = min(end_pfn, MAX_DMA_PFN);
|
|
|
|
|
|
|
|
nr_pages += end_pfn - start_pfn;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Iterate over free memory ranges to calculate the number of free
|
|
|
|
* pages in the DMA zone, while not counting potential partial
|
|
|
|
* pages at the beginning or the end of the range:
|
|
|
|
*/
|
|
|
|
nr_free_pages = 0;
|
|
|
|
for_each_free_mem_range(u, NUMA_NO_NODE, MEMBLOCK_NONE, &start_addr, &end_addr, NULL) {
|
|
|
|
start_pfn = min_t(unsigned long, PFN_UP(start_addr), MAX_DMA_PFN);
|
|
|
|
end_pfn = min_t(unsigned long, PFN_DOWN(end_addr), MAX_DMA_PFN);
|
|
|
|
|
|
|
|
if (start_pfn < end_pfn)
|
|
|
|
nr_free_pages += end_pfn - start_pfn;
|
|
|
|
}
|
|
|
|
|
|
|
|
set_dma_reserve(nr_pages - nr_free_pages);
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
2011-11-01 21:58:22 +08:00
|
|
|
void __init zone_sizes_init(void)
|
|
|
|
{
|
|
|
|
unsigned long max_zone_pfns[MAX_NR_ZONES];
|
|
|
|
|
|
|
|
memset(max_zone_pfns, 0, sizeof(max_zone_pfns));
|
|
|
|
|
|
|
|
#ifdef CONFIG_ZONE_DMA
|
2014-12-10 10:09:01 +08:00
|
|
|
max_zone_pfns[ZONE_DMA] = min(MAX_DMA_PFN, max_low_pfn);
|
2011-11-01 21:58:22 +08:00
|
|
|
#endif
|
|
|
|
#ifdef CONFIG_ZONE_DMA32
|
2014-12-10 10:09:01 +08:00
|
|
|
max_zone_pfns[ZONE_DMA32] = min(MAX_DMA32_PFN, max_low_pfn);
|
2011-11-01 21:58:22 +08:00
|
|
|
#endif
|
|
|
|
max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
|
|
|
|
#ifdef CONFIG_HIGHMEM
|
|
|
|
max_zone_pfns[ZONE_HIGHMEM] = max_pfn;
|
|
|
|
#endif
|
|
|
|
|
2020-06-04 06:57:10 +08:00
|
|
|
free_area_init(max_zone_pfns);
|
2011-11-01 21:58:22 +08:00
|
|
|
}
|
|
|
|
|
2021-02-21 07:17:08 +08:00
|
|
|
__visible DEFINE_PER_CPU_ALIGNED(struct tlb_state, cpu_tlbstate) = {
|
x86/mm: Rework lazy TLB to track the actual loaded mm
Lazy TLB state is currently managed in a rather baroque manner.
AFAICT, there are three possible states:
- Non-lazy. This means that we're running a user thread or a
kernel thread that has called use_mm(). current->mm ==
current->active_mm == cpu_tlbstate.active_mm and
cpu_tlbstate.state == TLBSTATE_OK.
- Lazy with user mm. We're running a kernel thread without an mm
and we're borrowing an mm_struct. We have current->mm == NULL,
current->active_mm == cpu_tlbstate.active_mm, cpu_tlbstate.state
!= TLBSTATE_OK (i.e. TLBSTATE_LAZY or 0). The current cpu is set
in mm_cpumask(current->active_mm). CR3 points to
current->active_mm->pgd. The TLB is up to date.
- Lazy with init_mm. This happens when we call leave_mm(). We
have current->mm == NULL, current->active_mm ==
cpu_tlbstate.active_mm, but that mm is only relelvant insofar as
the scheduler is tracking it for refcounting. cpu_tlbstate.state
!= TLBSTATE_OK. The current cpu is clear in
mm_cpumask(current->active_mm). CR3 points to swapper_pg_dir,
i.e. init_mm->pgd.
This patch simplifies the situation. Other than perf, x86 stops
caring about current->active_mm at all. We have
cpu_tlbstate.loaded_mm pointing to the mm that CR3 references. The
TLB is always up to date for that mm. leave_mm() just switches us
to init_mm. There are no longer any special cases for mm_cpumask,
and switch_mm() switches mms without worrying about laziness.
After this patch, cpu_tlbstate.state serves only to tell the TLB
flush code whether it may switch to init_mm instead of doing a
normal flush.
This makes fairly extensive changes to xen_exit_mmap(), which used
to look a bit like black magic.
Perf is unchanged. With or without this change, perf may behave a bit
erratically if it tries to read user memory in kernel thread context.
We should build on this patch to teach perf to never look at user
memory when cpu_tlbstate.loaded_mm != current->mm.
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Arjan van de Ven <arjan@linux.intel.com>
Cc: Borislav Petkov <bpetkov@suse.de>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Nadav Amit <nadav.amit@gmail.com>
Cc: Nadav Amit <namit@vmware.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-mm@kvack.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-05-29 01:00:15 +08:00
|
|
|
.loaded_mm = &init_mm,
|
x86/mm: Implement PCID based optimization: try to preserve old TLB entries using PCID
PCID is a "process context ID" -- it's what other architectures call
an address space ID. Every non-global TLB entry is tagged with a
PCID, only TLB entries that match the currently selected PCID are
used, and we can switch PGDs without flushing the TLB. x86's
PCID is 12 bits.
This is an unorthodox approach to using PCID. x86's PCID is far too
short to uniquely identify a process, and we can't even really
uniquely identify a running process because there are monster
systems with over 4096 CPUs. To make matters worse, past attempts
to use all 12 PCID bits have resulted in slowdowns instead of
speedups.
This patch uses PCID differently. We use a PCID to identify a
recently-used mm on a per-cpu basis. An mm has no fixed PCID
binding at all; instead, we give it a fresh PCID each time it's
loaded except in cases where we want to preserve the TLB, in which
case we reuse a recent value.
Here are some benchmark results, done on a Skylake laptop at 2.3 GHz
(turbo off, intel_pstate requesting max performance) under KVM with
the guest using idle=poll (to avoid artifacts when bouncing between
CPUs). I haven't done any real statistics here -- I just ran them
in a loop and picked the fastest results that didn't look like
outliers. Unpatched means commit a4eb8b993554, so all the
bookkeeping overhead is gone.
ping-pong between two mms on the same CPU using eventfd:
patched: 1.22µs
patched, nopcid: 1.33µs
unpatched: 1.34µs
Same ping-pong, but now touch 512 pages (all zero-page to minimize
cache misses) each iteration. dTLB misses are measured by
dtlb_load_misses.miss_causes_a_walk:
patched: 1.8µs 11M dTLB misses
patched, nopcid: 6.2µs, 207M dTLB misses
unpatched: 6.1µs, 190M dTLB misses
Signed-off-by: Andy Lutomirski <luto@kernel.org>
Reviewed-by: Nadav Amit <nadav.amit@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Arjan van de Ven <arjan@linux.intel.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Rik van Riel <riel@redhat.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-mm@kvack.org
Link: http://lkml.kernel.org/r/9ee75f17a81770feed616358e6860d98a2a5b1e7.1500957502.git.luto@kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-07-25 12:41:38 +08:00
|
|
|
.next_asid = 1,
|
2014-10-25 06:58:08 +08:00
|
|
|
.cr4 = ~0UL, /* fail hard if we screw up cr4 shadow initialization */
|
|
|
|
};
|
|
|
|
|
2014-11-03 21:02:03 +08:00
|
|
|
void update_cache_mode_entry(unsigned entry, enum page_cache_mode cache)
|
|
|
|
{
|
|
|
|
/* entry 0 MUST be WB (hardwired to speed up translations) */
|
|
|
|
BUG_ON(!entry && cache != _PAGE_CACHE_MODE_WB);
|
|
|
|
|
|
|
|
__cachemode2pte_tbl[cache] = __cm_idx2pte(entry);
|
|
|
|
__pte2cachemode_tbl[entry] = cache;
|
|
|
|
}
|
2018-06-14 06:48:28 +08:00
|
|
|
|
2018-08-15 02:50:47 +08:00
|
|
|
#ifdef CONFIG_SWAP
|
2022-08-12 00:13:30 +08:00
|
|
|
unsigned long arch_max_swapfile_size(void)
|
2018-06-14 06:48:28 +08:00
|
|
|
{
|
|
|
|
unsigned long pages;
|
|
|
|
|
|
|
|
pages = generic_max_swapfile_size();
|
|
|
|
|
2018-11-14 02:49:10 +08:00
|
|
|
if (boot_cpu_has_bug(X86_BUG_L1TF) && l1tf_mitigation != L1TF_MITIGATION_OFF) {
|
2018-06-14 06:48:28 +08:00
|
|
|
/* Limit the swap file size to MAX_PA/2 for L1TF workaround */
|
2018-08-23 21:44:18 +08:00
|
|
|
unsigned long long l1tf_limit = l1tf_pfn_limit();
|
2018-06-21 18:36:29 +08:00
|
|
|
/*
|
|
|
|
* We encode swap offsets also with 3 bits below those for pfn
|
|
|
|
* which makes the usable limit higher.
|
|
|
|
*/
|
2018-06-22 23:39:33 +08:00
|
|
|
#if CONFIG_PGTABLE_LEVELS > 2
|
2018-06-21 18:36:29 +08:00
|
|
|
l1tf_limit <<= PAGE_SHIFT - SWP_OFFSET_FIRST_BIT;
|
|
|
|
#endif
|
2018-08-20 17:58:35 +08:00
|
|
|
pages = min_t(unsigned long long, l1tf_limit, pages);
|
2018-06-14 06:48:28 +08:00
|
|
|
}
|
|
|
|
return pages;
|
|
|
|
}
|
2018-08-15 02:50:47 +08:00
|
|
|
#endif
|