UbiquitousOS
/
OpenCloudOS-Kernel
mirror of https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git
11
0
Fork 0
Code Issues Projects Releases Wiki Activity
OpenCloudOS-Kernel/arch/sh/mm/fault.c

498 lines
11 KiB
C
Raw Normal View History

sh: page table alloc cleanups and page fault optimizations. Cleanup of page table allocators, using generic folded PMD and PUD helpers. TLB flushing operations are moved to a more sensible spot. The page fault handler is also optimized slightly, we no longer waste cycles on IRQ disabling for flushing of the page from the ITLB, since we're already under CLI protection by the initial exception handler. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2006-09-27 14:13:36 +08:00
/*
* Page fault handler for SH with an MMU.
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
*
* Copyright (C) 1999 Niibe Yutaka
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
* Copyright (C) 2003 - 2012 Paul Mundt
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
*
* Based on linux/arch/i386/mm/fault.c:
* Copyright (C) 1995 Linus Torvalds
sh: page table alloc cleanups and page fault optimizations. Cleanup of page table allocators, using generic folded PMD and PUD helpers. TLB flushing operations are moved to a more sensible spot. The page fault handler is also optimized slightly, we no longer waste cycles on IRQ disabling for flushing of the page from the ITLB, since we're already under CLI protection by the initial exception handler. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2006-09-27 14:13:36 +08:00
*
* This file is subject to the terms and conditions of the GNU General Public
* License. See the file "COPYING" in the main directory of this archive
* for more details.
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
*/
#include <linux/kernel.h>
#include <linux/mm.h>
sched/headers: Prepare for new header dependencies before moving code to <linux/sched/signal.h> We are going to split <linux/sched/signal.h> out of <linux/sched.h>, which will have to be picked up from other headers and a couple of .c files. Create a trivial placeholder <linux/sched/signal.h> file that just maps to <linux/sched.h> to make this patch obviously correct and bisectable. Include the new header in the files that are going to need it. Acked-by: Linus Torvalds <torvalds@linux-foundation.org> Cc: Mike Galbraith <efault@gmx.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: linux-kernel@vger.kernel.org Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-02-09 01:51:30 +08:00
#include <linux/sched/signal.h>
sh: More cosmetic cleanups and trivial fixes. Nothing exciting here, just trivial fixes.. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2006-09-27 16:03:56 +08:00
#include <linux/hardirq.h>
#include <linux/kprobes.h>
perf: Do the big rename: Performance Counters -> Performance Events Bye-bye Performance Counters, welcome Performance Events! In the past few months the perfcounters subsystem has grown out its initial role of counting hardware events, and has become (and is becoming) a much broader generic event enumeration, reporting, logging, monitoring, analysis facility. Naming its core object 'perf_counter' and naming the subsystem 'perfcounters' has become more and more of a misnomer. With pending code like hw-breakpoints support the 'counter' name is less and less appropriate. All in one, we've decided to rename the subsystem to 'performance events' and to propagate this rename through all fields, variables and API names. (in an ABI compatible fashion) The word 'event' is also a bit shorter than 'counter' - which makes it slightly more convenient to write/handle as well. Thanks goes to Stephane Eranian who first observed this misnomer and suggested a rename. User-space tooling and ABI compatibility is not affected - this patch should be function-invariant. (Also, defconfigs were not touched to keep the size down.) This patch has been generated via the following script: FILES=$(find * -type f | grep -vE 'oprofile|[^K]config') sed -i \ -e 's/PERF_EVENT_/PERF_RECORD_/g' \ -e 's/PERF_COUNTER/PERF_EVENT/g' \ -e 's/perf_counter/perf_event/g' \ -e 's/nb_counters/nb_events/g' \ -e 's/swcounter/swevent/g' \ -e 's/tpcounter_event/tp_event/g' \ $FILES for N in $(find . -name perf_counter.[ch]); do M=$(echo $N | sed 's/perf_counter/perf_event/g') mv $N $M done FILES=$(find . -name perf_event.*) sed -i \ -e 's/COUNTER_MASK/REG_MASK/g' \ -e 's/COUNTER/EVENT/g' \ -e 's/\<event\>/event_id/g' \ -e 's/counter/event/g' \ -e 's/Counter/Event/g' \ $FILES ... to keep it as correct as possible. This script can also be used by anyone who has pending perfcounters patches - it converts a Linux kernel tree over to the new naming. We tried to time this change to the point in time where the amount of pending patches is the smallest: the end of the merge window. Namespace clashes were fixed up in a preparatory patch - and some stylistic fallout will be fixed up in a subsequent patch. ( NOTE: 'counters' are still the proper terminology when we deal with hardware registers - and these sed scripts are a bit over-eager in renaming them. I've undone some of that, but in case there's something left where 'counter' would be better than 'event' we can undo that on an individual basis instead of touching an otherwise nicely automated patch. ) Suggested-by: Stephane Eranian <eranian@google.com> Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Acked-by: Paul Mackerras <paulus@samba.org> Reviewed-by: Arjan van de Ven <arjan@linux.intel.com> Cc: Mike Galbraith <efault@gmx.de> Cc: Arnaldo Carvalho de Melo <acme@redhat.com> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: David Howells <dhowells@redhat.com> Cc: Kyle McMartin <kyle@mcmartin.ca> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> Cc: "David S. Miller" <davem@davemloft.net> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: <linux-arch@vger.kernel.org> LKML-Reference: <new-submission> Signed-off-by: Ingo Molnar <mingo@elte.hu>
2009-09-21 18:02:48 +08:00
#include <linux/perf_event.h>
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
#include <linux/kdebug.h>
mm/fault, arch: Use pagefault_disable() to check for disabled pagefaults in the handler Introduce faulthandler_disabled() and use it to check for irq context and disabled pagefaults (via pagefault_disable()) in the pagefault handlers. Please note that we keep the in_atomic() checks in place - to detect whether in irq context (in which case preemption is always properly disabled). In contrast, preempt_disable() should never be used to disable pagefaults. With !CONFIG_PREEMPT_COUNT, preempt_disable() doesn't modify the preempt counter, and therefore the result of in_atomic() differs. We validate that condition by using might_fault() checks when calling might_sleep(). Therefore, add a comment to faulthandler_disabled(), describing why this is needed. faulthandler_disabled() and pagefault_disable() are defined in linux/uaccess.h, so let's properly add that include to all relevant files. This patch is based on a patch from Thomas Gleixner. Reviewed-and-tested-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: David Hildenbrand <dahi@linux.vnet.ibm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: David.Laight@ACULAB.COM Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: airlied@linux.ie Cc: akpm@linux-foundation.org Cc: benh@kernel.crashing.org Cc: bigeasy@linutronix.de Cc: borntraeger@de.ibm.com Cc: daniel.vetter@intel.com Cc: heiko.carstens@de.ibm.com Cc: herbert@gondor.apana.org.au Cc: hocko@suse.cz Cc: hughd@google.com Cc: mst@redhat.com Cc: paulus@samba.org Cc: ralf@linux-mips.org Cc: schwidefsky@de.ibm.com Cc: yang.shi@windriver.com Link: http://lkml.kernel.org/r/1431359540-32227-7-git-send-email-dahi@linux.vnet.ibm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-05-11 23:52:11 +08:00
#include <linux/uaccess.h>
sh: trapped io support V2 The idea is that we want to get rid of the in/out/readb/writeb callbacks from the machvec and replace that with simple inline read and write operations to memory. Fast and simple for most hardware devices (think pci). Some devices require special treatment though - like 16-bit only CF devices - so we need to have some method to hook in callbacks. This patch makes it possible to add a per-device trap generating filter. This way we can get maximum performance of sane hardware - which doesn't need this filter - and crappy hardware works but gets punished by a performance hit. V2 changes things around a bit and replaces io access callbacks with a simple minimum_bus_width value. In the future we can add stride as well. Signed-off-by: Magnus Damm <damm@igel.co.jp> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2008-02-07 19:18:21 +08:00
#include <asm/io_trapped.h>
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
#include <asm/mmu_context.h>
sh: Revert TLB miss fast-path changes that broke PTEA parts. This ended up causing problems for older parts (particularly ones using PTEA). Revert this for now, it can be added back in once it's had some more testing. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2007-02-14 13:13:10 +08:00
#include <asm/tlbflush.h>
Disintegrate asm/system.h for SH Disintegrate asm/system.h for SH. Signed-off-by: David Howells <dhowells@redhat.com> cc: linux-sh@vger.kernel.org
2012-03-29 01:30:03 +08:00
#include <asm/traps.h>
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
static void
signal/sh: Remove tsk parameter from force_sig_info_fault The force_sig_info_fault function is always called with tsk == current. Make that explicit by removing the tsk parameter. This also makes it clear that the sh force_sig_info_fault passes current into force_sig_fault. Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
2019-02-06 08:38:15 +08:00
force_sig_info_fault(int si_signo, int si_code, unsigned long address)
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
{
signal: Remove the task parameter from force_sig_fault As synchronous exceptions really only make sense against the current task (otherwise how are you synchronous) remove the task parameter from from force_sig_fault to make it explicit that is what is going on. The two known exceptions that deliver a synchronous exception to a stopped ptraced task have already been changed to force_sig_fault_to_task. The callers have been changed with the following emacs regular expression (with obvious variations on the architectures that take more arguments) to avoid typos: force_sig_fault[(]\([^,]+\)[,]\([^,]+\)[,]\([^,]+\)[,]\W+current[)] -> force_sig_fault(\1,\2,\3) Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
2019-05-24 00:04:24 +08:00
force_sig_fault(si_signo, si_code, (void __user *)address);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
}
sh: Improve oops error reporting In some cases the opps error reporting doesn't give enough information to diagnose the problem, only printing information if it is thought to be valid. Replace the current code with more detailed output. This code is based on the ARM reporting, with minor changes for the SH. [lethal@linux-sh.org: fixed up for 64-bit PTEs and pte_offset_kernel()] Signed-off-by: Stuart Menefy <stuart.menefy@st.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-04-19 16:25:03 +08:00
/*
* This is useful to dump out the page tables associated with
* 'addr' in mm 'mm'.
*/
static void show_pte(struct mm_struct *mm, unsigned long addr)
{
pgd_t *pgd;
sh: Fix up recursive fault in oops with unset TTB. Presently the oops code looks for the pgd either from the mm context or the cached TTB value. There are presently cases where the TTB can be unset or otherwise cleared by hardware, which we weren't handling, resulting in recursive faults on the NULL pgd. In these cases we can simply reload from swapper_pg_dir and continue on as normal. Cc: stable@vger.kernel.org Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-07-24 12:15:54 +08:00
if (mm) {
sh: Improve oops error reporting In some cases the opps error reporting doesn't give enough information to diagnose the problem, only printing information if it is thought to be valid. Replace the current code with more detailed output. This code is based on the ARM reporting, with minor changes for the SH. [lethal@linux-sh.org: fixed up for 64-bit PTEs and pte_offset_kernel()] Signed-off-by: Stuart Menefy <stuart.menefy@st.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-04-19 16:25:03 +08:00
pgd = mm->pgd;
sh: Fix up recursive fault in oops with unset TTB. Presently the oops code looks for the pgd either from the mm context or the cached TTB value. There are presently cases where the TTB can be unset or otherwise cleared by hardware, which we weren't handling, resulting in recursive faults on the NULL pgd. In these cases we can simply reload from swapper_pg_dir and continue on as normal. Cc: stable@vger.kernel.org Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-07-24 12:15:54 +08:00
} else {
sh: Improve oops error reporting In some cases the opps error reporting doesn't give enough information to diagnose the problem, only printing information if it is thought to be valid. Replace the current code with more detailed output. This code is based on the ARM reporting, with minor changes for the SH. [lethal@linux-sh.org: fixed up for 64-bit PTEs and pte_offset_kernel()] Signed-off-by: Stuart Menefy <stuart.menefy@st.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-04-19 16:25:03 +08:00
pgd = get_TTB();
sh: Fix up recursive fault in oops with unset TTB. Presently the oops code looks for the pgd either from the mm context or the cached TTB value. There are presently cases where the TTB can be unset or otherwise cleared by hardware, which we weren't handling, resulting in recursive faults on the NULL pgd. In these cases we can simply reload from swapper_pg_dir and continue on as normal. Cc: stable@vger.kernel.org Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-07-24 12:15:54 +08:00
if (unlikely(!pgd))
pgd = swapper_pg_dir;
}
sh: Improve oops error reporting In some cases the opps error reporting doesn't give enough information to diagnose the problem, only printing information if it is thought to be valid. Replace the current code with more detailed output. This code is based on the ARM reporting, with minor changes for the SH. [lethal@linux-sh.org: fixed up for 64-bit PTEs and pte_offset_kernel()] Signed-off-by: Stuart Menefy <stuart.menefy@st.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-04-19 16:25:03 +08:00
printk(KERN_ALERT "pgd = %p\n", pgd);
pgd += pgd_index(addr);
printk(KERN_ALERT "[%08lx] *pgd=%0*Lx", addr,
sh: Enable shared page fault handler for _32/_64. This moves the now generic _32 page fault handling code to a shared place and adapts the _64 implementation to make use of it. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 14:33:28 +08:00
(u32)(sizeof(*pgd) * 2), (u64)pgd_val(*pgd));
sh: Improve oops error reporting In some cases the opps error reporting doesn't give enough information to diagnose the problem, only printing information if it is thought to be valid. Replace the current code with more detailed output. This code is based on the ARM reporting, with minor changes for the SH. [lethal@linux-sh.org: fixed up for 64-bit PTEs and pte_offset_kernel()] Signed-off-by: Stuart Menefy <stuart.menefy@st.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-04-19 16:25:03 +08:00
do {
pud_t *pud;
pmd_t *pmd;
pte_t *pte;
if (pgd_none(*pgd))
break;
if (pgd_bad(*pgd)) {
printk("(bad)");
break;
}
pud = pud_offset(pgd, addr);
if (PTRS_PER_PUD != 1)
sh: Enable shared page fault handler for _32/_64. This moves the now generic _32 page fault handling code to a shared place and adapts the _64 implementation to make use of it. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 14:33:28 +08:00
printk(", *pud=%0*Lx", (u32)(sizeof(*pud) * 2),
sh: Improve oops error reporting In some cases the opps error reporting doesn't give enough information to diagnose the problem, only printing information if it is thought to be valid. Replace the current code with more detailed output. This code is based on the ARM reporting, with minor changes for the SH. [lethal@linux-sh.org: fixed up for 64-bit PTEs and pte_offset_kernel()] Signed-off-by: Stuart Menefy <stuart.menefy@st.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-04-19 16:25:03 +08:00
(u64)pud_val(*pud));
if (pud_none(*pud))
break;
if (pud_bad(*pud)) {
printk("(bad)");
break;
}
pmd = pmd_offset(pud, addr);
if (PTRS_PER_PMD != 1)
sh: Enable shared page fault handler for _32/_64. This moves the now generic _32 page fault handling code to a shared place and adapts the _64 implementation to make use of it. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 14:33:28 +08:00
printk(", *pmd=%0*Lx", (u32)(sizeof(*pmd) * 2),
sh: Improve oops error reporting In some cases the opps error reporting doesn't give enough information to diagnose the problem, only printing information if it is thought to be valid. Replace the current code with more detailed output. This code is based on the ARM reporting, with minor changes for the SH. [lethal@linux-sh.org: fixed up for 64-bit PTEs and pte_offset_kernel()] Signed-off-by: Stuart Menefy <stuart.menefy@st.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-04-19 16:25:03 +08:00
(u64)pmd_val(*pmd));
if (pmd_none(*pmd))
break;
if (pmd_bad(*pmd)) {
printk("(bad)");
break;
}
/* We must not map this if we have highmem enabled */
if (PageHighMem(pfn_to_page(pmd_val(*pmd) >> PAGE_SHIFT)))
break;
pte = pte_offset_kernel(pmd, addr);
sh: Enable shared page fault handler for _32/_64. This moves the now generic _32 page fault handling code to a shared place and adapts the _64 implementation to make use of it. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 14:33:28 +08:00
printk(", *pte=%0*Lx", (u32)(sizeof(*pte) * 2),
(u64)pte_val(*pte));
sh: Improve oops error reporting In some cases the opps error reporting doesn't give enough information to diagnose the problem, only printing information if it is thought to be valid. Replace the current code with more detailed output. This code is based on the ARM reporting, with minor changes for the SH. [lethal@linux-sh.org: fixed up for 64-bit PTEs and pte_offset_kernel()] Signed-off-by: Stuart Menefy <stuart.menefy@st.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-04-19 16:25:03 +08:00
} while (0);
printk("\n");
}
sh: Tidy up vmalloc fault handling. This rewrites the vmalloc fault handling as per x86, which subsequently allows for easy future tie-in for vmalloc_sync_all(). Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2009-07-05 02:18:47 +08:00
static inline pmd_t *vmalloc_sync_one(pgd_t *pgd, unsigned long address)
{
unsigned index = pgd_index(address);
pgd_t *pgd_k;
pud_t *pud, *pud_k;
pmd_t *pmd, *pmd_k;
pgd += index;
pgd_k = init_mm.pgd + index;
if (!pgd_present(*pgd_k))
return NULL;
pud = pud_offset(pgd, address);
pud_k = pud_offset(pgd_k, address);
if (!pud_present(*pud_k))
return NULL;
sh: Definitions for 3-level page table layout If using 64-bit PTEs and 4K pages then each page table has 512 entries (as opposed to 1024 entries with 32-bit PTEs). Unlike MIPS, SH follows the convention that all structures in the page table (pgd_t, pmd_t, pgprot_t, etc) must be the same size. Therefore, 64-bit PTEs require 64-bit PGD entries, etc. Using 2-levels of page tables and 64-bit PTEs it is only possible to map 1GB of virtual address space. In order to map all 4GB of virtual address space we need to adopt a 3-level page table layout. This actually works out better for CONFIG_SUPERH32 because we only waste 2 PGD entries on the P1 and P2 areas (which are untranslated) instead of 256. Signed-off-by: Matt Fleming <matt@console-pimps.org> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2009-12-13 22:38:50 +08:00
if (!pud_present(*pud))
set_pud(pud, *pud_k);
sh: Tidy up vmalloc fault handling. This rewrites the vmalloc fault handling as per x86, which subsequently allows for easy future tie-in for vmalloc_sync_all(). Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2009-07-05 02:18:47 +08:00
pmd = pmd_offset(pud, address);
pmd_k = pmd_offset(pud_k, address);
if (!pmd_present(*pmd_k))
return NULL;
if (!pmd_present(*pmd))
set_pmd(pmd, *pmd_k);
sh: Restore previous behaviour on kernel fault The last commit changed the behaviour on kernel faults when we were doing something other than syncing the page tables. vmalloc_sync_one() needs to return NULL if the page tables are up to date, because the reason for the fault was not a missing/inconsitent page table entry. By returning NULL if the page tables are sync'd we signal to the calling function that further work must be done to resolve this fault. Also, remove the superfluous __va() around the first argument to vmalloc_sync_one(). The value of pgd_k is already a virtual address and using it wth __va() causes a NULL dereference. Signed-off-by: Matt Fleming <matt@console-pimps.org> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2009-07-13 19:38:04 +08:00
else {
/*
* The page tables are fully synchronised so there must
* be another reason for the fault. Return NULL here to
* signal that we have not taken care of the fault.
*/
sh: Tidy up vmalloc fault handling. This rewrites the vmalloc fault handling as per x86, which subsequently allows for easy future tie-in for vmalloc_sync_all(). Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2009-07-05 02:18:47 +08:00
BUG_ON(pmd_page(*pmd) != pmd_page(*pmd_k));
sh: Restore previous behaviour on kernel fault The last commit changed the behaviour on kernel faults when we were doing something other than syncing the page tables. vmalloc_sync_one() needs to return NULL if the page tables are up to date, because the reason for the fault was not a missing/inconsitent page table entry. By returning NULL if the page tables are sync'd we signal to the calling function that further work must be done to resolve this fault. Also, remove the superfluous __va() around the first argument to vmalloc_sync_one(). The value of pgd_k is already a virtual address and using it wth __va() causes a NULL dereference. Signed-off-by: Matt Fleming <matt@console-pimps.org> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2009-07-13 19:38:04 +08:00
return NULL;
}
sh: Tidy up vmalloc fault handling. This rewrites the vmalloc fault handling as per x86, which subsequently allows for easy future tie-in for vmalloc_sync_all(). Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2009-07-05 02:18:47 +08:00
return pmd_k;
}
sh64: Fix up vmalloc fault range check. With the previous attempt reverted this switches to conditionalizing the end address. Nominally VMALLOC_END, but extended for P3_ADDR_MAX in the store queue case. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-18 19:01:16 +08:00
#ifdef CONFIG_SH_STORE_QUEUES
#define __FAULT_ADDR_LIMIT P3_ADDR_MAX
#else
#define __FAULT_ADDR_LIMIT VMALLOC_END
#endif
sh: Tidy up vmalloc fault handling. This rewrites the vmalloc fault handling as per x86, which subsequently allows for easy future tie-in for vmalloc_sync_all(). Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2009-07-05 02:18:47 +08:00
/*
* Handle a fault on the vmalloc or module mapping area
*/
static noinline int vmalloc_fault(unsigned long address)
{
pgd_t *pgd_k;
pmd_t *pmd_k;
pte_t *pte_k;
Revert "sh: Ensure fixmap and store queue space can co-exist." This reverts commit 20e7c297efeca0861adcca073a0d283da659834b. With store queues enabled the area above P4SEG has special properties from the MMU's point of view, which was causing fixmap failure. We'll have to do something else to satisfy the vmalloc range check. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-18 18:30:05 +08:00
/* Make sure we are in vmalloc/module/P3 area: */
sh64: Fix up vmalloc fault range check. With the previous attempt reverted this switches to conditionalizing the end address. Nominally VMALLOC_END, but extended for P3_ADDR_MAX in the store queue case. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-18 19:01:16 +08:00
if (!(address >= VMALLOC_START && address < __FAULT_ADDR_LIMIT))
sh: Tidy up vmalloc fault handling. This rewrites the vmalloc fault handling as per x86, which subsequently allows for easy future tie-in for vmalloc_sync_all(). Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2009-07-05 02:18:47 +08:00
return -1;
/*
* Synchronize this task's top level page-table
* with the 'reference' page table.
*
* Do _not_ use "current" here. We might be inside
* an interrupt in the middle of a task switch..
*/
pgd_k = get_TTB();
sh: Restore previous behaviour on kernel fault The last commit changed the behaviour on kernel faults when we were doing something other than syncing the page tables. vmalloc_sync_one() needs to return NULL if the page tables are up to date, because the reason for the fault was not a missing/inconsitent page table entry. By returning NULL if the page tables are sync'd we signal to the calling function that further work must be done to resolve this fault. Also, remove the superfluous __va() around the first argument to vmalloc_sync_one(). The value of pgd_k is already a virtual address and using it wth __va() causes a NULL dereference. Signed-off-by: Matt Fleming <matt@console-pimps.org> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2009-07-13 19:38:04 +08:00
pmd_k = vmalloc_sync_one(pgd_k, address);
sh: Tidy up vmalloc fault handling. This rewrites the vmalloc fault handling as per x86, which subsequently allows for easy future tie-in for vmalloc_sync_all(). Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2009-07-05 02:18:47 +08:00
if (!pmd_k)
return -1;
pte_k = pte_offset_kernel(pmd_k, address);
if (!pte_present(*pte_k))
return -1;
return 0;
}
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
static void
show_fault_oops(struct pt_regs *regs, unsigned long address)
{
if (!oops_may_print())
return;
printk(KERN_ALERT "BUG: unable to handle kernel ");
if (address < PAGE_SIZE)
printk(KERN_CONT "NULL pointer dereference");
else
printk(KERN_CONT "paging request");
printk(KERN_CONT " at %08lx\n", address);
printk(KERN_ALERT "PC:");
printk_address(regs->pc, 1);
show_pte(NULL, address);
}
static noinline void
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
no_context(struct pt_regs *regs, unsigned long error_code,
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
unsigned long address)
{
/* Are we prepared to handle this kernel fault? */
if (fixup_exception(regs))
return;
if (handle_trapped_io(regs, address))
return;
/*
* Oops. The kernel tried to access some bad page. We'll have to
* terminate things with extreme prejudice.
*/
bust_spinlocks(1);
show_fault_oops(regs, address);
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
die("Oops", regs, error_code);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
bust_spinlocks(0);
do_exit(SIGKILL);
}
static void
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
__bad_area_nosemaphore(struct pt_regs *regs, unsigned long error_code,
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
unsigned long address, int si_code)
{
/* User mode accesses just cause a SIGSEGV */
if (user_mode(regs)) {
/*
* It's possible to have interrupts off here:
*/
local_irq_enable();
signal/sh: Remove tsk parameter from force_sig_info_fault The force_sig_info_fault function is always called with tsk == current. Make that explicit by removing the tsk parameter. This also makes it clear that the sh force_sig_info_fault passes current into force_sig_fault. Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
2019-02-06 08:38:15 +08:00
force_sig_info_fault(SIGSEGV, si_code, address);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
return;
}
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
no_context(regs, error_code, address);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
}
static noinline void
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
bad_area_nosemaphore(struct pt_regs *regs, unsigned long error_code,
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
unsigned long address)
{
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
__bad_area_nosemaphore(regs, error_code, address, SEGV_MAPERR);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
}
static void
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
__bad_area(struct pt_regs *regs, unsigned long error_code,
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
unsigned long address, int si_code)
{
struct mm_struct *mm = current->mm;
/*
* Something tried to access memory that isn't in our memory map..
* Fix it, but check if it's kernel or user first..
*/
up_read(&mm->mmap_sem);
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
__bad_area_nosemaphore(regs, error_code, address, si_code);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
}
static noinline void
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
bad_area(struct pt_regs *regs, unsigned long error_code, unsigned long address)
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
{
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
__bad_area(regs, error_code, address, SEGV_MAPERR);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
}
static noinline void
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
bad_area_access_error(struct pt_regs *regs, unsigned long error_code,
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
unsigned long address)
{
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
__bad_area(regs, error_code, address, SEGV_ACCERR);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
}
static void
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
do_sigbus(struct pt_regs *regs, unsigned long error_code, unsigned long address)
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
{
struct task_struct *tsk = current;
struct mm_struct *mm = tsk->mm;
up_read(&mm->mmap_sem);
/* Kernel mode? Handle exceptions or die: */
if (!user_mode(regs))
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
no_context(regs, error_code, address);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
signal/sh: Remove tsk parameter from force_sig_info_fault The force_sig_info_fault function is always called with tsk == current. Make that explicit by removing the tsk parameter. This also makes it clear that the sh force_sig_info_fault passes current into force_sig_fault. Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
2019-02-06 08:38:15 +08:00
force_sig_info_fault(SIGBUS, BUS_ADRERR, address);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
}
static noinline int
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
mm_fault_error(struct pt_regs *regs, unsigned long error_code,
mm: convert return type of handle_mm_fault() caller to vm_fault_t Use new return type vm_fault_t for fault handler. For now, this is just documenting that the function returns a VM_FAULT value rather than an errno. Once all instances are converted, vm_fault_t will become a distinct type. Ref-> commit 1c8f422059ae ("mm: change return type to vm_fault_t") In this patch all the caller of handle_mm_fault() are changed to return vm_fault_t type. Link: http://lkml.kernel.org/r/20180617084810.GA6730@jordon-HP-15-Notebook-PC Signed-off-by: Souptick Joarder <jrdr.linux@gmail.com> Cc: Matthew Wilcox <willy@infradead.org> Cc: Richard Henderson <rth@twiddle.net> Cc: Tony Luck <tony.luck@intel.com> Cc: Matt Turner <mattst88@gmail.com> Cc: Vineet Gupta <vgupta@synopsys.com> Cc: Russell King <linux@armlinux.org.uk> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will.deacon@arm.com> Cc: Richard Kuo <rkuo@codeaurora.org> Cc: Geert Uytterhoeven <geert@linux-m68k.org> Cc: Michal Simek <monstr@monstr.eu> Cc: James Hogan <jhogan@kernel.org> Cc: Ley Foon Tan <lftan@altera.com> Cc: Jonas Bonn <jonas@southpole.se> Cc: James E.J. Bottomley <jejb@parisc-linux.org> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: Palmer Dabbelt <palmer@sifive.com> Cc: Yoshinori Sato <ysato@users.sourceforge.jp> Cc: David S. Miller <davem@davemloft.net> Cc: Richard Weinberger <richard@nod.at> Cc: Guan Xuetao <gxt@pku.edu.cn> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: "Levin, Alexander (Sasha Levin)" <alexander.levin@verizon.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2018-08-18 06:44:47 +08:00
unsigned long address, vm_fault_t fault)
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
{
/*
* Pagefault was interrupted by SIGKILL. We have no reason to
* continue pagefault.
*/
if (fatal_signal_pending(current)) {
if (!(fault & VM_FAULT_RETRY))
up_read(&current->mm->mmap_sem);
if (!user_mode(regs))
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
no_context(regs, error_code, address);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
return 1;
}
if (!(fault & VM_FAULT_ERROR))
return 0;
if (fault & VM_FAULT_OOM) {
/* Kernel mode? Handle exceptions or die: */
if (!user_mode(regs)) {
up_read(&current->mm->mmap_sem);
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
no_context(regs, error_code, address);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
return 1;
}
mm, oom: remove statically defined arch functions of same name out_of_memory() is a globally defined function to call the oom killer. x86, sh, and powerpc all use a function of the same name within file scope in their respective fault.c unnecessarily. Inline the functions into the pagefault handlers to clean the code up. Signed-off-by: David Rientjes <rientjes@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: Paul Mackerras <paulus@samba.org> Cc: Paul Mundt <lethal@linux-sh.org> Reviewed-by: Michal Hocko <mhocko@suse.cz> Reviewed-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2012-12-13 05:52:10 +08:00
up_read(&current->mm->mmap_sem);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
mm, oom: remove statically defined arch functions of same name out_of_memory() is a globally defined function to call the oom killer. x86, sh, and powerpc all use a function of the same name within file scope in their respective fault.c unnecessarily. Inline the functions into the pagefault handlers to clean the code up. Signed-off-by: David Rientjes <rientjes@google.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: Paul Mackerras <paulus@samba.org> Cc: Paul Mundt <lethal@linux-sh.org> Reviewed-by: Michal Hocko <mhocko@suse.cz> Reviewed-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2012-12-13 05:52:10 +08:00
/*
* We ran out of memory, call the OOM killer, and return the
* userspace (which will retry the fault, or kill us if we got
* oom-killed):
*/
pagefault_out_of_memory();
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
} else {
if (fault & VM_FAULT_SIGBUS)
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
do_sigbus(regs, error_code, address);
vm: add VM_FAULT_SIGSEGV handling support The core VM already knows about VM_FAULT_SIGBUS, but cannot return a "you should SIGSEGV" error, because the SIGSEGV case was generally handled by the caller - usually the architecture fault handler. That results in lots of duplication - all the architecture fault handlers end up doing very similar "look up vma, check permissions, do retries etc" - but it generally works. However, there are cases where the VM actually wants to SIGSEGV, and applications _expect_ SIGSEGV. In particular, when accessing the stack guard page, libsigsegv expects a SIGSEGV. And it usually got one, because the stack growth is handled by that duplicated architecture fault handler. However, when the generic VM layer started propagating the error return from the stack expansion in commit fee7e49d4514 ("mm: propagate error from stack expansion even for guard page"), that now exposed the existing VM_FAULT_SIGBUS result to user space. And user space really expected SIGSEGV, not SIGBUS. To fix that case, we need to add a VM_FAULT_SIGSEGV, and teach all those duplicate architecture fault handlers about it. They all already have the code to handle SIGSEGV, so it's about just tying that new return value to the existing code, but it's all a bit annoying. This is the mindless minimal patch to do this. A more extensive patch would be to try to gather up the mostly shared fault handling logic into one generic helper routine, and long-term we really should do that cleanup. Just from this patch, you can generally see that most architectures just copied (directly or indirectly) the old x86 way of doing things, but in the meantime that original x86 model has been improved to hold the VM semaphore for shorter times etc and to handle VM_FAULT_RETRY and other "newer" things, so it would be a good idea to bring all those improvements to the generic case and teach other architectures about them too. Reported-and-tested-by: Takashi Iwai <tiwai@suse.de> Tested-by: Jan Engelhardt <jengelh@inai.de> Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com> # "s390 still compiles and boots" Cc: linux-arch@vger.kernel.org Cc: stable@vger.kernel.org Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2015-01-30 02:51:32 +08:00
else if (fault & VM_FAULT_SIGSEGV)
bad_area(regs, error_code, address);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
else
BUG();
}
return 1;
}
sh: Enable shared page fault handler for _32/_64. This moves the now generic _32 page fault handling code to a shared place and adapts the _64 implementation to make use of it. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 14:33:28 +08:00
static inline int access_error(int error_code, struct vm_area_struct *vma)
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
{
sh: Enable shared page fault handler for _32/_64. This moves the now generic _32 page fault handling code to a shared place and adapts the _64 implementation to make use of it. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 14:33:28 +08:00
if (error_code & FAULT_CODE_WRITE) {
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
/* write, present and write, not present: */
if (unlikely(!(vma->vm_flags & VM_WRITE)))
return 1;
return 0;
}
sh: Enable shared page fault handler for _32/_64. This moves the now generic _32 page fault handling code to a shared place and adapts the _64 implementation to make use of it. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 14:33:28 +08:00
/* ITLB miss on NX page */
if (unlikely((error_code & FAULT_CODE_ITLB) &&
!(vma->vm_flags & VM_EXEC)))
return 1;
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
/* read, not present: */
if (unlikely(!(vma->vm_flags & (VM_READ | VM_EXEC | VM_WRITE))))
return 1;
return 0;
}
sh: Tidy up vmalloc fault handling. This rewrites the vmalloc fault handling as per x86, which subsequently allows for easy future tie-in for vmalloc_sync_all(). Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2009-07-05 02:18:47 +08:00
static int fault_in_kernel_space(unsigned long address)
{
return address >= TASK_SIZE;
}
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
/*
* This routine handles page faults. It determines the address,
* and the problem, and then passes it off to one of the appropriate
* routines.
*/
sh: Set up correct siginfo structures for page faults. Remove the previous saving of fault codes into the thread_struct as they are never used, and appeared to be inherited from x86. Signed-off-by: Stuart Menefy <stuart.menefy@st.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2006-11-21 12:34:04 +08:00
asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
unsigned long error_code,
sh: Set up correct siginfo structures for page faults. Remove the previous saving of fault codes into the thread_struct as they are never used, and appeared to be inherited from x86. Signed-off-by: Stuart Menefy <stuart.menefy@st.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2006-11-21 12:34:04 +08:00
unsigned long address)
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
{
sh: Tidy up vmalloc fault handling. This rewrites the vmalloc fault handling as per x86, which subsequently allows for easy future tie-in for vmalloc_sync_all(). Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2009-07-05 02:18:47 +08:00
unsigned long vec;
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
struct task_struct *tsk;
struct mm_struct *mm;
struct vm_area_struct * vma;
mm: convert return type of handle_mm_fault() caller to vm_fault_t Use new return type vm_fault_t for fault handler. For now, this is just documenting that the function returns a VM_FAULT value rather than an errno. Once all instances are converted, vm_fault_t will become a distinct type. Ref-> commit 1c8f422059ae ("mm: change return type to vm_fault_t") In this patch all the caller of handle_mm_fault() are changed to return vm_fault_t type. Link: http://lkml.kernel.org/r/20180617084810.GA6730@jordon-HP-15-Notebook-PC Signed-off-by: Souptick Joarder <jrdr.linux@gmail.com> Cc: Matthew Wilcox <willy@infradead.org> Cc: Richard Henderson <rth@twiddle.net> Cc: Tony Luck <tony.luck@intel.com> Cc: Matt Turner <mattst88@gmail.com> Cc: Vineet Gupta <vgupta@synopsys.com> Cc: Russell King <linux@armlinux.org.uk> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will.deacon@arm.com> Cc: Richard Kuo <rkuo@codeaurora.org> Cc: Geert Uytterhoeven <geert@linux-m68k.org> Cc: Michal Simek <monstr@monstr.eu> Cc: James Hogan <jhogan@kernel.org> Cc: Ley Foon Tan <lftan@altera.com> Cc: Jonas Bonn <jonas@southpole.se> Cc: James E.J. Bottomley <jejb@parisc-linux.org> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: Palmer Dabbelt <palmer@sifive.com> Cc: Yoshinori Sato <ysato@users.sourceforge.jp> Cc: David S. Miller <davem@davemloft.net> Cc: Richard Weinberger <richard@nod.at> Cc: Guan Xuetao <gxt@pku.edu.cn> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: "Levin, Alexander (Sasha Levin)" <alexander.levin@verizon.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2018-08-18 06:44:47 +08:00
vm_fault_t fault;
arch: mm: pass userspace fault flag to generic fault handler Unlike global OOM handling, memory cgroup code will invoke the OOM killer in any OOM situation because it has no way of telling faults occuring in kernel context - which could be handled more gracefully - from user-triggered faults. Pass a flag that identifies faults originating in user space from the architecture-specific fault handlers to generic code so that memcg OOM handling can be improved. Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> Reviewed-by: Michal Hocko <mhocko@suse.cz> Cc: David Rientjes <rientjes@google.com> Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> Cc: azurIt <azurit@pobox.sk> Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-09-13 06:13:39 +08:00
unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
tsk = current;
sh: Tidy up vmalloc fault handling. This rewrites the vmalloc fault handling as per x86, which subsequently allows for easy future tie-in for vmalloc_sync_all(). Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2009-07-05 02:18:47 +08:00
mm = tsk->mm;
vec = lookup_exception_vector();
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
sh: Tidy up vmalloc fault handling. This rewrites the vmalloc fault handling as per x86, which subsequently allows for easy future tie-in for vmalloc_sync_all(). Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2009-07-05 02:18:47 +08:00
/*
* We fault-in kernel-space virtual memory on-demand. The
* 'reference' page table is init_mm.pgd.
*
* NOTE! We MUST NOT take any locks for this case. We may
* be in an interrupt or a critical region, and should
* only copy the information from the master page table,
* nothing more.
*/
if (unlikely(fault_in_kernel_space(address))) {
if (vmalloc_fault(address) >= 0)
sh: pmd rework. Remove extra bits from the pmd structure and store a kernel logical address rather than a physical address. This allows it to be directly dereferenced. Another piece of wierdness inherited from x86. Signed-off-by: Stuart Menefy <stuart.menefy@st.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2006-11-21 14:38:05 +08:00
return;
mm, kprobes: generalize and rename notify_page_fault() as kprobe_page_fault() Architectures which support kprobes have very similar boilerplate around calling kprobe_fault_handler(). Use a helper function in kprobes.h to unify them, based on the x86 code. This changes the behaviour for other architectures when preemption is enabled. Previously, they would have disabled preemption while calling the kprobe handler. However, preemption would be disabled if this fault was due to a kprobe, so we know the fault was not due to a kprobe handler and can simply return failure. This behaviour was introduced in commit a980c0ef9f6d ("x86/kprobes: Refactor kprobes_fault() like kprobe_exceptions_notify()") [anshuman.khandual@arm.com: export kprobe_fault_handler()] Link: http://lkml.kernel.org/r/1561133358-8876-1-git-send-email-anshuman.khandual@arm.com Link: http://lkml.kernel.org/r/1560420444-25737-1-git-send-email-anshuman.khandual@arm.com Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Matthew Wilcox <willy@infradead.org> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Christophe Leroy <christophe.leroy@c-s.fr> Cc: Stephen Rothwell <sfr@canb.auug.org.au> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Paul Mackerras <paulus@samba.org> Cc: Russell King <linux@armlinux.org.uk> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will.deacon@arm.com> Cc: Tony Luck <tony.luck@intel.com> Cc: Fenghua Yu <fenghua.yu@intel.com> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> Cc: Heiko Carstens <heiko.carstens@de.ibm.com> Cc: Yoshinori Sato <ysato@users.sourceforge.jp> Cc: "David S. Miller" <davem@davemloft.net> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Ingo Molnar <mingo@redhat.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: Vineet Gupta <vgupta@synopsys.com> Cc: James Hogan <jhogan@kernel.org> Cc: Paul Burton <paul.burton@mips.com> Cc: Ralf Baechle <ralf@linux-mips.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2019-07-17 07:28:00 +08:00
if (kprobe_page_fault(regs, vec))
sh: vmalloc pgtable sync fix. This fixes a problem in the code which copies the vmalloc portion of the kernel's page table into the current user space page table. The addition of the four level page table code breaks on folded page tables, because the pud level is always present (although folded). This updates the code to use the same style of updates for the pud as is used for the pgd level. Signed-off-by: Stuart Menefy <stuart.menefy@st.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2008-09-05 15:17:15 +08:00
return;
sh: pmd rework. Remove extra bits from the pmd structure and store a kernel logical address rather than a physical address. This allows it to be directly dereferenced. Another piece of wierdness inherited from x86. Signed-off-by: Stuart Menefy <stuart.menefy@st.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2006-11-21 14:38:05 +08:00
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
bad_area_nosemaphore(regs, error_code, address);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
return;
sh: pmd rework. Remove extra bits from the pmd structure and store a kernel logical address rather than a physical address. This allows it to be directly dereferenced. Another piece of wierdness inherited from x86. Signed-off-by: Stuart Menefy <stuart.menefy@st.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2006-11-21 14:38:05 +08:00
}
mm, kprobes: generalize and rename notify_page_fault() as kprobe_page_fault() Architectures which support kprobes have very similar boilerplate around calling kprobe_fault_handler(). Use a helper function in kprobes.h to unify them, based on the x86 code. This changes the behaviour for other architectures when preemption is enabled. Previously, they would have disabled preemption while calling the kprobe handler. However, preemption would be disabled if this fault was due to a kprobe, so we know the fault was not due to a kprobe handler and can simply return failure. This behaviour was introduced in commit a980c0ef9f6d ("x86/kprobes: Refactor kprobes_fault() like kprobe_exceptions_notify()") [anshuman.khandual@arm.com: export kprobe_fault_handler()] Link: http://lkml.kernel.org/r/1561133358-8876-1-git-send-email-anshuman.khandual@arm.com Link: http://lkml.kernel.org/r/1560420444-25737-1-git-send-email-anshuman.khandual@arm.com Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> Reviewed-by: Dave Hansen <dave.hansen@linux.intel.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Matthew Wilcox <willy@infradead.org> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Christophe Leroy <christophe.leroy@c-s.fr> Cc: Stephen Rothwell <sfr@canb.auug.org.au> Cc: Andrey Konovalov <andreyknvl@google.com> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Paul Mackerras <paulus@samba.org> Cc: Russell King <linux@armlinux.org.uk> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will.deacon@arm.com> Cc: Tony Luck <tony.luck@intel.com> Cc: Fenghua Yu <fenghua.yu@intel.com> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> Cc: Heiko Carstens <heiko.carstens@de.ibm.com> Cc: Yoshinori Sato <ysato@users.sourceforge.jp> Cc: "David S. Miller" <davem@davemloft.net> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Ingo Molnar <mingo@redhat.com> Cc: Andy Lutomirski <luto@kernel.org> Cc: Vineet Gupta <vgupta@synopsys.com> Cc: James Hogan <jhogan@kernel.org> Cc: Paul Burton <paul.burton@mips.com> Cc: Ralf Baechle <ralf@linux-mips.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2019-07-17 07:28:00 +08:00
if (unlikely(kprobe_page_fault(regs, vec)))
sh: Hook up page fault events for software perf counters. This adds page fault instrumentation for the software performance counters. Follows the x86 and powerpc changes. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2009-06-25 01:30:10 +08:00
return;
sh: Conditionally re-enable IRQs in fault path. The current kernel behaviour is to reenable interrupts unconditionally when taking a page fault. This patch changes this to only enable them if interrupts were previously enabled. It also fixes a problem seen with this fix in place: the kernel previously flushed the vsyscall page when handling a signal, which is not only unncessary, but caused a possible sleep with interrupts disabled. Signed-off-by: Stuart Menefy <stuart.menefy@st.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2008-07-02 16:51:23 +08:00
/* Only enable interrupts if they were on before the fault */
sh: Hook up page fault events for software perf counters. This adds page fault instrumentation for the software performance counters. Follows the x86 and powerpc changes. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2009-06-25 01:30:10 +08:00
if ((regs->sr & SR_IMASK) != SR_IMASK)
sh: Conditionally re-enable IRQs in fault path. The current kernel behaviour is to reenable interrupts unconditionally when taking a page fault. This patch changes this to only enable them if interrupts were previously enabled. It also fixes a problem seen with this fix in place: the kernel previously flushed the vsyscall page when handling a signal, which is not only unncessary, but caused a possible sleep with interrupts disabled. Signed-off-by: Stuart Menefy <stuart.menefy@st.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2008-07-02 16:51:23 +08:00
local_irq_enable();
perf: Remove the nmi parameter from the swevent and overflow interface The nmi parameter indicated if we could do wakeups from the current context, if not, we would set some state and self-IPI and let the resulting interrupt do the wakeup. For the various event classes: - hardware: nmi=0; PMI is in fact an NMI or we run irq_work_run from the PMI-tail (ARM etc.) - tracepoint: nmi=0; since tracepoint could be from NMI context. - software: nmi=[0,1]; some, like the schedule thing cannot perform wakeups, and hence need 0. As one can see, there is very little nmi=1 usage, and the down-side of not using it is that on some platforms some software events can have a jiffy delay in wakeup (when arch_irq_work_raise isn't implemented). The up-side however is that we can remove the nmi parameter and save a bunch of conditionals in fast paths. Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Michael Cree <mcree@orcon.net.nz> Cc: Will Deacon <will.deacon@arm.com> Cc: Deng-Cheng Zhu <dengcheng.zhu@gmail.com> Cc: Anton Blanchard <anton@samba.org> Cc: Eric B Munson <emunson@mgebm.net> Cc: Heiko Carstens <heiko.carstens@de.ibm.com> Cc: Paul Mundt <lethal@linux-sh.org> Cc: David S. Miller <davem@davemloft.net> Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Jason Wessel <jason.wessel@windriver.com> Cc: Don Zickus <dzickus@redhat.com> Link: http://lkml.kernel.org/n/tip-agjev8eu666tvknpb3iaj0fg@git.kernel.org Signed-off-by: Ingo Molnar <mingo@elte.hu>
2011-06-27 20:41:57 +08:00
perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, address);
sh: Conditionally re-enable IRQs in fault path. The current kernel behaviour is to reenable interrupts unconditionally when taking a page fault. This patch changes this to only enable them if interrupts were previously enabled. It also fixes a problem seen with this fix in place: the kernel previously flushed the vsyscall page when handling a signal, which is not only unncessary, but caused a possible sleep with interrupts disabled. Signed-off-by: Stuart Menefy <stuart.menefy@st.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2008-07-02 16:51:23 +08:00
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
/*
sh: Tidy up vmalloc fault handling. This rewrites the vmalloc fault handling as per x86, which subsequently allows for easy future tie-in for vmalloc_sync_all(). Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2009-07-05 02:18:47 +08:00
* If we're in an interrupt, have no user context or are running
mm/fault, arch: Use pagefault_disable() to check for disabled pagefaults in the handler Introduce faulthandler_disabled() and use it to check for irq context and disabled pagefaults (via pagefault_disable()) in the pagefault handlers. Please note that we keep the in_atomic() checks in place - to detect whether in irq context (in which case preemption is always properly disabled). In contrast, preempt_disable() should never be used to disable pagefaults. With !CONFIG_PREEMPT_COUNT, preempt_disable() doesn't modify the preempt counter, and therefore the result of in_atomic() differs. We validate that condition by using might_fault() checks when calling might_sleep(). Therefore, add a comment to faulthandler_disabled(), describing why this is needed. faulthandler_disabled() and pagefault_disable() are defined in linux/uaccess.h, so let's properly add that include to all relevant files. This patch is based on a patch from Thomas Gleixner. Reviewed-and-tested-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: David Hildenbrand <dahi@linux.vnet.ibm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: David.Laight@ACULAB.COM Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: airlied@linux.ie Cc: akpm@linux-foundation.org Cc: benh@kernel.crashing.org Cc: bigeasy@linutronix.de Cc: borntraeger@de.ibm.com Cc: daniel.vetter@intel.com Cc: heiko.carstens@de.ibm.com Cc: herbert@gondor.apana.org.au Cc: hocko@suse.cz Cc: hughd@google.com Cc: mst@redhat.com Cc: paulus@samba.org Cc: ralf@linux-mips.org Cc: schwidefsky@de.ibm.com Cc: yang.shi@windriver.com Link: http://lkml.kernel.org/r/1431359540-32227-7-git-send-email-dahi@linux.vnet.ibm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-05-11 23:52:11 +08:00
* with pagefaults disabled then we must not take the fault:
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
*/
mm/fault, arch: Use pagefault_disable() to check for disabled pagefaults in the handler Introduce faulthandler_disabled() and use it to check for irq context and disabled pagefaults (via pagefault_disable()) in the pagefault handlers. Please note that we keep the in_atomic() checks in place - to detect whether in irq context (in which case preemption is always properly disabled). In contrast, preempt_disable() should never be used to disable pagefaults. With !CONFIG_PREEMPT_COUNT, preempt_disable() doesn't modify the preempt counter, and therefore the result of in_atomic() differs. We validate that condition by using might_fault() checks when calling might_sleep(). Therefore, add a comment to faulthandler_disabled(), describing why this is needed. faulthandler_disabled() and pagefault_disable() are defined in linux/uaccess.h, so let's properly add that include to all relevant files. This patch is based on a patch from Thomas Gleixner. Reviewed-and-tested-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: David Hildenbrand <dahi@linux.vnet.ibm.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: David.Laight@ACULAB.COM Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: airlied@linux.ie Cc: akpm@linux-foundation.org Cc: benh@kernel.crashing.org Cc: bigeasy@linutronix.de Cc: borntraeger@de.ibm.com Cc: daniel.vetter@intel.com Cc: heiko.carstens@de.ibm.com Cc: herbert@gondor.apana.org.au Cc: hocko@suse.cz Cc: hughd@google.com Cc: mst@redhat.com Cc: paulus@samba.org Cc: ralf@linux-mips.org Cc: schwidefsky@de.ibm.com Cc: yang.shi@windriver.com Link: http://lkml.kernel.org/r/1431359540-32227-7-git-send-email-dahi@linux.vnet.ibm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-05-11 23:52:11 +08:00
if (unlikely(faulthandler_disabled() || !mm)) {
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
bad_area_nosemaphore(regs, error_code, address);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
return;
}
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
sh/mm/fault_32.c: Port OOM changes to do_page_fault Commit d065bd810b6deb67d4897a14bfe21f8eb526ba99 (mm: retry page fault when blocking on disk transfer) and commit 37b23e0525d393d48a7d59f870b3bc061a30ccdb (x86,mm: make pagefault killable) The above commits introduced changes into the x86 pagefault handler for making the page fault handler retryable as well as killable. These changes reduce the mmap_sem hold time, which is crucial during OOM killer invocation. Port these changes to the 32-bit SH platform. Signed-off-by: Kautuk Consul <consul.kautuk@gmail.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-03-31 20:06:11 +08:00
retry:
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
down_read(&mm->mmap_sem);
vma = find_vma(mm, address);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
if (unlikely(!vma)) {
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
bad_area(regs, error_code, address);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
return;
}
if (likely(vma->vm_start <= address))
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
goto good_area;
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
if (unlikely(!(vma->vm_flags & VM_GROWSDOWN))) {
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
bad_area(regs, error_code, address);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
return;
}
if (unlikely(expand_stack(vma, address))) {
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
bad_area(regs, error_code, address);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
return;
}
sh: Tidy up vmalloc fault handling. This rewrites the vmalloc fault handling as per x86, which subsequently allows for easy future tie-in for vmalloc_sync_all(). Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2009-07-05 02:18:47 +08:00
/*
* Ok, we have a good vm_area for this memory access, so
* we can handle it..
*/
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
good_area:
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
if (unlikely(access_error(error_code, vma))) {
bad_area_access_error(regs, error_code, address);
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
return;
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
}
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
set_thread_fault_code(error_code);
arch: mm: pass userspace fault flag to generic fault handler Unlike global OOM handling, memory cgroup code will invoke the OOM killer in any OOM situation because it has no way of telling faults occuring in kernel context - which could be handled more gracefully - from user-triggered faults. Pass a flag that identifies faults originating in user space from the architecture-specific fault handlers to generic code so that memcg OOM handling can be improved. Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> Reviewed-by: Michal Hocko <mhocko@suse.cz> Cc: David Rientjes <rientjes@google.com> Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> Cc: azurIt <azurit@pobox.sk> Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-09-13 06:13:39 +08:00
if (user_mode(regs))
flags |= FAULT_FLAG_USER;
if (error_code & FAULT_CODE_WRITE)
flags |= FAULT_FLAG_WRITE;
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
/*
* If for any reason at all we couldn't handle the fault,
* make sure we exit gracefully rather than endlessly redo
* the fault.
*/
mm: do not pass mm_struct into handle_mm_fault We always have vma->vm_mm around. Link: http://lkml.kernel.org/r/1466021202-61880-8-git-send-email-kirill.shutemov@linux.intel.com Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-07-27 06:25:18 +08:00
fault = handle_mm_fault(vma, address, flags);
sh/mm/fault_32.c: Port OOM changes to do_page_fault Commit d065bd810b6deb67d4897a14bfe21f8eb526ba99 (mm: retry page fault when blocking on disk transfer) and commit 37b23e0525d393d48a7d59f870b3bc061a30ccdb (x86,mm: make pagefault killable) The above commits introduced changes into the x86 pagefault handler for making the page fault handler retryable as well as killable. These changes reduce the mmap_sem hold time, which is crucial during OOM killer invocation. Port these changes to the 32-bit SH platform. Signed-off-by: Kautuk Consul <consul.kautuk@gmail.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-03-31 20:06:11 +08:00
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
if (unlikely(fault & (VM_FAULT_RETRY | VM_FAULT_ERROR)))
sh: Support thread fault code encoding. This provides a simple interface modelled after sparc64/m32r to encode the error code in the upper byte of thread_info for finer-grained handling in the page fault path. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 13:57:28 +08:00
if (mm_fault_error(regs, error_code, address, fault))
sh: Tidy up and generalize page fault error paths. This follows the x86 changes for tidying up the page fault error paths. We'll build on top of this for _32/_64 unification. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-05-14 09:27:34 +08:00
return;
sh/mm/fault_32.c: Port OOM changes to do_page_fault Commit d065bd810b6deb67d4897a14bfe21f8eb526ba99 (mm: retry page fault when blocking on disk transfer) and commit 37b23e0525d393d48a7d59f870b3bc061a30ccdb (x86,mm: make pagefault killable) The above commits introduced changes into the x86 pagefault handler for making the page fault handler retryable as well as killable. These changes reduce the mmap_sem hold time, which is crucial during OOM killer invocation. Port these changes to the 32-bit SH platform. Signed-off-by: Kautuk Consul <consul.kautuk@gmail.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-03-31 20:06:11 +08:00
if (flags & FAULT_FLAG_ALLOW_RETRY) {
if (fault & VM_FAULT_MAJOR) {
tsk->maj_flt++;
perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS_MAJ, 1,
regs, address);
} else {
tsk->min_flt++;
perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS_MIN, 1,
regs, address);
}
if (fault & VM_FAULT_RETRY) {
flags &= ~FAULT_FLAG_ALLOW_RETRY;
readahead: fault retry breaks mmap file read random detection .fault now can retry. The retry can break state machine of .fault. In filemap_fault, if page is miss, ra->mmap_miss is increased. In the second try, since the page is in page cache now, ra->mmap_miss is decreased. And these are done in one fault, so we can't detect random mmap file access. Add a new flag to indicate .fault is tried once. In the second try, skip ra->mmap_miss decreasing. The filemap_fault state machine is ok with it. I only tested x86, didn't test other archs, but looks the change for other archs is obvious, but who knows :) Signed-off-by: Shaohua Li <shaohua.li@fusionio.com> Cc: Rik van Riel <riel@redhat.com> Cc: Wu Fengguang <fengguang.wu@intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2012-10-09 07:32:19 +08:00
flags |= FAULT_FLAG_TRIED;
sh/mm/fault_32.c: Port OOM changes to do_page_fault Commit d065bd810b6deb67d4897a14bfe21f8eb526ba99 (mm: retry page fault when blocking on disk transfer) and commit 37b23e0525d393d48a7d59f870b3bc061a30ccdb (x86,mm: make pagefault killable) The above commits introduced changes into the x86 pagefault handler for making the page fault handler retryable as well as killable. These changes reduce the mmap_sem hold time, which is crucial during OOM killer invocation. Port these changes to the 32-bit SH platform. Signed-off-by: Kautuk Consul <consul.kautuk@gmail.com> Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2012-03-31 20:06:11 +08:00
/*
* No need to up_read(&mm->mmap_sem) as we would
* have already released it in __lock_page_or_retry
* in mm/filemap.c.
*/
goto retry;
}
sh: Hook up page fault events for software perf counters. This adds page fault instrumentation for the software performance counters. Follows the x86 and powerpc changes. Signed-off-by: Paul Mundt <lethal@linux-sh.org>
2009-06-25 01:30:10 +08:00
}
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
2005-04-17 06:20:36 +08:00
up_read(&mm->mmap_sem);
}