ASIDs have always been stored as unsigned longs, ie. 32 bits on MIPS32
kernels. This is problematic because it is feasible for the ASID version
to overflow & wrap around to zero.
We currently attempt to handle this overflow by simply setting the ASID
version to 1, using asid_first_version(), but we make no attempt to
account for the fact that there may be mm_structs with stale ASIDs that
have versions which we now reuse due to the overflow & wrap around.
Encountering this requires that:
1) A struct mm_struct X is active on CPU A using ASID (V,n).
2) That mm is not used on CPU A for the length of time that it takes
for CPU A's asid_cache to overflow & wrap around to the same
version V that the mm had in step 1. During this time tasks using
the mm could either be sleeping or only scheduled on other CPUs.
3) Some other mm Y becomes active on CPU A and is allocated the same
ASID (V,n).
4) mm X now becomes active on CPU A again, and now incorrectly has the
same ASID as mm Y.
Where struct mm_struct ASIDs are represented above in the format
(version, EntryHi.ASID), and on a typical MIPS32 system version will be
24 bits wide & EntryHi.ASID will be 8 bits wide.
The length of time required in step 2 is highly dependent upon the CPU &
workload, but for a hypothetical 2GHz CPU running a workload which
generates a new ASID every 10000 cycles this period is around 248 days.
Due to this long period of time & the fact that tasks need to be
scheduled in just the right (or wrong, depending upon your inclination)
way, this is obviously a difficult bug to encounter but it's entirely
possible as evidenced by reports.
In order to fix this, simply extend ASIDs to 64 bits even on MIPS32
builds. This will extend the period of time required for the
hypothetical system above to encounter the problem from 28 days to
around 3 trillion years, which feels safely outside of the realms of
possibility.
The cost of this is slightly more generated code in some commonly
executed paths, but this is pretty minimal:
| Code Size Gain | Percentage
-----------------------|----------------|-------------
decstation_defconfig | +270 | +0.00%
32r2el_defconfig | +652 | +0.01%
32r6el_defconfig | +1000 | +0.01%
I have been unable to measure any change in performance of the LMbench
lat_ctx or lat_proc tests resulting from the 64b ASIDs on either
32r2el_defconfig+interAptiv or 32r6el_defconfig+I6500 systems.
Signed-off-by: Paul Burton <paul.burton@mips.com>
Suggested-by: James Hogan <jhogan@kernel.org>
References: https://lore.kernel.org/linux-mips/80B78A8B8FEE6145A87579E8435D78C30205D5F3@fzex.ruijie.com.cn/
References: https://lore.kernel.org/linux-mips/1488684260-18867-1-git-send-email-jiwei.sun@windriver.com/
Cc: Jiwei Sun <jiwei.sun@windriver.com>
Cc: Yu Huabing <yhb@ruijie.com.cn>
Cc: stable@vger.kernel.org # 2.6.12+
Cc: linux-mips@vger.kernel.org
Introduce cpu_cluster() & cpu_set_cluster() accessor functions in the
same vein as cpu_core(), cpu_vpe_id() & their set variants. These will
be used in further patches to allow users to get or set a CPUs cluster
number.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/17012/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Up until now we have open-coded checks for whether CPUs are siblings,
with slight variations on whether we consider the package ID or not.
This will only get more complex when we introduce cluster support, so in
preparation for that this patch introduces a cpus_are_siblings()
function which can be used to check whether or not 2 CPUs are siblings
in a consistent manner.
By checking globalnumber with the VP ID masked out this also has the
neat side effect of being ready for multi-cluster systems already.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Acked-by: Rafael J. Wysocki <rjw@rjwysocki.net>
Acked-by: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/17011/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
This patch modifies the way we store core & VP IDs such that we store
them in a single 32 bit integer whose format matches that of the MIPSr6
GlobalNumber register. Whereas we have previously stored core & VP IDs
in separate fields, storing them in a single GlobalNumber-like field:
1) Reduces the size of struct cpuinfo_mips by 4 bytes, and will allow
it to not grow when cluster support is added.
2) Gives us a natural place to store cluster number, which matches up
with what the architecture provides.
3) Will be useful in the future as a parameter to the MIPSr6 GINVI
instruction to specify a target CPU whose icache that instruction
should operate on.
The cpu_set*() accessor functions are moved out of the asm/cpu-info.h
header in order to allow them to use the WARN_ON macro, which is
unusable in asm/cpu-info.h due to include ordering.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/17010/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
We currently have fields in struct cpuinfo_mips for the core & VP(E) ID
of a particular CPU, and various pieces of code directly access those
fields. This patch abstracts such access by introducing accessor
functions cpu_core(), cpu_set_core(), cpu_vpe_id() & cpu_set_vpe_id()
and having code that needs to access these values call those functions
rather than directly accessing the struct cpuinfo_mips fields. This
prepares us for changes to the way in which those values are stored in
later patches.
The cpu_vpe_id() function is introduced even though we already had a
cpu_vpe_id() macro for a couple of reasons:
1) It's more consistent with the core, and future cluster, accessors.
2) It ensures a sensible return type without explicit casts.
3) It's generally preferable to use functions rather than macros.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/17009/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
<linux/cache.h> already defines SMP_CACHE_BYTES as L1_CACHE_BYTES.
This change results in a build error in <asm/cpu-info.h> which directly
includes <asm/cache.h>. Fix this by including <linux/cache.h> instead.
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Add the main support for the MIPS Virtualization ASE (A.K.A. VZ) to MIPS
KVM. The bulk of this work is in vz.c, with various new state and
definitions elsewhere.
Enough is implemented to be able to run on a minimal VZ core. Further
patches will fill out support for guest features which are optional or
can be disabled.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Acked-by: Ralf Baechle <ralf@linux-mips.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: "Radim Krčmář" <rkrcmar@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: linux-mips@linux-mips.org
Cc: kvm@vger.kernel.org
Cc: linux-doc@vger.kernel.org
Add functions for MIPS VZ TLB management to tlb.c.
kvm_vz_host_tlb_inv() will be used for invalidating root TLB entries
after GPA page tables have been modified due to a KVM page fault. It
arranges for a root GPA mapping to be flushed from the TLB, using the
gpa_mm ASID or the current GuestID to do the probe.
kvm_vz_local_flush_roottlb_all_guests() and
kvm_vz_local_flush_guesttlb_all() flush all TLB entries in the
corresponding TLB for guest mappings (GPA->RPA for root TLB with
GuestID, and all entries for guest TLB). They will be used when starting
a new GuestID cycle, when VZ hardware is enabled/disabled, and also when
switching to a guest when the guest TLB contents may be stale or belong
to a different VM.
kvm_vz_guest_tlb_lookup() converts a guest virtual address to a guest
physical address using the guest TLB. This will be used to decode guest
virtual addresses which are sometimes provided by VZ hardware in
CP0_BadVAddr for certain exceptions when the guest physical address is
unavailable.
kvm_vz_save_guesttlb() and kvm_vz_load_guesttlb() will be used to
preserve wired guest VTLB entries while a guest isn't running.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Acked-by: Ralf Baechle <ralf@linux-mips.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: "Radim Krčmář" <rkrcmar@redhat.com>
Cc: linux-mips@linux-mips.org
Cc: kvm@vger.kernel.org
Add a few new cpu-features.h definitions for VZ sub-features, namely the
existence of the CP0_GuestCtl0Ext, CP0_GuestCtl1, and CP0_GuestCtl2
registers, and support for GuestID to dialias TLB entries belonging to
different guests.
Also add certain features present in the guest, with the naming scheme
cpu_guest_has_*. These are added separately to the main options bitfield
since they generally parallel similar features in the root context. A
few of these (FPU, MSA, watchpoints, perf counters, CP0_[X]ContextConfig
registers, MAAR registers, and probably others in future) can be
dynamically configured in the guest context, for which the
cpu_guest_has_dyn_* macros are added.
[ralf@linux-mips.org: Resolve merge conflict.]
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/13231/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Add support for extended ASIDs as determined by the Config4.AE bit.
Since the only supported CPUs known to implement this are Netlogic XLP
and MIPS I6400, select this variable ASID support based upon
CONFIG_CPU_XLP and CONFIG_CPU_MIPSR6.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Jayachandran C. <jchandra@broadcom.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Radim Krčmář <rkrcmar@redhat.com>
Cc: linux-mips@linux-mips.org
Cc: kvm@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/13211/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
In preparation for supporting variable ASID masks, retrieve ASID masks
using functions in asm/cpu-info.h which accept struct cpuinfo_mips. This
will allow those functions to determine the ASID mask based upon the CPU
in a later patch. This also allows for the r3k & r8k cases to be handled
in Kconfig, which is arguably cleaner than the previous #ifdefs.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Radim Krčmář <rkrcmar@redhat.com>
Cc: linux-mips@linux-mips.org
Cc: kvm@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/13210/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Introduce support for bringing up Virtual Processors in MIPSr6 systems
as CPUs, much like their VPE parallel from the now-deprecated MT ASE.
The existing mips_cps_boot_vpes function fits the MIPSr6 architecture
pretty well - it can now simply write the mask of running VPs to the
VC_RUN register, rather than looping through each & starting or stopping
as appropriate as is done for VPEs from the MT ASE. Thus the VP support
is in general an extension & simplification of the existing MT ASE VPE
(aka SMVP) support.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: Matt Redfearn <matt.redfearn@imgtec.com>
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Maciej W. Rozycki <macro@linux-mips.org>
Cc: Niklas Cassel <niklas.cassel@axis.com>
Cc: Ezequiel Garcia <ezequiel.garcia@imgtec.com>
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/12339/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
If cpu_name_string() is used in non-atomic context when preemption is
enabled, it can trigger a BUG such as this one:
BUG: using smp_processor_id() in preemptible [00000000] code: unaligned/156
caller is __show_regs+0x1e4/0x330
CPU: 2 PID: 156 Comm: unaligned Tainted: G W 4.3.0-00366-ga3592179816d-dirty #1501
Stack : ffffffff80900000 ffffffff8019bc18 000000000000005f ffffffff80a20000
0000000000000000 0000000000000009 ffffffff8019c0e0 ffffffff80835648
a8000000ff2bdec0 ffffffff80a1e628 000000000000009c 0000000000000002
ffffffff80840000 a8000000fff2ffb0 0000000000000020 ffffffff8020e43c
a8000000fff2fcf8 ffffffff80a20000 0000000000000000 ffffffff808f2607
ffffffff8082b138 ffffffff8019cd1c 0000000000000030 ffffffff8082b138
0000000000000002 000000000000009c 0000000000000000 0000000000000000
0000000000000000 a8000000fff2fc40 0000000000000000 ffffffff8044dbf4
0000000000000000 0000000000000000 0000000000000000 ffffffff8010c400
ffffffff80855bb0 ffffffff8010d008 0000000000000000 ffffffff8044dbf4
...
Call Trace:
[<ffffffff8010d008>] show_stack+0x90/0xb0
[<ffffffff8044dbf4>] dump_stack+0x84/0xe0
[<ffffffff8046d4ec>] check_preemption_disabled+0x10c/0x110
[<ffffffff8010c40c>] __show_regs+0x1e4/0x330
[<ffffffff8010d060>] show_registers+0x28/0xc0
[<ffffffff80110748>] do_ade+0xcc8/0xce0
[<ffffffff80105b84>] resume_userspace_check+0x0/0x10
This is possible because cpu_name_string() is used by __show_regs(),
which is used by both show_regs() and show_registers(). These two
functions are used by various exception handling functions, only some of
which ensure that interrupts or preemption is disabled.
However the following have interrupts explicitly enabled or not
explicitly disabled:
- do_reserved() (irqs enabled)
- do_ade() (irqs not disabled)
This can be hit by setting /sys/kernel/debug/mips/unaligned_action to 2,
and triggering an address error exception, e.g. an unaligned access or
access to kernel segment from user mode.
To fix the above cases, use raw_smp_processor_id() instead. It is
unusual for CPU names to be different in the same system, and even if
they were, its possible the process has migrated between the exception
of interest and the cpu_name_string() call anyway.
Signed-off-by: James Hogan <james.hogan@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/12212/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Define the central place the default FCSR value is set from, initialised
in `cpu_probe'. Determine the FCSR mask applied to values written to
the register with CTC1 in the full emulation mode and via ptrace(2),
according to the ISA level of processor hardware or the writability of
bits 31:18 if actual FPU hardware is used.
Software may rely on FCSR bits whose functions our emulator does not
implement, so it should not allow them to be set or software may get
confused. For ptrace(2) it's just sanity.
[ralf@linux-mips.org: Fixed double inclusion of <asm/current.h>.]
Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9711/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
activate_mm() and switch_mm() call get_new_mmu_context() which in turn
can enable the HTW before the entryhi is changed with the new ASID.
Since the latter will enable the HTW in local_flush_tlb_all(),
then there is a small timing window where the HTW is running with the
new ASID but with an old pgd since the TLBMISS_HANDLER_SETUP_PGD
hasn't assigned a new one yet. In order to prevent that, we introduce a
simple htw counter to avoid starting HTW accidentally due to nested
htw_{start,stop}() sequences. Moreover, since various IPI calls can
enforce TLB flushing operations on a different core, such an operation
may interrupt another htw_{stop,start} in progress leading inconsistent
updates of the htw_seq variable. In order to avoid that, we disable the
interrupts whenever we update that variable.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: <stable@vger.kernel.org> # 3.17+
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/9118/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Different cores use different CCA values to achieve write-combine
memory writes. For cores that do not support write-combine we
set the default value to CCA:2 (uncached, non-coherent) which is the
default value as set by the kernel.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/7402/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Long integers which are 4 bytes in MIPS32 can't hold new CPU
options anymore, so the type of the 'options' variable is changed
to unsigned long long which allows 32 more cpu options to be defined
for MIPS32
Also, re-arrange the 'options' struct member to avoid potential 4-byte
alignment gap in the middle of the struct.
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/7324/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
This patch is prepared for Loongson's NUMA support, it offer meaningful
sysfs files such as physical_package_id, core_id, core_siblings and
thread_siblings in /sys/devices/system/cpu/cpu?/topology.
Signed-off-by: Huacai Chen <chenhc@lemote.com>
Reviewed-by: Andreas Herrmann <andreas.herrmann@caviumnetworks.com>
Cc: John Crispin <john@phrozen.org>
Cc: Steven J. Hill <Steven.Hill@imgtec.com>
Cc: Aurelien Jarno <aurelien@aurel32.net>
Cc: linux-mips@linux-mips.org
Cc: Fuxin Zhang <zhangfx@lemote.com>
Cc: Zhangjin Wu <wuzhangjin@gmail.com>
Patchwork: https://patchwork.linux-mips.org/patch/7184/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
When used in a non-MT kernel, the cpu_vpe_id macro never made use of
its cpuinfo argument. It doesn't actually need to since it is returning
a constant 0. However not using the argument can lead to build failures
if the compiler then notices that a variable used as part of the
argument is unused. Prevent that problem by "using" the argument as far
as the compiler is concerned, whilst still returning 0 as before.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Reviewed-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/7280/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Nobody is maintaining SMTC anymore and there also seems to be no userbase.
Which is a pity - the SMTC technology primarily developed by Kevin D.
Kissell <kevink@paralogos.com> is an ingenious demonstration for the MT
ASE's power and elegance.
Based on Markos Chandras <Markos.Chandras@imgtec.com> patch
https://patchwork.linux-mips.org/patch/6719/ which while very similar did
no longer apply cleanly when I tried to merge it plus some additional
post-SMTC cleanup - SMTC was a feature as tricky to remove as it was to
merge once upon a time.
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
asid_cache must be unsigned long otherwise on 64 bit systems it will
become 0 if the value in get_new_mmu_context() reaches 0xffffffff and
in the end the assumption of ASID_FIRST_VERSION is not true anymore
thus leads to more dangerous things.
Initial patch by Yong Zhang <yong.zhang@windriver.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Reported-by: libin <huawei.libin@huawei.com>
And there are more CPUs or configurations that want to provide special
per-CPU information in /proc/cpuinfo. So I think there needs to be a
hook mechanism, such as a notifier.
This is a first cut only; I need to think about what sort of looking
the notifier needs to have. But I'd appreciate testing on MT hardware!
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Cc: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/6066/
This patch adds support for probing the MSAP bit within the Config3
register in order to detect the presence of the MSA ASE. Presence of the
ASE will be indicated in /proc/cpuinfo. The value of the MSA
implementation register will be displayed at boot to aid debugging and
verification of a correct setup, as is done for the FPU.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/6430/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
The vpe_id field of struct cpuinfo_mips is only present when one of
CONFIG_MIPS_MT_{SMP,SMTC} is enabled. That means that any code accessing
which may compile without MT is currently forced to use an #ifdef.
Instead this patch provides an accessor macro, #ifdef'd appropriately
to prevent further #ifdef's elsewhere.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/6646/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
The Fixed Page Size TLB (FTLB) is a set-associative dual entry TLB. Its
purpose is to reduce the number of TLB misses by increasing the effective
TLB size and keep the implementation complexity to minimum levels.
A supported core can have both VTLB and FTLB.
Reviewed-by: James Hogan <james.hogan@imgtec.com>
Reviewed-by: Paul Burton <paul.burton@imgtec.com>
Signed-off-by: Leonid Yegoshin <Leonid.Yegoshin@imgtec.com>
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Signed-off-by: John Crispin <blogic@openwrt.org>
Patchwork: http://patchwork.linux-mips.org/patch/6139/
boot_cpu_data is used the same as current_cpu_data but returns the CPU
data for CPU 0. This means it doesn't have to use smp_processor_id()
thus no need to disable preemption.
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Having received another series of whitespace patches I decided to do this
once and for all rather than dealing with this kind of patches trickling
in forever.
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Linux kernel 2.6.32 and later allocate address space from the top of the
kernel virtual memory address space.
This patch implements virtual memory size detection for 64 bit MIPS CPUs
to avoid resulting crashes.
Signed-off-by: Guenter Roeck <guenter.roeck@ericsson.com>
Cc: linux-mips@linux-mips.org
Patchwork: http://patchwork.linux-mips.org/patch/935/
Reviewed-by: David Daney <ddaney@caviumnetworks.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Outlining fixes the issue were on certain CPUs such as the R10000 family
the delay loop would need an extra cycle if it overlaps a cacheline
boundary.
The rewrite also fixes build errors with GCC 4.4 which was changed in
way incompatible with the kernel's inline assembly.
Relying on pure C for computation of the delay value removes the need for
explicit. The price we pay is a slight slowdown of the computation - to
be fixed on another day.
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
This is the main support code for the patch. Here we just add the
code, the following patches hook it up.
Signed-off-by: David Daney <ddaney@avtrex.com>
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
create mode 100644 arch/mips/include/asm/watch.h
create mode 100644 arch/mips/kernel/watch.c
Paul Burton