UbiquitousOS
/
OpenCloudOS-Kernel
mirror of https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git
11
0
Fork 0
Code Issues Projects Releases Wiki Activity
OpenCloudOS-Kernel/arch/riscv/lib/uaccess.S

237 lines
5.1 KiB
ArmAsm
Raw Normal View History

RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
#include <linux/linkage.h>
riscv: fix compile failure with EXPORT_SYMBOL() & !MMU When support for !MMU was added, the declaration of __asm_copy_to_user() & __asm_copy_from_user() were #ifdefed out hence their EXPORT_SYMBOL() give an error message like: .../riscv_ksyms.c:13:15: error: '__asm_copy_to_user' undeclared here .../riscv_ksyms.c:14:15: error: '__asm_copy_from_user' undeclared here Since these symbols are not defined with !MMU it's wrong to export them. Same for __clear_user() (even though this one is also declared in include/asm-generic/uaccess.h and thus doesn't give an error message). Fix this by doing the EXPORT_SYMBOL() directly where these symbols are defined: inside lib/uaccess.S itself. Fixes: 6bd33e1ece52 ("riscv: fix compile failure with EXPORT_SYMBOL() & !MMU") Reported-by: kbuild test robot <lkp@intel.com> Cc: Christoph Hellwig <hch@lst.de> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Paul Walmsley <paul.walmsley@sifive.com> Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com> Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com>
2019-12-22 17:26:04 +08:00
#include <asm-generic/export.h>
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
#include <asm/asm.h>
#include <asm/csr.h>
.macro fixup op reg addr lbl
RISC-V: Stop using LOCAL for the uaccess fixups LLVM's integrated assembler doesn't support the LOCAL directive, which we're using when generating our uaccess fixup tables. Luckily the table fragment is small enough that there's only one internal symbol, so using a relative symbol reference doesn't really complicate anything. Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2020-02-28 03:16:28 +08:00
100:
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
\op \reg, \addr
.section __ex_table,"a"
.balign RISCV_SZPTR
RISC-V: Stop using LOCAL for the uaccess fixups LLVM's integrated assembler doesn't support the LOCAL directive, which we're using when generating our uaccess fixup tables. Luckily the table fragment is small enough that there's only one internal symbol, so using a relative symbol reference doesn't really complicate anything. Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2020-02-28 03:16:28 +08:00
RISCV_PTR 100b, \lbl
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
.previous
.endm
riscv: split the declaration of __copy_user We use a single __copy_user assembly function to copy memory both from and to userspace. While this works, it triggers sparse errors because we're implicitly casting between the kernel and user address spaces by calling __copy_user. This patch splits the C declaration into a pair of functions, __asm_copy_{to,from}_user, that have sane semantics WRT __user. This split make things fine from sparse's point of view. The assembly implementation keeps a single definition but add a double ENTRY() for it, one for __asm_copy_to_user and another one for __asm_copy_from_user. The result is a spare-safe implementation that pays no performance or code size penalty. Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2018-06-09 08:33:51 +08:00
ENTRY(__asm_copy_to_user)
ENTRY(__asm_copy_from_user)
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
/* Enable access to user memory */
li t6, SR_SUM
riscv: abstract out CSR names for supervisor vs machine mode Many of the privileged CSRs exist in a supervisor and machine version that are used very similarly. Provide versions of the CSR names and fields that map to either the S-mode or M-mode variant depending on a new CONFIG_RISCV_M_MODE kconfig symbol. Contains contributions from Damien Le Moal <Damien.LeMoal@wdc.com> and Paul Walmsley <paul.walmsley@sifive.com>. Signed-off-by: Christoph Hellwig <hch@lst.de> Acked-by: Thomas Gleixner <tglx@linutronix.de> # for drivers/clocksource, drivers/irqchip [paul.walmsley@sifive.com: updated to apply] Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com>
2019-10-28 20:10:32 +08:00
csrs CSR_STATUS, t6
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
riscv: __asm_copy_to-from_user: Optimize unaligned memory access and pipeline stall This patch will reduce cpu usage dramatically in kernel space especially for application which use sys-call with large buffer size, such as network applications. The main reason behind this is that every unaligned memory access will raise exceptions and switch between s-mode and m-mode causing large overhead. First copy in bytes until reaches the first word aligned boundary in destination memory address. This is the preparation before the bulk aligned word copy. The destination address is aligned now, but oftentimes the source address is not in an aligned boundary. To reduce the unaligned memory access, it reads the data from source in aligned boundaries, which will cause the data to have an offset, and then combines the data in the next iteration by fixing offset with shifting before writing to destination. The majority of the improving copy speed comes from this shift copy. In the lucky situation that the both source and destination address are on the aligned boundary, perform load and store with register size to copy the data. Without the unrolling, it will reduce the speed since the next store instruction for the same register using from the load will stall the pipeline. At last, copying the remainder in one byte at a time. Signed-off-by: Akira Tsukamoto <akira.tsukamoto@gmail.com> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-06-23 20:40:39 +08:00
/* Save for return value */
mv t5, a2
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
/*
riscv: __asm_copy_to-from_user: Optimize unaligned memory access and pipeline stall This patch will reduce cpu usage dramatically in kernel space especially for application which use sys-call with large buffer size, such as network applications. The main reason behind this is that every unaligned memory access will raise exceptions and switch between s-mode and m-mode causing large overhead. First copy in bytes until reaches the first word aligned boundary in destination memory address. This is the preparation before the bulk aligned word copy. The destination address is aligned now, but oftentimes the source address is not in an aligned boundary. To reduce the unaligned memory access, it reads the data from source in aligned boundaries, which will cause the data to have an offset, and then combines the data in the next iteration by fixing offset with shifting before writing to destination. The majority of the improving copy speed comes from this shift copy. In the lucky situation that the both source and destination address are on the aligned boundary, perform load and store with register size to copy the data. Without the unrolling, it will reduce the speed since the next store instruction for the same register using from the load will stall the pipeline. At last, copying the remainder in one byte at a time. Signed-off-by: Akira Tsukamoto <akira.tsukamoto@gmail.com> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-06-23 20:40:39 +08:00
* Register allocation for code below:
* a0 - start of uncopied dst
* a1 - start of uncopied src
* a2 - size
* t0 - end of uncopied dst
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
*/
riscv: __asm_copy_to-from_user: Optimize unaligned memory access and pipeline stall This patch will reduce cpu usage dramatically in kernel space especially for application which use sys-call with large buffer size, such as network applications. The main reason behind this is that every unaligned memory access will raise exceptions and switch between s-mode and m-mode causing large overhead. First copy in bytes until reaches the first word aligned boundary in destination memory address. This is the preparation before the bulk aligned word copy. The destination address is aligned now, but oftentimes the source address is not in an aligned boundary. To reduce the unaligned memory access, it reads the data from source in aligned boundaries, which will cause the data to have an offset, and then combines the data in the next iteration by fixing offset with shifting before writing to destination. The majority of the improving copy speed comes from this shift copy. In the lucky situation that the both source and destination address are on the aligned boundary, perform load and store with register size to copy the data. Without the unrolling, it will reduce the speed since the next store instruction for the same register using from the load will stall the pipeline. At last, copying the remainder in one byte at a time. Signed-off-by: Akira Tsukamoto <akira.tsukamoto@gmail.com> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-06-23 20:40:39 +08:00
add t0, a0, a2
/*
* Use byte copy only if too small.
*/
riscv: __asm_copy_to-from_user: Fix: overrun copy There were two causes for the overrun memory access. The threshold size was too small. The aligning dst require one SZREG and unrolling word copy requires 8*SZREG, total have to be at least 9*SZREG. Inside the unrolling copy, the subtracting -(8*SZREG-1) would make iteration happening one extra loop. Proper value is -(8*SZREG). Signed-off-by: Akira Tsukamoto <akira.tsukamoto@gmail.com> Fixes: ca6eaaa210de ("riscv: __asm_copy_to-from_user: Optimize unaligned memory access and pipeline stall") Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-07-20 16:50:52 +08:00
li a3, 9*SZREG /* size must be larger than size in word_copy */
riscv: __asm_copy_to-from_user: Optimize unaligned memory access and pipeline stall This patch will reduce cpu usage dramatically in kernel space especially for application which use sys-call with large buffer size, such as network applications. The main reason behind this is that every unaligned memory access will raise exceptions and switch between s-mode and m-mode causing large overhead. First copy in bytes until reaches the first word aligned boundary in destination memory address. This is the preparation before the bulk aligned word copy. The destination address is aligned now, but oftentimes the source address is not in an aligned boundary. To reduce the unaligned memory access, it reads the data from source in aligned boundaries, which will cause the data to have an offset, and then combines the data in the next iteration by fixing offset with shifting before writing to destination. The majority of the improving copy speed comes from this shift copy. In the lucky situation that the both source and destination address are on the aligned boundary, perform load and store with register size to copy the data. Without the unrolling, it will reduce the speed since the next store instruction for the same register using from the load will stall the pipeline. At last, copying the remainder in one byte at a time. Signed-off-by: Akira Tsukamoto <akira.tsukamoto@gmail.com> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-06-23 20:40:39 +08:00
bltu a2, a3, .Lbyte_copy_tail
/*
* Copy first bytes until dst is align to word boundary.
* a0 - start of dst
* t1 - start of aligned dst
*/
addi t1, a0, SZREG-1
andi t1, t1, ~(SZREG-1)
/* dst is already aligned, skip */
beq a0, t1, .Lskip_first_bytes
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
1:
riscv: __asm_copy_to-from_user: Optimize unaligned memory access and pipeline stall This patch will reduce cpu usage dramatically in kernel space especially for application which use sys-call with large buffer size, such as network applications. The main reason behind this is that every unaligned memory access will raise exceptions and switch between s-mode and m-mode causing large overhead. First copy in bytes until reaches the first word aligned boundary in destination memory address. This is the preparation before the bulk aligned word copy. The destination address is aligned now, but oftentimes the source address is not in an aligned boundary. To reduce the unaligned memory access, it reads the data from source in aligned boundaries, which will cause the data to have an offset, and then combines the data in the next iteration by fixing offset with shifting before writing to destination. The majority of the improving copy speed comes from this shift copy. In the lucky situation that the both source and destination address are on the aligned boundary, perform load and store with register size to copy the data. Without the unrolling, it will reduce the speed since the next store instruction for the same register using from the load will stall the pipeline. At last, copying the remainder in one byte at a time. Signed-off-by: Akira Tsukamoto <akira.tsukamoto@gmail.com> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-06-23 20:40:39 +08:00
/* a5 - one byte for copying data */
fixup lb a5, 0(a1), 10f
addi a1, a1, 1 /* src */
fixup sb a5, 0(a0), 10f
addi a0, a0, 1 /* dst */
bltu a0, t1, 1b /* t1 - start of aligned dst */
.Lskip_first_bytes:
/*
* Now dst is aligned.
* Use shift-copy if src is misaligned.
* Use word-copy if both src and dst are aligned because
* can not use shift-copy which do not require shifting
*/
/* a1 - start of src */
andi a3, a1, SZREG-1
bnez a3, .Lshift_copy
.Lword_copy:
/*
* Both src and dst are aligned, unrolled word copy
*
* a0 - start of aligned dst
* a1 - start of aligned src
* a3 - a1 & mask:(SZREG-1)
* t0 - end of aligned dst
*/
riscv: __asm_copy_to-from_user: Fix: overrun copy There were two causes for the overrun memory access. The threshold size was too small. The aligning dst require one SZREG and unrolling word copy requires 8*SZREG, total have to be at least 9*SZREG. Inside the unrolling copy, the subtracting -(8*SZREG-1) would make iteration happening one extra loop. Proper value is -(8*SZREG). Signed-off-by: Akira Tsukamoto <akira.tsukamoto@gmail.com> Fixes: ca6eaaa210de ("riscv: __asm_copy_to-from_user: Optimize unaligned memory access and pipeline stall") Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-07-20 16:50:52 +08:00
addi t0, t0, -(8*SZREG) /* not to over run */
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
2:
riscv: __asm_copy_to-from_user: Optimize unaligned memory access and pipeline stall This patch will reduce cpu usage dramatically in kernel space especially for application which use sys-call with large buffer size, such as network applications. The main reason behind this is that every unaligned memory access will raise exceptions and switch between s-mode and m-mode causing large overhead. First copy in bytes until reaches the first word aligned boundary in destination memory address. This is the preparation before the bulk aligned word copy. The destination address is aligned now, but oftentimes the source address is not in an aligned boundary. To reduce the unaligned memory access, it reads the data from source in aligned boundaries, which will cause the data to have an offset, and then combines the data in the next iteration by fixing offset with shifting before writing to destination. The majority of the improving copy speed comes from this shift copy. In the lucky situation that the both source and destination address are on the aligned boundary, perform load and store with register size to copy the data. Without the unrolling, it will reduce the speed since the next store instruction for the same register using from the load will stall the pipeline. At last, copying the remainder in one byte at a time. Signed-off-by: Akira Tsukamoto <akira.tsukamoto@gmail.com> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-06-23 20:40:39 +08:00
fixup REG_L a4, 0(a1), 10f
fixup REG_L a5, SZREG(a1), 10f
fixup REG_L a6, 2*SZREG(a1), 10f
fixup REG_L a7, 3*SZREG(a1), 10f
fixup REG_L t1, 4*SZREG(a1), 10f
fixup REG_L t2, 5*SZREG(a1), 10f
fixup REG_L t3, 6*SZREG(a1), 10f
fixup REG_L t4, 7*SZREG(a1), 10f
fixup REG_S a4, 0(a0), 10f
fixup REG_S a5, SZREG(a0), 10f
fixup REG_S a6, 2*SZREG(a0), 10f
fixup REG_S a7, 3*SZREG(a0), 10f
fixup REG_S t1, 4*SZREG(a0), 10f
fixup REG_S t2, 5*SZREG(a0), 10f
fixup REG_S t3, 6*SZREG(a0), 10f
fixup REG_S t4, 7*SZREG(a0), 10f
addi a0, a0, 8*SZREG
addi a1, a1, 8*SZREG
bltu a0, t0, 2b
riscv: __asm_copy_to-from_user: Fix: overrun copy There were two causes for the overrun memory access. The threshold size was too small. The aligning dst require one SZREG and unrolling word copy requires 8*SZREG, total have to be at least 9*SZREG. Inside the unrolling copy, the subtracting -(8*SZREG-1) would make iteration happening one extra loop. Proper value is -(8*SZREG). Signed-off-by: Akira Tsukamoto <akira.tsukamoto@gmail.com> Fixes: ca6eaaa210de ("riscv: __asm_copy_to-from_user: Optimize unaligned memory access and pipeline stall") Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-07-20 16:50:52 +08:00
addi t0, t0, 8*SZREG /* revert to original value */
riscv: __asm_copy_to-from_user: Optimize unaligned memory access and pipeline stall This patch will reduce cpu usage dramatically in kernel space especially for application which use sys-call with large buffer size, such as network applications. The main reason behind this is that every unaligned memory access will raise exceptions and switch between s-mode and m-mode causing large overhead. First copy in bytes until reaches the first word aligned boundary in destination memory address. This is the preparation before the bulk aligned word copy. The destination address is aligned now, but oftentimes the source address is not in an aligned boundary. To reduce the unaligned memory access, it reads the data from source in aligned boundaries, which will cause the data to have an offset, and then combines the data in the next iteration by fixing offset with shifting before writing to destination. The majority of the improving copy speed comes from this shift copy. In the lucky situation that the both source and destination address are on the aligned boundary, perform load and store with register size to copy the data. Without the unrolling, it will reduce the speed since the next store instruction for the same register using from the load will stall the pipeline. At last, copying the remainder in one byte at a time. Signed-off-by: Akira Tsukamoto <akira.tsukamoto@gmail.com> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-06-23 20:40:39 +08:00
j .Lbyte_copy_tail
.Lshift_copy:
/*
* Word copy with shifting.
* For misaligned copy we still perform aligned word copy, but
* we need to use the value fetched from the previous iteration and
* do some shifts.
* This is safe because reading less than a word size.
*
* a0 - start of aligned dst
* a1 - start of src
* a3 - a1 & mask:(SZREG-1)
* t0 - end of uncopied dst
* t1 - end of aligned dst
*/
/* calculating aligned word boundary for dst */
andi t1, t0, ~(SZREG-1)
/* Converting unaligned src to aligned arc */
andi a1, a1, ~(SZREG-1)
/*
* Calculate shifts
* t3 - prev shift
* t4 - current shift
*/
riscv: __asm_copy_to-from_user: Fix: fail on RV32 Had a bug when converting bytes to bits when the cpu was rv32. The a3 contains the number of bytes and multiple of 8 would be the bits. The LGREG is holding 2 for RV32 and 3 for RV32, so to achieve multiple of 8 it must always be constant 3. The 2 was mistakenly used for rv32. Signed-off-by: Akira Tsukamoto <akira.tsukamoto@gmail.com> Fixes: ca6eaaa210de ("riscv: __asm_copy_to-from_user: Optimize unaligned memory access and pipeline stall") Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-07-20 16:51:45 +08:00
slli t3, a3, 3 /* converting bytes in a3 to bits */
riscv: __asm_copy_to-from_user: Optimize unaligned memory access and pipeline stall This patch will reduce cpu usage dramatically in kernel space especially for application which use sys-call with large buffer size, such as network applications. The main reason behind this is that every unaligned memory access will raise exceptions and switch between s-mode and m-mode causing large overhead. First copy in bytes until reaches the first word aligned boundary in destination memory address. This is the preparation before the bulk aligned word copy. The destination address is aligned now, but oftentimes the source address is not in an aligned boundary. To reduce the unaligned memory access, it reads the data from source in aligned boundaries, which will cause the data to have an offset, and then combines the data in the next iteration by fixing offset with shifting before writing to destination. The majority of the improving copy speed comes from this shift copy. In the lucky situation that the both source and destination address are on the aligned boundary, perform load and store with register size to copy the data. Without the unrolling, it will reduce the speed since the next store instruction for the same register using from the load will stall the pipeline. At last, copying the remainder in one byte at a time. Signed-off-by: Akira Tsukamoto <akira.tsukamoto@gmail.com> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-06-23 20:40:39 +08:00
li a5, SZREG*8
sub t4, a5, t3
/* Load the first word to combine with seceond word */
fixup REG_L a5, 0(a1), 10f
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
3:
riscv: __asm_copy_to-from_user: Optimize unaligned memory access and pipeline stall This patch will reduce cpu usage dramatically in kernel space especially for application which use sys-call with large buffer size, such as network applications. The main reason behind this is that every unaligned memory access will raise exceptions and switch between s-mode and m-mode causing large overhead. First copy in bytes until reaches the first word aligned boundary in destination memory address. This is the preparation before the bulk aligned word copy. The destination address is aligned now, but oftentimes the source address is not in an aligned boundary. To reduce the unaligned memory access, it reads the data from source in aligned boundaries, which will cause the data to have an offset, and then combines the data in the next iteration by fixing offset with shifting before writing to destination. The majority of the improving copy speed comes from this shift copy. In the lucky situation that the both source and destination address are on the aligned boundary, perform load and store with register size to copy the data. Without the unrolling, it will reduce the speed since the next store instruction for the same register using from the load will stall the pipeline. At last, copying the remainder in one byte at a time. Signed-off-by: Akira Tsukamoto <akira.tsukamoto@gmail.com> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-06-23 20:40:39 +08:00
/* Main shifting copy
*
* a0 - start of aligned dst
* a1 - start of aligned src
* t1 - end of aligned dst
*/
/* At least one iteration will be executed */
srl a4, a5, t3
fixup REG_L a5, SZREG(a1), 10f
addi a1, a1, SZREG
sll a2, a5, t4
or a2, a2, a4
fixup REG_S a2, 0(a0), 10f
addi a0, a0, SZREG
bltu a0, t1, 3b
/* Revert src to original unaligned value */
add a1, a1, a3
.Lbyte_copy_tail:
/*
* Byte copy anything left.
*
* a0 - start of remaining dst
* a1 - start of remaining src
* t0 - end of remaining dst
*/
bgeu a0, t0, 5f
4:
fixup lb a5, 0(a1), 10f
addi a1, a1, 1 /* src */
fixup sb a5, 0(a0), 10f
addi a0, a0, 1 /* dst */
bltu a0, t0, 4b /* t0 - end of dst */
5:
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
/* Disable access to user memory */
riscv: abstract out CSR names for supervisor vs machine mode Many of the privileged CSRs exist in a supervisor and machine version that are used very similarly. Provide versions of the CSR names and fields that map to either the S-mode or M-mode variant depending on a new CONFIG_RISCV_M_MODE kconfig symbol. Contains contributions from Damien Le Moal <Damien.LeMoal@wdc.com> and Paul Walmsley <paul.walmsley@sifive.com>. Signed-off-by: Christoph Hellwig <hch@lst.de> Acked-by: Thomas Gleixner <tglx@linutronix.de> # for drivers/clocksource, drivers/irqchip [paul.walmsley@sifive.com: updated to apply] Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com>
2019-10-28 20:10:32 +08:00
csrc CSR_STATUS, t6
riscv: __asm_copy_to-from_user: Optimize unaligned memory access and pipeline stall This patch will reduce cpu usage dramatically in kernel space especially for application which use sys-call with large buffer size, such as network applications. The main reason behind this is that every unaligned memory access will raise exceptions and switch between s-mode and m-mode causing large overhead. First copy in bytes until reaches the first word aligned boundary in destination memory address. This is the preparation before the bulk aligned word copy. The destination address is aligned now, but oftentimes the source address is not in an aligned boundary. To reduce the unaligned memory access, it reads the data from source in aligned boundaries, which will cause the data to have an offset, and then combines the data in the next iteration by fixing offset with shifting before writing to destination. The majority of the improving copy speed comes from this shift copy. In the lucky situation that the both source and destination address are on the aligned boundary, perform load and store with register size to copy the data. Without the unrolling, it will reduce the speed since the next store instruction for the same register using from the load will stall the pipeline. At last, copying the remainder in one byte at a time. Signed-off-by: Akira Tsukamoto <akira.tsukamoto@gmail.com> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-06-23 20:40:39 +08:00
li a0, 0
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
ret
riscv: split the declaration of __copy_user We use a single __copy_user assembly function to copy memory both from and to userspace. While this works, it triggers sparse errors because we're implicitly casting between the kernel and user address spaces by calling __copy_user. This patch splits the C declaration into a pair of functions, __asm_copy_{to,from}_user, that have sane semantics WRT __user. This split make things fine from sparse's point of view. The assembly implementation keeps a single definition but add a double ENTRY() for it, one for __asm_copy_to_user and another one for __asm_copy_from_user. The result is a spare-safe implementation that pays no performance or code size penalty. Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2018-06-09 08:33:51 +08:00
ENDPROC(__asm_copy_to_user)
ENDPROC(__asm_copy_from_user)
riscv: fix compile failure with EXPORT_SYMBOL() & !MMU When support for !MMU was added, the declaration of __asm_copy_to_user() & __asm_copy_from_user() were #ifdefed out hence their EXPORT_SYMBOL() give an error message like: .../riscv_ksyms.c:13:15: error: '__asm_copy_to_user' undeclared here .../riscv_ksyms.c:14:15: error: '__asm_copy_from_user' undeclared here Since these symbols are not defined with !MMU it's wrong to export them. Same for __clear_user() (even though this one is also declared in include/asm-generic/uaccess.h and thus doesn't give an error message). Fix this by doing the EXPORT_SYMBOL() directly where these symbols are defined: inside lib/uaccess.S itself. Fixes: 6bd33e1ece52 ("riscv: fix compile failure with EXPORT_SYMBOL() & !MMU") Reported-by: kbuild test robot <lkp@intel.com> Cc: Christoph Hellwig <hch@lst.de> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Paul Walmsley <paul.walmsley@sifive.com> Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com> Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com>
2019-12-22 17:26:04 +08:00
EXPORT_SYMBOL(__asm_copy_to_user)
EXPORT_SYMBOL(__asm_copy_from_user)
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
ENTRY(__clear_user)
/* Enable access to user memory */
li t6, SR_SUM
riscv: abstract out CSR names for supervisor vs machine mode Many of the privileged CSRs exist in a supervisor and machine version that are used very similarly. Provide versions of the CSR names and fields that map to either the S-mode or M-mode variant depending on a new CONFIG_RISCV_M_MODE kconfig symbol. Contains contributions from Damien Le Moal <Damien.LeMoal@wdc.com> and Paul Walmsley <paul.walmsley@sifive.com>. Signed-off-by: Christoph Hellwig <hch@lst.de> Acked-by: Thomas Gleixner <tglx@linutronix.de> # for drivers/clocksource, drivers/irqchip [paul.walmsley@sifive.com: updated to apply] Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com>
2019-10-28 20:10:32 +08:00
csrs CSR_STATUS, t6
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
add a3, a0, a1
addi t0, a0, SZREG-1
andi t1, a3, ~(SZREG-1)
andi t0, t0, ~(SZREG-1)
/*
* a3: terminal address of target region
* t0: lowest doubleword-aligned address in target region
* t1: highest doubleword-aligned address in target region
*/
bgeu t0, t1, 2f
bltu a0, t0, 4f
1:
riscv: Fix the bug in memory access fixup code A piece of fixup code is currently shared by __copy_user and __clear_user. It first disables the access to user-space memory and then returns the "n" argument, which represents #(bytes not processed). However,__copy_user's "n" is in register a2, while __clear_user's in a1, and thus it causes errors for programs like setdomainname02 testcase in LTP. This patch fixes this issue by separating their fixup code and returning the right value for the kernel to handle a relative fault properly. Signed-off-by: Alan Kao <alankao@andestech.com> Cc: Greentime Hu <greentime@andestech.com> Cc: Zong Li <zong@andestech.com> Cc: Vincent Chen <vincentc@andestech.com> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2018-05-08 10:59:33 +08:00
fixup REG_S, zero, (a0), 11f
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
addi a0, a0, SZREG
bltu a0, t1, 1b
2:
bltu a0, a3, 5f
3:
/* Disable access to user memory */
riscv: abstract out CSR names for supervisor vs machine mode Many of the privileged CSRs exist in a supervisor and machine version that are used very similarly. Provide versions of the CSR names and fields that map to either the S-mode or M-mode variant depending on a new CONFIG_RISCV_M_MODE kconfig symbol. Contains contributions from Damien Le Moal <Damien.LeMoal@wdc.com> and Paul Walmsley <paul.walmsley@sifive.com>. Signed-off-by: Christoph Hellwig <hch@lst.de> Acked-by: Thomas Gleixner <tglx@linutronix.de> # for drivers/clocksource, drivers/irqchip [paul.walmsley@sifive.com: updated to apply] Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com>
2019-10-28 20:10:32 +08:00
csrc CSR_STATUS, t6
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
li a0, 0
ret
4: /* Edge case: unalignment */
riscv: Fix the bug in memory access fixup code A piece of fixup code is currently shared by __copy_user and __clear_user. It first disables the access to user-space memory and then returns the "n" argument, which represents #(bytes not processed). However,__copy_user's "n" is in register a2, while __clear_user's in a1, and thus it causes errors for programs like setdomainname02 testcase in LTP. This patch fixes this issue by separating their fixup code and returning the right value for the kernel to handle a relative fault properly. Signed-off-by: Alan Kao <alankao@andestech.com> Cc: Greentime Hu <greentime@andestech.com> Cc: Zong Li <zong@andestech.com> Cc: Vincent Chen <vincentc@andestech.com> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2018-05-08 10:59:33 +08:00
fixup sb, zero, (a0), 11f
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
addi a0, a0, 1
bltu a0, t0, 4b
j 1b
5: /* Edge case: remainder */
riscv: Fix the bug in memory access fixup code A piece of fixup code is currently shared by __copy_user and __clear_user. It first disables the access to user-space memory and then returns the "n" argument, which represents #(bytes not processed). However,__copy_user's "n" is in register a2, while __clear_user's in a1, and thus it causes errors for programs like setdomainname02 testcase in LTP. This patch fixes this issue by separating their fixup code and returning the right value for the kernel to handle a relative fault properly. Signed-off-by: Alan Kao <alankao@andestech.com> Cc: Greentime Hu <greentime@andestech.com> Cc: Zong Li <zong@andestech.com> Cc: Vincent Chen <vincentc@andestech.com> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2018-05-08 10:59:33 +08:00
fixup sb, zero, (a0), 11f
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
addi a0, a0, 1
bltu a0, a3, 5b
j 3b
ENDPROC(__clear_user)
riscv: fix compile failure with EXPORT_SYMBOL() & !MMU When support for !MMU was added, the declaration of __asm_copy_to_user() & __asm_copy_from_user() were #ifdefed out hence their EXPORT_SYMBOL() give an error message like: .../riscv_ksyms.c:13:15: error: '__asm_copy_to_user' undeclared here .../riscv_ksyms.c:14:15: error: '__asm_copy_from_user' undeclared here Since these symbols are not defined with !MMU it's wrong to export them. Same for __clear_user() (even though this one is also declared in include/asm-generic/uaccess.h and thus doesn't give an error message). Fix this by doing the EXPORT_SYMBOL() directly where these symbols are defined: inside lib/uaccess.S itself. Fixes: 6bd33e1ece52 ("riscv: fix compile failure with EXPORT_SYMBOL() & !MMU") Reported-by: kbuild test robot <lkp@intel.com> Cc: Christoph Hellwig <hch@lst.de> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Paul Walmsley <paul.walmsley@sifive.com> Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com> Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com>
2019-12-22 17:26:04 +08:00
EXPORT_SYMBOL(__clear_user)
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
.section .fixup,"ax"
.balign 4
riscv: Fix the bug in memory access fixup code A piece of fixup code is currently shared by __copy_user and __clear_user. It first disables the access to user-space memory and then returns the "n" argument, which represents #(bytes not processed). However,__copy_user's "n" is in register a2, while __clear_user's in a1, and thus it causes errors for programs like setdomainname02 testcase in LTP. This patch fixes this issue by separating their fixup code and returning the right value for the kernel to handle a relative fault properly. Signed-off-by: Alan Kao <alankao@andestech.com> Cc: Greentime Hu <greentime@andestech.com> Cc: Zong Li <zong@andestech.com> Cc: Vincent Chen <vincentc@andestech.com> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2018-05-08 10:59:33 +08:00
/* Fixup code for __copy_user(10) and __clear_user(11) */
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
10:
/* Disable access to user memory */
riscv: abstract out CSR names for supervisor vs machine mode Many of the privileged CSRs exist in a supervisor and machine version that are used very similarly. Provide versions of the CSR names and fields that map to either the S-mode or M-mode variant depending on a new CONFIG_RISCV_M_MODE kconfig symbol. Contains contributions from Damien Le Moal <Damien.LeMoal@wdc.com> and Paul Walmsley <paul.walmsley@sifive.com>. Signed-off-by: Christoph Hellwig <hch@lst.de> Acked-by: Thomas Gleixner <tglx@linutronix.de> # for drivers/clocksource, drivers/irqchip [paul.walmsley@sifive.com: updated to apply] Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com>
2019-10-28 20:10:32 +08:00
csrs CSR_STATUS, t6
riscv: __asm_copy_to-from_user: Optimize unaligned memory access and pipeline stall This patch will reduce cpu usage dramatically in kernel space especially for application which use sys-call with large buffer size, such as network applications. The main reason behind this is that every unaligned memory access will raise exceptions and switch between s-mode and m-mode causing large overhead. First copy in bytes until reaches the first word aligned boundary in destination memory address. This is the preparation before the bulk aligned word copy. The destination address is aligned now, but oftentimes the source address is not in an aligned boundary. To reduce the unaligned memory access, it reads the data from source in aligned boundaries, which will cause the data to have an offset, and then combines the data in the next iteration by fixing offset with shifting before writing to destination. The majority of the improving copy speed comes from this shift copy. In the lucky situation that the both source and destination address are on the aligned boundary, perform load and store with register size to copy the data. Without the unrolling, it will reduce the speed since the next store instruction for the same register using from the load will stall the pipeline. At last, copying the remainder in one byte at a time. Signed-off-by: Akira Tsukamoto <akira.tsukamoto@gmail.com> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-06-23 20:40:39 +08:00
mv a0, t5
riscv: Fix the bug in memory access fixup code A piece of fixup code is currently shared by __copy_user and __clear_user. It first disables the access to user-space memory and then returns the "n" argument, which represents #(bytes not processed). However,__copy_user's "n" is in register a2, while __clear_user's in a1, and thus it causes errors for programs like setdomainname02 testcase in LTP. This patch fixes this issue by separating their fixup code and returning the right value for the kernel to handle a relative fault properly. Signed-off-by: Alan Kao <alankao@andestech.com> Cc: Greentime Hu <greentime@andestech.com> Cc: Zong Li <zong@andestech.com> Cc: Vincent Chen <vincentc@andestech.com> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2018-05-08 10:59:33 +08:00
ret
11:
riscv: abstract out CSR names for supervisor vs machine mode Many of the privileged CSRs exist in a supervisor and machine version that are used very similarly. Provide versions of the CSR names and fields that map to either the S-mode or M-mode variant depending on a new CONFIG_RISCV_M_MODE kconfig symbol. Contains contributions from Damien Le Moal <Damien.LeMoal@wdc.com> and Paul Walmsley <paul.walmsley@sifive.com>. Signed-off-by: Christoph Hellwig <hch@lst.de> Acked-by: Thomas Gleixner <tglx@linutronix.de> # for drivers/clocksource, drivers/irqchip [paul.walmsley@sifive.com: updated to apply] Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com>
2019-10-28 20:10:32 +08:00
csrs CSR_STATUS, t6
riscv: Fix the bug in memory access fixup code A piece of fixup code is currently shared by __copy_user and __clear_user. It first disables the access to user-space memory and then returns the "n" argument, which represents #(bytes not processed). However,__copy_user's "n" is in register a2, while __clear_user's in a1, and thus it causes errors for programs like setdomainname02 testcase in LTP. This patch fixes this issue by separating their fixup code and returning the right value for the kernel to handle a relative fault properly. Signed-off-by: Alan Kao <alankao@andestech.com> Cc: Greentime Hu <greentime@andestech.com> Cc: Zong Li <zong@andestech.com> Cc: Vincent Chen <vincentc@andestech.com> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2018-05-08 10:59:33 +08:00
mv a0, a1
RISC-V: Generic library routines and assembly This patch contains code that is more specific to the RISC-V ISA than it is to Linux. It contains string and math operations, C wrappers for various assembly instructions, stack walking code, and uaccess. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 09:03:19 +08:00
ret
.previous