UbiquitousOS
/
OpenCloudOS-Kernel
mirror of https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git
11
0
Fork 0
Code Issues Projects Releases Wiki Activity

Arm SMMU updates for 5.12 

- Support for MT8192 IOMMU from Mediatek
 
 - Arm v7s io-pgtable extensions for MT8192
 
 - Removal of TLBI_ON_MAP quirk
 
 - New Qualcomm compatible strings
 
 - Allow SVA without hardware broadcast TLB maintenance on SMMUv3
 
 - Virtualization Host Extension support for SMMUv3 (SVA)
 
 - Allow SMMUv3 PMU (perf) driver to be built independently from IOMMU
 
 - Misc cleanups
 -----BEGIN PGP SIGNATURE-----
 
 iQFEBAABCgAuFiEEPxTL6PPUbjXGY88ct6xw3ITBYzQFAmAYD/QQHHdpbGxAa2Vy
 bmVsLm9yZwAKCRC3rHDchMFjNLZjCACBK2bCmkVJfs39YeeMhHWVRyiiMpIuCvja
 7WuIqeXnbtAg19hkwPqns+XYeTTnoeo/aSZUS9z8DIDUqADMES+1UUMfVf2mfL3A
 VD1pGRtQezYS+rDOc5dL9m1CcJ2RRY5tfQEjK8Fnm5CItvylNXbURH1nfu1tMVW7
 fn1tMNKCTnY7FkABttKyR4a1XIyFX+fPdSrqd5PjQ7D6rK2ZYrorMf55WpmQEfFu
 3er4F3Xz8nJeTvswPUMNB3ibvcdICM9VcSwFHpElTq2dbHUDDz3uFQwyiXrutEmb
 Xz4UGabjS/ZEPaty6y7VxdYEbD4Q2dzii57jKKtV49N36iCvkSR2
 =5KXT
 -----END PGP SIGNATURE-----

Merge tag 'arm-smmu-updates' of git://git.kernel.org/pub/scm/linux/kernel/git/will/linux into arm/smmu

Arm SMMU updates for 5.12

- Support for MT8192 IOMMU from Mediatek

- Arm v7s io-pgtable extensions for MT8192

- Removal of TLBI_ON_MAP quirk

- New Qualcomm compatible strings

- Allow SVA without hardware broadcast TLB maintenance on SMMUv3

- Virtualization Host Extension support for SMMUv3 (SVA)

- Allow SMMUv3 PMU (perf) driver to be built independently from IOMMU

- Misc cleanups
This commit is contained in:
Joerg Roedel 2021-02-02 13:35:40 +01:00
parent 6ee1d745b7 7060377ce0
commit d1e3306ba8
27 changed files with 981 additions and 356 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
Documentation/devicetree/bindings/iommu/arm,smmu.yaml
View File

@ -34,9 +34,11 @@ properties:
items:
- enum:
- qcom,sc7180-smmu-500
- qcom,sc8180x-smmu-500
- qcom,sdm845-smmu-500
- qcom,sm8150-smmu-500
- qcom,sm8250-smmu-500
- qcom,sm8350-smmu-500
- const: arm,mmu-500
- description: Qcom Adreno GPUs implementing "arm,smmu-v2"
items:

105
Documentation/devicetree/bindings/iommu/mediatek,iommu.txt
View File

@ -1,105 +0,0 @@
* Mediatek IOMMU Architecture Implementation
Some Mediatek SOCs contain a Multimedia Memory Management Unit (M4U), and
this M4U have two generations of HW architecture. Generation one uses flat
pagetable, and only supports 4K size page mapping. Generation two uses the
ARM Short-Descriptor translation table format for address translation.
About the M4U Hardware Block Diagram, please check below:
EMI (External Memory Interface)
|
m4u (Multimedia Memory Management Unit)
|
+--------+
| |
gals0-rx gals1-rx (Global Async Local Sync rx)
| |
| |
gals0-tx gals1-tx (Global Async Local Sync tx)
| | Some SoCs may have GALS.
+--------+
|
SMI Common(Smart Multimedia Interface Common)
|
+----------------+-------
| |
| gals-rx There may be GALS in some larbs.
| |
| |
| gals-tx
| |
SMI larb0 SMI larb1 ... SoCs have several SMI local arbiter(larb).
(display) (vdec)
| |
| |
+-----+-----+ +----+----+
| | | | | |
| | |... | | | ... There are different ports in each larb.
| | | | | |
OVL0 RDMA0 WDMA0 MC PP VLD
As above, The Multimedia HW will go through SMI and M4U while it
access EMI. SMI is a bridge between m4u and the Multimedia HW. It contain
smi local arbiter and smi common. It will control whether the Multimedia
HW should go though the m4u for translation or bypass it and talk
directly with EMI. And also SMI help control the power domain and clocks for
each local arbiter.
Normally we specify a local arbiter(larb) for each multimedia HW
like display, video decode, and camera. And there are different ports
in each larb. Take a example, There are many ports like MC, PP, VLD in the
video decode local arbiter, all these ports are according to the video HW.
In some SoCs, there may be a GALS(Global Async Local Sync) module between
smi-common and m4u, and additional GALS module between smi-larb and
smi-common. GALS can been seen as a "asynchronous fifo" which could help
synchronize for the modules in different clock frequency.
Required properties:
- compatible : must be one of the following string:
"mediatek,mt2701-m4u" for mt2701 which uses generation one m4u HW.
"mediatek,mt2712-m4u" for mt2712 which uses generation two m4u HW.
"mediatek,mt6779-m4u" for mt6779 which uses generation two m4u HW.
"mediatek,mt7623-m4u", "mediatek,mt2701-m4u" for mt7623 which uses
generation one m4u HW.
"mediatek,mt8167-m4u" for mt8167 which uses generation two m4u HW.
"mediatek,mt8173-m4u" for mt8173 which uses generation two m4u HW.
"mediatek,mt8183-m4u" for mt8183 which uses generation two m4u HW.
- reg : m4u register base and size.
- interrupts : the interrupt of m4u.
- clocks : must contain one entry for each clock-names.
- clock-names : Only 1 optional clock:
- "bclk": the block clock of m4u.
Here is the list which require this "bclk":
- mt2701, mt2712, mt7623 and mt8173.
Note that m4u use the EMI clock which always has been enabled before kernel
if there is no this "bclk".
- mediatek,larbs : List of phandle to the local arbiters in the current Socs.
Refer to bindings/memory-controllers/mediatek,smi-larb.txt. It must sort
according to the local arbiter index, like larb0, larb1, larb2...
- iommu-cells : must be 1. This is the mtk_m4u_id according to the HW.
Specifies the mtk_m4u_id as defined in
dt-binding/memory/mt2701-larb-port.h for mt2701, mt7623
dt-binding/memory/mt2712-larb-port.h for mt2712,
dt-binding/memory/mt6779-larb-port.h for mt6779,
dt-binding/memory/mt8167-larb-port.h for mt8167,
dt-binding/memory/mt8173-larb-port.h for mt8173, and
dt-binding/memory/mt8183-larb-port.h for mt8183.
Example:
iommu: iommu@10205000 {
compatible = "mediatek,mt8173-m4u";
reg = <0 0x10205000 0 0x1000>;
interrupts = <GIC_SPI 139 IRQ_TYPE_LEVEL_LOW>;
clocks = <&infracfg CLK_INFRA_M4U>;
clock-names = "bclk";
mediatek,larbs = <&larb0 &larb1 &larb2 &larb3 &larb4 &larb5>;
#iommu-cells = <1>;
};
Example for a client device:
display {
compatible = "mediatek,mt8173-disp";
iommus = <&iommu M4U_PORT_DISP_OVL0>,
<&iommu M4U_PORT_DISP_RDMA0>;
...
};

183
Documentation/devicetree/bindings/iommu/mediatek,iommu.yaml Normal file
View File

@ -0,0 +1,183 @@
# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
%YAML 1.2
---
$id: http://devicetree.org/schemas/iommu/mediatek,iommu.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#
title: MediaTek IOMMU Architecture Implementation
maintainers:
- Yong Wu <yong.wu@mediatek.com>
description: |+
Some MediaTek SOCs contain a Multimedia Memory Management Unit (M4U), and
this M4U have two generations of HW architecture. Generation one uses flat
pagetable, and only supports 4K size page mapping. Generation two uses the
ARM Short-Descriptor translation table format for address translation.
About the M4U Hardware Block Diagram, please check below:
EMI (External Memory Interface)
|
m4u (Multimedia Memory Management Unit)
|
+--------+
| |
gals0-rx gals1-rx (Global Async Local Sync rx)
| |
| |
gals0-tx gals1-tx (Global Async Local Sync tx)
| | Some SoCs may have GALS.
+--------+
|
SMI Common(Smart Multimedia Interface Common)
|
+----------------+-------
| |
| gals-rx There may be GALS in some larbs.
| |
| |
| gals-tx
| |
SMI larb0 SMI larb1 ... SoCs have several SMI local arbiter(larb).
(display) (vdec)
| |
| |
+-----+-----+ +----+----+
| | | | | |
| | |... | | | ... There are different ports in each larb.
| | | | | |
OVL0 RDMA0 WDMA0 MC PP VLD
As above, The Multimedia HW will go through SMI and M4U while it
access EMI. SMI is a bridge between m4u and the Multimedia HW. It contain
smi local arbiter and smi common. It will control whether the Multimedia
HW should go though the m4u for translation or bypass it and talk
directly with EMI. And also SMI help control the power domain and clocks for
each local arbiter.
Normally we specify a local arbiter(larb) for each multimedia HW
like display, video decode, and camera. And there are different ports
in each larb. Take a example, There are many ports like MC, PP, VLD in the
video decode local arbiter, all these ports are according to the video HW.
In some SoCs, there may be a GALS(Global Async Local Sync) module between
smi-common and m4u, and additional GALS module between smi-larb and
smi-common. GALS can been seen as a "asynchronous fifo" which could help
synchronize for the modules in different clock frequency.
properties:
compatible:
oneOf:
- enum:
- mediatek,mt2701-m4u # generation one
- mediatek,mt2712-m4u # generation two
- mediatek,mt6779-m4u # generation two
- mediatek,mt8167-m4u # generation two
- mediatek,mt8173-m4u # generation two
- mediatek,mt8183-m4u # generation two
- mediatek,mt8192-m4u # generation two
- description: mt7623 generation one
items:
- const: mediatek,mt7623-m4u
- const: mediatek,mt2701-m4u
reg:
maxItems: 1
interrupts:
maxItems: 1
clocks:
items:
- description: bclk is the block clock.
clock-names:
items:
- const: bclk
mediatek,larbs:
$ref: /schemas/types.yaml#/definitions/phandle-array
minItems: 1
maxItems: 32
description: |
List of phandle to the local arbiters in the current Socs.
Refer to bindings/memory-controllers/mediatek,smi-larb.yaml. It must sort
according to the local arbiter index, like larb0, larb1, larb2...
'#iommu-cells':
const: 1
description: |
This is the mtk_m4u_id according to the HW. Specifies the mtk_m4u_id as
defined in
dt-binding/memory/mt2701-larb-port.h for mt2701 and mt7623,
dt-binding/memory/mt2712-larb-port.h for mt2712,
dt-binding/memory/mt6779-larb-port.h for mt6779,
dt-binding/memory/mt8167-larb-port.h for mt8167,
dt-binding/memory/mt8173-larb-port.h for mt8173,
dt-binding/memory/mt8183-larb-port.h for mt8183,
dt-binding/memory/mt8192-larb-port.h for mt8192.
power-domains:
maxItems: 1
required:
- compatible
- reg
- interrupts
- mediatek,larbs
- '#iommu-cells'
allOf:
- if:
properties:
compatible:
contains:
enum:
- mediatek,mt2701-m4u
- mediatek,mt2712-m4u
- mediatek,mt8173-m4u
- mediatek,mt8192-m4u
then:
required:
- clocks
- if:
properties:
compatible:
enum:
- mediatek,mt8192-m4u
then:
required:
- power-domains
additionalProperties: false
examples:
- |
#include <dt-bindings/clock/mt8173-clk.h>
#include <dt-bindings/interrupt-controller/arm-gic.h>
iommu: iommu@10205000 {
compatible = "mediatek,mt8173-m4u";
reg = <0x10205000 0x1000>;
interrupts = <GIC_SPI 139 IRQ_TYPE_LEVEL_LOW>;
clocks = <&infracfg CLK_INFRA_M4U>;
clock-names = "bclk";
mediatek,larbs = <&larb0 &larb1 &larb2
&larb3 &larb4 &larb5>;
#iommu-cells = <1>;
};
- |
#include <dt-bindings/memory/mt8173-larb-port.h>
/* Example for a client device */
display {
compatible = "mediatek,mt8173-disp";
iommus = <&iommu M4U_PORT_DISP_OVL0>,
<&iommu M4U_PORT_DISP_RDMA0>;
};

9
MAINTAINERS
View File

@ -11178,6 +11178,15 @@ S: Maintained
F: Documentation/devicetree/bindings/i2c/i2c-mt65xx.txt
F: drivers/i2c/busses/i2c-mt65xx.c
MEDIATEK IOMMU DRIVER
M: Yong Wu <yong.wu@mediatek.com>
L: iommu@lists.linux-foundation.org
L: linux-mediatek@lists.infradead.org (moderated for non-subscribers)
S: Supported
F: Documentation/devicetree/bindings/iommu/mediatek*
F: drivers/iommu/mtk-iommu*
F: include/dt-bindings/memory/mt*-port.h
MEDIATEK JPEG DRIVER
M: Rick Chang <rick.chang@mediatek.com>
M: Bin Liu <bin.liu@mediatek.com>

10
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c
View File

@ -182,9 +182,13 @@ static void arm_smmu_mm_invalidate_range(struct mmu_notifier *mn,
unsigned long start, unsigned long end)
{
struct arm_smmu_mmu_notifier *smmu_mn = mn_to_smmu(mn);
struct arm_smmu_domain *smmu_domain = smmu_mn->domain;
size_t size = end - start + 1;
arm_smmu_atc_inv_domain(smmu_mn->domain, mm->pasid, start,
end - start + 1);
if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_BTM))
arm_smmu_tlb_inv_range_asid(start, size, smmu_mn->cd->asid,
PAGE_SIZE, false, smmu_domain);
arm_smmu_atc_inv_domain(smmu_domain, mm->pasid, start, size);
}
static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm)
@ -391,7 +395,7 @@ bool arm_smmu_sva_supported(struct arm_smmu_device *smmu)
unsigned long reg, fld;
unsigned long oas;
unsigned long asid_bits;
u32 feat_mask = ARM_SMMU_FEAT_BTM | ARM_SMMU_FEAT_COHERENCY;
u32 feat_mask = ARM_SMMU_FEAT_COHERENCY;
if (vabits_actual == 52)
feat_mask |= ARM_SMMU_FEAT_VAX;

150
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
View File

@ -88,15 +88,6 @@ static struct arm_smmu_option_prop arm_smmu_options[] = {
{ 0, NULL},
};
static inline void __iomem *arm_smmu_page1_fixup(unsigned long offset,
struct arm_smmu_device *smmu)
{
if (offset > SZ_64K)
return smmu->page1 + offset - SZ_64K;
return smmu->base + offset;
}
static void parse_driver_options(struct arm_smmu_device *smmu)
{
int i = 0;
@ -272,9 +263,11 @@ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent)
cmd[1] |= FIELD_PREP(CMDQ_CFGI_1_RANGE, 31);
break;
case CMDQ_OP_TLBI_NH_VA:
cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid);
fallthrough;
case CMDQ_OP_TLBI_EL2_VA:
cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_NUM, ent->tlbi.num);
cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_SCALE, ent->tlbi.scale);
cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid);
cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid);
cmd[1] |= FIELD_PREP(CMDQ_TLBI_1_LEAF, ent->tlbi.leaf);
cmd[1] |= FIELD_PREP(CMDQ_TLBI_1_TTL, ent->tlbi.ttl);
@ -296,6 +289,9 @@ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent)
case CMDQ_OP_TLBI_S12_VMALL:
cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid);
break;
case CMDQ_OP_TLBI_EL2_ASID:
cmd[0] |= FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid);
break;
case CMDQ_OP_ATC_INV:
cmd[0] |= FIELD_PREP(CMDQ_0_SSV, ent->substream_valid);
cmd[0] |= FIELD_PREP(CMDQ_ATC_0_GLOBAL, ent->atc.global);
@ -886,7 +882,8 @@ static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu,
void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid)
{
struct arm_smmu_cmdq_ent cmd = {
.opcode = CMDQ_OP_TLBI_NH_ASID,
.opcode = smmu->features & ARM_SMMU_FEAT_E2H ?
CMDQ_OP_TLBI_EL2_ASID : CMDQ_OP_TLBI_NH_ASID,
.tlbi.asid = asid,
};
@ -1269,13 +1266,16 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid,
}
if (s1_cfg) {
u64 strw = smmu->features & ARM_SMMU_FEAT_E2H ?
STRTAB_STE_1_STRW_EL2 : STRTAB_STE_1_STRW_NSEL1;
BUG_ON(ste_live);
dst[1] = cpu_to_le64(
FIELD_PREP(STRTAB_STE_1_S1DSS, STRTAB_STE_1_S1DSS_SSID0) |
FIELD_PREP(STRTAB_STE_1_S1CIR, STRTAB_STE_1_S1C_CACHE_WBRA) |
FIELD_PREP(STRTAB_STE_1_S1COR, STRTAB_STE_1_S1C_CACHE_WBRA) |
FIELD_PREP(STRTAB_STE_1_S1CSH, ARM_SMMU_SH_ISH) |
FIELD_PREP(STRTAB_STE_1_STRW, STRTAB_STE_1_STRW_NSEL1));
FIELD_PREP(STRTAB_STE_1_STRW, strw));
if (smmu->features & ARM_SMMU_FEAT_STALLS &&
!(smmu->features & ARM_SMMU_FEAT_STALL_FORCE))
@ -1667,40 +1667,28 @@ static void arm_smmu_tlb_inv_context(void *cookie)
arm_smmu_atc_inv_domain(smmu_domain, 0, 0, 0);
}
static void arm_smmu_tlb_inv_range(unsigned long iova, size_t size,
size_t granule, bool leaf,
static void __arm_smmu_tlb_inv_range(struct arm_smmu_cmdq_ent *cmd,
unsigned long iova, size_t size,
size_t granule,
struct arm_smmu_domain *smmu_domain)
{
struct arm_smmu_device *smmu = smmu_domain->smmu;
unsigned long start = iova, end = iova + size, num_pages = 0, tg = 0;
unsigned long end = iova + size, num_pages = 0, tg = 0;
size_t inv_range = granule;
struct arm_smmu_cmdq_batch cmds = {};
struct arm_smmu_cmdq_ent cmd = {
.tlbi = {
.leaf = leaf,
},
};
if (!size)
return;
if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
cmd.opcode = CMDQ_OP_TLBI_NH_VA;
cmd.tlbi.asid = smmu_domain->s1_cfg.cd.asid;
} else {
cmd.opcode = CMDQ_OP_TLBI_S2_IPA;
cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid;
}
if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) {
/* Get the leaf page size */
tg = __ffs(smmu_domain->domain.pgsize_bitmap);
/* Convert page size of 12,14,16 (log2) to 1,2,3 */
cmd.tlbi.tg = (tg - 10) / 2;
cmd->tlbi.tg = (tg - 10) / 2;
/* Determine what level the granule is at */
cmd.tlbi.ttl = 4 - ((ilog2(granule) - 3) / (tg - 3));
cmd->tlbi.ttl = 4 - ((ilog2(granule) - 3) / (tg - 3));
num_pages = size >> tg;
}
@ -1718,11 +1706,11 @@ static void arm_smmu_tlb_inv_range(unsigned long iova, size_t size,
/* Determine the power of 2 multiple number of pages */
scale = __ffs(num_pages);
cmd.tlbi.scale = scale;
cmd->tlbi.scale = scale;
/* Determine how many chunks of 2^scale size we have */
num = (num_pages >> scale) & CMDQ_TLBI_RANGE_NUM_MAX;
cmd.tlbi.num = num - 1;
cmd->tlbi.num = num - 1;
/* range is num * 2^scale * pgsize */
inv_range = num << (scale + tg);
@ -1731,17 +1719,54 @@ static void arm_smmu_tlb_inv_range(unsigned long iova, size_t size,
num_pages -= num << scale;
}
cmd.tlbi.addr = iova;
arm_smmu_cmdq_batch_add(smmu, &cmds, &cmd);
cmd->tlbi.addr = iova;
arm_smmu_cmdq_batch_add(smmu, &cmds, cmd);
iova += inv_range;
}
arm_smmu_cmdq_batch_submit(smmu, &cmds);
}
static void arm_smmu_tlb_inv_range_domain(unsigned long iova, size_t size,
size_t granule, bool leaf,
struct arm_smmu_domain *smmu_domain)
{
struct arm_smmu_cmdq_ent cmd = {
.tlbi = {
.leaf = leaf,
},
};
if (smmu_domain->stage == ARM_SMMU_DOMAIN_S1) {
cmd.opcode = smmu_domain->smmu->features & ARM_SMMU_FEAT_E2H ?
CMDQ_OP_TLBI_EL2_VA : CMDQ_OP_TLBI_NH_VA;
cmd.tlbi.asid = smmu_domain->s1_cfg.cd.asid;
} else {
cmd.opcode = CMDQ_OP_TLBI_S2_IPA;
cmd.tlbi.vmid = smmu_domain->s2_cfg.vmid;
}
__arm_smmu_tlb_inv_range(&cmd, iova, size, granule, smmu_domain);
/*
* Unfortunately, this can't be leaf-only since we may have
* zapped an entire table.
*/
arm_smmu_atc_inv_domain(smmu_domain, 0, start, size);
arm_smmu_atc_inv_domain(smmu_domain, 0, iova, size);
}
void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
size_t granule, bool leaf,
struct arm_smmu_domain *smmu_domain)
{
struct arm_smmu_cmdq_ent cmd = {
.opcode = smmu_domain->smmu->features & ARM_SMMU_FEAT_E2H ?
CMDQ_OP_TLBI_EL2_VA : CMDQ_OP_TLBI_NH_VA,
.tlbi = {
.asid = asid,
.leaf = leaf,
},
};
__arm_smmu_tlb_inv_range(&cmd, iova, size, granule, smmu_domain);
}
static void arm_smmu_tlb_inv_page_nosync(struct iommu_iotlb_gather *gather,
@ -1757,7 +1782,7 @@ static void arm_smmu_tlb_inv_page_nosync(struct iommu_iotlb_gather *gather,
static void arm_smmu_tlb_inv_walk(unsigned long iova, size_t size,
size_t granule, void *cookie)
{
arm_smmu_tlb_inv_range(iova, size, granule, false, cookie);
arm_smmu_tlb_inv_range_domain(iova, size, granule, false, cookie);
}
static const struct iommu_flush_ops arm_smmu_flush_ops = {
@ -2280,7 +2305,8 @@ static void arm_smmu_iotlb_sync(struct iommu_domain *domain,
{
struct arm_smmu_domain *smmu_domain = to_smmu_domain(domain);
arm_smmu_tlb_inv_range(gather->start, gather->end - gather->start,
arm_smmu_tlb_inv_range_domain(gather->start,
gather->end - gather->start + 1,
gather->pgsize, true, smmu_domain);
}
@ -2611,6 +2637,7 @@ static struct iommu_ops arm_smmu_ops = {
/* Probing and initialisation functions */
static int arm_smmu_init_one_queue(struct arm_smmu_device *smmu,
struct arm_smmu_queue *q,
void __iomem *page,
unsigned long prod_off,
unsigned long cons_off,
size_t dwords, const char *name)
@ -2639,8 +2666,8 @@ static int arm_smmu_init_one_queue(struct arm_smmu_device *smmu,
1 << q->llq.max_n_shift, name);
}
q->prod_reg = arm_smmu_page1_fixup(prod_off, smmu);
q->cons_reg = arm_smmu_page1_fixup(cons_off, smmu);
q->prod_reg = page + prod_off;
q->cons_reg = page + cons_off;
q->ent_dwords = dwords;
q->q_base = Q_BASE_RWA;
@ -2684,9 +2711,9 @@ static int arm_smmu_init_queues(struct arm_smmu_device *smmu)
int ret;
/* cmdq */
ret = arm_smmu_init_one_queue(smmu, &smmu->cmdq.q, ARM_SMMU_CMDQ_PROD,
ARM_SMMU_CMDQ_CONS, CMDQ_ENT_DWORDS,
"cmdq");
ret = arm_smmu_init_one_queue(smmu, &smmu->cmdq.q, smmu->base,
ARM_SMMU_CMDQ_PROD, ARM_SMMU_CMDQ_CONS,
CMDQ_ENT_DWORDS, "cmdq");
if (ret)
return ret;
@ -2695,9 +2722,9 @@ static int arm_smmu_init_queues(struct arm_smmu_device *smmu)
return ret;
/* evtq */
ret = arm_smmu_init_one_queue(smmu, &smmu->evtq.q, ARM_SMMU_EVTQ_PROD,
ARM_SMMU_EVTQ_CONS, EVTQ_ENT_DWORDS,
"evtq");
ret = arm_smmu_init_one_queue(smmu, &smmu->evtq.q, smmu->page1,
ARM_SMMU_EVTQ_PROD, ARM_SMMU_EVTQ_CONS,
EVTQ_ENT_DWORDS, "evtq");
if (ret)
return ret;
@ -2705,9 +2732,9 @@ static int arm_smmu_init_queues(struct arm_smmu_device *smmu)
if (!(smmu->features & ARM_SMMU_FEAT_PRI))
return 0;
return arm_smmu_init_one_queue(smmu, &smmu->priq.q, ARM_SMMU_PRIQ_PROD,
ARM_SMMU_PRIQ_CONS, PRIQ_ENT_DWORDS,
"priq");
return arm_smmu_init_one_queue(smmu, &smmu->priq.q, smmu->page1,
ARM_SMMU_PRIQ_PROD, ARM_SMMU_PRIQ_CONS,
PRIQ_ENT_DWORDS, "priq");
}
static int arm_smmu_init_l1_strtab(struct arm_smmu_device *smmu)
@ -3060,7 +3087,11 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu, bool bypass)
writel_relaxed(reg, smmu->base + ARM_SMMU_CR1);
/* CR2 (random crap) */
reg = CR2_PTM | CR2_RECINVSID | CR2_E2H;
reg = CR2_PTM | CR2_RECINVSID;
if (smmu->features & ARM_SMMU_FEAT_E2H)
reg |= CR2_E2H;
writel_relaxed(reg, smmu->base + ARM_SMMU_CR2);
/* Stream table */
@ -3099,10 +3130,8 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu, bool bypass)
/* Event queue */
writeq_relaxed(smmu->evtq.q.q_base, smmu->base + ARM_SMMU_EVTQ_BASE);
writel_relaxed(smmu->evtq.q.llq.prod,
arm_smmu_page1_fixup(ARM_SMMU_EVTQ_PROD, smmu));
writel_relaxed(smmu->evtq.q.llq.cons,
arm_smmu_page1_fixup(ARM_SMMU_EVTQ_CONS, smmu));
writel_relaxed(smmu->evtq.q.llq.prod, smmu->page1 + ARM_SMMU_EVTQ_PROD);
writel_relaxed(smmu->evtq.q.llq.cons, smmu->page1 + ARM_SMMU_EVTQ_CONS);
enables |= CR0_EVTQEN;
ret = arm_smmu_write_reg_sync(smmu, enables, ARM_SMMU_CR0,
@ -3117,9 +3146,9 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu, bool bypass)
writeq_relaxed(smmu->priq.q.q_base,
smmu->base + ARM_SMMU_PRIQ_BASE);
writel_relaxed(smmu->priq.q.llq.prod,
arm_smmu_page1_fixup(ARM_SMMU_PRIQ_PROD, smmu));
smmu->page1 + ARM_SMMU_PRIQ_PROD);
writel_relaxed(smmu->priq.q.llq.cons,
arm_smmu_page1_fixup(ARM_SMMU_PRIQ_CONS, smmu));
smmu->page1 + ARM_SMMU_PRIQ_CONS);
enables |= CR0_PRIQEN;
ret = arm_smmu_write_reg_sync(smmu, enables, ARM_SMMU_CR0,
@ -3221,8 +3250,11 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
smmu->options |= ARM_SMMU_OPT_MSIPOLL;
}
if (reg & IDR0_HYP)
if (reg & IDR0_HYP) {
smmu->features |= ARM_SMMU_FEAT_HYP;
if (cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN))
smmu->features |= ARM_SMMU_FEAT_E2H;
}
/*
* The coherency feature as set by FW is used in preference to the ID
@ -3489,11 +3521,7 @@ err_reset_pci_ops: __maybe_unused;
static void __iomem *arm_smmu_ioremap(struct device *dev, resource_size_t start,
resource_size_t size)
{
struct resource res = {
.flags = IORESOURCE_MEM,
.start = start,
.end = start + size - 1,
};
struct resource res = DEFINE_RES_MEM(start, size);
return devm_ioremap_resource(dev, &res);
}

14
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h
View File

@ -139,15 +139,15 @@
#define ARM_SMMU_CMDQ_CONS 0x9c
#define ARM_SMMU_EVTQ_BASE 0xa0
#define ARM_SMMU_EVTQ_PROD 0x100a8
#define ARM_SMMU_EVTQ_CONS 0x100ac
#define ARM_SMMU_EVTQ_PROD 0xa8
#define ARM_SMMU_EVTQ_CONS 0xac
#define ARM_SMMU_EVTQ_IRQ_CFG0 0xb0
#define ARM_SMMU_EVTQ_IRQ_CFG1 0xb8
#define ARM_SMMU_EVTQ_IRQ_CFG2 0xbc
#define ARM_SMMU_PRIQ_BASE 0xc0
#define ARM_SMMU_PRIQ_PROD 0x100c8
#define ARM_SMMU_PRIQ_CONS 0x100cc
#define ARM_SMMU_PRIQ_PROD 0xc8
#define ARM_SMMU_PRIQ_CONS 0xcc
#define ARM_SMMU_PRIQ_IRQ_CFG0 0xd0
#define ARM_SMMU_PRIQ_IRQ_CFG1 0xd8
#define ARM_SMMU_PRIQ_IRQ_CFG2 0xdc
@ -430,6 +430,8 @@ struct arm_smmu_cmdq_ent {
#define CMDQ_OP_TLBI_NH_ASID 0x11
#define CMDQ_OP_TLBI_NH_VA 0x12
#define CMDQ_OP_TLBI_EL2_ALL 0x20
#define CMDQ_OP_TLBI_EL2_ASID 0x21
#define CMDQ_OP_TLBI_EL2_VA 0x22
#define CMDQ_OP_TLBI_S12_VMALL 0x28
#define CMDQ_OP_TLBI_S2_IPA 0x2a
#define CMDQ_OP_TLBI_NSNH_ALL 0x30
@ -604,6 +606,7 @@ struct arm_smmu_device {
#define ARM_SMMU_FEAT_RANGE_INV (1 << 15)
#define ARM_SMMU_FEAT_BTM (1 << 16)
#define ARM_SMMU_FEAT_SVA (1 << 17)
#define ARM_SMMU_FEAT_E2H (1 << 18)
u32 features;
#define ARM_SMMU_OPT_SKIP_PREFETCH (1 << 0)
@ -694,6 +697,9 @@ extern struct arm_smmu_ctx_desc quiet_cd;
int arm_smmu_write_ctx_desc(struct arm_smmu_domain *smmu_domain, int ssid,
struct arm_smmu_ctx_desc *cd);
void arm_smmu_tlb_inv_asid(struct arm_smmu_device *smmu, u16 asid);
void arm_smmu_tlb_inv_range_asid(unsigned long iova, size_t size, int asid,
size_t granule, bool leaf,
struct arm_smmu_domain *smmu_domain);
bool arm_smmu_free_asid(struct arm_smmu_ctx_desc *cd);
int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, int ssid,
unsigned long iova, size_t size);

5
drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
View File

@ -166,6 +166,7 @@ static const struct of_device_id qcom_smmu_client_of_match[] __maybe_unused = {
{ .compatible = "qcom,mdss" },
{ .compatible = "qcom,sc7180-mdss" },
{ .compatible = "qcom,sc7180-mss-pil" },
{ .compatible = "qcom,sc8180x-mdss" },
{ .compatible = "qcom,sdm845-mdss" },
{ .compatible = "qcom,sdm845-mss-pil" },
{ }
@ -206,6 +207,8 @@ static int qcom_smmu_cfg_probe(struct arm_smmu_device *smmu)
smr = arm_smmu_gr0_read(smmu, ARM_SMMU_GR0_SMR(i));
if (FIELD_GET(ARM_SMMU_SMR_VALID, smr)) {
/* Ignore valid bit for SMR mask extraction. */
smr &= ~ARM_SMMU_SMR_VALID;
smmu->smrs[i].id = FIELD_GET(ARM_SMMU_SMR_ID, smr);
smmu->smrs[i].mask = FIELD_GET(ARM_SMMU_SMR_MASK, smr);
smmu->smrs[i].valid = true;
@ -327,10 +330,12 @@ static struct arm_smmu_device *qcom_smmu_create(struct arm_smmu_device *smmu,
static const struct of_device_id __maybe_unused qcom_smmu_impl_of_match[] = {
{ .compatible = "qcom,msm8998-smmu-v2" },
{ .compatible = "qcom,sc7180-smmu-500" },
{ .compatible = "qcom,sc8180x-smmu-500" },
{ .compatible = "qcom,sdm630-smmu-v2" },
{ .compatible = "qcom,sdm845-smmu-500" },
{ .compatible = "qcom,sm8150-smmu-500" },
{ .compatible = "qcom,sm8250-smmu-500" },
{ .compatible = "qcom,sm8350-smmu-500" },
{ }
};

63
drivers/iommu/io-pgtable-arm-v7s.c
View File

@ -44,26 +44,25 @@
/*
* We have 32 bits total; 12 bits resolved at level 1, 8 bits at level 2,
* and 12 bits in a page. With some carefully-chosen coefficients we can
* hide the ugly inconsistencies behind these macros and at least let the
* rest of the code pretend to be somewhat sane.
* and 12 bits in a page.
* MediaTek extend 2 bits to reach 34bits, 14 bits at lvl1 and 8 bits at lvl2.
*/
#define ARM_V7S_ADDR_BITS 32
#define _ARM_V7S_LVL_BITS(lvl) (16 - (lvl) * 4)
#define ARM_V7S_LVL_SHIFT(lvl) (ARM_V7S_ADDR_BITS - (4 + 8 * (lvl)))
#define _ARM_V7S_LVL_BITS(lvl, cfg) ((lvl) == 1 ? ((cfg)->ias - 20) : 8)
#define ARM_V7S_LVL_SHIFT(lvl) ((lvl) == 1 ? 20 : 12)
#define ARM_V7S_TABLE_SHIFT 10
#define ARM_V7S_PTES_PER_LVL(lvl) (1 << _ARM_V7S_LVL_BITS(lvl))
#define ARM_V7S_TABLE_SIZE(lvl) \
(ARM_V7S_PTES_PER_LVL(lvl) * sizeof(arm_v7s_iopte))
#define ARM_V7S_PTES_PER_LVL(lvl, cfg) (1 << _ARM_V7S_LVL_BITS(lvl, cfg))
#define ARM_V7S_TABLE_SIZE(lvl, cfg) \
(ARM_V7S_PTES_PER_LVL(lvl, cfg) * sizeof(arm_v7s_iopte))
#define ARM_V7S_BLOCK_SIZE(lvl) (1UL << ARM_V7S_LVL_SHIFT(lvl))
#define ARM_V7S_LVL_MASK(lvl) ((u32)(~0U << ARM_V7S_LVL_SHIFT(lvl)))
#define ARM_V7S_TABLE_MASK ((u32)(~0U << ARM_V7S_TABLE_SHIFT))
#define _ARM_V7S_IDX_MASK(lvl) (ARM_V7S_PTES_PER_LVL(lvl) - 1)
#define ARM_V7S_LVL_IDX(addr, lvl) ({ \
#define _ARM_V7S_IDX_MASK(lvl, cfg) (ARM_V7S_PTES_PER_LVL(lvl, cfg) - 1)
#define ARM_V7S_LVL_IDX(addr, lvl, cfg) ({ \
int _l = lvl; \
((u32)(addr) >> ARM_V7S_LVL_SHIFT(_l)) & _ARM_V7S_IDX_MASK(_l); \
((addr) >> ARM_V7S_LVL_SHIFT(_l)) & _ARM_V7S_IDX_MASK(_l, cfg); \
})
/*
@ -112,9 +111,10 @@
#define ARM_V7S_TEX_MASK 0x7
#define ARM_V7S_ATTR_TEX(val) (((val) & ARM_V7S_TEX_MASK) << ARM_V7S_TEX_SHIFT)
/* MediaTek extend the two bits for PA 32bit/33bit */
/* MediaTek extend the bits below for PA 32bit/33bit/34bit */
#define ARM_V7S_ATTR_MTK_PA_BIT32 BIT(9)
#define ARM_V7S_ATTR_MTK_PA_BIT33 BIT(4)
#define ARM_V7S_ATTR_MTK_PA_BIT34 BIT(5)
/* *well, except for TEX on level 2 large pages, of course :( */
#define ARM_V7S_CONT_PAGE_TEX_SHIFT 6
@ -194,6 +194,8 @@ static arm_v7s_iopte paddr_to_iopte(phys_addr_t paddr, int lvl,
pte |= ARM_V7S_ATTR_MTK_PA_BIT32;
if (paddr & BIT_ULL(33))
pte |= ARM_V7S_ATTR_MTK_PA_BIT33;
if (paddr & BIT_ULL(34))
pte |= ARM_V7S_ATTR_MTK_PA_BIT34;
return pte;
}
@ -218,6 +220,8 @@ static phys_addr_t iopte_to_paddr(arm_v7s_iopte pte, int lvl,
paddr |= BIT_ULL(32);
if (pte & ARM_V7S_ATTR_MTK_PA_BIT33)
paddr |= BIT_ULL(33);
if (pte & ARM_V7S_ATTR_MTK_PA_BIT34)
paddr |= BIT_ULL(34);
return paddr;
}
@ -234,7 +238,7 @@ static void *__arm_v7s_alloc_table(int lvl, gfp_t gfp,
struct device *dev = cfg->iommu_dev;
phys_addr_t phys;
dma_addr_t dma;
size_t size = ARM_V7S_TABLE_SIZE(lvl);
size_t size = ARM_V7S_TABLE_SIZE(lvl, cfg);
void *table = NULL;
if (lvl == 1)
@ -280,7 +284,7 @@ static void __arm_v7s_free_table(void *table, int lvl,
{
struct io_pgtable_cfg *cfg = &data->iop.cfg;
struct device *dev = cfg->iommu_dev;
size_t size = ARM_V7S_TABLE_SIZE(lvl);
size_t size = ARM_V7S_TABLE_SIZE(lvl, cfg);
if (!cfg->coherent_walk)
dma_unmap_single(dev, __arm_v7s_dma_addr(table), size,
@ -424,7 +428,7 @@ static int arm_v7s_init_pte(struct arm_v7s_io_pgtable *data,
arm_v7s_iopte *tblp;
size_t sz = ARM_V7S_BLOCK_SIZE(lvl);
tblp = ptep - ARM_V7S_LVL_IDX(iova, lvl);
tblp = ptep - ARM_V7S_LVL_IDX(iova, lvl, cfg);
if (WARN_ON(__arm_v7s_unmap(data, NULL, iova + i * sz,
sz, lvl, tblp) != sz))
return -EINVAL;
@ -477,7 +481,7 @@ static int __arm_v7s_map(struct arm_v7s_io_pgtable *data, unsigned long iova,
int num_entries = size >> ARM_V7S_LVL_SHIFT(lvl);
/* Find our entry at the current level */
ptep += ARM_V7S_LVL_IDX(iova, lvl);
ptep += ARM_V7S_LVL_IDX(iova, lvl, cfg);
/* If we can install a leaf entry at this level, then do so */
if (num_entries)
@ -519,7 +523,6 @@ static int arm_v7s_map(struct io_pgtable_ops *ops, unsigned long iova,
phys_addr_t paddr, size_t size, int prot, gfp_t gfp)
{
struct arm_v7s_io_pgtable *data = io_pgtable_ops_to_data(ops);
struct io_pgtable *iop = &data->iop;
int ret;
if (WARN_ON(iova >= (1ULL << data->iop.cfg.ias) ||
@ -535,12 +538,7 @@ static int arm_v7s_map(struct io_pgtable_ops *ops, unsigned long iova,
* Synchronise all PTE updates for the new mapping before there's
* a chance for anything to kick off a table walk for the new iova.
*/
if (iop->cfg.quirks & IO_PGTABLE_QUIRK_TLBI_ON_MAP) {
io_pgtable_tlb_flush_walk(iop, iova, size,
ARM_V7S_BLOCK_SIZE(2));
} else {
wmb();
}
return ret;
}
@ -550,7 +548,7 @@ static void arm_v7s_free_pgtable(struct io_pgtable *iop)
struct arm_v7s_io_pgtable *data = io_pgtable_to_data(iop);
int i;
for (i = 0; i < ARM_V7S_PTES_PER_LVL(1); i++) {
for (i = 0; i < ARM_V7S_PTES_PER_LVL(1, &data->iop.cfg); i++) {
arm_v7s_iopte pte = data->pgd[i];
if (ARM_V7S_PTE_IS_TABLE(pte, 1))
@ -602,9 +600,9 @@ static size_t arm_v7s_split_blk_unmap(struct arm_v7s_io_pgtable *data,
if (!tablep)
return 0; /* Bytes unmapped */
num_ptes = ARM_V7S_PTES_PER_LVL(2);
num_ptes = ARM_V7S_PTES_PER_LVL(2, cfg);
num_entries = size >> ARM_V7S_LVL_SHIFT(2);
unmap_idx = ARM_V7S_LVL_IDX(iova, 2);
unmap_idx = ARM_V7S_LVL_IDX(iova, 2, cfg);
pte = arm_v7s_prot_to_pte(arm_v7s_pte_to_prot(blk_pte, 1), 2, cfg);
if (num_entries > 1)
@ -646,7 +644,7 @@ static size_t __arm_v7s_unmap(struct arm_v7s_io_pgtable *data,
if (WARN_ON(lvl > 2))
return 0;
idx = ARM_V7S_LVL_IDX(iova, lvl);
idx = ARM_V7S_LVL_IDX(iova, lvl, &iop->cfg);
ptep += idx;
do {
pte[i] = READ_ONCE(ptep[i]);
@ -717,7 +715,7 @@ static size_t arm_v7s_unmap(struct io_pgtable_ops *ops, unsigned long iova,
{
struct arm_v7s_io_pgtable *data = io_pgtable_ops_to_data(ops);
if (WARN_ON(upper_32_bits(iova)))
if (WARN_ON(iova >= (1ULL << data->iop.cfg.ias)))
return 0;
return __arm_v7s_unmap(data, gather, iova, size, 1, data->pgd);
@ -732,7 +730,7 @@ static phys_addr_t arm_v7s_iova_to_phys(struct io_pgtable_ops *ops,
u32 mask;
do {
ptep += ARM_V7S_LVL_IDX(iova, ++lvl);
ptep += ARM_V7S_LVL_IDX(iova, ++lvl, &data->iop.cfg);
pte = READ_ONCE(*ptep);
ptep = iopte_deref(pte, lvl, data);
} while (ARM_V7S_PTE_IS_TABLE(pte, lvl));
@ -751,15 +749,14 @@ static struct io_pgtable *arm_v7s_alloc_pgtable(struct io_pgtable_cfg *cfg,
{
struct arm_v7s_io_pgtable *data;
if (cfg->ias > ARM_V7S_ADDR_BITS)
if (cfg->ias > (arm_v7s_is_mtk_enabled(cfg) ? 34 : ARM_V7S_ADDR_BITS))
return NULL;
if (cfg->oas > (arm_v7s_is_mtk_enabled(cfg) ? 34 : ARM_V7S_ADDR_BITS))
if (cfg->oas > (arm_v7s_is_mtk_enabled(cfg) ? 35 : ARM_V7S_ADDR_BITS))
return NULL;
if (cfg->quirks & ~(IO_PGTABLE_QUIRK_ARM_NS |
IO_PGTABLE_QUIRK_NO_PERMS |
IO_PGTABLE_QUIRK_TLBI_ON_MAP |
IO_PGTABLE_QUIRK_ARM_MTK_EXT |
IO_PGTABLE_QUIRK_NON_STRICT))
return NULL;
@ -775,8 +772,8 @@ static struct io_pgtable *arm_v7s_alloc_pgtable(struct io_pgtable_cfg *cfg,
spin_lock_init(&data->split_lock);
data->l2_tables = kmem_cache_create("io-pgtable_armv7s_l2",
ARM_V7S_TABLE_SIZE(2),
ARM_V7S_TABLE_SIZE(2),
ARM_V7S_TABLE_SIZE(2, cfg),
ARM_V7S_TABLE_SIZE(2, cfg),
ARM_V7S_TABLE_SLAB_FLAGS, NULL);
if (!data->l2_tables)
goto out_free_data;

23
drivers/iommu/iommu.c
View File

@ -2426,9 +2426,6 @@ static int __iommu_map(struct iommu_domain *domain, unsigned long iova,
size -= pgsize;
}
if (ops->iotlb_sync_map)
ops->iotlb_sync_map(domain);
/* unroll mapping in case something went wrong */
if (ret)
iommu_unmap(domain, orig_iova, orig_size - size);
@ -2438,18 +2435,31 @@ static int __iommu_map(struct iommu_domain *domain, unsigned long iova,
return ret;
}
static int _iommu_map(struct iommu_domain *domain, unsigned long iova,
phys_addr_t paddr, size_t size, int prot, gfp_t gfp)
{
const struct iommu_ops *ops = domain->ops;
int ret;
ret = __iommu_map(domain, iova, paddr, size, prot, GFP_KERNEL);
if (ret == 0 && ops->iotlb_sync_map)
ops->iotlb_sync_map(domain, iova, size);
return ret;
}
int iommu_map(struct iommu_domain *domain, unsigned long iova,
phys_addr_t paddr, size_t size, int prot)
{
might_sleep();
return __iommu_map(domain, iova, paddr, size, prot, GFP_KERNEL);
return _iommu_map(domain, iova, paddr, size, prot, GFP_KERNEL);
}
EXPORT_SYMBOL_GPL(iommu_map);
int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova,
phys_addr_t paddr, size_t size, int prot)
{
return __iommu_map(domain, iova, paddr, size, prot, GFP_ATOMIC);
return _iommu_map(domain, iova, paddr, size, prot, GFP_ATOMIC);
}
EXPORT_SYMBOL_GPL(iommu_map_atomic);
@ -2533,6 +2543,7 @@ static size_t __iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
struct scatterlist *sg, unsigned int nents, int prot,
gfp_t gfp)
{
const struct iommu_ops *ops = domain->ops;
size_t len = 0, mapped = 0;
phys_addr_t start;
unsigned int i = 0;
@ -2563,6 +2574,8 @@ static size_t __iommu_map_sg(struct iommu_domain *domain, unsigned long iova,
sg = sg_next(sg);
}
if (ops->iotlb_sync_map)
ops->iotlb_sync_map(domain, iova, mapped);
return mapped;
out_err:

10
drivers/iommu/msm_iommu.c
View File

@ -343,7 +343,6 @@ static int msm_iommu_domain_config(struct msm_priv *priv)
spin_lock_init(&priv->pgtlock);
priv->cfg = (struct io_pgtable_cfg) {
.quirks = IO_PGTABLE_QUIRK_TLBI_ON_MAP,
.pgsize_bitmap = msm_iommu_ops.pgsize_bitmap,
.ias = 32,
.oas = 32,
@ -490,6 +489,14 @@ static int msm_iommu_map(struct iommu_domain *domain, unsigned long iova,
return ret;
}
static void msm_iommu_sync_map(struct iommu_domain *domain, unsigned long iova,
size_t size)
{
struct msm_priv *priv = to_msm_priv(domain);
__flush_iotlb_range(iova, size, SZ_4K, false, priv);
}
static size_t msm_iommu_unmap(struct iommu_domain *domain, unsigned long iova,
size_t len, struct iommu_iotlb_gather *gather)
{
@ -680,6 +687,7 @@ static struct iommu_ops msm_iommu_ops = {
* kick starting the other master.
*/
.iotlb_sync = NULL,
.iotlb_sync_map = msm_iommu_sync_map,
.iova_to_phys = msm_iommu_iova_to_phys,
.probe_device = msm_iommu_probe_device,
.release_device = msm_iommu_release_device,

407
drivers/iommu/mtk_iommu.c
View File

@ -3,10 +3,12 @@
* Copyright (c) 2015-2016 MediaTek Inc.
* Author: Yong Wu <yong.wu@mediatek.com>
*/
#include <linux/bitfield.h>
#include <linux/bug.h>
#include <linux/clk.h>
#include <linux/component.h>
#include <linux/device.h>
#include <linux/dma-direct.h>
#include <linux/dma-iommu.h>
#include <linux/err.h>
#include <linux/interrupt.h>
@ -20,6 +22,7 @@
#include <linux/of_irq.h>
#include <linux/of_platform.h>
#include <linux/platform_device.h>
#include <linux/pm_runtime.h>
#include <linux/regmap.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
@ -88,6 +91,9 @@
#define F_REG_MMU1_FAULT_MASK GENMASK(13, 7)
#define REG_MMU0_FAULT_VA 0x13c
#define F_MMU_INVAL_VA_31_12_MASK GENMASK(31, 12)
#define F_MMU_INVAL_VA_34_32_MASK GENMASK(11, 9)
#define F_MMU_INVAL_PA_34_32_MASK GENMASK(8, 6)
#define F_MMU_FAULT_VA_WRITE_BIT BIT(1)
#define F_MMU_FAULT_VA_LAYER_BIT BIT(0)
@ -103,13 +109,6 @@
#define MTK_PROTECT_PA_ALIGN 256
/*
* Get the local arbiter ID and the portid within the larb arbiter
* from mtk_m4u_id which is defined by MTK_M4U_ID.
*/
#define MTK_M4U_TO_LARB(id) (((id) >> 5) & 0xf)
#define MTK_M4U_TO_PORT(id) ((id) & 0x1f)
#define HAS_4GB_MODE BIT(0)
/* HW will use the EMI clock if there isn't the "bclk". */
#define HAS_BCLK BIT(1)
@ -119,6 +118,7 @@
#define HAS_SUB_COMM BIT(5)
#define WR_THROT_EN BIT(6)
#define HAS_LEGACY_IVRP_PADDR BIT(7)
#define IOVA_34_EN BIT(8)
#define MTK_IOMMU_HAS_FLAG(pdata, _x) \
((((pdata)->flags) & (_x)) == (_x))
@ -127,11 +127,19 @@ struct mtk_iommu_domain {
struct io_pgtable_cfg cfg;
struct io_pgtable_ops *iop;
struct mtk_iommu_data *data;
struct iommu_domain domain;
};
static const struct iommu_ops mtk_iommu_ops;
static int mtk_iommu_hw_init(const struct mtk_iommu_data *data);
#define MTK_IOMMU_TLB_ADDR(iova) ({ \
dma_addr_t _addr = iova; \
((lower_32_bits(_addr) & GENMASK(31, 12)) | upper_32_bits(_addr));\
})
/*
* In M4U 4GB mode, the physical address is remapped as below:
*
@ -160,6 +168,25 @@ static LIST_HEAD(m4ulist); /* List all the M4U HWs */
#define for_each_m4u(data) list_for_each_entry(data, &m4ulist, list)
struct mtk_iommu_iova_region {
dma_addr_t iova_base;
unsigned long long size;
};
static const struct mtk_iommu_iova_region single_domain[] = {
{.iova_base = 0, .size = SZ_4G},
};
static const struct mtk_iommu_iova_region mt8192_multi_dom[] = {
{ .iova_base = 0x0, .size = SZ_4G}, /* disp: 0 ~ 4G */
#if IS_ENABLED(CONFIG_ARCH_DMA_ADDR_T_64BIT)
{ .iova_base = SZ_4G, .size = SZ_4G}, /* vdec: 4G ~ 8G */
{ .iova_base = SZ_4G * 2, .size = SZ_4G}, /* CAM/MDP: 8G ~ 12G */
{ .iova_base = 0x240000000ULL, .size = 0x4000000}, /* CCU0 */
{ .iova_base = 0x244000000ULL, .size = 0x4000000}, /* CCU1 */
#endif
};
/*
* There may be 1 or 2 M4U HWs, But we always expect they are in the same domain
* for the performance.
@ -182,33 +209,43 @@ static struct mtk_iommu_domain *to_mtk_domain(struct iommu_domain *dom)
return container_of(dom, struct mtk_iommu_domain, domain);
}
static void mtk_iommu_tlb_flush_all(void *cookie)
static void mtk_iommu_tlb_flush_all(struct mtk_iommu_data *data)
{
struct mtk_iommu_data *data = cookie;
for_each_m4u(data) {
if (pm_runtime_get_if_in_use(data->dev) <= 0)
continue;
writel_relaxed(F_INVLD_EN1 | F_INVLD_EN0,
data->base + data->plat_data->inv_sel_reg);
writel_relaxed(F_ALL_INVLD, data->base + REG_MMU_INVALIDATE);
wmb(); /* Make sure the tlb flush all done */
pm_runtime_put(data->dev);
}
}
static void mtk_iommu_tlb_flush_range_sync(unsigned long iova, size_t size,
size_t granule, void *cookie)
size_t granule,
struct mtk_iommu_data *data)
{
struct mtk_iommu_data *data = cookie;
bool has_pm = !!data->dev->pm_domain;
unsigned long flags;
int ret;
u32 tmp;
for_each_m4u(data) {
if (has_pm) {
if (pm_runtime_get_if_in_use(data->dev) <= 0)
continue;
}
spin_lock_irqsave(&data->tlb_lock, flags);
writel_relaxed(F_INVLD_EN1 | F_INVLD_EN0,
data->base + data->plat_data->inv_sel_reg);
writel_relaxed(iova, data->base + REG_MMU_INVLD_START_A);
writel_relaxed(iova + size - 1,
writel_relaxed(MTK_IOMMU_TLB_ADDR(iova),
data->base + REG_MMU_INVLD_START_A);
writel_relaxed(MTK_IOMMU_TLB_ADDR(iova + size - 1),
data->base + REG_MMU_INVLD_END_A);
writel_relaxed(F_MMU_INV_RANGE,
data->base + REG_MMU_INVALIDATE);
@ -219,36 +256,24 @@ static void mtk_iommu_tlb_flush_range_sync(unsigned long iova, size_t size,
if (ret) {
dev_warn(data->dev,
"Partial TLB flush timed out, falling back to full flush\n");
mtk_iommu_tlb_flush_all(cookie);
mtk_iommu_tlb_flush_all(data);
}
/* Clear the CPE status */
writel_relaxed(0, data->base + REG_MMU_CPE_DONE);
spin_unlock_irqrestore(&data->tlb_lock, flags);
if (has_pm)
pm_runtime_put(data->dev);
}
}
static void mtk_iommu_tlb_flush_page_nosync(struct iommu_iotlb_gather *gather,
unsigned long iova, size_t granule,
void *cookie)
{
struct mtk_iommu_data *data = cookie;
struct iommu_domain *domain = &data->m4u_dom->domain;
iommu_iotlb_gather_add_page(domain, gather, iova, granule);
}
static const struct iommu_flush_ops mtk_iommu_flush_ops = {
.tlb_flush_all = mtk_iommu_tlb_flush_all,
.tlb_flush_walk = mtk_iommu_tlb_flush_range_sync,
.tlb_add_page = mtk_iommu_tlb_flush_page_nosync,
};
static irqreturn_t mtk_iommu_isr(int irq, void *dev_id)
{
struct mtk_iommu_data *data = dev_id;
struct mtk_iommu_domain *dom = data->m4u_dom;
u32 int_state, regval, fault_iova, fault_pa;
unsigned int fault_larb, fault_port, sub_comm = 0;
u32 int_state, regval, va34_32, pa34_32;
u64 fault_iova, fault_pa;
bool layer, write;
/* Read error info from registers */
@ -264,6 +289,14 @@ static irqreturn_t mtk_iommu_isr(int irq, void *dev_id)
}
layer = fault_iova & F_MMU_FAULT_VA_LAYER_BIT;
write = fault_iova & F_MMU_FAULT_VA_WRITE_BIT;
if (MTK_IOMMU_HAS_FLAG(data->plat_data, IOVA_34_EN)) {
va34_32 = FIELD_GET(F_MMU_INVAL_VA_34_32_MASK, fault_iova);
pa34_32 = FIELD_GET(F_MMU_INVAL_PA_34_32_MASK, fault_iova);
fault_iova = fault_iova & F_MMU_INVAL_VA_31_12_MASK;
fault_iova |= (u64)va34_32 << 32;
fault_pa |= (u64)pa34_32 << 32;
}
fault_port = F_MMU_INT_ID_PORT_ID(regval);
if (MTK_IOMMU_HAS_FLAG(data->plat_data, HAS_SUB_COMM)) {
fault_larb = F_MMU_INT_ID_COMM_ID(regval);
@ -277,7 +310,7 @@ static irqreturn_t mtk_iommu_isr(int irq, void *dev_id)
write ? IOMMU_FAULT_WRITE : IOMMU_FAULT_READ)) {
dev_err_ratelimited(
data->dev,
"fault type=0x%x iova=0x%x pa=0x%x larb=%d port=%d layer=%d %s\n",
"fault type=0x%x iova=0x%llx pa=0x%llx larb=%d port=%d layer=%d %s\n",
int_state, fault_iova, fault_pa, fault_larb, fault_port,
layer, write ? "write" : "read");
}
@ -292,21 +325,57 @@ static irqreturn_t mtk_iommu_isr(int irq, void *dev_id)
return IRQ_HANDLED;
}
static void mtk_iommu_config(struct mtk_iommu_data *data,
struct device *dev, bool enable)
static int mtk_iommu_get_domain_id(struct device *dev,
const struct mtk_iommu_plat_data *plat_data)
{
const struct mtk_iommu_iova_region *rgn = plat_data->iova_region;
const struct bus_dma_region *dma_rgn = dev->dma_range_map;
int i, candidate = -1;
dma_addr_t dma_end;
if (!dma_rgn || plat_data->iova_region_nr == 1)
return 0;
dma_end = dma_rgn->dma_start + dma_rgn->size - 1;
for (i = 0; i < plat_data->iova_region_nr; i++, rgn++) {
/* Best fit. */
if (dma_rgn->dma_start == rgn->iova_base &&
dma_end == rgn->iova_base + rgn->size - 1)
return i;
/* ok if it is inside this region. */
if (dma_rgn->dma_start >= rgn->iova_base &&
dma_end < rgn->iova_base + rgn->size)
candidate = i;
}
if (candidate >= 0)
return candidate;
dev_err(dev, "Can NOT find the iommu domain id(%pad 0x%llx).\n",
&dma_rgn->dma_start, dma_rgn->size);
return -EINVAL;
}
static void mtk_iommu_config(struct mtk_iommu_data *data, struct device *dev,
bool enable, unsigned int domid)
{
struct mtk_smi_larb_iommu *larb_mmu;
unsigned int larbid, portid;
struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
const struct mtk_iommu_iova_region *region;
int i;
for (i = 0; i < fwspec->num_ids; ++i) {
larbid = MTK_M4U_TO_LARB(fwspec->ids[i]);
portid = MTK_M4U_TO_PORT(fwspec->ids[i]);
larb_mmu = &data->larb_imu[larbid];
dev_dbg(dev, "%s iommu port: %d\n",
enable ? "enable" : "disable", portid);
region = data->plat_data->iova_region + domid;
larb_mmu->bank[portid] = upper_32_bits(region->iova_base);
dev_dbg(dev, "%s iommu for larb(%s) port %d dom %d bank %d.\n",
enable ? "enable" : "disable", dev_name(larb_mmu->dev),
portid, domid, larb_mmu->bank[portid]);
if (enable)
larb_mmu->mmu |= MTK_SMI_MMU_EN(portid);
@ -315,22 +384,34 @@ static void mtk_iommu_config(struct mtk_iommu_data *data,
}
}
static int mtk_iommu_domain_finalise(struct mtk_iommu_domain *dom)
static int mtk_iommu_domain_finalise(struct mtk_iommu_domain *dom,
struct mtk_iommu_data *data,
unsigned int domid)
{
struct mtk_iommu_data *data = mtk_iommu_get_m4u_data();
const struct mtk_iommu_iova_region *region;
/* Use the exist domain as there is only one pgtable here. */
if (data->m4u_dom) {
dom->iop = data->m4u_dom->iop;
dom->cfg = data->m4u_dom->cfg;
dom->domain.pgsize_bitmap = data->m4u_dom->cfg.pgsize_bitmap;
goto update_iova_region;
}
dom->cfg = (struct io_pgtable_cfg) {
.quirks = IO_PGTABLE_QUIRK_ARM_NS |
IO_PGTABLE_QUIRK_NO_PERMS |
IO_PGTABLE_QUIRK_TLBI_ON_MAP |
IO_PGTABLE_QUIRK_ARM_MTK_EXT,
.pgsize_bitmap = mtk_iommu_ops.pgsize_bitmap,
.ias = 32,
.oas = 34,
.tlb = &mtk_iommu_flush_ops,
.ias = MTK_IOMMU_HAS_FLAG(data->plat_data, IOVA_34_EN) ? 34 : 32,
.iommu_dev = data->dev,
};
if (MTK_IOMMU_HAS_FLAG(data->plat_data, HAS_4GB_MODE))
dom->cfg.oas = data->enable_4GB ? 33 : 32;
else
dom->cfg.oas = 35;
dom->iop = alloc_io_pgtable_ops(ARM_V7S, &dom->cfg, data);
if (!dom->iop) {
dev_err(data->dev, "Failed to alloc io pgtable\n");
@ -339,6 +420,13 @@ static int mtk_iommu_domain_finalise(struct mtk_iommu_domain *dom)
/* Update our support page sizes bitmap */
dom->domain.pgsize_bitmap = dom->cfg.pgsize_bitmap;
update_iova_region:
/* Update the iova region for this domain */
region = data->plat_data->iova_region + domid;
dom->domain.geometry.aperture_start = region->iova_base;
dom->domain.geometry.aperture_end = region->iova_base + region->size - 1;
dom->domain.geometry.force_aperture = true;
return 0;
}
@ -353,30 +441,16 @@ static struct iommu_domain *mtk_iommu_domain_alloc(unsigned type)
if (!dom)
return NULL;
if (iommu_get_dma_cookie(&dom->domain))
goto free_dom;
if (mtk_iommu_domain_finalise(dom))
goto put_dma_cookie;
dom->domain.geometry.aperture_start = 0;
dom->domain.geometry.aperture_end = DMA_BIT_MASK(32);
dom->domain.geometry.force_aperture = true;
return &dom->domain;
put_dma_cookie:
iommu_put_dma_cookie(&dom->domain);
free_dom:
if (iommu_get_dma_cookie(&dom->domain)) {
kfree(dom);
return NULL;
}
return &dom->domain;
}
static void mtk_iommu_domain_free(struct iommu_domain *domain)
{
struct mtk_iommu_domain *dom = to_mtk_domain(domain);
free_io_pgtable_ops(dom->iop);
iommu_put_dma_cookie(domain);
kfree(to_mtk_domain(domain));
}
@ -386,18 +460,37 @@ static int mtk_iommu_attach_device(struct iommu_domain *domain,
{
struct mtk_iommu_data *data = dev_iommu_priv_get(dev);
struct mtk_iommu_domain *dom = to_mtk_domain(domain);
struct device *m4udev = data->dev;
int ret, domid;
if (!data)
domid = mtk_iommu_get_domain_id(dev, data->plat_data);
if (domid < 0)
return domid;
if (!dom->data) {
if (mtk_iommu_domain_finalise(dom, data, domid))
return -ENODEV;
dom->data = data;
}
/* Update the pgtable base address register of the M4U HW */
if (!data->m4u_dom) {
if (!data->m4u_dom) { /* Initialize the M4U HW */
ret = pm_runtime_resume_and_get(m4udev);
if (ret < 0)
return ret;
ret = mtk_iommu_hw_init(data);
if (ret) {
pm_runtime_put(m4udev);
return ret;
}
data->m4u_dom = dom;
writel(dom->cfg.arm_v7s_cfg.ttbr & MMU_PT_ADDR_MASK,
data->base + REG_MMU_PT_BASE_ADDR);
pm_runtime_put(m4udev);
}
mtk_iommu_config(data, dev, true);
mtk_iommu_config(data, dev, true, domid);
return 0;
}
@ -406,20 +499,16 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain,
{
struct mtk_iommu_data *data = dev_iommu_priv_get(dev);
if (!data)
return;
mtk_iommu_config(data, dev, false);
mtk_iommu_config(data, dev, false, 0);
}
static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova,
phys_addr_t paddr, size_t size, int prot, gfp_t gfp)
{
struct mtk_iommu_domain *dom = to_mtk_domain(domain);
struct mtk_iommu_data *data = mtk_iommu_get_m4u_data();
/* The "4GB mode" M4U physically can not use the lower remap of Dram. */
if (data->enable_4GB)
if (dom->data->enable_4GB)
paddr |= BIT_ULL(32);
/* Synchronize with the tlb_lock */
@ -431,37 +520,48 @@ static size_t mtk_iommu_unmap(struct iommu_domain *domain,
struct iommu_iotlb_gather *gather)
{
struct mtk_iommu_domain *dom = to_mtk_domain(domain);
unsigned long end = iova + size - 1;
if (gather->start > iova)
gather->start = iova;
if (gather->end < end)
gather->end = end;
return dom->iop->unmap(dom->iop, iova, size, gather);
}
static void mtk_iommu_flush_iotlb_all(struct iommu_domain *domain)
{
mtk_iommu_tlb_flush_all(mtk_iommu_get_m4u_data());
struct mtk_iommu_domain *dom = to_mtk_domain(domain);
mtk_iommu_tlb_flush_all(dom->data);
}
static void mtk_iommu_iotlb_sync(struct iommu_domain *domain,
struct iommu_iotlb_gather *gather)
{
struct mtk_iommu_data *data = mtk_iommu_get_m4u_data();
size_t length = gather->end - gather->start;
if (gather->start == ULONG_MAX)
return;
struct mtk_iommu_domain *dom = to_mtk_domain(domain);
size_t length = gather->end - gather->start + 1;
mtk_iommu_tlb_flush_range_sync(gather->start, length, gather->pgsize,
data);
dom->data);
}
static void mtk_iommu_sync_map(struct iommu_domain *domain, unsigned long iova,
size_t size)
{
struct mtk_iommu_domain *dom = to_mtk_domain(domain);
mtk_iommu_tlb_flush_range_sync(iova, size, size, dom->data);
}
static phys_addr_t mtk_iommu_iova_to_phys(struct iommu_domain *domain,
dma_addr_t iova)
{
struct mtk_iommu_domain *dom = to_mtk_domain(domain);
struct mtk_iommu_data *data = mtk_iommu_get_m4u_data();
phys_addr_t pa;
pa = dom->iop->iova_to_phys(dom->iop, iova);
if (data->enable_4GB && pa >= MTK_IOMMU_4GB_MODE_REMAP_BASE)
if (dom->data->enable_4GB && pa >= MTK_IOMMU_4GB_MODE_REMAP_BASE)
pa &= ~BIT_ULL(32);
return pa;
@ -493,19 +593,25 @@ static void mtk_iommu_release_device(struct device *dev)
static struct iommu_group *mtk_iommu_device_group(struct device *dev)
{
struct mtk_iommu_data *data = mtk_iommu_get_m4u_data();
struct iommu_group *group;
int domid;
if (!data)
return ERR_PTR(-ENODEV);
/* All the client devices are in the same m4u iommu-group */
if (!data->m4u_group) {
data->m4u_group = iommu_group_alloc();
if (IS_ERR(data->m4u_group))
dev_err(dev, "Failed to allocate M4U IOMMU group\n");
domid = mtk_iommu_get_domain_id(dev, data->plat_data);
if (domid < 0)
return ERR_PTR(domid);
group = data->m4u_group[domid];
if (!group) {
group = iommu_group_alloc();
if (!IS_ERR(group))
data->m4u_group[domid] = group;
} else {
iommu_group_ref_get(data->m4u_group);
iommu_group_ref_get(group);
}
return data->m4u_group;
return group;
}
static int mtk_iommu_of_xlate(struct device *dev, struct of_phandle_args *args)
@ -530,6 +636,35 @@ static int mtk_iommu_of_xlate(struct device *dev, struct of_phandle_args *args)
return iommu_fwspec_add_ids(dev, args->args, 1);
}
static void mtk_iommu_get_resv_regions(struct device *dev,
struct list_head *head)
{
struct mtk_iommu_data *data = dev_iommu_priv_get(dev);
unsigned int domid = mtk_iommu_get_domain_id(dev, data->plat_data), i;
const struct mtk_iommu_iova_region *resv, *curdom;
struct iommu_resv_region *region;
int prot = IOMMU_WRITE | IOMMU_READ;
if (domid < 0)
return;
curdom = data->plat_data->iova_region + domid;
for (i = 0; i < data->plat_data->iova_region_nr; i++) {
resv = data->plat_data->iova_region + i;
/* Only reserve when the region is inside the current domain */
if (resv->iova_base <= curdom->iova_base ||
resv->iova_base + resv->size >= curdom->iova_base + curdom->size)
continue;
region = iommu_alloc_resv_region(resv->iova_base, resv->size,
prot, IOMMU_RESV_RESERVED);
if (!region)
return;
list_add_tail(&region->list, head);
}
}
static const struct iommu_ops mtk_iommu_ops = {
.domain_alloc = mtk_iommu_domain_alloc,
.domain_free = mtk_iommu_domain_free,
@ -539,11 +674,14 @@ static const struct iommu_ops mtk_iommu_ops = {
.unmap = mtk_iommu_unmap,
.flush_iotlb_all = mtk_iommu_flush_iotlb_all,
.iotlb_sync = mtk_iommu_iotlb_sync,
.iotlb_sync_map = mtk_iommu_sync_map,
.iova_to_phys = mtk_iommu_iova_to_phys,
.probe_device = mtk_iommu_probe_device,
.release_device = mtk_iommu_release_device,
.device_group = mtk_iommu_device_group,
.of_xlate = mtk_iommu_of_xlate,
.get_resv_regions = mtk_iommu_get_resv_regions,
.put_resv_regions = generic_iommu_put_resv_regions,
.pgsize_bitmap = SZ_4K | SZ_64K | SZ_1M | SZ_16M,
};
@ -639,6 +777,9 @@ static int mtk_iommu_probe(struct platform_device *pdev)
{
struct mtk_iommu_data *data;
struct device *dev = &pdev->dev;
struct device_node *larbnode, *smicomm_node;
struct platform_device *plarbdev;
struct device_link *link;
struct resource *res;
resource_size_t ioaddr;
struct component_match *match = NULL;
@ -705,8 +846,6 @@ static int mtk_iommu_probe(struct platform_device *pdev)
return larb_nr;
for (i = 0; i < larb_nr; i++) {
struct device_node *larbnode;
struct platform_device *plarbdev;
u32 id;
larbnode = of_parse_phandle(dev->of_node, "mediatek,larbs", i);
@ -733,31 +872,64 @@ static int mtk_iommu_probe(struct platform_device *pdev)
compare_of, larbnode);
}
platform_set_drvdata(pdev, data);
/* Get smi-common dev from the last larb. */
smicomm_node = of_parse_phandle(larbnode, "mediatek,smi", 0);
if (!smicomm_node)
return -EINVAL;
ret = mtk_iommu_hw_init(data);
if (ret)
return ret;
plarbdev = of_find_device_by_node(smicomm_node);
of_node_put(smicomm_node);
data->smicomm_dev = &plarbdev->dev;
pm_runtime_enable(dev);
link = device_link_add(data->smicomm_dev, dev,
DL_FLAG_STATELESS | DL_FLAG_PM_RUNTIME);
if (!link) {
dev_err(dev, "Unable link %s.\n", dev_name(data->smicomm_dev));
goto out_runtime_disable;
}
platform_set_drvdata(pdev, data);
ret = iommu_device_sysfs_add(&data->iommu, dev, NULL,
"mtk-iommu.%pa", &ioaddr);
if (ret)
return ret;
goto out_link_remove;
iommu_device_set_ops(&data->iommu, &mtk_iommu_ops);
iommu_device_set_fwnode(&data->iommu, &pdev->dev.of_node->fwnode);
ret = iommu_device_register(&data->iommu);
if (ret)
return ret;
goto out_sysfs_remove;
spin_lock_init(&data->tlb_lock);
list_add_tail(&data->list, &m4ulist);
if (!iommu_present(&platform_bus_type))
bus_set_iommu(&platform_bus_type, &mtk_iommu_ops);
if (!iommu_present(&platform_bus_type)) {
ret = bus_set_iommu(&platform_bus_type, &mtk_iommu_ops);
if (ret)
goto out_list_del;
}
return component_master_add_with_match(dev, &mtk_iommu_com_ops, match);
ret = component_master_add_with_match(dev, &mtk_iommu_com_ops, match);
if (ret)
goto out_bus_set_null;
return ret;
out_bus_set_null:
bus_set_iommu(&platform_bus_type, NULL);
out_list_del:
list_del(&data->list);
iommu_device_unregister(&data->iommu);
out_sysfs_remove:
iommu_device_sysfs_remove(&data->iommu);
out_link_remove:
device_link_remove(data->smicomm_dev, dev);
out_runtime_disable:
pm_runtime_disable(dev);
return ret;
}
static int mtk_iommu_remove(struct platform_device *pdev)
@ -771,12 +943,14 @@ static int mtk_iommu_remove(struct platform_device *pdev)
bus_set_iommu(&platform_bus_type, NULL);
clk_disable_unprepare(data->bclk);
device_link_remove(data->smicomm_dev, &pdev->dev);
pm_runtime_disable(&pdev->dev);
devm_free_irq(&pdev->dev, data->irq, data);
component_master_del(&pdev->dev, &mtk_iommu_com_ops);
return 0;
}
static int __maybe_unused mtk_iommu_suspend(struct device *dev)
static int __maybe_unused mtk_iommu_runtime_suspend(struct device *dev)
{
struct mtk_iommu_data *data = dev_get_drvdata(dev);
struct mtk_iommu_suspend_reg *reg = &data->reg;
@ -794,7 +968,7 @@ static int __maybe_unused mtk_iommu_suspend(struct device *dev)
return 0;
}
static int __maybe_unused mtk_iommu_resume(struct device *dev)
static int __maybe_unused mtk_iommu_runtime_resume(struct device *dev)
{
struct mtk_iommu_data *data = dev_get_drvdata(dev);
struct mtk_iommu_suspend_reg *reg = &data->reg;
@ -802,6 +976,9 @@ static int __maybe_unused mtk_iommu_resume(struct device *dev)
void __iomem *base = data->base;
int ret;
/* Avoid first resume to affect the default value of registers below. */
if (!m4u_dom)
return 0;
ret = clk_prepare_enable(data->bclk);
if (ret) {
dev_err(data->dev, "Failed to enable clk(%d) in resume\n", ret);
@ -815,20 +992,22 @@ static int __maybe_unused mtk_iommu_resume(struct device *dev)
writel_relaxed(reg->int_main_control, base + REG_MMU_INT_MAIN_CONTROL);
writel_relaxed(reg->ivrp_paddr, base + REG_MMU_IVRP_PADDR);
writel_relaxed(reg->vld_pa_rng, base + REG_MMU_VLD_PA_RNG);
if (m4u_dom)
writel(m4u_dom->cfg.arm_v7s_cfg.ttbr & MMU_PT_ADDR_MASK,
base + REG_MMU_PT_BASE_ADDR);
writel(m4u_dom->cfg.arm_v7s_cfg.ttbr & MMU_PT_ADDR_MASK, base + REG_MMU_PT_BASE_ADDR);
return 0;
}
static const struct dev_pm_ops mtk_iommu_pm_ops = {
SET_NOIRQ_SYSTEM_SLEEP_PM_OPS(mtk_iommu_suspend, mtk_iommu_resume)
SET_RUNTIME_PM_OPS(mtk_iommu_runtime_suspend, mtk_iommu_runtime_resume, NULL)
SET_LATE_SYSTEM_SLEEP_PM_OPS(pm_runtime_force_suspend,
pm_runtime_force_resume)
};
static const struct mtk_iommu_plat_data mt2712_data = {
.m4u_plat = M4U_MT2712,
.flags = HAS_4GB_MODE | HAS_BCLK | HAS_VLD_PA_RNG,
.inv_sel_reg = REG_MMU_INV_SEL_GEN1,
.iova_region = single_domain,
.iova_region_nr = ARRAY_SIZE(single_domain),
.larbid_remap = {{0}, {1}, {2}, {3}, {4}, {5}, {6}, {7}},
};
@ -836,6 +1015,8 @@ static const struct mtk_iommu_plat_data mt6779_data = {
.m4u_plat = M4U_MT6779,
.flags = HAS_SUB_COMM | OUT_ORDER_WR_EN | WR_THROT_EN,
.inv_sel_reg = REG_MMU_INV_SEL_GEN2,
.iova_region = single_domain,
.iova_region_nr = ARRAY_SIZE(single_domain),
.larbid_remap = {{0}, {1}, {2}, {3}, {5}, {7, 8}, {10}, {9}},
};
@ -843,6 +1024,8 @@ static const struct mtk_iommu_plat_data mt8167_data = {
.m4u_plat = M4U_MT8167,
.flags = RESET_AXI | HAS_LEGACY_IVRP_PADDR,
.inv_sel_reg = REG_MMU_INV_SEL_GEN1,
.iova_region = single_domain,
.iova_region_nr = ARRAY_SIZE(single_domain),
.larbid_remap = {{0}, {1}, {2}}, /* Linear mapping. */
};
@ -851,6 +1034,8 @@ static const struct mtk_iommu_plat_data mt8173_data = {
.flags = HAS_4GB_MODE | HAS_BCLK | RESET_AXI |
HAS_LEGACY_IVRP_PADDR,
.inv_sel_reg = REG_MMU_INV_SEL_GEN1,
.iova_region = single_domain,
.iova_region_nr = ARRAY_SIZE(single_domain),
.larbid_remap = {{0}, {1}, {2}, {3}, {4}, {5}}, /* Linear mapping. */
};
@ -858,15 +1043,29 @@ static const struct mtk_iommu_plat_data mt8183_data = {
.m4u_plat = M4U_MT8183,
.flags = RESET_AXI,
.inv_sel_reg = REG_MMU_INV_SEL_GEN1,
.iova_region = single_domain,
.iova_region_nr = ARRAY_SIZE(single_domain),
.larbid_remap = {{0}, {4}, {5}, {6}, {7}, {2}, {3}, {1}},
};
static const struct mtk_iommu_plat_data mt8192_data = {
.m4u_plat = M4U_MT8192,
.flags = HAS_BCLK | HAS_SUB_COMM | OUT_ORDER_WR_EN |
WR_THROT_EN | IOVA_34_EN,
.inv_sel_reg = REG_MMU_INV_SEL_GEN2,
.iova_region = mt8192_multi_dom,
.iova_region_nr = ARRAY_SIZE(mt8192_multi_dom),
.larbid_remap = {{0}, {1}, {4, 5}, {7}, {2}, {9, 11, 19, 20},
{0, 14, 16}, {0, 13, 18, 17}},
};
static const struct of_device_id mtk_iommu_of_ids[] = {
{ .compatible = "mediatek,mt2712-m4u", .data = &mt2712_data},
{ .compatible = "mediatek,mt6779-m4u", .data = &mt6779_data},
{ .compatible = "mediatek,mt8167-m4u", .data = &mt8167_data},
{ .compatible = "mediatek,mt8173-m4u", .data = &mt8173_data},
{ .compatible = "mediatek,mt8183-m4u", .data = &mt8183_data},
{ .compatible = "mediatek,mt8192-m4u", .data = &mt8192_data},
{}
};

12
drivers/iommu/mtk_iommu.h
View File

@ -17,10 +17,13 @@
#include <linux/spinlock.h>
#include <linux/dma-mapping.h>
#include <soc/mediatek/smi.h>
#include <dt-bindings/memory/mtk-memory-port.h>
#define MTK_LARB_COM_MAX 8
#define MTK_LARB_SUBCOM_MAX 4
#define MTK_IOMMU_GROUP_MAX 8
struct mtk_iommu_suspend_reg {
union {
u32 standard_axi_mode;/* v1 */
@ -42,12 +45,18 @@ enum mtk_iommu_plat {
M4U_MT8167,
M4U_MT8173,
M4U_MT8183,
M4U_MT8192,
};
struct mtk_iommu_iova_region;
struct mtk_iommu_plat_data {
enum mtk_iommu_plat m4u_plat;
u32 flags;
u32 inv_sel_reg;
unsigned int iova_region_nr;
const struct mtk_iommu_iova_region *iova_region;
unsigned char larbid_remap[MTK_LARB_COM_MAX][MTK_LARB_SUBCOM_MAX];
};
@ -61,12 +70,13 @@ struct mtk_iommu_data {
phys_addr_t protect_base; /* protect memory base */
struct mtk_iommu_suspend_reg reg;
struct mtk_iommu_domain *m4u_dom;
struct iommu_group *m4u_group;
struct iommu_group *m4u_group[MTK_IOMMU_GROUP_MAX];
bool enable_4GB;
spinlock_t tlb_lock; /* lock for tlb range flush */
struct iommu_device iommu;
const struct mtk_iommu_plat_data *plat_data;
struct device *smicomm_dev;
struct dma_iommu_mapping *mapping; /* For mtk_iommu_v1.c */

7
drivers/iommu/tegra-gart.c
View File

@ -261,7 +261,8 @@ static int gart_iommu_of_xlate(struct device *dev,
return 0;
}
static void gart_iommu_sync_map(struct iommu_domain *domain)
static void gart_iommu_sync_map(struct iommu_domain *domain, unsigned long iova,
size_t size)
{
FLUSH_GART_REGS(gart_handle);
}
@ -269,7 +270,9 @@ static void gart_iommu_sync_map(struct iommu_domain *domain)
static void gart_iommu_sync(struct iommu_domain *domain,
struct iommu_iotlb_gather *gather)
{
gart_iommu_sync_map(domain);
size_t length = gather->end - gather->start + 1;
gart_iommu_sync_map(domain, gather->start, length);
}
static const struct iommu_ops gart_iommu_ops = {

8
drivers/memory/mtk-smi.c
View File

@ -15,6 +15,7 @@
#include <linux/pm_runtime.h>
#include <soc/mediatek/smi.h>
#include <dt-bindings/memory/mt2701-larb-port.h>
#include <dt-bindings/memory/mtk-memory-port.h>
/* mt8173 */
#define SMI_LARB_MMU_EN 0xf00
@ -43,6 +44,10 @@
/* mt2712 */
#define SMI_LARB_NONSEC_CON(id) (0x380 + ((id) * 4))
#define F_MMU_EN BIT(0)
#define BANK_SEL(id) ({ \
u32 _id = (id) & 0x3; \
(_id << 8 | _id << 10 | _id << 12 | _id << 14); \
})
/* SMI COMMON */
#define SMI_BUS_SEL 0x220
@ -87,6 +92,7 @@ struct mtk_smi_larb { /* larb: local arbiter */
const struct mtk_smi_larb_gen *larb_gen;
int larbid;
u32 *mmu;
unsigned char *bank;
};
static int mtk_smi_clk_enable(const struct mtk_smi *smi)
@ -153,6 +159,7 @@ mtk_smi_larb_bind(struct device *dev, struct device *master, void *data)
if (dev == larb_mmu[i].dev) {
larb->larbid = i;
larb->mmu = &larb_mmu[i].mmu;
larb->bank = larb_mmu[i].bank;
return 0;
}
}
@ -171,6 +178,7 @@ static void mtk_smi_larb_config_port_gen2_general(struct device *dev)
for_each_set_bit(i, (unsigned long *)larb->mmu, 32) {
reg = readl_relaxed(larb->base + SMI_LARB_NONSEC_CON(i));
reg |= F_MMU_EN;
reg |= BANK_SEL(larb->bank[i]);
writel(reg, larb->base + SMI_LARB_NONSEC_CON(i));
}
}

2
drivers/perf/Kconfig
View File

@ -62,7 +62,7 @@ config ARM_PMU_ACPI
config ARM_SMMU_V3_PMU
tristate "ARM SMMUv3 Performance Monitors Extension"
depends on ARM64 && ACPI && ARM_SMMU_V3
depends on ARM64 && ACPI
help
Provides support for the ARM SMMUv3 Performance Monitor Counter
Groups (PMCG), which provide monitoring of transactions passing

4
include/dt-bindings/memory/mt2701-larb-port.h
View File

@ -4,8 +4,8 @@
* Author: Honghui Zhang <honghui.zhang@mediatek.com>
*/
#ifndef _MT2701_LARB_PORT_H_
#define _MT2701_LARB_PORT_H_
#ifndef _DT_BINDINGS_MEMORY_MT2701_LARB_PORT_H_
#define _DT_BINDINGS_MEMORY_MT2701_LARB_PORT_H_
/*
* Mediatek m4u generation 1 such as mt2701 has flat m4u port numbers,

6
include/dt-bindings/memory/mt2712-larb-port.h
View File

@ -3,10 +3,10 @@
* Copyright (c) 2017 MediaTek Inc.
* Author: Yong Wu <yong.wu@mediatek.com>
*/
#ifndef __DTS_IOMMU_PORT_MT2712_H
#define __DTS_IOMMU_PORT_MT2712_H
#ifndef _DT_BINDINGS_MEMORY_MT2712_LARB_PORT_H_
#define _DT_BINDINGS_MEMORY_MT2712_LARB_PORT_H_
#define MTK_M4U_ID(larb, port) (((larb) << 5) | (port))
#include <dt-bindings/memory/mtk-memory-port.h>
#define M4U_LARB0_ID 0
#define M4U_LARB1_ID 1

6
include/dt-bindings/memory/mt6779-larb-port.h
View File

@ -4,10 +4,10 @@
* Author: Chao Hao <chao.hao@mediatek.com>
*/
#ifndef _DTS_IOMMU_PORT_MT6779_H_
#define _DTS_IOMMU_PORT_MT6779_H_
#ifndef _DT_BINDINGS_MEMORY_MT6779_LARB_PORT_H_
#define _DT_BINDINGS_MEMORY_MT6779_LARB_PORT_H_
#define MTK_M4U_ID(larb, port) (((larb) << 5) | (port))
#include <dt-bindings/memory/mtk-memory-port.h>
#define M4U_LARB0_ID 0
#define M4U_LARB1_ID 1

6
include/dt-bindings/memory/mt8167-larb-port.h
View File

@ -5,10 +5,10 @@
* Author: Honghui Zhang <honghui.zhang@mediatek.com>
* Author: Fabien Parent <fparent@baylibre.com>
*/
#ifndef __DTS_IOMMU_PORT_MT8167_H
#define __DTS_IOMMU_PORT_MT8167_H
#ifndef _DT_BINDINGS_MEMORY_MT8167_LARB_PORT_H_
#define _DT_BINDINGS_MEMORY_MT8167_LARB_PORT_H_
#define MTK_M4U_ID(larb, port) (((larb) << 5) | (port))
#include <dt-bindings/memory/mtk-memory-port.h>
#define M4U_LARB0_ID 0
#define M4U_LARB1_ID 1

6
include/dt-bindings/memory/mt8173-larb-port.h
View File

@ -3,10 +3,10 @@
* Copyright (c) 2015-2016 MediaTek Inc.
* Author: Yong Wu <yong.wu@mediatek.com>
*/
#ifndef __DTS_IOMMU_PORT_MT8173_H
#define __DTS_IOMMU_PORT_MT8173_H
#ifndef _DT_BINDINGS_MEMORY_MT8173_LARB_PORT_H_
#define _DT_BINDINGS_MEMORY_MT8173_LARB_PORT_H_
#define MTK_M4U_ID(larb, port) (((larb) << 5) | (port))
#include <dt-bindings/memory/mtk-memory-port.h>
#define M4U_LARB0_ID 0
#define M4U_LARB1_ID 1

6
include/dt-bindings/memory/mt8183-larb-port.h
View File

@ -3,10 +3,10 @@
* Copyright (c) 2018 MediaTek Inc.
* Author: Yong Wu <yong.wu@mediatek.com>
*/
#ifndef __DTS_IOMMU_PORT_MT8183_H
#define __DTS_IOMMU_PORT_MT8183_H
#ifndef _DT_BINDINGS_MEMORY_MT8183_LARB_PORT_H_
#define _DT_BINDINGS_MEMORY_MT8183_LARB_PORT_H_
#define MTK_M4U_ID(larb, port) (((larb) << 5) | (port))
#include <dt-bindings/memory/mtk-memory-port.h>
#define M4U_LARB0_ID 0
#define M4U_LARB1_ID 1

243
include/dt-bindings/memory/mt8192-larb-port.h Normal file
View File

@ -0,0 +1,243 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* Copyright (c) 2020 MediaTek Inc.
*
* Author: Chao Hao <chao.hao@mediatek.com>
* Author: Yong Wu <yong.wu@mediatek.com>
*/
#ifndef _DT_BINDINGS_MEMORY_MT8192_LARB_PORT_H_
#define _DT_BINDINGS_MEMORY_MT8192_LARB_PORT_H_
#include <dt-bindings/memory/mtk-memory-port.h>
/*
* MM IOMMU supports 16GB dma address.
*
* The address will preassign like this:
*
* modules dma-address-region larbs-ports
* disp 0 ~ 4G larb0/1
* vcodec 4G ~ 8G larb4/5/7
* cam/mdp 8G ~ 12G larb2/9/11/13/14/16/17/18/19/20
* CCU0 0x4000_0000 ~ 0x43ff_ffff larb13: port 9/10
* CCU1 0x4400_0000 ~ 0x47ff_ffff larb14: port 4/5
*
* larb3/6/8/10/12/15 is null.
*/
/* larb0 */
#define M4U_PORT_L0_DISP_POSTMASK0 MTK_M4U_ID(0, 0)
#define M4U_PORT_L0_OVL_RDMA0_HDR MTK_M4U_ID(0, 1)
#define M4U_PORT_L0_OVL_RDMA0 MTK_M4U_ID(0, 2)
#define M4U_PORT_L0_DISP_RDMA0 MTK_M4U_ID(0, 3)
#define M4U_PORT_L0_DISP_WDMA0 MTK_M4U_ID(0, 4)
#define M4U_PORT_L0_DISP_FAKE0 MTK_M4U_ID(0, 5)
/* larb1 */
#define M4U_PORT_L1_OVL_2L_RDMA0_HDR MTK_M4U_ID(1, 0)
#define M4U_PORT_L1_OVL_2L_RDMA2_HDR MTK_M4U_ID(1, 1)
#define M4U_PORT_L1_OVL_2L_RDMA0 MTK_M4U_ID(1, 2)
#define M4U_PORT_L1_OVL_2L_RDMA2 MTK_M4U_ID(1, 3)
#define M4U_PORT_L1_DISP_MDP_RDMA4 MTK_M4U_ID(1, 4)
#define M4U_PORT_L1_DISP_RDMA4 MTK_M4U_ID(1, 5)
#define M4U_PORT_L1_DISP_UFBC_WDMA0 MTK_M4U_ID(1, 6)
#define M4U_PORT_L1_DISP_FAKE1 MTK_M4U_ID(1, 7)
/* larb2 */
#define M4U_PORT_L2_MDP_RDMA0 MTK_M4U_ID(2, 0)
#define M4U_PORT_L2_MDP_RDMA1 MTK_M4U_ID(2, 1)
#define M4U_PORT_L2_MDP_WROT0 MTK_M4U_ID(2, 2)
#define M4U_PORT_L2_MDP_WROT1 MTK_M4U_ID(2, 3)
#define M4U_PORT_L2_MDP_DISP_FAKE0 MTK_M4U_ID(2, 4)
/* larb3: null */
/* larb4 */
#define M4U_PORT_L4_VDEC_MC_EXT MTK_M4U_ID(4, 0)
#define M4U_PORT_L4_VDEC_UFO_EXT MTK_M4U_ID(4, 1)
#define M4U_PORT_L4_VDEC_PP_EXT MTK_M4U_ID(4, 2)
#define M4U_PORT_L4_VDEC_PRED_RD_EXT MTK_M4U_ID(4, 3)
#define M4U_PORT_L4_VDEC_PRED_WR_EXT MTK_M4U_ID(4, 4)
#define M4U_PORT_L4_VDEC_PPWRAP_EXT MTK_M4U_ID(4, 5)
#define M4U_PORT_L4_VDEC_TILE_EXT MTK_M4U_ID(4, 6)
#define M4U_PORT_L4_VDEC_VLD_EXT MTK_M4U_ID(4, 7)
#define M4U_PORT_L4_VDEC_VLD2_EXT MTK_M4U_ID(4, 8)
#define M4U_PORT_L4_VDEC_AVC_MV_EXT MTK_M4U_ID(4, 9)
#define M4U_PORT_L4_VDEC_RG_CTRL_DMA_EXT MTK_M4U_ID(4, 10)
/* larb5 */
#define M4U_PORT_L5_VDEC_LAT0_VLD_EXT MTK_M4U_ID(5, 0)
#define M4U_PORT_L5_VDEC_LAT0_VLD2_EXT MTK_M4U_ID(5, 1)
#define M4U_PORT_L5_VDEC_LAT0_AVC_MV_EXT MTK_M4U_ID(5, 2)
#define M4U_PORT_L5_VDEC_LAT0_PRED_RD_EXT MTK_M4U_ID(5, 3)
#define M4U_PORT_L5_VDEC_LAT0_TILE_EXT MTK_M4U_ID(5, 4)
#define M4U_PORT_L5_VDEC_LAT0_WDMA_EXT MTK_M4U_ID(5, 5)
#define M4U_PORT_L5_VDEC_LAT0_RG_CTRL_DMA_EXT MTK_M4U_ID(5, 6)
#define M4U_PORT_L5_VDEC_UFO_ENC_EXT MTK_M4U_ID(5, 7)
/* larb6: null */
/* larb7 */
#define M4U_PORT_L7_VENC_RCPU MTK_M4U_ID(7, 0)
#define M4U_PORT_L7_VENC_REC MTK_M4U_ID(7, 1)
#define M4U_PORT_L7_VENC_BSDMA MTK_M4U_ID(7, 2)
#define M4U_PORT_L7_VENC_SV_COMV MTK_M4U_ID(7, 3)
#define M4U_PORT_L7_VENC_RD_COMV MTK_M4U_ID(7, 4)
#define M4U_PORT_L7_VENC_CUR_LUMA MTK_M4U_ID(7, 5)
#define M4U_PORT_L7_VENC_CUR_CHROMA MTK_M4U_ID(7, 6)
#define M4U_PORT_L7_VENC_REF_LUMA MTK_M4U_ID(7, 7)
#define M4U_PORT_L7_VENC_REF_CHROMA MTK_M4U_ID(7, 8)
#define M4U_PORT_L7_JPGENC_Y_RDMA MTK_M4U_ID(7, 9)
#define M4U_PORT_L7_JPGENC_Q_RDMA MTK_M4U_ID(7, 10)
#define M4U_PORT_L7_JPGENC_C_TABLE MTK_M4U_ID(7, 11)
#define M4U_PORT_L7_JPGENC_BSDMA MTK_M4U_ID(7, 12)
#define M4U_PORT_L7_VENC_SUB_R_LUMA MTK_M4U_ID(7, 13)
#define M4U_PORT_L7_VENC_SUB_W_LUMA MTK_M4U_ID(7, 14)
/* larb8: null */
/* larb9 */
#define M4U_PORT_L9_IMG_IMGI_D1 MTK_M4U_ID(9, 0)
#define M4U_PORT_L9_IMG_IMGBI_D1 MTK_M4U_ID(9, 1)
#define M4U_PORT_L9_IMG_DMGI_D1 MTK_M4U_ID(9, 2)
#define M4U_PORT_L9_IMG_DEPI_D1 MTK_M4U_ID(9, 3)
#define M4U_PORT_L9_IMG_ICE_D1 MTK_M4U_ID(9, 4)
#define M4U_PORT_L9_IMG_SMTI_D1 MTK_M4U_ID(9, 5)
#define M4U_PORT_L9_IMG_SMTO_D2 MTK_M4U_ID(9, 6)
#define M4U_PORT_L9_IMG_SMTO_D1 MTK_M4U_ID(9, 7)
#define M4U_PORT_L9_IMG_CRZO_D1 MTK_M4U_ID(9, 8)
#define M4U_PORT_L9_IMG_IMG3O_D1 MTK_M4U_ID(9, 9)
#define M4U_PORT_L9_IMG_VIPI_D1 MTK_M4U_ID(9, 10)
#define M4U_PORT_L9_IMG_SMTI_D5 MTK_M4U_ID(9, 11)
#define M4U_PORT_L9_IMG_TIMGO_D1 MTK_M4U_ID(9, 12)
#define M4U_PORT_L9_IMG_UFBC_W0 MTK_M4U_ID(9, 13)
#define M4U_PORT_L9_IMG_UFBC_R0 MTK_M4U_ID(9, 14)
/* larb10: null */
/* larb11 */
#define M4U_PORT_L11_IMG_IMGI_D1 MTK_M4U_ID(11, 0)
#define M4U_PORT_L11_IMG_IMGBI_D1 MTK_M4U_ID(11, 1)
#define M4U_PORT_L11_IMG_DMGI_D1 MTK_M4U_ID(11, 2)
#define M4U_PORT_L11_IMG_DEPI_D1 MTK_M4U_ID(11, 3)
#define M4U_PORT_L11_IMG_ICE_D1 MTK_M4U_ID(11, 4)
#define M4U_PORT_L11_IMG_SMTI_D1 MTK_M4U_ID(11, 5)
#define M4U_PORT_L11_IMG_SMTO_D2 MTK_M4U_ID(11, 6)
#define M4U_PORT_L11_IMG_SMTO_D1 MTK_M4U_ID(11, 7)
#define M4U_PORT_L11_IMG_CRZO_D1 MTK_M4U_ID(11, 8)
#define M4U_PORT_L11_IMG_IMG3O_D1 MTK_M4U_ID(11, 9)
#define M4U_PORT_L11_IMG_VIPI_D1 MTK_M4U_ID(11, 10)
#define M4U_PORT_L11_IMG_SMTI_D5 MTK_M4U_ID(11, 11)
#define M4U_PORT_L11_IMG_TIMGO_D1 MTK_M4U_ID(11, 12)
#define M4U_PORT_L11_IMG_UFBC_W0 MTK_M4U_ID(11, 13)
#define M4U_PORT_L11_IMG_UFBC_R0 MTK_M4U_ID(11, 14)
#define M4U_PORT_L11_IMG_WPE_RDMA1 MTK_M4U_ID(11, 15)
#define M4U_PORT_L11_IMG_WPE_RDMA0 MTK_M4U_ID(11, 16)
#define M4U_PORT_L11_IMG_WPE_WDMA MTK_M4U_ID(11, 17)
#define M4U_PORT_L11_IMG_MFB_RDMA0 MTK_M4U_ID(11, 18)
#define M4U_PORT_L11_IMG_MFB_RDMA1 MTK_M4U_ID(11, 19)
#define M4U_PORT_L11_IMG_MFB_RDMA2 MTK_M4U_ID(11, 20)
#define M4U_PORT_L11_IMG_MFB_RDMA3 MTK_M4U_ID(11, 21)
#define M4U_PORT_L11_IMG_MFB_RDMA4 MTK_M4U_ID(11, 22)
#define M4U_PORT_L11_IMG_MFB_RDMA5 MTK_M4U_ID(11, 23)
#define M4U_PORT_L11_IMG_MFB_WDMA0 MTK_M4U_ID(11, 24)
#define M4U_PORT_L11_IMG_MFB_WDMA1 MTK_M4U_ID(11, 25)
/* larb12: null */
/* larb13 */
#define M4U_PORT_L13_CAM_MRAWI MTK_M4U_ID(13, 0)
#define M4U_PORT_L13_CAM_MRAWO0 MTK_M4U_ID(13, 1)
#define M4U_PORT_L13_CAM_MRAWO1 MTK_M4U_ID(13, 2)
#define M4U_PORT_L13_CAM_CAMSV1 MTK_M4U_ID(13, 3)
#define M4U_PORT_L13_CAM_CAMSV2 MTK_M4U_ID(13, 4)
#define M4U_PORT_L13_CAM_CAMSV3 MTK_M4U_ID(13, 5)
#define M4U_PORT_L13_CAM_CAMSV4 MTK_M4U_ID(13, 6)
#define M4U_PORT_L13_CAM_CAMSV5 MTK_M4U_ID(13, 7)
#define M4U_PORT_L13_CAM_CAMSV6 MTK_M4U_ID(13, 8)
#define M4U_PORT_L13_CAM_CCUI MTK_M4U_ID(13, 9)
#define M4U_PORT_L13_CAM_CCUO MTK_M4U_ID(13, 10)
#define M4U_PORT_L13_CAM_FAKE MTK_M4U_ID(13, 11)
/* larb14 */
#define M4U_PORT_L14_CAM_RESERVE1 MTK_M4U_ID(14, 0)
#define M4U_PORT_L14_CAM_RESERVE2 MTK_M4U_ID(14, 1)
#define M4U_PORT_L14_CAM_RESERVE3 MTK_M4U_ID(14, 2)
#define M4U_PORT_L14_CAM_CAMSV0 MTK_M4U_ID(14, 3)
#define M4U_PORT_L14_CAM_CCUI MTK_M4U_ID(14, 4)
#define M4U_PORT_L14_CAM_CCUO MTK_M4U_ID(14, 5)
/* larb15: null */
/* larb16 */
#define M4U_PORT_L16_CAM_IMGO_R1_A MTK_M4U_ID(16, 0)
#define M4U_PORT_L16_CAM_RRZO_R1_A MTK_M4U_ID(16, 1)
#define M4U_PORT_L16_CAM_CQI_R1_A MTK_M4U_ID(16, 2)
#define M4U_PORT_L16_CAM_BPCI_R1_A MTK_M4U_ID(16, 3)
#define M4U_PORT_L16_CAM_YUVO_R1_A MTK_M4U_ID(16, 4)
#define M4U_PORT_L16_CAM_UFDI_R2_A MTK_M4U_ID(16, 5)
#define M4U_PORT_L16_CAM_RAWI_R2_A MTK_M4U_ID(16, 6)
#define M4U_PORT_L16_CAM_RAWI_R3_A MTK_M4U_ID(16, 7)
#define M4U_PORT_L16_CAM_AAO_R1_A MTK_M4U_ID(16, 8)
#define M4U_PORT_L16_CAM_AFO_R1_A MTK_M4U_ID(16, 9)
#define M4U_PORT_L16_CAM_FLKO_R1_A MTK_M4U_ID(16, 10)
#define M4U_PORT_L16_CAM_LCESO_R1_A MTK_M4U_ID(16, 11)
#define M4U_PORT_L16_CAM_CRZO_R1_A MTK_M4U_ID(16, 12)
#define M4U_PORT_L16_CAM_LTMSO_R1_A MTK_M4U_ID(16, 13)
#define M4U_PORT_L16_CAM_RSSO_R1_A MTK_M4U_ID(16, 14)
#define M4U_PORT_L16_CAM_AAHO_R1_A MTK_M4U_ID(16, 15)
#define M4U_PORT_L16_CAM_LSCI_R1_A MTK_M4U_ID(16, 16)
/* larb17 */
#define M4U_PORT_L17_CAM_IMGO_R1_B MTK_M4U_ID(17, 0)
#define M4U_PORT_L17_CAM_RRZO_R1_B MTK_M4U_ID(17, 1)
#define M4U_PORT_L17_CAM_CQI_R1_B MTK_M4U_ID(17, 2)
#define M4U_PORT_L17_CAM_BPCI_R1_B MTK_M4U_ID(17, 3)
#define M4U_PORT_L17_CAM_YUVO_R1_B MTK_M4U_ID(17, 4)
#define M4U_PORT_L17_CAM_UFDI_R2_B MTK_M4U_ID(17, 5)
#define M4U_PORT_L17_CAM_RAWI_R2_B MTK_M4U_ID(17, 6)
#define M4U_PORT_L17_CAM_RAWI_R3_B MTK_M4U_ID(17, 7)
#define M4U_PORT_L17_CAM_AAO_R1_B MTK_M4U_ID(17, 8)
#define M4U_PORT_L17_CAM_AFO_R1_B MTK_M4U_ID(17, 9)
#define M4U_PORT_L17_CAM_FLKO_R1_B MTK_M4U_ID(17, 10)
#define M4U_PORT_L17_CAM_LCESO_R1_B MTK_M4U_ID(17, 11)
#define M4U_PORT_L17_CAM_CRZO_R1_B MTK_M4U_ID(17, 12)
#define M4U_PORT_L17_CAM_LTMSO_R1_B MTK_M4U_ID(17, 13)
#define M4U_PORT_L17_CAM_RSSO_R1_B MTK_M4U_ID(17, 14)
#define M4U_PORT_L17_CAM_AAHO_R1_B MTK_M4U_ID(17, 15)
#define M4U_PORT_L17_CAM_LSCI_R1_B MTK_M4U_ID(17, 16)
/* larb18 */
#define M4U_PORT_L18_CAM_IMGO_R1_C MTK_M4U_ID(18, 0)
#define M4U_PORT_L18_CAM_RRZO_R1_C MTK_M4U_ID(18, 1)
#define M4U_PORT_L18_CAM_CQI_R1_C MTK_M4U_ID(18, 2)
#define M4U_PORT_L18_CAM_BPCI_R1_C MTK_M4U_ID(18, 3)
#define M4U_PORT_L18_CAM_YUVO_R1_C MTK_M4U_ID(18, 4)
#define M4U_PORT_L18_CAM_UFDI_R2_C MTK_M4U_ID(18, 5)
#define M4U_PORT_L18_CAM_RAWI_R2_C MTK_M4U_ID(18, 6)
#define M4U_PORT_L18_CAM_RAWI_R3_C MTK_M4U_ID(18, 7)
#define M4U_PORT_L18_CAM_AAO_R1_C MTK_M4U_ID(18, 8)
#define M4U_PORT_L18_CAM_AFO_R1_C MTK_M4U_ID(18, 9)
#define M4U_PORT_L18_CAM_FLKO_R1_C MTK_M4U_ID(18, 10)
#define M4U_PORT_L18_CAM_LCESO_R1_C MTK_M4U_ID(18, 11)
#define M4U_PORT_L18_CAM_CRZO_R1_C MTK_M4U_ID(18, 12)
#define M4U_PORT_L18_CAM_LTMSO_R1_C MTK_M4U_ID(18, 13)
#define M4U_PORT_L18_CAM_RSSO_R1_C MTK_M4U_ID(18, 14)
#define M4U_PORT_L18_CAM_AAHO_R1_C MTK_M4U_ID(18, 15)
#define M4U_PORT_L18_CAM_LSCI_R1_C MTK_M4U_ID(18, 16)
/* larb19 */
#define M4U_PORT_L19_IPE_DVS_RDMA MTK_M4U_ID(19, 0)
#define M4U_PORT_L19_IPE_DVS_WDMA MTK_M4U_ID(19, 1)
#define M4U_PORT_L19_IPE_DVP_RDMA MTK_M4U_ID(19, 2)
#define M4U_PORT_L19_IPE_DVP_WDMA MTK_M4U_ID(19, 3)
/* larb20 */
#define M4U_PORT_L20_IPE_FDVT_RDA MTK_M4U_ID(20, 0)
#define M4U_PORT_L20_IPE_FDVT_RDB MTK_M4U_ID(20, 1)
#define M4U_PORT_L20_IPE_FDVT_WRA MTK_M4U_ID(20, 2)
#define M4U_PORT_L20_IPE_FDVT_WRB MTK_M4U_ID(20, 3)
#define M4U_PORT_L20_IPE_RSC_RDMA0 MTK_M4U_ID(20, 4)
#define M4U_PORT_L20_IPE_RSC_WDMA MTK_M4U_ID(20, 5)
#endif

15
include/dt-bindings/memory/mtk-memory-port.h Normal file
View File

@ -0,0 +1,15 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* Copyright (c) 2020 MediaTek Inc.
* Author: Yong Wu <yong.wu@mediatek.com>
*/
#ifndef __DT_BINDINGS_MEMORY_MTK_MEMORY_PORT_H_
#define __DT_BINDINGS_MEMORY_MTK_MEMORY_PORT_H_
#define MTK_LARB_NR_MAX 32
#define MTK_M4U_ID(larb, port) (((larb) << 5) | (port))
#define MTK_M4U_TO_LARB(id) (((id) >> 5) & 0x1f)
#define MTK_M4U_TO_PORT(id) ((id) & 0x1f)
#endif

13
include/linux/io-pgtable.h
View File

@ -68,13 +68,9 @@ struct io_pgtable_cfg {
* hardware which does not implement the permissions of a given
* format, and/or requires some format-specific default value.
*
* IO_PGTABLE_QUIRK_TLBI_ON_MAP: If the format forbids caching invalid
* (unmapped) entries but the hardware might do so anyway, perform
* TLB maintenance when mapping as well as when unmapping.
*
* IO_PGTABLE_QUIRK_ARM_MTK_EXT: (ARM v7s format) MediaTek IOMMUs extend
* to support up to 34 bits PA where the bit32 and bit33 are
* encoded in the bit9 and bit4 of the PTE respectively.
* to support up to 35 bits PA where the bit32, bit33 and bit34 are
* encoded in the bit9, bit4 and bit5 of the PTE respectively.
*
* IO_PGTABLE_QUIRK_NON_STRICT: Skip issuing synchronous leaf TLBIs
* on unmap, for DMA domains using the flush queue mechanism for
@ -88,7 +84,6 @@ struct io_pgtable_cfg {
*/
#define IO_PGTABLE_QUIRK_ARM_NS BIT(0)
#define IO_PGTABLE_QUIRK_NO_PERMS BIT(1)
#define IO_PGTABLE_QUIRK_TLBI_ON_MAP BIT(2)
#define IO_PGTABLE_QUIRK_ARM_MTK_EXT BIT(3)
#define IO_PGTABLE_QUIRK_NON_STRICT BIT(4)
#define IO_PGTABLE_QUIRK_ARM_TTBR1 BIT(5)
@ -214,6 +209,7 @@ struct io_pgtable_domain_attr {
static inline void io_pgtable_tlb_flush_all(struct io_pgtable *iop)
{
if (iop->cfg.tlb && iop->cfg.tlb->tlb_flush_all)
iop->cfg.tlb->tlb_flush_all(iop->cookie);
}
@ -221,6 +217,7 @@ static inline void
io_pgtable_tlb_flush_walk(struct io_pgtable *iop, unsigned long iova,
size_t size, size_t granule)
{
if (iop->cfg.tlb && iop->cfg.tlb->tlb_flush_walk)
iop->cfg.tlb->tlb_flush_walk(iova, size, granule, iop->cookie);
}
@ -229,7 +226,7 @@ io_pgtable_tlb_add_page(struct io_pgtable *iop,
struct iommu_iotlb_gather * gather, unsigned long iova,
size_t granule)
{
if (iop->cfg.tlb->tlb_add_page)
if (iop->cfg.tlb && iop->cfg.tlb->tlb_add_page)
iop->cfg.tlb->tlb_add_page(gather, iova, granule, iop->cookie);
}

7
include/linux/iommu.h
View File

@ -170,7 +170,7 @@ enum iommu_dev_features {
* struct iommu_iotlb_gather - Range information for a pending IOTLB flush
*
* @start: IOVA representing the start of the range to be flushed
* @end: IOVA representing the end of the range to be flushed (exclusive)
* @end: IOVA representing the end of the range to be flushed (inclusive)
* @pgsize: The interval at which to perform the flush
*
* This structure is intended to be updated by multiple calls to the
@ -246,7 +246,8 @@ struct iommu_ops {
size_t (*unmap)(struct iommu_domain *domain, unsigned long iova,
size_t size, struct iommu_iotlb_gather *iotlb_gather);
void (*flush_iotlb_all)(struct iommu_domain *domain);
void (*iotlb_sync_map)(struct iommu_domain *domain);
void (*iotlb_sync_map)(struct iommu_domain *domain, unsigned long iova,
size_t size);
void (*iotlb_sync)(struct iommu_domain *domain,
struct iommu_iotlb_gather *iotlb_gather);
phys_addr_t (*iova_to_phys)(struct iommu_domain *domain, dma_addr_t iova);
@ -538,7 +539,7 @@ static inline void iommu_iotlb_gather_add_page(struct iommu_domain *domain,
struct iommu_iotlb_gather *gather,
unsigned long iova, size_t size)
{
unsigned long start = iova, end = start + size;
unsigned long start = iova, end = start + size - 1;
/*
* If the new page is disjoint from the current range or is mapped at

3
include/soc/mediatek/smi.h
View File

@ -11,13 +11,12 @@
#ifdef CONFIG_MTK_SMI
#define MTK_LARB_NR_MAX 16
#define MTK_SMI_MMU_EN(port) BIT(port)
struct mtk_smi_larb_iommu {
struct device *dev;
unsigned int mmu;
unsigned char bank[32];
};
/*