UbiquitousOS
/
OpenCloudOS-Kernel
mirror of https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git
11
0
Fork 0
Code Issues Projects Releases Wiki Activity
OpenCloudOS-Kernel/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c

455 lines
13 KiB
C
Raw Normal View History

iommu: arm-smmu-impl: Add sdm845 implementation hook Add reset hook for sdm845 based platforms to turn off the wait-for-safe sequence. Understanding how wait-for-safe logic affects USB and UFS performance on MTP845 and DB845 boards: Qcom's implementation of arm,mmu-500 adds a WAIT-FOR-SAFE logic to address under-performance issues in real-time clients, such as Display, and Camera. On receiving an invalidation requests, the SMMU forwards SAFE request to these clients and waits for SAFE ack signal from real-time clients. The SAFE signal from such clients is used to qualify the start of invalidation. This logic is controlled by chicken bits, one for each - MDP (display), IFE0, and IFE1 (camera), that can be accessed only from secure software on sdm845. This configuration, however, degrades the performance of non-real time clients, such as USB, and UFS etc. This happens because, with wait-for-safe logic enabled the hardware tries to throttle non-real time clients while waiting for SAFE ack signals from real-time clients. On mtp845 and db845 devices, with wait-for-safe logic enabled by the bootloaders we see degraded performance of USB and UFS when kernel enables the smmu stage-1 translations for these clients. Turn off this wait-for-safe logic from the kernel gets us back the perf of USB and UFS devices until we re-visit this when we start seeing perf issues on display/camera on upstream supported SDM845 platforms. The bootloaders on these boards implement secure monitor callbacks to handle a specific command - QCOM_SCM_SVC_SMMU_PROGRAM with which the logic can be toggled. There are other boards such as cheza whose bootloaders don't enable this logic. Such boards don't implement callbacks to handle the specific SCM call so disabling this logic for such boards will be a no-op. This change is inspired by the downstream change from Patrick Daly to address performance issues with display and camera by handling this wait-for-safe within separte io-pagetable ops to do TLB maintenance. So a big thanks to him for the change and for all the offline discussions. Without this change the UFS reads are pretty slow: $ time dd if=/dev/sda of=/dev/zero bs=1048576 count=10 conv=sync 10+0 records in 10+0 records out 10485760 bytes (10.0MB) copied, 22.394903 seconds, 457.2KB/s real 0m 22.39s user 0m 0.00s sys 0m 0.01s With this change they are back to rock! $ time dd if=/dev/sda of=/dev/zero bs=1048576 count=300 conv=sync 300+0 records in 300+0 records out 314572800 bytes (300.0MB) copied, 1.030541 seconds, 291.1MB/s real 0m 1.03s user 0m 0.00s sys 0m 0.54s Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Reviewed-by: Stephen Boyd <swboyd@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-09-20 16:04:29 +08:00
// SPDX-License-Identifier: GPL-2.0-only
/*
* Copyright (c) 2019, The Linux Foundation. All rights reserved.
*/
iommu/arm-smmu-qcom: hook up qcom_smmu_impl for ACPI boot The hookup with qcom_smmu_impl is required to do ACPI boot on SC8180X based devices like Lenovo Flex 5G laptop and Microsoft Surface Pro X. Define acpi_platform_list for these platforms and match them using acpi_match_platform_list() call, and create qcom_smmu_impl accordingly. (np == NULL) is used to check ACPI boot, because fwnode of SMMU device is a static allocation and thus helpers like has_acpi_companion() don't work here. Signed-off-by: Shawn Guo <shawn.guo@linaro.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20210509022607.17534-1-shawn.guo@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2021-05-09 10:26:07 +08:00
#include <linux/acpi.h>
iommu/arm-smmu-qcom: Add implementation for the adreno GPU SMMU Add a special implementation for the SMMU attached to most Adreno GPU target triggered from the qcom,adreno-smmu compatible string. The new Adreno SMMU implementation will enable split pagetables (TTBR1) for the domain attached to the GPU device (SID 0) and hard code it context bank 0 so the GPU hardware can implement per-instance pagetables. Co-developed-by: Rob Clark <robdclark@chromium.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Rob Clark <robdclark@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20201109184728.2463097-2-jcrouse@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-10 02:47:25 +08:00
#include <linux/adreno-smmu-priv.h>
iommu/arm-smmu: Allow client devices to select direct mapping Some client devices want to directly map the IOMMU themselves instead of using the DMA domain. Allow those devices to opt in to direct mapping by way of a list of compatible strings. Co-developed-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/7cf1f64167b5545b7f42275395be1f1e2ea3a6ac.1587407458.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-04-21 02:33:51 +08:00
#include <linux/of_device.h>
iommu: arm-smmu-impl: Add sdm845 implementation hook Add reset hook for sdm845 based platforms to turn off the wait-for-safe sequence. Understanding how wait-for-safe logic affects USB and UFS performance on MTP845 and DB845 boards: Qcom's implementation of arm,mmu-500 adds a WAIT-FOR-SAFE logic to address under-performance issues in real-time clients, such as Display, and Camera. On receiving an invalidation requests, the SMMU forwards SAFE request to these clients and waits for SAFE ack signal from real-time clients. The SAFE signal from such clients is used to qualify the start of invalidation. This logic is controlled by chicken bits, one for each - MDP (display), IFE0, and IFE1 (camera), that can be accessed only from secure software on sdm845. This configuration, however, degrades the performance of non-real time clients, such as USB, and UFS etc. This happens because, with wait-for-safe logic enabled the hardware tries to throttle non-real time clients while waiting for SAFE ack signals from real-time clients. On mtp845 and db845 devices, with wait-for-safe logic enabled by the bootloaders we see degraded performance of USB and UFS when kernel enables the smmu stage-1 translations for these clients. Turn off this wait-for-safe logic from the kernel gets us back the perf of USB and UFS devices until we re-visit this when we start seeing perf issues on display/camera on upstream supported SDM845 platforms. The bootloaders on these boards implement secure monitor callbacks to handle a specific command - QCOM_SCM_SVC_SMMU_PROGRAM with which the logic can be toggled. There are other boards such as cheza whose bootloaders don't enable this logic. Such boards don't implement callbacks to handle the specific SCM call so disabling this logic for such boards will be a no-op. This change is inspired by the downstream change from Patrick Daly to address performance issues with display and camera by handling this wait-for-safe within separte io-pagetable ops to do TLB maintenance. So a big thanks to him for the change and for all the offline discussions. Without this change the UFS reads are pretty slow: $ time dd if=/dev/sda of=/dev/zero bs=1048576 count=10 conv=sync 10+0 records in 10+0 records out 10485760 bytes (10.0MB) copied, 22.394903 seconds, 457.2KB/s real 0m 22.39s user 0m 0.00s sys 0m 0.01s With this change they are back to rock! $ time dd if=/dev/sda of=/dev/zero bs=1048576 count=300 conv=sync 300+0 records in 300+0 records out 314572800 bytes (300.0MB) copied, 1.030541 seconds, 291.1MB/s real 0m 1.03s user 0m 0.00s sys 0m 0.54s Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Reviewed-by: Stephen Boyd <swboyd@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-09-20 16:04:29 +08:00
#include <linux/qcom_scm.h>
#include "arm-smmu.h"
struct qcom_smmu {
struct arm_smmu_device smmu;
iommu/arm-smmu-qcom: Implement S2CR quirk The firmware found in some Qualcomm platforms intercepts writes to S2CR in order to replace bypass type streams with fault; and ignore S2CR updates of type fault. Detect this behavior and implement a custom write_s2cr function in order to trick the firmware into supporting bypass streams by the means of configuring the stream for translation using a reserved and disabled context bank. Also circumvent the problem of configuring faulting streams by configuring the stream as bypass. Cc: <stable@vger.kernel.org> Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org> Tested-by: Steev Klimaszewski <steev@kali.org> Acked-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/20201019182323.3162386-4-bjorn.andersson@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2020-10-20 02:23:23 +08:00
bool bypass_quirk;
u8 bypass_cbndx;
iommu/arm-smmu-qcom: Add stall support Add, via the adreno-smmu-priv interface, a way for the GPU to request the SMMU to stall translation on faults, and then later resume the translation, either retrying or terminating the current translation. This will be used on the GPU side to "freeze" the GPU while we snapshot useful state for devcoredump. Signed-off-by: Rob Clark <robdclark@chromium.org> Acked-by: Jordan Crouse <jordan@cosmicpenguin.net> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20210610214431.539029-5-robdclark@gmail.com Signed-off-by: Rob Clark <robdclark@chromium.org>
2021-06-11 05:44:12 +08:00
u32 stall_enabled;
iommu: arm-smmu-impl: Add sdm845 implementation hook Add reset hook for sdm845 based platforms to turn off the wait-for-safe sequence. Understanding how wait-for-safe logic affects USB and UFS performance on MTP845 and DB845 boards: Qcom's implementation of arm,mmu-500 adds a WAIT-FOR-SAFE logic to address under-performance issues in real-time clients, such as Display, and Camera. On receiving an invalidation requests, the SMMU forwards SAFE request to these clients and waits for SAFE ack signal from real-time clients. The SAFE signal from such clients is used to qualify the start of invalidation. This logic is controlled by chicken bits, one for each - MDP (display), IFE0, and IFE1 (camera), that can be accessed only from secure software on sdm845. This configuration, however, degrades the performance of non-real time clients, such as USB, and UFS etc. This happens because, with wait-for-safe logic enabled the hardware tries to throttle non-real time clients while waiting for SAFE ack signals from real-time clients. On mtp845 and db845 devices, with wait-for-safe logic enabled by the bootloaders we see degraded performance of USB and UFS when kernel enables the smmu stage-1 translations for these clients. Turn off this wait-for-safe logic from the kernel gets us back the perf of USB and UFS devices until we re-visit this when we start seeing perf issues on display/camera on upstream supported SDM845 platforms. The bootloaders on these boards implement secure monitor callbacks to handle a specific command - QCOM_SCM_SVC_SMMU_PROGRAM with which the logic can be toggled. There are other boards such as cheza whose bootloaders don't enable this logic. Such boards don't implement callbacks to handle the specific SCM call so disabling this logic for such boards will be a no-op. This change is inspired by the downstream change from Patrick Daly to address performance issues with display and camera by handling this wait-for-safe within separte io-pagetable ops to do TLB maintenance. So a big thanks to him for the change and for all the offline discussions. Without this change the UFS reads are pretty slow: $ time dd if=/dev/sda of=/dev/zero bs=1048576 count=10 conv=sync 10+0 records in 10+0 records out 10485760 bytes (10.0MB) copied, 22.394903 seconds, 457.2KB/s real 0m 22.39s user 0m 0.00s sys 0m 0.01s With this change they are back to rock! $ time dd if=/dev/sda of=/dev/zero bs=1048576 count=300 conv=sync 300+0 records in 300+0 records out 314572800 bytes (300.0MB) copied, 1.030541 seconds, 291.1MB/s real 0m 1.03s user 0m 0.00s sys 0m 0.54s Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Reviewed-by: Stephen Boyd <swboyd@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-09-20 16:04:29 +08:00
};
iommu/arm-smmu-qcom: Implement S2CR quirk The firmware found in some Qualcomm platforms intercepts writes to S2CR in order to replace bypass type streams with fault; and ignore S2CR updates of type fault. Detect this behavior and implement a custom write_s2cr function in order to trick the firmware into supporting bypass streams by the means of configuring the stream for translation using a reserved and disabled context bank. Also circumvent the problem of configuring faulting streams by configuring the stream as bypass. Cc: <stable@vger.kernel.org> Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org> Tested-by: Steev Klimaszewski <steev@kali.org> Acked-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/20201019182323.3162386-4-bjorn.andersson@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2020-10-20 02:23:23 +08:00
static struct qcom_smmu *to_qcom_smmu(struct arm_smmu_device *smmu)
{
return container_of(smmu, struct qcom_smmu, smmu);
}
iommu/arm-smmu: Add a way for implementations to influence SCTLR For the Adreno GPU's SMMU, we want SCTLR.HUPCF set to ensure that pending translations are not terminated on iova fault. Otherwise a terminated CP read could hang the GPU by returning invalid command-stream data. Add a hook to for the implementation to modify the sctlr value if it wishes. Co-developed-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Rob Clark <robdclark@chromium.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Link: https://lore.kernel.org/r/20201109184728.2463097-3-jcrouse@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-10 02:47:26 +08:00
static void qcom_adreno_smmu_write_sctlr(struct arm_smmu_device *smmu, int idx,
u32 reg)
{
iommu/arm-smmu-qcom: Add stall support Add, via the adreno-smmu-priv interface, a way for the GPU to request the SMMU to stall translation on faults, and then later resume the translation, either retrying or terminating the current translation. This will be used on the GPU side to "freeze" the GPU while we snapshot useful state for devcoredump. Signed-off-by: Rob Clark <robdclark@chromium.org> Acked-by: Jordan Crouse <jordan@cosmicpenguin.net> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20210610214431.539029-5-robdclark@gmail.com Signed-off-by: Rob Clark <robdclark@chromium.org>
2021-06-11 05:44:12 +08:00
struct qcom_smmu *qsmmu = to_qcom_smmu(smmu);
iommu/arm-smmu: Add a way for implementations to influence SCTLR For the Adreno GPU's SMMU, we want SCTLR.HUPCF set to ensure that pending translations are not terminated on iova fault. Otherwise a terminated CP read could hang the GPU by returning invalid command-stream data. Add a hook to for the implementation to modify the sctlr value if it wishes. Co-developed-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Rob Clark <robdclark@chromium.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Link: https://lore.kernel.org/r/20201109184728.2463097-3-jcrouse@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-10 02:47:26 +08:00
/*
* On the GPU device we want to process subsequent transactions after a
* fault to keep the GPU from hanging
*/
reg |= ARM_SMMU_SCTLR_HUPCF;
iommu/arm-smmu-qcom: Add stall support Add, via the adreno-smmu-priv interface, a way for the GPU to request the SMMU to stall translation on faults, and then later resume the translation, either retrying or terminating the current translation. This will be used on the GPU side to "freeze" the GPU while we snapshot useful state for devcoredump. Signed-off-by: Rob Clark <robdclark@chromium.org> Acked-by: Jordan Crouse <jordan@cosmicpenguin.net> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20210610214431.539029-5-robdclark@gmail.com Signed-off-by: Rob Clark <robdclark@chromium.org>
2021-06-11 05:44:12 +08:00
if (qsmmu->stall_enabled & BIT(idx))
reg |= ARM_SMMU_SCTLR_CFCFG;
iommu/arm-smmu: Add a way for implementations to influence SCTLR For the Adreno GPU's SMMU, we want SCTLR.HUPCF set to ensure that pending translations are not terminated on iova fault. Otherwise a terminated CP read could hang the GPU by returning invalid command-stream data. Add a hook to for the implementation to modify the sctlr value if it wishes. Co-developed-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Rob Clark <robdclark@chromium.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Link: https://lore.kernel.org/r/20201109184728.2463097-3-jcrouse@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-10 02:47:26 +08:00
arm_smmu_cb_write(smmu, idx, ARM_SMMU_CB_SCTLR, reg);
}
iommu/arm-smmu-qcom: Add an adreno-smmu-priv callback to get pagefault info Add a callback in adreno-smmu-priv to read interesting SMMU registers to provide an opportunity for a richer debug experience in the GPU driver. Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Rob Clark <robdclark@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20210610214431.539029-3-robdclark@gmail.com Signed-off-by: Rob Clark <robdclark@chromium.org>
2021-06-11 05:44:10 +08:00
static void qcom_adreno_smmu_get_fault_info(const void *cookie,
struct adreno_smmu_fault_info *info)
{
struct arm_smmu_domain *smmu_domain = (void *)cookie;
struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
struct arm_smmu_device *smmu = smmu_domain->smmu;
info->fsr = arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_FSR);
info->fsynr0 = arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_FSYNR0);
info->fsynr1 = arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_FSYNR1);
info->far = arm_smmu_cb_readq(smmu, cfg->cbndx, ARM_SMMU_CB_FAR);
info->cbfrsynra = arm_smmu_gr1_read(smmu, ARM_SMMU_GR1_CBFRSYNRA(cfg->cbndx));
info->ttbr0 = arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_TTBR0);
info->contextidr = arm_smmu_cb_read(smmu, cfg->cbndx, ARM_SMMU_CB_CONTEXTIDR);
}
iommu/arm-smmu-qcom: Add stall support Add, via the adreno-smmu-priv interface, a way for the GPU to request the SMMU to stall translation on faults, and then later resume the translation, either retrying or terminating the current translation. This will be used on the GPU side to "freeze" the GPU while we snapshot useful state for devcoredump. Signed-off-by: Rob Clark <robdclark@chromium.org> Acked-by: Jordan Crouse <jordan@cosmicpenguin.net> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20210610214431.539029-5-robdclark@gmail.com Signed-off-by: Rob Clark <robdclark@chromium.org>
2021-06-11 05:44:12 +08:00
static void qcom_adreno_smmu_set_stall(const void *cookie, bool enabled)
{
struct arm_smmu_domain *smmu_domain = (void *)cookie;
struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
struct qcom_smmu *qsmmu = to_qcom_smmu(smmu_domain->smmu);
if (enabled)
qsmmu->stall_enabled |= BIT(cfg->cbndx);
else
qsmmu->stall_enabled &= ~BIT(cfg->cbndx);
}
static void qcom_adreno_smmu_resume_translation(const void *cookie, bool terminate)
{
struct arm_smmu_domain *smmu_domain = (void *)cookie;
struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
struct arm_smmu_device *smmu = smmu_domain->smmu;
u32 reg = 0;
if (terminate)
reg |= ARM_SMMU_RESUME_TERMINATE;
arm_smmu_cb_write(smmu, cfg->cbndx, ARM_SMMU_CB_RESUME, reg);
}
iommu/arm-smmu-qcom: Add implementation for the adreno GPU SMMU Add a special implementation for the SMMU attached to most Adreno GPU target triggered from the qcom,adreno-smmu compatible string. The new Adreno SMMU implementation will enable split pagetables (TTBR1) for the domain attached to the GPU device (SID 0) and hard code it context bank 0 so the GPU hardware can implement per-instance pagetables. Co-developed-by: Rob Clark <robdclark@chromium.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Rob Clark <robdclark@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20201109184728.2463097-2-jcrouse@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-10 02:47:25 +08:00
#define QCOM_ADRENO_SMMU_GPU_SID 0
static bool qcom_adreno_smmu_is_gpu_device(struct device *dev)
{
struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
int i;
/*
* The GPU will always use SID 0 so that is a handy way to uniquely
* identify it and configure it for per-instance pagetables
*/
for (i = 0; i < fwspec->num_ids; i++) {
u16 sid = FIELD_GET(ARM_SMMU_SMR_ID, fwspec->ids[i]);
if (sid == QCOM_ADRENO_SMMU_GPU_SID)
return true;
}
return false;
}
static const struct io_pgtable_cfg *qcom_adreno_smmu_get_ttbr1_cfg(
const void *cookie)
{
struct arm_smmu_domain *smmu_domain = (void *)cookie;
struct io_pgtable *pgtable =
io_pgtable_ops_to_pgtable(smmu_domain->pgtbl_ops);
return &pgtable->cfg;
}
/*
* Local implementation to configure TTBR0 with the specified pagetable config.
* The GPU driver will call this to enable TTBR0 when per-instance pagetables
* are active
*/
static int qcom_adreno_smmu_set_ttbr0_cfg(const void *cookie,
const struct io_pgtable_cfg *pgtbl_cfg)
{
struct arm_smmu_domain *smmu_domain = (void *)cookie;
struct io_pgtable *pgtable = io_pgtable_ops_to_pgtable(smmu_domain->pgtbl_ops);
struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
struct arm_smmu_cb *cb = &smmu_domain->smmu->cbs[cfg->cbndx];
/* The domain must have split pagetables already enabled */
if (cb->tcr[0] & ARM_SMMU_TCR_EPD1)
return -EINVAL;
/* If the pagetable config is NULL, disable TTBR0 */
if (!pgtbl_cfg) {
/* Do nothing if it is already disabled */
if ((cb->tcr[0] & ARM_SMMU_TCR_EPD0))
return -EINVAL;
/* Set TCR to the original configuration */
cb->tcr[0] = arm_smmu_lpae_tcr(&pgtable->cfg);
cb->ttbr[0] = FIELD_PREP(ARM_SMMU_TTBRn_ASID, cb->cfg->asid);
} else {
u32 tcr = cb->tcr[0];
/* Don't call this again if TTBR0 is already enabled */
if (!(cb->tcr[0] & ARM_SMMU_TCR_EPD0))
return -EINVAL;
tcr |= arm_smmu_lpae_tcr(pgtbl_cfg);
tcr &= ~(ARM_SMMU_TCR_EPD0 | ARM_SMMU_TCR_EPD1);
cb->tcr[0] = tcr;
cb->ttbr[0] = pgtbl_cfg->arm_lpae_s1_cfg.ttbr;
cb->ttbr[0] |= FIELD_PREP(ARM_SMMU_TTBRn_ASID, cb->cfg->asid);
}
arm_smmu_write_context_bank(smmu_domain->smmu, cb->cfg->cbndx);
return 0;
}
static int qcom_adreno_smmu_alloc_context_bank(struct arm_smmu_domain *smmu_domain,
struct arm_smmu_device *smmu,
struct device *dev, int start)
{
int count;
/*
* Assign context bank 0 to the GPU device so the GPU hardware can
* switch pagetables
*/
if (qcom_adreno_smmu_is_gpu_device(dev)) {
start = 0;
count = 1;
} else {
start = 1;
count = smmu->num_context_banks;
}
return __arm_smmu_alloc_bitmap(smmu->context_map, start, count);
}
iommu/arm-smmu-qcom: Skip the TTBR1 quirk for db820c. db820c wants to use the qcom smmu path to get HUPCF set (which keeps the GPU from wedging and then sometimes wedging the kernel after a page fault), but it doesn't have separate pagetables support yet in drm/msm so we can't go all the way to the TTBR1 path. Signed-off-by: Eric Anholt <eric@anholt.net> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20210326231303.3071950-1-eric@anholt.net Signed-off-by: Will Deacon <will@kernel.org>
2021-03-27 07:13:02 +08:00
static bool qcom_adreno_can_do_ttbr1(struct arm_smmu_device *smmu)
{
const struct device_node *np = smmu->dev->of_node;
if (of_device_is_compatible(np, "qcom,msm8996-smmu-v2"))
return false;
return true;
}
iommu/arm-smmu-qcom: Add implementation for the adreno GPU SMMU Add a special implementation for the SMMU attached to most Adreno GPU target triggered from the qcom,adreno-smmu compatible string. The new Adreno SMMU implementation will enable split pagetables (TTBR1) for the domain attached to the GPU device (SID 0) and hard code it context bank 0 so the GPU hardware can implement per-instance pagetables. Co-developed-by: Rob Clark <robdclark@chromium.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Rob Clark <robdclark@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20201109184728.2463097-2-jcrouse@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-10 02:47:25 +08:00
static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
{
struct adreno_smmu_priv *priv;
iommu/arm-smmu: Optimize ->tlb_flush_walk() for qcom implementation Currently for iommu_unmap() of large scatter-gather list with page size elements, the majority of time is spent in flushing of partial walks in __arm_lpae_unmap() which is a VA based TLB invalidation invalidating page-by-page on iommus like arm-smmu-v2 (TLBIVA). For example: to unmap a 32MB scatter-gather list with page size elements (8192 entries), there are 16->2MB buffer unmaps based on the pgsize (2MB for 4K granule) and each of 2MB will further result in 512 TLBIVAs (2MB/4K) resulting in a total of 8192 TLBIVAs (512*16) for 16->2MB causing a huge overhead. On qcom implementation, there are several performance improvements for TLB cache invalidations in HW like wait-for-safe (for realtime clients such as camera and display) and few others to allow for cache lookups/updates when TLBI is in progress for the same context bank. So the cost of over-invalidation is less compared to the unmap latency on several usecases like camera which deals with large buffers. So, ASID based TLB invalidations (TLBIASID) can be used to invalidate the entire context for partial walk flush thereby improving the unmap latency. For this example of 32MB scatter-gather list unmap, this change results in just 16 ASID based TLB invalidations (TLBIASIDs) as opposed to 8192 TLBIVAs thereby increasing the performance of unmaps drastically. Test on QTI SM8150 SoC for 10 iterations of iommu_{map_sg}/unmap: (average over 10 iterations) Before this optimization: size iommu_map_sg iommu_unmap 4K 2.067 us 1.854 us 64K 9.598 us 8.802 us 1M 148.890 us 130.718 us 2M 305.864 us 67.291 us 12M 1793.604 us 390.838 us 16M 2386.848 us 518.187 us 24M 3563.296 us 775.989 us 32M 4747.171 us 1033.364 us After this optimization: size iommu_map_sg iommu_unmap 4K 1.723 us 1.765 us 64K 9.880 us 8.869 us 1M 155.364 us 135.223 us 2M 303.906 us 5.385 us 12M 1786.557 us 21.250 us 16M 2391.890 us 27.437 us 24M 3570.895 us 39.937 us 32M 4755.234 us 51.797 us Real world data also shows big difference in unmap performance as below: There were reports of camera frame drops because of high overhead in iommu unmap without this optimization because of frequent unmaps issued by camera of about 100MB/s taking more than 100ms thereby causing frame drops. Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Link: https://lore.kernel.org/r/20210811160426.10312-1-saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2021-08-12 00:04:26 +08:00
smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;
iommu/arm-smmu-qcom: Add implementation for the adreno GPU SMMU Add a special implementation for the SMMU attached to most Adreno GPU target triggered from the qcom,adreno-smmu compatible string. The new Adreno SMMU implementation will enable split pagetables (TTBR1) for the domain attached to the GPU device (SID 0) and hard code it context bank 0 so the GPU hardware can implement per-instance pagetables. Co-developed-by: Rob Clark <robdclark@chromium.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Rob Clark <robdclark@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20201109184728.2463097-2-jcrouse@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-10 02:47:25 +08:00
/* Only enable split pagetables for the GPU device (SID 0) */
if (!qcom_adreno_smmu_is_gpu_device(dev))
return 0;
/*
* All targets that use the qcom,adreno-smmu compatible string *should*
* be AARCH64 stage 1 but double check because the arm-smmu code assumes
* that is the case when the TTBR1 quirk is enabled
*/
iommu/arm-smmu-qcom: Skip the TTBR1 quirk for db820c. db820c wants to use the qcom smmu path to get HUPCF set (which keeps the GPU from wedging and then sometimes wedging the kernel after a page fault), but it doesn't have separate pagetables support yet in drm/msm so we can't go all the way to the TTBR1 path. Signed-off-by: Eric Anholt <eric@anholt.net> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20210326231303.3071950-1-eric@anholt.net Signed-off-by: Will Deacon <will@kernel.org>
2021-03-27 07:13:02 +08:00
if (qcom_adreno_can_do_ttbr1(smmu_domain->smmu) &&
(smmu_domain->stage == ARM_SMMU_DOMAIN_S1) &&
iommu/arm-smmu-qcom: Add implementation for the adreno GPU SMMU Add a special implementation for the SMMU attached to most Adreno GPU target triggered from the qcom,adreno-smmu compatible string. The new Adreno SMMU implementation will enable split pagetables (TTBR1) for the domain attached to the GPU device (SID 0) and hard code it context bank 0 so the GPU hardware can implement per-instance pagetables. Co-developed-by: Rob Clark <robdclark@chromium.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Rob Clark <robdclark@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20201109184728.2463097-2-jcrouse@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-10 02:47:25 +08:00
(smmu_domain->cfg.fmt == ARM_SMMU_CTX_FMT_AARCH64))
pgtbl_cfg->quirks |= IO_PGTABLE_QUIRK_ARM_TTBR1;
/*
* Initialize private interface with GPU:
*/
priv = dev_get_drvdata(dev);
priv->cookie = smmu_domain;
priv->get_ttbr1_cfg = qcom_adreno_smmu_get_ttbr1_cfg;
priv->set_ttbr0_cfg = qcom_adreno_smmu_set_ttbr0_cfg;
iommu/arm-smmu-qcom: Add an adreno-smmu-priv callback to get pagefault info Add a callback in adreno-smmu-priv to read interesting SMMU registers to provide an opportunity for a richer debug experience in the GPU driver. Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Rob Clark <robdclark@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20210610214431.539029-3-robdclark@gmail.com Signed-off-by: Rob Clark <robdclark@chromium.org>
2021-06-11 05:44:10 +08:00
priv->get_fault_info = qcom_adreno_smmu_get_fault_info;
iommu/arm-smmu-qcom: Add stall support Add, via the adreno-smmu-priv interface, a way for the GPU to request the SMMU to stall translation on faults, and then later resume the translation, either retrying or terminating the current translation. This will be used on the GPU side to "freeze" the GPU while we snapshot useful state for devcoredump. Signed-off-by: Rob Clark <robdclark@chromium.org> Acked-by: Jordan Crouse <jordan@cosmicpenguin.net> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20210610214431.539029-5-robdclark@gmail.com Signed-off-by: Rob Clark <robdclark@chromium.org>
2021-06-11 05:44:12 +08:00
priv->set_stall = qcom_adreno_smmu_set_stall;
priv->resume_translation = qcom_adreno_smmu_resume_translation;
iommu/arm-smmu-qcom: Add implementation for the adreno GPU SMMU Add a special implementation for the SMMU attached to most Adreno GPU target triggered from the qcom,adreno-smmu compatible string. The new Adreno SMMU implementation will enable split pagetables (TTBR1) for the domain attached to the GPU device (SID 0) and hard code it context bank 0 so the GPU hardware can implement per-instance pagetables. Co-developed-by: Rob Clark <robdclark@chromium.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Rob Clark <robdclark@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20201109184728.2463097-2-jcrouse@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-10 02:47:25 +08:00
return 0;
}
iommu/arm-smmu: Mark qcom_smmu_client_of_match as possibly unused When CONFIG_OF=n of_match_device() gets pre-processed out of existence leaving qcom-smmu_client_of_match unused. Mark it as possibly unused to keep the compiler from warning in that case. Fixes: 0e764a01015d ("iommu/arm-smmu: Allow client devices to select direct mapping") Reported-by: kbuild test robot <lkp@intel.com> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/20200604203905.31964-1-jcrouse@codeaurora.org Signed-off-by: Joerg Roedel <jroedel@suse.de>
2020-06-05 04:39:04 +08:00
static const struct of_device_id qcom_smmu_client_of_match[] __maybe_unused = {
iommu/arm-smmu: Allow client devices to select direct mapping Some client devices want to directly map the IOMMU themselves instead of using the DMA domain. Allow those devices to opt in to direct mapping by way of a list of compatible strings. Co-developed-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/7cf1f64167b5545b7f42275395be1f1e2ea3a6ac.1587407458.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-04-21 02:33:51 +08:00
{ .compatible = "qcom,adreno" },
{ .compatible = "qcom,mdp4" },
{ .compatible = "qcom,mdss" },
{ .compatible = "qcom,sc7180-mdss" },
iommu/arm-smmu-qcom: Request direct mapping for modem device The modem remote processor has two access paths to DDR. One path is directly connected to DDR and another path goes through an SMMU. The SMMU path is configured to be a direct mapping because it's used by various peripherals in the modem subsystem. Typically this direct mapping is configured statically at EL2 by QHEE (Qualcomm's Hypervisor Execution Environment) before the kernel is entered. In certain firmware configuration, especially when the kernel is already in full control of the SMMU, defer programming the modem SIDs to the kernel. Let's add compatibles here so that we can have the kernel program the SIDs for the modem in these cases. Signed-off-by: Sibi Sankar <sibis@codeaurora.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Reviewed-by: Stephen Boyd <swboyd@chromium.org> Link: https://lore.kernel.org/r/20200511175532.25874-1-sibis@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-05-12 01:55:32 +08:00
{ .compatible = "qcom,sc7180-mss-pil" },
iommu/arm-smmu-qcom: Add SC7280 SMMU compatible Add compatible for SC7280 SMMU to use the Qualcomm Technologies, Inc. specific implementation. Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/53a50cd91c97b5b598a73941985b79b51acefa14.1623155117.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2021-06-08 20:30:06 +08:00
{ .compatible = "qcom,sc7280-mdss" },
iommu/arm-smmu-qcom: Request direct mapping for modem device The SID configuration requirement for Modem on SC7280 is similar to the ones found on SC7180/SDM845 SoCs. So, add the SC7280 modem compatible to the list to defer the programming of the modem SIDs to the kernel. Signed-off-by: Sibi Sankar <sibis@codeaurora.org> Reviewed-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Reviewed-by: Stephen Boyd <swboyd@chromium.org> Link: https://lore.kernel.org/r/1631886935-14691-5-git-send-email-sibis@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2021-09-17 21:55:29 +08:00
{ .compatible = "qcom,sc7280-mss-pil" },
iommu/arm-smmu-qcom: Add Qualcomm SC8180X impl The primary SMMU found in Qualcomm SC8180X platform needs to use the Qualcomm implementation, so add a specific compatible for this. Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20210121014005.1612382-2-bjorn.andersson@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2021-01-21 09:40:05 +08:00
{ .compatible = "qcom,sc8180x-mdss" },
iommu/arm-smmu: Allow client devices to select direct mapping Some client devices want to directly map the IOMMU themselves instead of using the DMA domain. Allow those devices to opt in to direct mapping by way of a list of compatible strings. Co-developed-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/7cf1f64167b5545b7f42275395be1f1e2ea3a6ac.1587407458.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-04-21 02:33:51 +08:00
{ .compatible = "qcom,sdm845-mdss" },
iommu/arm-smmu-qcom: Request direct mapping for modem device The modem remote processor has two access paths to DDR. One path is directly connected to DDR and another path goes through an SMMU. The SMMU path is configured to be a direct mapping because it's used by various peripherals in the modem subsystem. Typically this direct mapping is configured statically at EL2 by QHEE (Qualcomm's Hypervisor Execution Environment) before the kernel is entered. In certain firmware configuration, especially when the kernel is already in full control of the SMMU, defer programming the modem SIDs to the kernel. Let's add compatibles here so that we can have the kernel program the SIDs for the modem in these cases. Signed-off-by: Sibi Sankar <sibis@codeaurora.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Reviewed-by: Stephen Boyd <swboyd@chromium.org> Link: https://lore.kernel.org/r/20200511175532.25874-1-sibis@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-05-12 01:55:32 +08:00
{ .compatible = "qcom,sdm845-mss-pil" },
iommu/arm-smmu: Allow client devices to select direct mapping Some client devices want to directly map the IOMMU themselves instead of using the DMA domain. Allow those devices to opt in to direct mapping by way of a list of compatible strings. Co-developed-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/7cf1f64167b5545b7f42275395be1f1e2ea3a6ac.1587407458.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-04-21 02:33:51 +08:00
{ }
};
iommu/arm-smmu: Optimize ->tlb_flush_walk() for qcom implementation Currently for iommu_unmap() of large scatter-gather list with page size elements, the majority of time is spent in flushing of partial walks in __arm_lpae_unmap() which is a VA based TLB invalidation invalidating page-by-page on iommus like arm-smmu-v2 (TLBIVA). For example: to unmap a 32MB scatter-gather list with page size elements (8192 entries), there are 16->2MB buffer unmaps based on the pgsize (2MB for 4K granule) and each of 2MB will further result in 512 TLBIVAs (2MB/4K) resulting in a total of 8192 TLBIVAs (512*16) for 16->2MB causing a huge overhead. On qcom implementation, there are several performance improvements for TLB cache invalidations in HW like wait-for-safe (for realtime clients such as camera and display) and few others to allow for cache lookups/updates when TLBI is in progress for the same context bank. So the cost of over-invalidation is less compared to the unmap latency on several usecases like camera which deals with large buffers. So, ASID based TLB invalidations (TLBIASID) can be used to invalidate the entire context for partial walk flush thereby improving the unmap latency. For this example of 32MB scatter-gather list unmap, this change results in just 16 ASID based TLB invalidations (TLBIASIDs) as opposed to 8192 TLBIVAs thereby increasing the performance of unmaps drastically. Test on QTI SM8150 SoC for 10 iterations of iommu_{map_sg}/unmap: (average over 10 iterations) Before this optimization: size iommu_map_sg iommu_unmap 4K 2.067 us 1.854 us 64K 9.598 us 8.802 us 1M 148.890 us 130.718 us 2M 305.864 us 67.291 us 12M 1793.604 us 390.838 us 16M 2386.848 us 518.187 us 24M 3563.296 us 775.989 us 32M 4747.171 us 1033.364 us After this optimization: size iommu_map_sg iommu_unmap 4K 1.723 us 1.765 us 64K 9.880 us 8.869 us 1M 155.364 us 135.223 us 2M 303.906 us 5.385 us 12M 1786.557 us 21.250 us 16M 2391.890 us 27.437 us 24M 3570.895 us 39.937 us 32M 4755.234 us 51.797 us Real world data also shows big difference in unmap performance as below: There were reports of camera frame drops because of high overhead in iommu unmap without this optimization because of frequent unmaps issued by camera of about 100MB/s taking more than 100ms thereby causing frame drops. Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Link: https://lore.kernel.org/r/20210811160426.10312-1-saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2021-08-12 00:04:26 +08:00
static int qcom_smmu_init_context(struct arm_smmu_domain *smmu_domain,
struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
{
smmu_domain->cfg.flush_walk_prefer_tlbiasid = true;
return 0;
}
iommu/arm-smmu-qcom: Read back stream mappings The Qualcomm boot loader configures stream mapping for the peripherals that it accesses and in particular it sets up the stream mapping for the display controller to be allowed to scan out a splash screen or EFI framebuffer. Read back the stream mappings during initialization and make the arm-smmu driver maintain the streams in bypass mode. Cc: <stable@vger.kernel.org> Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org> Tested-by: Steev Klimaszewski <steev@kali.org> Acked-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/20201019182323.3162386-3-bjorn.andersson@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2020-10-20 02:23:22 +08:00
static int qcom_smmu_cfg_probe(struct arm_smmu_device *smmu)
{
iommu/arm-smmu-qcom: Implement S2CR quirk The firmware found in some Qualcomm platforms intercepts writes to S2CR in order to replace bypass type streams with fault; and ignore S2CR updates of type fault. Detect this behavior and implement a custom write_s2cr function in order to trick the firmware into supporting bypass streams by the means of configuring the stream for translation using a reserved and disabled context bank. Also circumvent the problem of configuring faulting streams by configuring the stream as bypass. Cc: <stable@vger.kernel.org> Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org> Tested-by: Steev Klimaszewski <steev@kali.org> Acked-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/20201019182323.3162386-4-bjorn.andersson@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2020-10-20 02:23:23 +08:00
unsigned int last_s2cr = ARM_SMMU_GR0_S2CR(smmu->num_mapping_groups - 1);
struct qcom_smmu *qsmmu = to_qcom_smmu(smmu);
u32 reg;
iommu/arm-smmu-qcom: Read back stream mappings The Qualcomm boot loader configures stream mapping for the peripherals that it accesses and in particular it sets up the stream mapping for the display controller to be allowed to scan out a splash screen or EFI framebuffer. Read back the stream mappings during initialization and make the arm-smmu driver maintain the streams in bypass mode. Cc: <stable@vger.kernel.org> Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org> Tested-by: Steev Klimaszewski <steev@kali.org> Acked-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/20201019182323.3162386-3-bjorn.andersson@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2020-10-20 02:23:22 +08:00
u32 smr;
int i;
iommu/arm-smmu-qcom: Implement S2CR quirk The firmware found in some Qualcomm platforms intercepts writes to S2CR in order to replace bypass type streams with fault; and ignore S2CR updates of type fault. Detect this behavior and implement a custom write_s2cr function in order to trick the firmware into supporting bypass streams by the means of configuring the stream for translation using a reserved and disabled context bank. Also circumvent the problem of configuring faulting streams by configuring the stream as bypass. Cc: <stable@vger.kernel.org> Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org> Tested-by: Steev Klimaszewski <steev@kali.org> Acked-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/20201019182323.3162386-4-bjorn.andersson@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2020-10-20 02:23:23 +08:00
/*
* With some firmware versions writes to S2CR of type FAULT are
* ignored, and writing BYPASS will end up written as FAULT in the
* register. Perform a write to S2CR to detect if this is the case and
* if so reserve a context bank to emulate bypass streams.
*/
reg = FIELD_PREP(ARM_SMMU_S2CR_TYPE, S2CR_TYPE_BYPASS) |
FIELD_PREP(ARM_SMMU_S2CR_CBNDX, 0xff) |
FIELD_PREP(ARM_SMMU_S2CR_PRIVCFG, S2CR_PRIVCFG_DEFAULT);
arm_smmu_gr0_write(smmu, last_s2cr, reg);
reg = arm_smmu_gr0_read(smmu, last_s2cr);
if (FIELD_GET(ARM_SMMU_S2CR_TYPE, reg) != S2CR_TYPE_BYPASS) {
qsmmu->bypass_quirk = true;
qsmmu->bypass_cbndx = smmu->num_context_banks - 1;
set_bit(qsmmu->bypass_cbndx, smmu->context_map);
iommu/arm-smmu-qcom: Initialize SCTLR of the bypass context On SM8150 it's occasionally observed that the boot hangs in between the writing of SMEs and context banks in arm_smmu_device_reset(). The problem seems to coincide with a display refresh happening after updating the stream mapping, but before clearing - and there by disabling translation - the context bank picked to emulate translation bypass. Resolve this by explicitly disabling the bypass context already in cfg_probe. Fixes: f9081b8ff593 ("iommu/arm-smmu-qcom: Implement S2CR quirk") Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20210106005038.4152731-1-bjorn.andersson@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2021-01-06 08:50:38 +08:00
arm_smmu_cb_write(smmu, qsmmu->bypass_cbndx, ARM_SMMU_CB_SCTLR, 0);
iommu/arm-smmu-qcom: Implement S2CR quirk The firmware found in some Qualcomm platforms intercepts writes to S2CR in order to replace bypass type streams with fault; and ignore S2CR updates of type fault. Detect this behavior and implement a custom write_s2cr function in order to trick the firmware into supporting bypass streams by the means of configuring the stream for translation using a reserved and disabled context bank. Also circumvent the problem of configuring faulting streams by configuring the stream as bypass. Cc: <stable@vger.kernel.org> Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org> Tested-by: Steev Klimaszewski <steev@kali.org> Acked-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/20201019182323.3162386-4-bjorn.andersson@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2020-10-20 02:23:23 +08:00
reg = FIELD_PREP(ARM_SMMU_CBAR_TYPE, CBAR_TYPE_S1_TRANS_S2_BYPASS);
arm_smmu_gr1_write(smmu, ARM_SMMU_GR1_CBAR(qsmmu->bypass_cbndx), reg);
}
iommu/arm-smmu-qcom: Read back stream mappings The Qualcomm boot loader configures stream mapping for the peripherals that it accesses and in particular it sets up the stream mapping for the display controller to be allowed to scan out a splash screen or EFI framebuffer. Read back the stream mappings during initialization and make the arm-smmu driver maintain the streams in bypass mode. Cc: <stable@vger.kernel.org> Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org> Tested-by: Steev Klimaszewski <steev@kali.org> Acked-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/20201019182323.3162386-3-bjorn.andersson@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2020-10-20 02:23:22 +08:00
for (i = 0; i < smmu->num_mapping_groups; i++) {
smr = arm_smmu_gr0_read(smmu, ARM_SMMU_GR0_SMR(i));
if (FIELD_GET(ARM_SMMU_SMR_VALID, smr)) {
iommu/arm-smmu-qcom: Fix mask extraction for bootloader programmed SMRs When extracting the mask for a SMR that was programmed by the bootloader, the SMR's valid bit is also extracted and is treated as part of the mask, which is not correct. Consider the scenario where an SMMU master whose context is determined by a bootloader programmed SMR is removed (omitting parts of device/driver core): ->iommu_release_device() -> arm_smmu_release_device() -> arm_smmu_master_free_smes() -> arm_smmu_free_sme() /* Assume that the SME is now free */ -> arm_smmu_write_sme() -> arm_smmu_write_smr() /* Construct SMR value using mask and SID */ Since the valid bit was considered as part of the mask, the SMR will be programmed as valid. Fix the SMR mask extraction step for bootloader programmed SMRs by masking out the valid bit when we know that we're already working with a valid SMR. Fixes: 07a7f2caaa5a ("iommu/arm-smmu-qcom: Read back stream mappings") Signed-off-by: Isaac J. Manjarres <isaacm@codeaurora.org> Cc: stable@vger.kernel.org Reviewed-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/1611611545-19055-1-git-send-email-isaacm@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2021-01-26 05:52:25 +08:00
/* Ignore valid bit for SMR mask extraction. */
smr &= ~ARM_SMMU_SMR_VALID;
iommu/arm-smmu-qcom: Read back stream mappings The Qualcomm boot loader configures stream mapping for the peripherals that it accesses and in particular it sets up the stream mapping for the display controller to be allowed to scan out a splash screen or EFI framebuffer. Read back the stream mappings during initialization and make the arm-smmu driver maintain the streams in bypass mode. Cc: <stable@vger.kernel.org> Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org> Tested-by: Steev Klimaszewski <steev@kali.org> Acked-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/20201019182323.3162386-3-bjorn.andersson@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2020-10-20 02:23:22 +08:00
smmu->smrs[i].id = FIELD_GET(ARM_SMMU_SMR_ID, smr);
smmu->smrs[i].mask = FIELD_GET(ARM_SMMU_SMR_MASK, smr);
smmu->smrs[i].valid = true;
smmu->s2crs[i].type = S2CR_TYPE_BYPASS;
smmu->s2crs[i].privcfg = S2CR_PRIVCFG_DEFAULT;
smmu->s2crs[i].cbndx = 0xff;
}
}
return 0;
}
iommu/arm-smmu-qcom: Implement S2CR quirk The firmware found in some Qualcomm platforms intercepts writes to S2CR in order to replace bypass type streams with fault; and ignore S2CR updates of type fault. Detect this behavior and implement a custom write_s2cr function in order to trick the firmware into supporting bypass streams by the means of configuring the stream for translation using a reserved and disabled context bank. Also circumvent the problem of configuring faulting streams by configuring the stream as bypass. Cc: <stable@vger.kernel.org> Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org> Tested-by: Steev Klimaszewski <steev@kali.org> Acked-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/20201019182323.3162386-4-bjorn.andersson@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2020-10-20 02:23:23 +08:00
static void qcom_smmu_write_s2cr(struct arm_smmu_device *smmu, int idx)
{
struct arm_smmu_s2cr *s2cr = smmu->s2crs + idx;
struct qcom_smmu *qsmmu = to_qcom_smmu(smmu);
u32 cbndx = s2cr->cbndx;
u32 type = s2cr->type;
u32 reg;
if (qsmmu->bypass_quirk) {
if (type == S2CR_TYPE_BYPASS) {
/*
* Firmware with quirky S2CR handling will substitute
* BYPASS writes with FAULT, so point the stream to the
* reserved context bank and ask for translation on the
* stream
*/
type = S2CR_TYPE_TRANS;
cbndx = qsmmu->bypass_cbndx;
} else if (type == S2CR_TYPE_FAULT) {
/*
* Firmware with quirky S2CR handling will ignore FAULT
* writes, so trick it to write FAULT by asking for a
* BYPASS.
*/
type = S2CR_TYPE_BYPASS;
cbndx = 0xff;
}
}
reg = FIELD_PREP(ARM_SMMU_S2CR_TYPE, type) |
FIELD_PREP(ARM_SMMU_S2CR_CBNDX, cbndx) |
FIELD_PREP(ARM_SMMU_S2CR_PRIVCFG, s2cr->privcfg);
arm_smmu_gr0_write(smmu, ARM_SMMU_GR0_S2CR(idx), reg);
}
iommu/arm-smmu: Allow client devices to select direct mapping Some client devices want to directly map the IOMMU themselves instead of using the DMA domain. Allow those devices to opt in to direct mapping by way of a list of compatible strings. Co-developed-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/7cf1f64167b5545b7f42275395be1f1e2ea3a6ac.1587407458.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-04-21 02:33:51 +08:00
static int qcom_smmu_def_domain_type(struct device *dev)
{
const struct of_device_id *match =
of_match_device(qcom_smmu_client_of_match, dev);
return match ? IOMMU_DOMAIN_IDENTITY : 0;
}
iommu: arm-smmu-impl: Add sdm845 implementation hook Add reset hook for sdm845 based platforms to turn off the wait-for-safe sequence. Understanding how wait-for-safe logic affects USB and UFS performance on MTP845 and DB845 boards: Qcom's implementation of arm,mmu-500 adds a WAIT-FOR-SAFE logic to address under-performance issues in real-time clients, such as Display, and Camera. On receiving an invalidation requests, the SMMU forwards SAFE request to these clients and waits for SAFE ack signal from real-time clients. The SAFE signal from such clients is used to qualify the start of invalidation. This logic is controlled by chicken bits, one for each - MDP (display), IFE0, and IFE1 (camera), that can be accessed only from secure software on sdm845. This configuration, however, degrades the performance of non-real time clients, such as USB, and UFS etc. This happens because, with wait-for-safe logic enabled the hardware tries to throttle non-real time clients while waiting for SAFE ack signals from real-time clients. On mtp845 and db845 devices, with wait-for-safe logic enabled by the bootloaders we see degraded performance of USB and UFS when kernel enables the smmu stage-1 translations for these clients. Turn off this wait-for-safe logic from the kernel gets us back the perf of USB and UFS devices until we re-visit this when we start seeing perf issues on display/camera on upstream supported SDM845 platforms. The bootloaders on these boards implement secure monitor callbacks to handle a specific command - QCOM_SCM_SVC_SMMU_PROGRAM with which the logic can be toggled. There are other boards such as cheza whose bootloaders don't enable this logic. Such boards don't implement callbacks to handle the specific SCM call so disabling this logic for such boards will be a no-op. This change is inspired by the downstream change from Patrick Daly to address performance issues with display and camera by handling this wait-for-safe within separte io-pagetable ops to do TLB maintenance. So a big thanks to him for the change and for all the offline discussions. Without this change the UFS reads are pretty slow: $ time dd if=/dev/sda of=/dev/zero bs=1048576 count=10 conv=sync 10+0 records in 10+0 records out 10485760 bytes (10.0MB) copied, 22.394903 seconds, 457.2KB/s real 0m 22.39s user 0m 0.00s sys 0m 0.01s With this change they are back to rock! $ time dd if=/dev/sda of=/dev/zero bs=1048576 count=300 conv=sync 300+0 records in 300+0 records out 314572800 bytes (300.0MB) copied, 1.030541 seconds, 291.1MB/s real 0m 1.03s user 0m 0.00s sys 0m 0.54s Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Reviewed-by: Stephen Boyd <swboyd@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-09-20 16:04:29 +08:00
static int qcom_sdm845_smmu500_reset(struct arm_smmu_device *smmu)
{
int ret;
/*
* To address performance degradation in non-real time clients,
* such as USB and UFS, turn off wait-for-safe on sdm845 based boards,
* such as MTP and db845, whose firmwares implement secure monitor
* call handlers to turn on/off the wait-for-safe logic.
*/
ret = qcom_scm_qsmmu500_wait_safe_toggle(0);
if (ret)
dev_warn(smmu->dev, "Failed to turn off SAFE logic\n");
return ret;
}
iommu: arm-smmu-impl: Convert to a generic reset implementation Currently the QCOM specific smmu reset implementation is very specific to SDM845 SoC and has a wait-for-safe logic which may not be required for other SoCs. So move the SDM845 specific logic to its specific reset function. Also add SC7180 SMMU compatible for calling into QCOM specific implementation. Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Reviewed-by: Stephen Boyd <swboyd@chromium.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/d24a0278021bc0b2732636c5728efe55e7318a8b.1587407458.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-04-21 02:33:49 +08:00
static int qcom_smmu500_reset(struct arm_smmu_device *smmu)
{
const struct device_node *np = smmu->dev->of_node;
arm_mmu500_reset(smmu);
if (of_device_is_compatible(np, "qcom,sdm845-smmu-500"))
return qcom_sdm845_smmu500_reset(smmu);
return 0;
}
iommu: arm-smmu-impl: Add sdm845 implementation hook Add reset hook for sdm845 based platforms to turn off the wait-for-safe sequence. Understanding how wait-for-safe logic affects USB and UFS performance on MTP845 and DB845 boards: Qcom's implementation of arm,mmu-500 adds a WAIT-FOR-SAFE logic to address under-performance issues in real-time clients, such as Display, and Camera. On receiving an invalidation requests, the SMMU forwards SAFE request to these clients and waits for SAFE ack signal from real-time clients. The SAFE signal from such clients is used to qualify the start of invalidation. This logic is controlled by chicken bits, one for each - MDP (display), IFE0, and IFE1 (camera), that can be accessed only from secure software on sdm845. This configuration, however, degrades the performance of non-real time clients, such as USB, and UFS etc. This happens because, with wait-for-safe logic enabled the hardware tries to throttle non-real time clients while waiting for SAFE ack signals from real-time clients. On mtp845 and db845 devices, with wait-for-safe logic enabled by the bootloaders we see degraded performance of USB and UFS when kernel enables the smmu stage-1 translations for these clients. Turn off this wait-for-safe logic from the kernel gets us back the perf of USB and UFS devices until we re-visit this when we start seeing perf issues on display/camera on upstream supported SDM845 platforms. The bootloaders on these boards implement secure monitor callbacks to handle a specific command - QCOM_SCM_SVC_SMMU_PROGRAM with which the logic can be toggled. There are other boards such as cheza whose bootloaders don't enable this logic. Such boards don't implement callbacks to handle the specific SCM call so disabling this logic for such boards will be a no-op. This change is inspired by the downstream change from Patrick Daly to address performance issues with display and camera by handling this wait-for-safe within separte io-pagetable ops to do TLB maintenance. So a big thanks to him for the change and for all the offline discussions. Without this change the UFS reads are pretty slow: $ time dd if=/dev/sda of=/dev/zero bs=1048576 count=10 conv=sync 10+0 records in 10+0 records out 10485760 bytes (10.0MB) copied, 22.394903 seconds, 457.2KB/s real 0m 22.39s user 0m 0.00s sys 0m 0.01s With this change they are back to rock! $ time dd if=/dev/sda of=/dev/zero bs=1048576 count=300 conv=sync 300+0 records in 300+0 records out 314572800 bytes (300.0MB) copied, 1.030541 seconds, 291.1MB/s real 0m 1.03s user 0m 0.00s sys 0m 0.54s Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Reviewed-by: Stephen Boyd <swboyd@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-09-20 16:04:29 +08:00
static const struct arm_smmu_impl qcom_smmu_impl = {
iommu/arm-smmu: Optimize ->tlb_flush_walk() for qcom implementation Currently for iommu_unmap() of large scatter-gather list with page size elements, the majority of time is spent in flushing of partial walks in __arm_lpae_unmap() which is a VA based TLB invalidation invalidating page-by-page on iommus like arm-smmu-v2 (TLBIVA). For example: to unmap a 32MB scatter-gather list with page size elements (8192 entries), there are 16->2MB buffer unmaps based on the pgsize (2MB for 4K granule) and each of 2MB will further result in 512 TLBIVAs (2MB/4K) resulting in a total of 8192 TLBIVAs (512*16) for 16->2MB causing a huge overhead. On qcom implementation, there are several performance improvements for TLB cache invalidations in HW like wait-for-safe (for realtime clients such as camera and display) and few others to allow for cache lookups/updates when TLBI is in progress for the same context bank. So the cost of over-invalidation is less compared to the unmap latency on several usecases like camera which deals with large buffers. So, ASID based TLB invalidations (TLBIASID) can be used to invalidate the entire context for partial walk flush thereby improving the unmap latency. For this example of 32MB scatter-gather list unmap, this change results in just 16 ASID based TLB invalidations (TLBIASIDs) as opposed to 8192 TLBIVAs thereby increasing the performance of unmaps drastically. Test on QTI SM8150 SoC for 10 iterations of iommu_{map_sg}/unmap: (average over 10 iterations) Before this optimization: size iommu_map_sg iommu_unmap 4K 2.067 us 1.854 us 64K 9.598 us 8.802 us 1M 148.890 us 130.718 us 2M 305.864 us 67.291 us 12M 1793.604 us 390.838 us 16M 2386.848 us 518.187 us 24M 3563.296 us 775.989 us 32M 4747.171 us 1033.364 us After this optimization: size iommu_map_sg iommu_unmap 4K 1.723 us 1.765 us 64K 9.880 us 8.869 us 1M 155.364 us 135.223 us 2M 303.906 us 5.385 us 12M 1786.557 us 21.250 us 16M 2391.890 us 27.437 us 24M 3570.895 us 39.937 us 32M 4755.234 us 51.797 us Real world data also shows big difference in unmap performance as below: There were reports of camera frame drops because of high overhead in iommu unmap without this optimization because of frequent unmaps issued by camera of about 100MB/s taking more than 100ms thereby causing frame drops. Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Link: https://lore.kernel.org/r/20210811160426.10312-1-saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2021-08-12 00:04:26 +08:00
.init_context = qcom_smmu_init_context,
iommu/arm-smmu-qcom: Read back stream mappings The Qualcomm boot loader configures stream mapping for the peripherals that it accesses and in particular it sets up the stream mapping for the display controller to be allowed to scan out a splash screen or EFI framebuffer. Read back the stream mappings during initialization and make the arm-smmu driver maintain the streams in bypass mode. Cc: <stable@vger.kernel.org> Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org> Tested-by: Steev Klimaszewski <steev@kali.org> Acked-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/20201019182323.3162386-3-bjorn.andersson@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2020-10-20 02:23:22 +08:00
.cfg_probe = qcom_smmu_cfg_probe,
iommu/arm-smmu: Allow client devices to select direct mapping Some client devices want to directly map the IOMMU themselves instead of using the DMA domain. Allow those devices to opt in to direct mapping by way of a list of compatible strings. Co-developed-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/7cf1f64167b5545b7f42275395be1f1e2ea3a6ac.1587407458.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-04-21 02:33:51 +08:00
.def_domain_type = qcom_smmu_def_domain_type,
iommu: arm-smmu-impl: Convert to a generic reset implementation Currently the QCOM specific smmu reset implementation is very specific to SDM845 SoC and has a wait-for-safe logic which may not be required for other SoCs. So move the SDM845 specific logic to its specific reset function. Also add SC7180 SMMU compatible for calling into QCOM specific implementation. Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Reviewed-by: Stephen Boyd <swboyd@chromium.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/d24a0278021bc0b2732636c5728efe55e7318a8b.1587407458.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-04-21 02:33:49 +08:00
.reset = qcom_smmu500_reset,
iommu/arm-smmu-qcom: Implement S2CR quirk The firmware found in some Qualcomm platforms intercepts writes to S2CR in order to replace bypass type streams with fault; and ignore S2CR updates of type fault. Detect this behavior and implement a custom write_s2cr function in order to trick the firmware into supporting bypass streams by the means of configuring the stream for translation using a reserved and disabled context bank. Also circumvent the problem of configuring faulting streams by configuring the stream as bypass. Cc: <stable@vger.kernel.org> Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org> Tested-by: Steev Klimaszewski <steev@kali.org> Acked-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/20201019182323.3162386-4-bjorn.andersson@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2020-10-20 02:23:23 +08:00
.write_s2cr = qcom_smmu_write_s2cr,
iommu: arm-smmu-impl: Add sdm845 implementation hook Add reset hook for sdm845 based platforms to turn off the wait-for-safe sequence. Understanding how wait-for-safe logic affects USB and UFS performance on MTP845 and DB845 boards: Qcom's implementation of arm,mmu-500 adds a WAIT-FOR-SAFE logic to address under-performance issues in real-time clients, such as Display, and Camera. On receiving an invalidation requests, the SMMU forwards SAFE request to these clients and waits for SAFE ack signal from real-time clients. The SAFE signal from such clients is used to qualify the start of invalidation. This logic is controlled by chicken bits, one for each - MDP (display), IFE0, and IFE1 (camera), that can be accessed only from secure software on sdm845. This configuration, however, degrades the performance of non-real time clients, such as USB, and UFS etc. This happens because, with wait-for-safe logic enabled the hardware tries to throttle non-real time clients while waiting for SAFE ack signals from real-time clients. On mtp845 and db845 devices, with wait-for-safe logic enabled by the bootloaders we see degraded performance of USB and UFS when kernel enables the smmu stage-1 translations for these clients. Turn off this wait-for-safe logic from the kernel gets us back the perf of USB and UFS devices until we re-visit this when we start seeing perf issues on display/camera on upstream supported SDM845 platforms. The bootloaders on these boards implement secure monitor callbacks to handle a specific command - QCOM_SCM_SVC_SMMU_PROGRAM with which the logic can be toggled. There are other boards such as cheza whose bootloaders don't enable this logic. Such boards don't implement callbacks to handle the specific SCM call so disabling this logic for such boards will be a no-op. This change is inspired by the downstream change from Patrick Daly to address performance issues with display and camera by handling this wait-for-safe within separte io-pagetable ops to do TLB maintenance. So a big thanks to him for the change and for all the offline discussions. Without this change the UFS reads are pretty slow: $ time dd if=/dev/sda of=/dev/zero bs=1048576 count=10 conv=sync 10+0 records in 10+0 records out 10485760 bytes (10.0MB) copied, 22.394903 seconds, 457.2KB/s real 0m 22.39s user 0m 0.00s sys 0m 0.01s With this change they are back to rock! $ time dd if=/dev/sda of=/dev/zero bs=1048576 count=300 conv=sync 300+0 records in 300+0 records out 314572800 bytes (300.0MB) copied, 1.030541 seconds, 291.1MB/s real 0m 1.03s user 0m 0.00s sys 0m 0.54s Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Reviewed-by: Stephen Boyd <swboyd@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-09-20 16:04:29 +08:00
};
iommu/arm-smmu-qcom: Add implementation for the adreno GPU SMMU Add a special implementation for the SMMU attached to most Adreno GPU target triggered from the qcom,adreno-smmu compatible string. The new Adreno SMMU implementation will enable split pagetables (TTBR1) for the domain attached to the GPU device (SID 0) and hard code it context bank 0 so the GPU hardware can implement per-instance pagetables. Co-developed-by: Rob Clark <robdclark@chromium.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Rob Clark <robdclark@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20201109184728.2463097-2-jcrouse@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-10 02:47:25 +08:00
static const struct arm_smmu_impl qcom_adreno_smmu_impl = {
.init_context = qcom_adreno_smmu_init_context,
.def_domain_type = qcom_smmu_def_domain_type,
.reset = qcom_smmu500_reset,
.alloc_context_bank = qcom_adreno_smmu_alloc_context_bank,
iommu/arm-smmu: Add a way for implementations to influence SCTLR For the Adreno GPU's SMMU, we want SCTLR.HUPCF set to ensure that pending translations are not terminated on iova fault. Otherwise a terminated CP read could hang the GPU by returning invalid command-stream data. Add a hook to for the implementation to modify the sctlr value if it wishes. Co-developed-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Rob Clark <robdclark@chromium.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Link: https://lore.kernel.org/r/20201109184728.2463097-3-jcrouse@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-10 02:47:26 +08:00
.write_sctlr = qcom_adreno_smmu_write_sctlr,
iommu/arm-smmu-qcom: Add implementation for the adreno GPU SMMU Add a special implementation for the SMMU attached to most Adreno GPU target triggered from the qcom,adreno-smmu compatible string. The new Adreno SMMU implementation will enable split pagetables (TTBR1) for the domain attached to the GPU device (SID 0) and hard code it context bank 0 so the GPU hardware can implement per-instance pagetables. Co-developed-by: Rob Clark <robdclark@chromium.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Rob Clark <robdclark@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20201109184728.2463097-2-jcrouse@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-10 02:47:25 +08:00
};
static struct arm_smmu_device *qcom_smmu_create(struct arm_smmu_device *smmu,
const struct arm_smmu_impl *impl)
iommu: arm-smmu-impl: Add sdm845 implementation hook Add reset hook for sdm845 based platforms to turn off the wait-for-safe sequence. Understanding how wait-for-safe logic affects USB and UFS performance on MTP845 and DB845 boards: Qcom's implementation of arm,mmu-500 adds a WAIT-FOR-SAFE logic to address under-performance issues in real-time clients, such as Display, and Camera. On receiving an invalidation requests, the SMMU forwards SAFE request to these clients and waits for SAFE ack signal from real-time clients. The SAFE signal from such clients is used to qualify the start of invalidation. This logic is controlled by chicken bits, one for each - MDP (display), IFE0, and IFE1 (camera), that can be accessed only from secure software on sdm845. This configuration, however, degrades the performance of non-real time clients, such as USB, and UFS etc. This happens because, with wait-for-safe logic enabled the hardware tries to throttle non-real time clients while waiting for SAFE ack signals from real-time clients. On mtp845 and db845 devices, with wait-for-safe logic enabled by the bootloaders we see degraded performance of USB and UFS when kernel enables the smmu stage-1 translations for these clients. Turn off this wait-for-safe logic from the kernel gets us back the perf of USB and UFS devices until we re-visit this when we start seeing perf issues on display/camera on upstream supported SDM845 platforms. The bootloaders on these boards implement secure monitor callbacks to handle a specific command - QCOM_SCM_SVC_SMMU_PROGRAM with which the logic can be toggled. There are other boards such as cheza whose bootloaders don't enable this logic. Such boards don't implement callbacks to handle the specific SCM call so disabling this logic for such boards will be a no-op. This change is inspired by the downstream change from Patrick Daly to address performance issues with display and camera by handling this wait-for-safe within separte io-pagetable ops to do TLB maintenance. So a big thanks to him for the change and for all the offline discussions. Without this change the UFS reads are pretty slow: $ time dd if=/dev/sda of=/dev/zero bs=1048576 count=10 conv=sync 10+0 records in 10+0 records out 10485760 bytes (10.0MB) copied, 22.394903 seconds, 457.2KB/s real 0m 22.39s user 0m 0.00s sys 0m 0.01s With this change they are back to rock! $ time dd if=/dev/sda of=/dev/zero bs=1048576 count=300 conv=sync 300+0 records in 300+0 records out 314572800 bytes (300.0MB) copied, 1.030541 seconds, 291.1MB/s real 0m 1.03s user 0m 0.00s sys 0m 0.54s Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Reviewed-by: Stephen Boyd <swboyd@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-09-20 16:04:29 +08:00
{
struct qcom_smmu *qsmmu;
arm-smmu-qcom: Ensure the qcom_scm driver has finished probing Robin Murphy pointed out that if the arm-smmu driver probes before the qcom_scm driver, we may call qcom_scm_qsmmu500_wait_safe_toggle() before the __scm is initialized. Now, getting this to happen is a bit contrived, as in my efforts it required enabling asynchronous probing for both drivers, moving the firmware dts node to the end of the dtsi file, as well as forcing a long delay in the qcom_scm_probe function. With those tweaks we ran into the following crash: [ 2.631040] arm-smmu 15000000.iommu: Stage-1: 48-bit VA -> 48-bit IPA [ 2.633372] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000 ... [ 2.633402] [0000000000000000] user address but active_mm is swapper [ 2.633409] Internal error: Oops: 96000005 [#1] PREEMPT SMP [ 2.633415] Modules linked in: [ 2.633427] CPU: 5 PID: 117 Comm: kworker/u16:2 Tainted: G W 5.10.0-rc1-mainline-00025-g272a618fc36-dirty #3971 [ 2.633430] Hardware name: Thundercomm Dragonboard 845c (DT) [ 2.633448] Workqueue: events_unbound async_run_entry_fn [ 2.633456] pstate: 80c00005 (Nzcv daif +PAN +UAO -TCO BTYPE=--) [ 2.633465] pc : qcom_scm_qsmmu500_wait_safe_toggle+0x78/0xb0 [ 2.633473] lr : qcom_smmu500_reset+0x58/0x78 [ 2.633476] sp : ffffffc0105a3b60 ... [ 2.633567] Call trace: [ 2.633572] qcom_scm_qsmmu500_wait_safe_toggle+0x78/0xb0 [ 2.633576] qcom_smmu500_reset+0x58/0x78 [ 2.633581] arm_smmu_device_reset+0x194/0x270 [ 2.633585] arm_smmu_device_probe+0xc94/0xeb8 [ 2.633592] platform_drv_probe+0x58/0xa8 [ 2.633597] really_probe+0xec/0x398 [ 2.633601] driver_probe_device+0x5c/0xb8 [ 2.633606] __driver_attach_async_helper+0x64/0x88 [ 2.633610] async_run_entry_fn+0x4c/0x118 [ 2.633617] process_one_work+0x20c/0x4b0 [ 2.633621] worker_thread+0x48/0x460 [ 2.633628] kthread+0x14c/0x158 [ 2.633634] ret_from_fork+0x10/0x18 [ 2.633642] Code: a9034fa0 d0007f73 29107fa0 91342273 (f9400020) To avoid this, this patch adds a check on qcom_scm_is_available() in the qcom_smmu_impl_init() function, returning -EPROBE_DEFER if its not ready. This allows the driver to try to probe again later after qcom_scm has finished probing. Reported-by: Robin Murphy <robin.murphy@arm.com> Signed-off-by: John Stultz <john.stultz@linaro.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Cc: Robin Murphy <robin.murphy@arm.com> Cc: Will Deacon <will@kernel.org> Cc: Andy Gross <agross@kernel.org> Cc: Maulik Shah <mkshah@codeaurora.org> Cc: Bjorn Andersson <bjorn.andersson@linaro.org> Cc: Saravana Kannan <saravanak@google.com> Cc: Marc Zyngier <maz@kernel.org> Cc: Lina Iyer <ilina@codeaurora.org> Cc: iommu@lists.linux-foundation.org Cc: linux-arm-msm <linux-arm-msm@vger.kernel.org> Link: https://lore.kernel.org/r/20201112220520.48159-1-john.stultz@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-13 06:05:19 +08:00
/* Check to make sure qcom_scm has finished probing */
if (!qcom_scm_is_available())
return ERR_PTR(-EPROBE_DEFER);
iommu/arm-smmu: Use new devm_krealloc() The implementation-specific subclassing of struct arm_smmu_device really wanted an appropriate version of realloc(). Now that one exists, take full advantage of it to clarify what's actually being done here. Signed-off-by: Robin Murphy <robin.murphy@arm.com> Link: https://lore.kernel.org/r/355e8d70c7f47d462d85b386aa09f2b5c655f023.1603713428.git.robin.murphy@arm.com Signed-off-by: Will Deacon <will@kernel.org>
2020-10-26 20:00:22 +08:00
qsmmu = devm_krealloc(smmu->dev, smmu, sizeof(*qsmmu), GFP_KERNEL);
iommu: arm-smmu-impl: Add sdm845 implementation hook Add reset hook for sdm845 based platforms to turn off the wait-for-safe sequence. Understanding how wait-for-safe logic affects USB and UFS performance on MTP845 and DB845 boards: Qcom's implementation of arm,mmu-500 adds a WAIT-FOR-SAFE logic to address under-performance issues in real-time clients, such as Display, and Camera. On receiving an invalidation requests, the SMMU forwards SAFE request to these clients and waits for SAFE ack signal from real-time clients. The SAFE signal from such clients is used to qualify the start of invalidation. This logic is controlled by chicken bits, one for each - MDP (display), IFE0, and IFE1 (camera), that can be accessed only from secure software on sdm845. This configuration, however, degrades the performance of non-real time clients, such as USB, and UFS etc. This happens because, with wait-for-safe logic enabled the hardware tries to throttle non-real time clients while waiting for SAFE ack signals from real-time clients. On mtp845 and db845 devices, with wait-for-safe logic enabled by the bootloaders we see degraded performance of USB and UFS when kernel enables the smmu stage-1 translations for these clients. Turn off this wait-for-safe logic from the kernel gets us back the perf of USB and UFS devices until we re-visit this when we start seeing perf issues on display/camera on upstream supported SDM845 platforms. The bootloaders on these boards implement secure monitor callbacks to handle a specific command - QCOM_SCM_SVC_SMMU_PROGRAM with which the logic can be toggled. There are other boards such as cheza whose bootloaders don't enable this logic. Such boards don't implement callbacks to handle the specific SCM call so disabling this logic for such boards will be a no-op. This change is inspired by the downstream change from Patrick Daly to address performance issues with display and camera by handling this wait-for-safe within separte io-pagetable ops to do TLB maintenance. So a big thanks to him for the change and for all the offline discussions. Without this change the UFS reads are pretty slow: $ time dd if=/dev/sda of=/dev/zero bs=1048576 count=10 conv=sync 10+0 records in 10+0 records out 10485760 bytes (10.0MB) copied, 22.394903 seconds, 457.2KB/s real 0m 22.39s user 0m 0.00s sys 0m 0.01s With this change they are back to rock! $ time dd if=/dev/sda of=/dev/zero bs=1048576 count=300 conv=sync 300+0 records in 300+0 records out 314572800 bytes (300.0MB) copied, 1.030541 seconds, 291.1MB/s real 0m 1.03s user 0m 0.00s sys 0m 0.54s Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Reviewed-by: Stephen Boyd <swboyd@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-09-20 16:04:29 +08:00
if (!qsmmu)
return ERR_PTR(-ENOMEM);
iommu/arm-smmu-qcom: Add implementation for the adreno GPU SMMU Add a special implementation for the SMMU attached to most Adreno GPU target triggered from the qcom,adreno-smmu compatible string. The new Adreno SMMU implementation will enable split pagetables (TTBR1) for the domain attached to the GPU device (SID 0) and hard code it context bank 0 so the GPU hardware can implement per-instance pagetables. Co-developed-by: Rob Clark <robdclark@chromium.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Rob Clark <robdclark@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20201109184728.2463097-2-jcrouse@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-10 02:47:25 +08:00
qsmmu->smmu.impl = impl;
iommu: arm-smmu-impl: Add sdm845 implementation hook Add reset hook for sdm845 based platforms to turn off the wait-for-safe sequence. Understanding how wait-for-safe logic affects USB and UFS performance on MTP845 and DB845 boards: Qcom's implementation of arm,mmu-500 adds a WAIT-FOR-SAFE logic to address under-performance issues in real-time clients, such as Display, and Camera. On receiving an invalidation requests, the SMMU forwards SAFE request to these clients and waits for SAFE ack signal from real-time clients. The SAFE signal from such clients is used to qualify the start of invalidation. This logic is controlled by chicken bits, one for each - MDP (display), IFE0, and IFE1 (camera), that can be accessed only from secure software on sdm845. This configuration, however, degrades the performance of non-real time clients, such as USB, and UFS etc. This happens because, with wait-for-safe logic enabled the hardware tries to throttle non-real time clients while waiting for SAFE ack signals from real-time clients. On mtp845 and db845 devices, with wait-for-safe logic enabled by the bootloaders we see degraded performance of USB and UFS when kernel enables the smmu stage-1 translations for these clients. Turn off this wait-for-safe logic from the kernel gets us back the perf of USB and UFS devices until we re-visit this when we start seeing perf issues on display/camera on upstream supported SDM845 platforms. The bootloaders on these boards implement secure monitor callbacks to handle a specific command - QCOM_SCM_SVC_SMMU_PROGRAM with which the logic can be toggled. There are other boards such as cheza whose bootloaders don't enable this logic. Such boards don't implement callbacks to handle the specific SCM call so disabling this logic for such boards will be a no-op. This change is inspired by the downstream change from Patrick Daly to address performance issues with display and camera by handling this wait-for-safe within separte io-pagetable ops to do TLB maintenance. So a big thanks to him for the change and for all the offline discussions. Without this change the UFS reads are pretty slow: $ time dd if=/dev/sda of=/dev/zero bs=1048576 count=10 conv=sync 10+0 records in 10+0 records out 10485760 bytes (10.0MB) copied, 22.394903 seconds, 457.2KB/s real 0m 22.39s user 0m 0.00s sys 0m 0.01s With this change they are back to rock! $ time dd if=/dev/sda of=/dev/zero bs=1048576 count=300 conv=sync 300+0 records in 300+0 records out 314572800 bytes (300.0MB) copied, 1.030541 seconds, 291.1MB/s real 0m 1.03s user 0m 0.00s sys 0m 0.54s Signed-off-by: Vivek Gautam <vivek.gautam@codeaurora.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Reviewed-by: Stephen Boyd <swboyd@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Signed-off-by: Will Deacon <will@kernel.org>
2019-09-20 16:04:29 +08:00
return &qsmmu->smmu;
}
iommu/arm-smmu-qcom: Add implementation for the adreno GPU SMMU Add a special implementation for the SMMU attached to most Adreno GPU target triggered from the qcom,adreno-smmu compatible string. The new Adreno SMMU implementation will enable split pagetables (TTBR1) for the domain attached to the GPU device (SID 0) and hard code it context bank 0 so the GPU hardware can implement per-instance pagetables. Co-developed-by: Rob Clark <robdclark@chromium.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Rob Clark <robdclark@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20201109184728.2463097-2-jcrouse@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-10 02:47:25 +08:00
iommu: arm-smmu-impl: Use table to list QCOM implementations Use table and of_match_node() to match qcom implementation instead of multiple of_device_compatible() calls for each QCOM SMMU implementation. Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/4e11899bc02102a6e6155db215911e8b5aaba950.1606287059.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-25 15:00:17 +08:00
static const struct of_device_id __maybe_unused qcom_smmu_impl_of_match[] = {
iommu: arm-smmu-qcom: Add sdm630/msm8998 compatibles for qcom quirks SDM630 and MSM8998 are among the SoCs that use Qualcomm's implementation of SMMUv2 which has already proven to be problematic over the years. Add their compatibles to the lookup list to prevent the platforms from being shut down by the hypervisor at MMU probe. Signed-off-by: Konrad Dybcio <konrad.dybcio@somainline.org> Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org> Link: https://lore.kernel.org/r/20210109165622.149777-1-konrad.dybcio@somainline.org Signed-off-by: Will Deacon <will@kernel.org>
2021-01-10 00:56:21 +08:00
{ .compatible = "qcom,msm8998-smmu-v2" },
iommu: arm-smmu-qcom: Add compatible for QCM2290 Add compatible for QCM2290 SMMU to use the Qualcomm Technologies, Inc. specific implementation. Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Signed-off-by: Loic Poulain <loic.poulain@linaro.org> Link: https://lore.kernel.org/r/1633096832-7762-1-git-send-email-loic.poulain@linaro.org [will: Sort by alphabetical order, add commit message] Signed-off-by: Will Deacon <will@kernel.org>
2021-10-01 22:00:31 +08:00
{ .compatible = "qcom,qcm2290-smmu-500" },
iommu: arm-smmu-impl: Use table to list QCOM implementations Use table and of_match_node() to match qcom implementation instead of multiple of_device_compatible() calls for each QCOM SMMU implementation. Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/4e11899bc02102a6e6155db215911e8b5aaba950.1606287059.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-25 15:00:17 +08:00
{ .compatible = "qcom,sc7180-smmu-500" },
iommu/arm-smmu-qcom: Add SC7280 SMMU compatible Add compatible for SC7280 SMMU to use the Qualcomm Technologies, Inc. specific implementation. Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/53a50cd91c97b5b598a73941985b79b51acefa14.1623155117.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2021-06-08 20:30:06 +08:00
{ .compatible = "qcom,sc7280-smmu-500" },
iommu/arm-smmu-qcom: Add Qualcomm SC8180X impl The primary SMMU found in Qualcomm SC8180X platform needs to use the Qualcomm implementation, so add a specific compatible for this. Signed-off-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20210121014005.1612382-2-bjorn.andersson@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2021-01-21 09:40:05 +08:00
{ .compatible = "qcom,sc8180x-smmu-500" },
iommu: arm-smmu-qcom: Add sdm630/msm8998 compatibles for qcom quirks SDM630 and MSM8998 are among the SoCs that use Qualcomm's implementation of SMMUv2 which has already proven to be problematic over the years. Add their compatibles to the lookup list to prevent the platforms from being shut down by the hypervisor at MMU probe. Signed-off-by: Konrad Dybcio <konrad.dybcio@somainline.org> Signed-off-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@somainline.org> Link: https://lore.kernel.org/r/20210109165622.149777-1-konrad.dybcio@somainline.org Signed-off-by: Will Deacon <will@kernel.org>
2021-01-10 00:56:21 +08:00
{ .compatible = "qcom,sdm630-smmu-v2" },
iommu: arm-smmu-impl: Use table to list QCOM implementations Use table and of_match_node() to match qcom implementation instead of multiple of_device_compatible() calls for each QCOM SMMU implementation. Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/4e11899bc02102a6e6155db215911e8b5aaba950.1606287059.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-25 15:00:17 +08:00
{ .compatible = "qcom,sdm845-smmu-500" },
iommu/arm-smmu-qcom: Add sm6125 compatible Add compatible for SM6125 SoC Signed-off-by: Martin Botka <martin.botka@somainline.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20210523212535.740979-1-martin.botka@somainline.org Signed-off-by: Will Deacon <will@kernel.org>
2021-05-24 05:25:33 +08:00
{ .compatible = "qcom,sm6125-smmu-500" },
iommu/arm-smmu-qcom: Add SM6350 SMMU compatible Add compatible for SM6350 SMMU to use the Qualcomm Technologies, Inc. specific implementation. Signed-off-by: Konrad Dybcio <konrad.dybcio@somainline.org> Link: https://lore.kernel.org/r/20210820202906.229292-2-konrad.dybcio@somainline.org Signed-off-by: Will Deacon <will@kernel.org>
2021-08-21 04:29:05 +08:00
{ .compatible = "qcom,sm6350-smmu-500" },
iommu: arm-smmu-impl: Use table to list QCOM implementations Use table and of_match_node() to match qcom implementation instead of multiple of_device_compatible() calls for each QCOM SMMU implementation. Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/4e11899bc02102a6e6155db215911e8b5aaba950.1606287059.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-25 15:00:17 +08:00
{ .compatible = "qcom,sm8150-smmu-500" },
{ .compatible = "qcom,sm8250-smmu-500" },
iommu: arm-smmu-impl: Add SM8350 qcom iommu implementation Add SM8350 qcom iommu implementation to the table of qcom_smmu_impl_of_match table which brings in iommu support for SM8350 SoC Signed-off-by: Vinod Koul <vkoul@kernel.org> Link: https://lore.kernel.org/r/20210115090322.2287538-2-vkoul@kernel.org Signed-off-by: Will Deacon <will@kernel.org>
2021-01-15 17:03:22 +08:00
{ .compatible = "qcom,sm8350-smmu-500" },
iommu: arm-smmu-impl: Use table to list QCOM implementations Use table and of_match_node() to match qcom implementation instead of multiple of_device_compatible() calls for each QCOM SMMU implementation. Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/4e11899bc02102a6e6155db215911e8b5aaba950.1606287059.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-25 15:00:17 +08:00
{ }
};
iommu/arm-smmu-qcom: Protect acpi_match_platform_list() call with CONFIG_ACPI The struct acpi_platform_list and function acpi_match_platform_list() defined in include/linux/acpi.h are available only when CONFIG_ACPI is enabled. Add protection to fix the build issues with !CONFIG_ACPI. Reported-by: kernel test robot <lkp@intel.com> Signed-off-by: Shawn Guo <shawn.guo@linaro.org> Link: https://lore.kernel.org/r/20210609015511.3955-1-shawn.guo@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2021-06-09 09:55:11 +08:00
#ifdef CONFIG_ACPI
iommu/arm-smmu-qcom: hook up qcom_smmu_impl for ACPI boot The hookup with qcom_smmu_impl is required to do ACPI boot on SC8180X based devices like Lenovo Flex 5G laptop and Microsoft Surface Pro X. Define acpi_platform_list for these platforms and match them using acpi_match_platform_list() call, and create qcom_smmu_impl accordingly. (np == NULL) is used to check ACPI boot, because fwnode of SMMU device is a static allocation and thus helpers like has_acpi_companion() don't work here. Signed-off-by: Shawn Guo <shawn.guo@linaro.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20210509022607.17534-1-shawn.guo@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2021-05-09 10:26:07 +08:00
static struct acpi_platform_list qcom_acpi_platlist[] = {
{ "LENOVO", "CB-01 ", 0x8180, ACPI_SIG_IORT, equal, "QCOM SMMU" },
{ "QCOM ", "QCOMEDK2", 0x8180, ACPI_SIG_IORT, equal, "QCOM SMMU" },
{ }
};
iommu/arm-smmu-qcom: Protect acpi_match_platform_list() call with CONFIG_ACPI The struct acpi_platform_list and function acpi_match_platform_list() defined in include/linux/acpi.h are available only when CONFIG_ACPI is enabled. Add protection to fix the build issues with !CONFIG_ACPI. Reported-by: kernel test robot <lkp@intel.com> Signed-off-by: Shawn Guo <shawn.guo@linaro.org> Link: https://lore.kernel.org/r/20210609015511.3955-1-shawn.guo@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2021-06-09 09:55:11 +08:00
#endif
iommu/arm-smmu-qcom: hook up qcom_smmu_impl for ACPI boot The hookup with qcom_smmu_impl is required to do ACPI boot on SC8180X based devices like Lenovo Flex 5G laptop and Microsoft Surface Pro X. Define acpi_platform_list for these platforms and match them using acpi_match_platform_list() call, and create qcom_smmu_impl accordingly. (np == NULL) is used to check ACPI boot, because fwnode of SMMU device is a static allocation and thus helpers like has_acpi_companion() don't work here. Signed-off-by: Shawn Guo <shawn.guo@linaro.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20210509022607.17534-1-shawn.guo@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2021-05-09 10:26:07 +08:00
iommu/arm-smmu-qcom: Add implementation for the adreno GPU SMMU Add a special implementation for the SMMU attached to most Adreno GPU target triggered from the qcom,adreno-smmu compatible string. The new Adreno SMMU implementation will enable split pagetables (TTBR1) for the domain attached to the GPU device (SID 0) and hard code it context bank 0 so the GPU hardware can implement per-instance pagetables. Co-developed-by: Rob Clark <robdclark@chromium.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Rob Clark <robdclark@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20201109184728.2463097-2-jcrouse@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-10 02:47:25 +08:00
struct arm_smmu_device *qcom_smmu_impl_init(struct arm_smmu_device *smmu)
{
iommu: arm-smmu-impl: Use table to list QCOM implementations Use table and of_match_node() to match qcom implementation instead of multiple of_device_compatible() calls for each QCOM SMMU implementation. Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/4e11899bc02102a6e6155db215911e8b5aaba950.1606287059.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-25 15:00:17 +08:00
const struct device_node *np = smmu->dev->of_node;
iommu/arm-smmu-qcom: Add implementation for the adreno GPU SMMU Add a special implementation for the SMMU attached to most Adreno GPU target triggered from the qcom,adreno-smmu compatible string. The new Adreno SMMU implementation will enable split pagetables (TTBR1) for the domain attached to the GPU device (SID 0) and hard code it context bank 0 so the GPU hardware can implement per-instance pagetables. Co-developed-by: Rob Clark <robdclark@chromium.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Rob Clark <robdclark@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20201109184728.2463097-2-jcrouse@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-10 02:47:25 +08:00
iommu/arm-smmu-qcom: Protect acpi_match_platform_list() call with CONFIG_ACPI The struct acpi_platform_list and function acpi_match_platform_list() defined in include/linux/acpi.h are available only when CONFIG_ACPI is enabled. Add protection to fix the build issues with !CONFIG_ACPI. Reported-by: kernel test robot <lkp@intel.com> Signed-off-by: Shawn Guo <shawn.guo@linaro.org> Link: https://lore.kernel.org/r/20210609015511.3955-1-shawn.guo@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2021-06-09 09:55:11 +08:00
#ifdef CONFIG_ACPI
iommu/arm-smmu-qcom: hook up qcom_smmu_impl for ACPI boot The hookup with qcom_smmu_impl is required to do ACPI boot on SC8180X based devices like Lenovo Flex 5G laptop and Microsoft Surface Pro X. Define acpi_platform_list for these platforms and match them using acpi_match_platform_list() call, and create qcom_smmu_impl accordingly. (np == NULL) is used to check ACPI boot, because fwnode of SMMU device is a static allocation and thus helpers like has_acpi_companion() don't work here. Signed-off-by: Shawn Guo <shawn.guo@linaro.org> Reviewed-by: Robin Murphy <robin.murphy@arm.com> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20210509022607.17534-1-shawn.guo@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2021-05-09 10:26:07 +08:00
if (np == NULL) {
/* Match platform for ACPI boot */
if (acpi_match_platform_list(qcom_acpi_platlist) >= 0)
return qcom_smmu_create(smmu, &qcom_smmu_impl);
}
iommu/arm-smmu-qcom: Protect acpi_match_platform_list() call with CONFIG_ACPI The struct acpi_platform_list and function acpi_match_platform_list() defined in include/linux/acpi.h are available only when CONFIG_ACPI is enabled. Add protection to fix the build issues with !CONFIG_ACPI. Reported-by: kernel test robot <lkp@intel.com> Signed-off-by: Shawn Guo <shawn.guo@linaro.org> Link: https://lore.kernel.org/r/20210609015511.3955-1-shawn.guo@linaro.org Signed-off-by: Will Deacon <will@kernel.org>
2021-06-09 09:55:11 +08:00
#endif
iommu: arm-smmu-impl: Use table to list QCOM implementations Use table and of_match_node() to match qcom implementation instead of multiple of_device_compatible() calls for each QCOM SMMU implementation. Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/4e11899bc02102a6e6155db215911e8b5aaba950.1606287059.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-25 15:00:17 +08:00
iommu/arm-smmu-qcom: Move the adreno smmu specific impl Adreno(GPU) SMMU and APSS(Application Processor SubSystem) SMMU both implement "arm,mmu-500" in some QTI SoCs and to run through adreno smmu specific implementation such as enabling split pagetables support, we need to match the "qcom,adreno-smmu" compatible first before apss smmu or else we will be running apps smmu implementation for adreno smmu and the additional features for adreno smmu is never set. For ex: we have "qcom,sc7280-smmu-500" compatible for both apps and adreno smmu implementing "arm,mmu-500", so the adreno smmu implementation is never reached because the current sequence checks for apps smmu compatible(qcom,sc7280-smmu-500) first and runs that specific impl and we never reach adreno smmu specific implementation. Suggested-by: Akhil P Oommen <akhilpo@codeaurora.org> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Acked-by: Jordan Crouse <jordan@cosmicpenguin.net> Link: https://lore.kernel.org/r/c42181d313fdd440011541a28cde8cd10fffb9d3.1623155117.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2021-06-08 20:30:07 +08:00
/*
* Do not change this order of implementation, i.e., first adreno
* smmu impl and then apss smmu since we can have both implementing
* arm,mmu-500 in which case we will miss setting adreno smmu specific
* features if the order is changed.
*/
iommu: arm-smmu-impl: Use table to list QCOM implementations Use table and of_match_node() to match qcom implementation instead of multiple of_device_compatible() calls for each QCOM SMMU implementation. Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/4e11899bc02102a6e6155db215911e8b5aaba950.1606287059.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-25 15:00:17 +08:00
if (of_device_is_compatible(np, "qcom,adreno-smmu"))
return qcom_smmu_create(smmu, &qcom_adreno_smmu_impl);
iommu/arm-smmu-qcom: Move the adreno smmu specific impl Adreno(GPU) SMMU and APSS(Application Processor SubSystem) SMMU both implement "arm,mmu-500" in some QTI SoCs and to run through adreno smmu specific implementation such as enabling split pagetables support, we need to match the "qcom,adreno-smmu" compatible first before apss smmu or else we will be running apps smmu implementation for adreno smmu and the additional features for adreno smmu is never set. For ex: we have "qcom,sc7280-smmu-500" compatible for both apps and adreno smmu implementing "arm,mmu-500", so the adreno smmu implementation is never reached because the current sequence checks for apps smmu compatible(qcom,sc7280-smmu-500) first and runs that specific impl and we never reach adreno smmu specific implementation. Suggested-by: Akhil P Oommen <akhilpo@codeaurora.org> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Acked-by: Jordan Crouse <jordan@cosmicpenguin.net> Link: https://lore.kernel.org/r/c42181d313fdd440011541a28cde8cd10fffb9d3.1623155117.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2021-06-08 20:30:07 +08:00
if (of_match_node(qcom_smmu_impl_of_match, np))
return qcom_smmu_create(smmu, &qcom_smmu_impl);
iommu: arm-smmu-impl: Use table to list QCOM implementations Use table and of_match_node() to match qcom implementation instead of multiple of_device_compatible() calls for each QCOM SMMU implementation. Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> Acked-by: Will Deacon <will@kernel.org> Link: https://lore.kernel.org/r/4e11899bc02102a6e6155db215911e8b5aaba950.1606287059.git.saiprakash.ranjan@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-25 15:00:17 +08:00
return smmu;
iommu/arm-smmu-qcom: Add implementation for the adreno GPU SMMU Add a special implementation for the SMMU attached to most Adreno GPU target triggered from the qcom,adreno-smmu compatible string. The new Adreno SMMU implementation will enable split pagetables (TTBR1) for the domain attached to the GPU device (SID 0) and hard code it context bank 0 so the GPU hardware can implement per-instance pagetables. Co-developed-by: Rob Clark <robdclark@chromium.org> Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org> Signed-off-by: Rob Clark <robdclark@chromium.org> Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org> Link: https://lore.kernel.org/r/20201109184728.2463097-2-jcrouse@codeaurora.org Signed-off-by: Will Deacon <will@kernel.org>
2020-11-10 02:47:25 +08:00
}