2007-10-22 07:41:41 +08:00
|
|
|
/*
|
|
|
|
* Copyright (c) 2006, Intel Corporation.
|
|
|
|
*
|
|
|
|
* This program is free software; you can redistribute it and/or modify it
|
|
|
|
* under the terms and conditions of the GNU General Public License,
|
|
|
|
* version 2, as published by the Free Software Foundation.
|
|
|
|
*
|
|
|
|
* This program is distributed in the hope it will be useful, but WITHOUT
|
|
|
|
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
|
|
|
|
* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
|
|
|
|
* more details.
|
|
|
|
*
|
|
|
|
* You should have received a copy of the GNU General Public License along with
|
|
|
|
* this program; if not, write to the Free Software Foundation, Inc., 59 Temple
|
|
|
|
* Place - Suite 330, Boston, MA 02111-1307 USA.
|
|
|
|
*
|
2008-02-24 07:23:35 +08:00
|
|
|
* Copyright (C) 2006-2008 Intel Corporation
|
|
|
|
* Author: Ashok Raj <ashok.raj@intel.com>
|
|
|
|
* Author: Shaohua Li <shaohua.li@intel.com>
|
|
|
|
* Author: Anil S Keshavamurthy <anil.s.keshavamurthy@intel.com>
|
2007-10-22 07:41:41 +08:00
|
|
|
*
|
2008-07-11 02:16:35 +08:00
|
|
|
* This file implements early detection/parsing of Remapping Devices
|
2007-10-22 07:41:41 +08:00
|
|
|
* reported to OS through BIOS via DMA remapping reporting (DMAR) ACPI
|
|
|
|
* tables.
|
2008-07-11 02:16:35 +08:00
|
|
|
*
|
|
|
|
* These routines are used by both DMA-remapping and Interrupt-remapping
|
2007-10-22 07:41:41 +08:00
|
|
|
*/
|
|
|
|
|
2015-06-12 15:57:06 +08:00
|
|
|
#define pr_fmt(fmt) "DMAR: " fmt
|
2012-06-09 05:13:11 +08:00
|
|
|
|
2007-10-22 07:41:41 +08:00
|
|
|
#include <linux/pci.h>
|
|
|
|
#include <linux/dmar.h>
|
2008-09-09 23:37:29 +08:00
|
|
|
#include <linux/iova.h>
|
|
|
|
#include <linux/intel-iommu.h>
|
2008-07-11 02:16:42 +08:00
|
|
|
#include <linux/timer.h>
|
2009-03-17 08:04:54 +08:00
|
|
|
#include <linux/irq.h>
|
|
|
|
#include <linux/interrupt.h>
|
2009-09-02 09:25:07 +08:00
|
|
|
#include <linux/tboot.h>
|
2009-07-07 11:40:19 +08:00
|
|
|
#include <linux/dmi.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 16:04:11 +08:00
|
|
|
#include <linux/slab.h>
|
iommu/vt-d: Make use of IOMMU sysfs support
Register our DRHD IOMMUs, cross link devices, and provide a base set
of attributes for the IOMMU. Note that IRQ remapping support parses
the DMAR table very early in boot, well before the iommu_class can
reasonably be setup, so our registration is split between
intel_iommu_init(), which occurs later, and alloc_iommu(), which
typically occurs much earlier, but may happen at any time later
with IOMMU hot-add support.
On a typical desktop system, this provides the following (pruned):
$ find /sys | grep dmar
/sys/devices/virtual/iommu/dmar0
/sys/devices/virtual/iommu/dmar0/devices
/sys/devices/virtual/iommu/dmar0/devices/0000:00:02.0
/sys/devices/virtual/iommu/dmar0/intel-iommu
/sys/devices/virtual/iommu/dmar0/intel-iommu/cap
/sys/devices/virtual/iommu/dmar0/intel-iommu/ecap
/sys/devices/virtual/iommu/dmar0/intel-iommu/address
/sys/devices/virtual/iommu/dmar0/intel-iommu/version
/sys/devices/virtual/iommu/dmar1
/sys/devices/virtual/iommu/dmar1/devices
/sys/devices/virtual/iommu/dmar1/devices/0000:00:00.0
/sys/devices/virtual/iommu/dmar1/devices/0000:00:01.0
/sys/devices/virtual/iommu/dmar1/devices/0000:00:16.0
/sys/devices/virtual/iommu/dmar1/devices/0000:00:1a.0
/sys/devices/virtual/iommu/dmar1/devices/0000:00:1b.0
/sys/devices/virtual/iommu/dmar1/devices/0000:00:1c.0
...
/sys/devices/virtual/iommu/dmar1/intel-iommu
/sys/devices/virtual/iommu/dmar1/intel-iommu/cap
/sys/devices/virtual/iommu/dmar1/intel-iommu/ecap
/sys/devices/virtual/iommu/dmar1/intel-iommu/address
/sys/devices/virtual/iommu/dmar1/intel-iommu/version
/sys/class/iommu/dmar0
/sys/class/iommu/dmar1
(devices also link back to the dmar units)
This makes address, version, capabilities, and extended capabilities
available, just like printed on boot. I've tried not to duplicate
data that can be found in the DMAR table, with the exception of the
address, which provides an easy way to associate the sysfs device with
a DRHD entry in the DMAR. It's tempting to add scopes and RMRR data
here, but the full DMAR table is already exposed under /sys/firmware/
and therefore already provides a way for userspace to learn such
details.
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
2014-06-13 06:12:31 +08:00
|
|
|
#include <linux/iommu.h>
|
2012-03-31 02:47:08 +08:00
|
|
|
#include <asm/irq_remapping.h>
|
2010-08-27 01:58:04 +08:00
|
|
|
#include <asm/iommu_table.h>
|
2007-10-22 07:41:41 +08:00
|
|
|
|
2012-09-26 18:44:43 +08:00
|
|
|
#include "irq_remapping.h"
|
|
|
|
|
2014-11-09 22:47:56 +08:00
|
|
|
typedef int (*dmar_res_handler_t)(struct acpi_dmar_header *, void *);
|
|
|
|
struct dmar_res_callback {
|
|
|
|
dmar_res_handler_t cb[ACPI_DMAR_TYPE_RESERVED];
|
|
|
|
void *arg[ACPI_DMAR_TYPE_RESERVED];
|
|
|
|
bool ignore_unhandled;
|
|
|
|
bool print_entry;
|
|
|
|
};
|
|
|
|
|
iommu/vt-d: Introduce a rwsem to protect global data structures
Introduce a global rwsem dmar_global_lock, which will be used to
protect DMAR related global data structures from DMAR/PCI/memory
device hotplug operations in process context.
DMA and interrupt remapping related data structures are read most,
and only change when memory/PCI/DMAR hotplug event happens.
So a global rwsem solution is adopted for balance between simplicity
and performance.
For interrupt remapping driver, function intel_irq_remapping_supported(),
dmar_table_init(), intel_enable_irq_remapping(), disable_irq_remapping(),
reenable_irq_remapping() and enable_drhd_fault_handling() etc
are called during booting, suspending and resuming with interrupt
disabled, so no need to take the global lock.
For interrupt remapping entry allocation, the locking model is:
down_read(&dmar_global_lock);
/* Find corresponding iommu */
iommu = map_hpet_to_ir(id);
if (iommu)
/*
* Allocate remapping entry and mark entry busy,
* the IOMMU won't be hot-removed until the
* allocated entry has been released.
*/
index = alloc_irte(iommu, irq, 1);
up_read(&dmar_global_lock);
For DMA remmaping driver, we only uses the dmar_global_lock rwsem to
protect functions which are only called in process context. For any
function which may be called in interrupt context, we will use RCU
to protect them in following patches.
Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
Signed-off-by: Joerg Roedel <joro@8bytes.org>
2014-02-19 14:07:33 +08:00
|
|
|
/*
|
|
|
|
* Assumptions:
|
|
|
|
* 1) The hotplug framework guarentees that DMAR unit will be hot-added
|
|
|
|
* before IO devices managed by that unit.
|
|
|
|
* 2) The hotplug framework guarantees that DMAR unit will be hot-removed
|
|
|
|
* after IO devices managed by that unit.
|
|
|
|
* 3) Hotplug events are rare.
|
|
|
|
*
|
|
|
|
* Locking rules for DMA and interrupt remapping related global data structures:
|
|
|
|
* 1) Use dmar_global_lock in process context
|
|
|
|
* 2) Use RCU in interrupt context
|
2007-10-22 07:41:41 +08:00
|
|
|
*/
|
iommu/vt-d: Introduce a rwsem to protect global data structures
Introduce a global rwsem dmar_global_lock, which will be used to
protect DMAR related global data structures from DMAR/PCI/memory
device hotplug operations in process context.
DMA and interrupt remapping related data structures are read most,
and only change when memory/PCI/DMAR hotplug event happens.
So a global rwsem solution is adopted for balance between simplicity
and performance.
For interrupt remapping driver, function intel_irq_remapping_supported(),
dmar_table_init(), intel_enable_irq_remapping(), disable_irq_remapping(),
reenable_irq_remapping() and enable_drhd_fault_handling() etc
are called during booting, suspending and resuming with interrupt
disabled, so no need to take the global lock.
For interrupt remapping entry allocation, the locking model is:
down_read(&dmar_global_lock);
/* Find corresponding iommu */
iommu = map_hpet_to_ir(id);
if (iommu)
/*
* Allocate remapping entry and mark entry busy,
* the IOMMU won't be hot-removed until the
* allocated entry has been released.
*/
index = alloc_irte(iommu, irq, 1);
up_read(&dmar_global_lock);
For DMA remmaping driver, we only uses the dmar_global_lock rwsem to
protect functions which are only called in process context. For any
function which may be called in interrupt context, we will use RCU
to protect them in following patches.
Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
Signed-off-by: Joerg Roedel <joro@8bytes.org>
2014-02-19 14:07:33 +08:00
|
|
|
DECLARE_RWSEM(dmar_global_lock);
|
2007-10-22 07:41:41 +08:00
|
|
|
LIST_HEAD(dmar_drhd_units);
|
|
|
|
|
2011-08-24 08:05:18 +08:00
|
|
|
struct acpi_table_header * __initdata dmar_tbl;
|
2014-02-19 14:07:36 +08:00
|
|
|
static int dmar_dev_scope_status = 1;
|
2014-11-09 22:47:57 +08:00
|
|
|
static unsigned long dmar_seq_ids[BITS_TO_LONGS(DMAR_UNITS_SUPPORTED)];
|
2007-10-22 07:41:41 +08:00
|
|
|
|
2014-01-06 14:18:16 +08:00
|
|
|
static int alloc_iommu(struct dmar_drhd_unit *drhd);
|
2014-01-06 14:18:20 +08:00
|
|
|
static void free_iommu(struct intel_iommu *iommu);
|
2014-01-06 14:18:16 +08:00
|
|
|
|
2017-02-01 20:23:08 +08:00
|
|
|
extern const struct iommu_ops intel_iommu_ops;
|
|
|
|
|
2014-11-09 22:47:58 +08:00
|
|
|
static void dmar_register_drhd_unit(struct dmar_drhd_unit *drhd)
|
2007-10-22 07:41:41 +08:00
|
|
|
{
|
|
|
|
/*
|
|
|
|
* add INCLUDE_ALL at the tail, so scan the list will find it at
|
|
|
|
* the very end.
|
|
|
|
*/
|
|
|
|
if (drhd->include_all)
|
2014-02-19 14:07:34 +08:00
|
|
|
list_add_tail_rcu(&drhd->list, &dmar_drhd_units);
|
2007-10-22 07:41:41 +08:00
|
|
|
else
|
2014-02-19 14:07:34 +08:00
|
|
|
list_add_rcu(&drhd->list, &dmar_drhd_units);
|
2007-10-22 07:41:41 +08:00
|
|
|
}
|
|
|
|
|
2014-02-19 14:07:24 +08:00
|
|
|
void *dmar_alloc_dev_scope(void *start, void *end, int *cnt)
|
2007-10-22 07:41:41 +08:00
|
|
|
{
|
|
|
|
struct acpi_dmar_device_scope *scope;
|
|
|
|
|
|
|
|
*cnt = 0;
|
|
|
|
while (start < end) {
|
|
|
|
scope = start;
|
2014-07-30 12:21:00 +08:00
|
|
|
if (scope->entry_type == ACPI_DMAR_SCOPE_TYPE_NAMESPACE ||
|
2014-03-07 22:39:27 +08:00
|
|
|
scope->entry_type == ACPI_DMAR_SCOPE_TYPE_ENDPOINT ||
|
2007-10-22 07:41:41 +08:00
|
|
|
scope->entry_type == ACPI_DMAR_SCOPE_TYPE_BRIDGE)
|
|
|
|
(*cnt)++;
|
2013-04-24 02:26:45 +08:00
|
|
|
else if (scope->entry_type != ACPI_DMAR_SCOPE_TYPE_IOAPIC &&
|
|
|
|
scope->entry_type != ACPI_DMAR_SCOPE_TYPE_HPET) {
|
2012-06-09 05:13:11 +08:00
|
|
|
pr_warn("Unsupported device scope\n");
|
2010-04-09 02:58:22 +08:00
|
|
|
}
|
2007-10-22 07:41:41 +08:00
|
|
|
start += scope->length;
|
|
|
|
}
|
|
|
|
if (*cnt == 0)
|
2014-02-19 14:07:24 +08:00
|
|
|
return NULL;
|
|
|
|
|
2014-03-07 23:08:36 +08:00
|
|
|
return kcalloc(*cnt, sizeof(struct dmar_dev_scope), GFP_KERNEL);
|
2014-02-19 14:07:24 +08:00
|
|
|
}
|
|
|
|
|
2014-03-07 23:08:36 +08:00
|
|
|
void dmar_free_dev_scope(struct dmar_dev_scope **devices, int *cnt)
|
2014-01-06 14:18:09 +08:00
|
|
|
{
|
2014-02-19 14:07:32 +08:00
|
|
|
int i;
|
2014-03-07 23:08:36 +08:00
|
|
|
struct device *tmp_dev;
|
2014-02-19 14:07:32 +08:00
|
|
|
|
2014-01-06 14:18:09 +08:00
|
|
|
if (*devices && *cnt) {
|
2014-02-19 14:07:32 +08:00
|
|
|
for_each_active_dev_scope(*devices, *cnt, i, tmp_dev)
|
2014-03-07 23:08:36 +08:00
|
|
|
put_device(tmp_dev);
|
2014-01-06 14:18:09 +08:00
|
|
|
kfree(*devices);
|
|
|
|
}
|
2014-02-19 14:07:34 +08:00
|
|
|
|
|
|
|
*devices = NULL;
|
|
|
|
*cnt = 0;
|
2014-01-06 14:18:09 +08:00
|
|
|
}
|
|
|
|
|
2014-02-19 14:07:35 +08:00
|
|
|
/* Optimize out kzalloc()/kfree() for normal cases */
|
|
|
|
static char dmar_pci_notify_info_buf[64];
|
|
|
|
|
|
|
|
static struct dmar_pci_notify_info *
|
|
|
|
dmar_alloc_pci_notify_info(struct pci_dev *dev, unsigned long event)
|
|
|
|
{
|
|
|
|
int level = 0;
|
|
|
|
size_t size;
|
|
|
|
struct pci_dev *tmp;
|
|
|
|
struct dmar_pci_notify_info *info;
|
|
|
|
|
|
|
|
BUG_ON(dev->is_virtfn);
|
|
|
|
|
|
|
|
/* Only generate path[] for device addition event */
|
|
|
|
if (event == BUS_NOTIFY_ADD_DEVICE)
|
|
|
|
for (tmp = dev; tmp; tmp = tmp->bus->self)
|
|
|
|
level++;
|
|
|
|
|
|
|
|
size = sizeof(*info) + level * sizeof(struct acpi_dmar_pci_path);
|
|
|
|
if (size <= sizeof(dmar_pci_notify_info_buf)) {
|
|
|
|
info = (struct dmar_pci_notify_info *)dmar_pci_notify_info_buf;
|
|
|
|
} else {
|
|
|
|
info = kzalloc(size, GFP_KERNEL);
|
|
|
|
if (!info) {
|
|
|
|
pr_warn("Out of memory when allocating notify_info "
|
|
|
|
"for %s.\n", pci_name(dev));
|
2014-02-19 14:07:36 +08:00
|
|
|
if (dmar_dev_scope_status == 0)
|
|
|
|
dmar_dev_scope_status = -ENOMEM;
|
2014-02-19 14:07:35 +08:00
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
info->event = event;
|
|
|
|
info->dev = dev;
|
|
|
|
info->seg = pci_domain_nr(dev->bus);
|
|
|
|
info->level = level;
|
|
|
|
if (event == BUS_NOTIFY_ADD_DEVICE) {
|
2014-04-15 10:35:35 +08:00
|
|
|
for (tmp = dev; tmp; tmp = tmp->bus->self) {
|
|
|
|
level--;
|
2014-10-02 17:50:25 +08:00
|
|
|
info->path[level].bus = tmp->bus->number;
|
2014-02-19 14:07:35 +08:00
|
|
|
info->path[level].device = PCI_SLOT(tmp->devfn);
|
|
|
|
info->path[level].function = PCI_FUNC(tmp->devfn);
|
|
|
|
if (pci_is_root_bus(tmp->bus))
|
|
|
|
info->bus = tmp->bus->number;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return info;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void dmar_free_pci_notify_info(struct dmar_pci_notify_info *info)
|
|
|
|
{
|
|
|
|
if ((void *)info != dmar_pci_notify_info_buf)
|
|
|
|
kfree(info);
|
|
|
|
}
|
|
|
|
|
|
|
|
static bool dmar_match_pci_path(struct dmar_pci_notify_info *info, int bus,
|
|
|
|
struct acpi_dmar_pci_path *path, int count)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
if (info->bus != bus)
|
2014-09-22 22:30:22 +08:00
|
|
|
goto fallback;
|
2014-02-19 14:07:35 +08:00
|
|
|
if (info->level != count)
|
2014-09-22 22:30:22 +08:00
|
|
|
goto fallback;
|
2014-02-19 14:07:35 +08:00
|
|
|
|
|
|
|
for (i = 0; i < count; i++) {
|
|
|
|
if (path[i].device != info->path[i].device ||
|
|
|
|
path[i].function != info->path[i].function)
|
2014-09-22 22:30:22 +08:00
|
|
|
goto fallback;
|
2014-02-19 14:07:35 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
return true;
|
2014-09-22 22:30:22 +08:00
|
|
|
|
|
|
|
fallback:
|
|
|
|
|
|
|
|
if (count != 1)
|
|
|
|
return false;
|
|
|
|
|
|
|
|
i = info->level - 1;
|
|
|
|
if (bus == info->path[i].bus &&
|
|
|
|
path[0].device == info->path[i].device &&
|
|
|
|
path[0].function == info->path[i].function) {
|
|
|
|
pr_info(FW_BUG "RMRR entry for device %02x:%02x.%x is broken - applying workaround\n",
|
|
|
|
bus, path[0].device, path[0].function);
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
return false;
|
2014-02-19 14:07:35 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Return: > 0 if match found, 0 if no match found, < 0 if error happens */
|
|
|
|
int dmar_insert_dev_scope(struct dmar_pci_notify_info *info,
|
|
|
|
void *start, void*end, u16 segment,
|
2014-03-07 23:08:36 +08:00
|
|
|
struct dmar_dev_scope *devices,
|
|
|
|
int devices_cnt)
|
2014-02-19 14:07:35 +08:00
|
|
|
{
|
|
|
|
int i, level;
|
2014-03-07 23:08:36 +08:00
|
|
|
struct device *tmp, *dev = &info->dev->dev;
|
2014-02-19 14:07:35 +08:00
|
|
|
struct acpi_dmar_device_scope *scope;
|
|
|
|
struct acpi_dmar_pci_path *path;
|
|
|
|
|
|
|
|
if (segment != info->seg)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
for (; start < end; start += scope->length) {
|
|
|
|
scope = start;
|
|
|
|
if (scope->entry_type != ACPI_DMAR_SCOPE_TYPE_ENDPOINT &&
|
|
|
|
scope->entry_type != ACPI_DMAR_SCOPE_TYPE_BRIDGE)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
path = (struct acpi_dmar_pci_path *)(scope + 1);
|
|
|
|
level = (scope->length - sizeof(*scope)) / sizeof(*path);
|
|
|
|
if (!dmar_match_pci_path(info, scope->bus, path, level))
|
|
|
|
continue;
|
|
|
|
|
2016-06-03 08:46:10 +08:00
|
|
|
/*
|
|
|
|
* We expect devices with endpoint scope to have normal PCI
|
|
|
|
* headers, and devices with bridge scope to have bridge PCI
|
|
|
|
* headers. However PCI NTB devices may be listed in the
|
|
|
|
* DMAR table with bridge scope, even though they have a
|
|
|
|
* normal PCI header. NTB devices are identified by class
|
|
|
|
* "BRIDGE_OTHER" (0680h) - we don't declare a socpe mismatch
|
|
|
|
* for this special case.
|
|
|
|
*/
|
|
|
|
if ((scope->entry_type == ACPI_DMAR_SCOPE_TYPE_ENDPOINT &&
|
|
|
|
info->dev->hdr_type != PCI_HEADER_TYPE_NORMAL) ||
|
|
|
|
(scope->entry_type == ACPI_DMAR_SCOPE_TYPE_BRIDGE &&
|
|
|
|
(info->dev->hdr_type == PCI_HEADER_TYPE_NORMAL &&
|
|
|
|
info->dev->class >> 8 != PCI_CLASS_BRIDGE_OTHER))) {
|
2014-02-19 14:07:35 +08:00
|
|
|
pr_warn("Device scope type does not match for %s\n",
|
2014-03-07 23:08:36 +08:00
|
|
|
pci_name(info->dev));
|
2014-02-19 14:07:35 +08:00
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
|
|
|
for_each_dev_scope(devices, devices_cnt, i, tmp)
|
|
|
|
if (tmp == NULL) {
|
2014-03-07 23:08:36 +08:00
|
|
|
devices[i].bus = info->dev->bus->number;
|
|
|
|
devices[i].devfn = info->dev->devfn;
|
|
|
|
rcu_assign_pointer(devices[i].dev,
|
|
|
|
get_device(dev));
|
2014-02-19 14:07:35 +08:00
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
BUG_ON(i >= devices_cnt);
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
int dmar_remove_dev_scope(struct dmar_pci_notify_info *info, u16 segment,
|
2014-03-07 23:08:36 +08:00
|
|
|
struct dmar_dev_scope *devices, int count)
|
2014-02-19 14:07:35 +08:00
|
|
|
{
|
|
|
|
int index;
|
2014-03-07 23:08:36 +08:00
|
|
|
struct device *tmp;
|
2014-02-19 14:07:35 +08:00
|
|
|
|
|
|
|
if (info->seg != segment)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
for_each_active_dev_scope(devices, count, index, tmp)
|
2014-03-07 23:08:36 +08:00
|
|
|
if (tmp == &info->dev->dev) {
|
2014-08-18 20:20:56 +08:00
|
|
|
RCU_INIT_POINTER(devices[index].dev, NULL);
|
2014-02-19 14:07:35 +08:00
|
|
|
synchronize_rcu();
|
2014-03-07 23:08:36 +08:00
|
|
|
put_device(tmp);
|
2014-02-19 14:07:35 +08:00
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int dmar_pci_bus_add_dev(struct dmar_pci_notify_info *info)
|
|
|
|
{
|
|
|
|
int ret = 0;
|
|
|
|
struct dmar_drhd_unit *dmaru;
|
|
|
|
struct acpi_dmar_hardware_unit *drhd;
|
|
|
|
|
|
|
|
for_each_drhd_unit(dmaru) {
|
|
|
|
if (dmaru->include_all)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
drhd = container_of(dmaru->hdr,
|
|
|
|
struct acpi_dmar_hardware_unit, header);
|
|
|
|
ret = dmar_insert_dev_scope(info, (void *)(drhd + 1),
|
|
|
|
((void *)drhd) + drhd->header.length,
|
|
|
|
dmaru->segment,
|
|
|
|
dmaru->devices, dmaru->devices_cnt);
|
2017-03-16 22:23:54 +08:00
|
|
|
if (ret)
|
2014-02-19 14:07:35 +08:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
if (ret >= 0)
|
|
|
|
ret = dmar_iommu_notify_scope_dev(info);
|
2014-02-19 14:07:36 +08:00
|
|
|
if (ret < 0 && dmar_dev_scope_status == 0)
|
|
|
|
dmar_dev_scope_status = ret;
|
2014-02-19 14:07:35 +08:00
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void dmar_pci_bus_del_dev(struct dmar_pci_notify_info *info)
|
|
|
|
{
|
|
|
|
struct dmar_drhd_unit *dmaru;
|
|
|
|
|
|
|
|
for_each_drhd_unit(dmaru)
|
|
|
|
if (dmar_remove_dev_scope(info, dmaru->segment,
|
|
|
|
dmaru->devices, dmaru->devices_cnt))
|
|
|
|
break;
|
|
|
|
dmar_iommu_notify_scope_dev(info);
|
|
|
|
}
|
|
|
|
|
|
|
|
static int dmar_pci_bus_notifier(struct notifier_block *nb,
|
|
|
|
unsigned long action, void *data)
|
|
|
|
{
|
|
|
|
struct pci_dev *pdev = to_pci_dev(data);
|
|
|
|
struct dmar_pci_notify_info *info;
|
|
|
|
|
2016-10-22 06:32:05 +08:00
|
|
|
/* Only care about add/remove events for physical functions.
|
|
|
|
* For VFs we actually do the lookup based on the corresponding
|
|
|
|
* PF in device_to_iommu() anyway. */
|
2014-02-19 14:07:35 +08:00
|
|
|
if (pdev->is_virtfn)
|
|
|
|
return NOTIFY_DONE;
|
2016-03-01 06:49:47 +08:00
|
|
|
if (action != BUS_NOTIFY_ADD_DEVICE &&
|
|
|
|
action != BUS_NOTIFY_REMOVED_DEVICE)
|
2014-02-19 14:07:35 +08:00
|
|
|
return NOTIFY_DONE;
|
|
|
|
|
|
|
|
info = dmar_alloc_pci_notify_info(pdev, action);
|
|
|
|
if (!info)
|
|
|
|
return NOTIFY_DONE;
|
|
|
|
|
|
|
|
down_write(&dmar_global_lock);
|
|
|
|
if (action == BUS_NOTIFY_ADD_DEVICE)
|
|
|
|
dmar_pci_bus_add_dev(info);
|
2016-03-01 06:49:47 +08:00
|
|
|
else if (action == BUS_NOTIFY_REMOVED_DEVICE)
|
2014-02-19 14:07:35 +08:00
|
|
|
dmar_pci_bus_del_dev(info);
|
|
|
|
up_write(&dmar_global_lock);
|
|
|
|
|
|
|
|
dmar_free_pci_notify_info(info);
|
|
|
|
|
|
|
|
return NOTIFY_OK;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct notifier_block dmar_pci_bus_nb = {
|
|
|
|
.notifier_call = dmar_pci_bus_notifier,
|
|
|
|
.priority = INT_MIN,
|
|
|
|
};
|
|
|
|
|
2014-11-09 22:47:58 +08:00
|
|
|
static struct dmar_drhd_unit *
|
|
|
|
dmar_find_dmaru(struct acpi_dmar_hardware_unit *drhd)
|
|
|
|
{
|
|
|
|
struct dmar_drhd_unit *dmaru;
|
|
|
|
|
|
|
|
list_for_each_entry_rcu(dmaru, &dmar_drhd_units, list)
|
|
|
|
if (dmaru->segment == drhd->segment &&
|
|
|
|
dmaru->reg_base_addr == drhd->address)
|
|
|
|
return dmaru;
|
|
|
|
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
2007-10-22 07:41:41 +08:00
|
|
|
/**
|
|
|
|
* dmar_parse_one_drhd - parses exactly one DMA remapping hardware definition
|
|
|
|
* structure which uniquely represent one DMA remapping hardware unit
|
|
|
|
* present in the platform
|
|
|
|
*/
|
2014-11-09 22:47:58 +08:00
|
|
|
static int dmar_parse_one_drhd(struct acpi_dmar_header *header, void *arg)
|
2007-10-22 07:41:41 +08:00
|
|
|
{
|
|
|
|
struct acpi_dmar_hardware_unit *drhd;
|
|
|
|
struct dmar_drhd_unit *dmaru;
|
2017-03-16 22:23:53 +08:00
|
|
|
int ret;
|
2007-10-22 07:41:41 +08:00
|
|
|
|
2009-04-11 13:27:48 +08:00
|
|
|
drhd = (struct acpi_dmar_hardware_unit *)header;
|
2014-11-09 22:47:58 +08:00
|
|
|
dmaru = dmar_find_dmaru(drhd);
|
|
|
|
if (dmaru)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
dmaru = kzalloc(sizeof(*dmaru) + header->length, GFP_KERNEL);
|
2007-10-22 07:41:41 +08:00
|
|
|
if (!dmaru)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
2014-11-09 22:47:58 +08:00
|
|
|
/*
|
|
|
|
* If header is allocated from slab by ACPI _DSM method, we need to
|
|
|
|
* copy the content because the memory buffer will be freed on return.
|
|
|
|
*/
|
|
|
|
dmaru->hdr = (void *)(dmaru + 1);
|
|
|
|
memcpy(dmaru->hdr, header, header->length);
|
2007-10-22 07:41:41 +08:00
|
|
|
dmaru->reg_base_addr = drhd->address;
|
2009-04-04 08:45:37 +08:00
|
|
|
dmaru->segment = drhd->segment;
|
2007-10-22 07:41:41 +08:00
|
|
|
dmaru->include_all = drhd->flags & 0x1; /* BIT0: INCLUDE_ALL */
|
2014-03-07 22:39:27 +08:00
|
|
|
dmaru->devices = dmar_alloc_dev_scope((void *)(drhd + 1),
|
|
|
|
((void *)drhd) + drhd->header.length,
|
|
|
|
&dmaru->devices_cnt);
|
|
|
|
if (dmaru->devices_cnt && dmaru->devices == NULL) {
|
|
|
|
kfree(dmaru);
|
|
|
|
return -ENOMEM;
|
2014-02-19 14:07:36 +08:00
|
|
|
}
|
2007-10-22 07:41:41 +08:00
|
|
|
|
2008-07-11 02:16:37 +08:00
|
|
|
ret = alloc_iommu(dmaru);
|
|
|
|
if (ret) {
|
2014-03-07 22:39:27 +08:00
|
|
|
dmar_free_dev_scope(&dmaru->devices,
|
|
|
|
&dmaru->devices_cnt);
|
2008-07-11 02:16:37 +08:00
|
|
|
kfree(dmaru);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
dmar_register_drhd_unit(dmaru);
|
2014-11-09 22:47:56 +08:00
|
|
|
|
2014-11-09 22:47:58 +08:00
|
|
|
out:
|
2014-11-09 22:47:56 +08:00
|
|
|
if (arg)
|
|
|
|
(*(int *)arg)++;
|
|
|
|
|
2008-07-11 02:16:37 +08:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2014-01-06 14:18:20 +08:00
|
|
|
static void dmar_free_drhd(struct dmar_drhd_unit *dmaru)
|
|
|
|
{
|
|
|
|
if (dmaru->devices && dmaru->devices_cnt)
|
|
|
|
dmar_free_dev_scope(&dmaru->devices, &dmaru->devices_cnt);
|
|
|
|
if (dmaru->iommu)
|
|
|
|
free_iommu(dmaru->iommu);
|
|
|
|
kfree(dmaru);
|
|
|
|
}
|
|
|
|
|
2014-11-09 22:47:56 +08:00
|
|
|
static int __init dmar_parse_one_andd(struct acpi_dmar_header *header,
|
|
|
|
void *arg)
|
2014-03-07 22:34:38 +08:00
|
|
|
{
|
|
|
|
struct acpi_dmar_andd *andd = (void *)header;
|
|
|
|
|
|
|
|
/* Check for NUL termination within the designated length */
|
2014-07-30 12:21:00 +08:00
|
|
|
if (strnlen(andd->device_name, header->length - 8) == header->length - 8) {
|
2014-03-07 22:34:38 +08:00
|
|
|
WARN_TAINT(1, TAINT_FIRMWARE_WORKAROUND,
|
|
|
|
"Your BIOS is broken; ANDD object name is not NUL-terminated\n"
|
|
|
|
"BIOS vendor: %s; Ver: %s; Product Version: %s\n",
|
|
|
|
dmi_get_system_info(DMI_BIOS_VENDOR),
|
|
|
|
dmi_get_system_info(DMI_BIOS_VERSION),
|
|
|
|
dmi_get_system_info(DMI_PRODUCT_VERSION));
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
pr_info("ANDD device: %x name: %s\n", andd->device_number,
|
2014-07-30 12:21:00 +08:00
|
|
|
andd->device_name);
|
2014-03-07 22:34:38 +08:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2009-10-07 19:18:00 +08:00
|
|
|
#ifdef CONFIG_ACPI_NUMA
|
2014-11-09 22:47:58 +08:00
|
|
|
static int dmar_parse_one_rhsa(struct acpi_dmar_header *header, void *arg)
|
2009-10-03 02:01:21 +08:00
|
|
|
{
|
|
|
|
struct acpi_dmar_rhsa *rhsa;
|
|
|
|
struct dmar_drhd_unit *drhd;
|
|
|
|
|
|
|
|
rhsa = (struct acpi_dmar_rhsa *)header;
|
2009-10-07 19:18:00 +08:00
|
|
|
for_each_drhd_unit(drhd) {
|
2009-10-03 02:01:21 +08:00
|
|
|
if (drhd->reg_base_addr == rhsa->base_address) {
|
|
|
|
int node = acpi_map_pxm_to_node(rhsa->proximity_domain);
|
|
|
|
|
|
|
|
if (!node_online(node))
|
|
|
|
node = -1;
|
|
|
|
drhd->iommu->node = node;
|
2009-10-07 19:18:00 +08:00
|
|
|
return 0;
|
|
|
|
}
|
2009-10-03 02:01:21 +08:00
|
|
|
}
|
2010-04-04 02:38:43 +08:00
|
|
|
WARN_TAINT(
|
|
|
|
1, TAINT_FIRMWARE_WORKAROUND,
|
|
|
|
"Your BIOS is broken; RHSA refers to non-existent DMAR unit at %llx\n"
|
|
|
|
"BIOS vendor: %s; Ver: %s; Product Version: %s\n",
|
|
|
|
drhd->reg_base_addr,
|
|
|
|
dmi_get_system_info(DMI_BIOS_VENDOR),
|
|
|
|
dmi_get_system_info(DMI_BIOS_VERSION),
|
|
|
|
dmi_get_system_info(DMI_PRODUCT_VERSION));
|
2009-10-03 02:01:21 +08:00
|
|
|
|
2009-10-07 19:18:00 +08:00
|
|
|
return 0;
|
2009-10-03 02:01:21 +08:00
|
|
|
}
|
2014-11-09 22:47:56 +08:00
|
|
|
#else
|
|
|
|
#define dmar_parse_one_rhsa dmar_res_noop
|
2009-10-07 19:18:00 +08:00
|
|
|
#endif
|
2009-10-03 02:01:21 +08:00
|
|
|
|
2017-09-13 04:10:21 +08:00
|
|
|
static void
|
2007-10-22 07:41:41 +08:00
|
|
|
dmar_table_print_dmar_entry(struct acpi_dmar_header *header)
|
|
|
|
{
|
|
|
|
struct acpi_dmar_hardware_unit *drhd;
|
|
|
|
struct acpi_dmar_reserved_memory *rmrr;
|
2009-05-18 13:51:34 +08:00
|
|
|
struct acpi_dmar_atsr *atsr;
|
2009-09-25 03:14:00 +08:00
|
|
|
struct acpi_dmar_rhsa *rhsa;
|
2007-10-22 07:41:41 +08:00
|
|
|
|
|
|
|
switch (header->type) {
|
|
|
|
case ACPI_DMAR_TYPE_HARDWARE_UNIT:
|
2009-05-18 13:51:34 +08:00
|
|
|
drhd = container_of(header, struct acpi_dmar_hardware_unit,
|
|
|
|
header);
|
2012-06-09 05:13:11 +08:00
|
|
|
pr_info("DRHD base: %#016Lx flags: %#x\n",
|
2009-05-18 13:51:34 +08:00
|
|
|
(unsigned long long)drhd->address, drhd->flags);
|
2007-10-22 07:41:41 +08:00
|
|
|
break;
|
|
|
|
case ACPI_DMAR_TYPE_RESERVED_MEMORY:
|
2009-05-18 13:51:34 +08:00
|
|
|
rmrr = container_of(header, struct acpi_dmar_reserved_memory,
|
|
|
|
header);
|
2012-06-09 05:13:11 +08:00
|
|
|
pr_info("RMRR base: %#016Lx end: %#016Lx\n",
|
2008-10-17 09:02:32 +08:00
|
|
|
(unsigned long long)rmrr->base_address,
|
|
|
|
(unsigned long long)rmrr->end_address);
|
2007-10-22 07:41:41 +08:00
|
|
|
break;
|
2014-07-30 12:21:00 +08:00
|
|
|
case ACPI_DMAR_TYPE_ROOT_ATS:
|
2009-05-18 13:51:34 +08:00
|
|
|
atsr = container_of(header, struct acpi_dmar_atsr, header);
|
2012-06-09 05:13:11 +08:00
|
|
|
pr_info("ATSR flags: %#x\n", atsr->flags);
|
2009-05-18 13:51:34 +08:00
|
|
|
break;
|
2014-07-30 12:21:00 +08:00
|
|
|
case ACPI_DMAR_TYPE_HARDWARE_AFFINITY:
|
2009-09-25 03:14:00 +08:00
|
|
|
rhsa = container_of(header, struct acpi_dmar_rhsa, header);
|
2012-06-09 05:13:11 +08:00
|
|
|
pr_info("RHSA base: %#016Lx proximity domain: %#x\n",
|
2009-09-25 03:14:00 +08:00
|
|
|
(unsigned long long)rhsa->base_address,
|
|
|
|
rhsa->proximity_domain);
|
|
|
|
break;
|
2014-07-30 12:21:00 +08:00
|
|
|
case ACPI_DMAR_TYPE_NAMESPACE:
|
2014-03-07 22:34:38 +08:00
|
|
|
/* We don't print this here because we need to sanity-check
|
|
|
|
it first. So print it in dmar_parse_one_andd() instead. */
|
|
|
|
break;
|
2007-10-22 07:41:41 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-09-04 07:58:32 +08:00
|
|
|
/**
|
|
|
|
* dmar_table_detect - checks to see if the platform supports DMAR devices
|
|
|
|
*/
|
|
|
|
static int __init dmar_table_detect(void)
|
|
|
|
{
|
|
|
|
acpi_status status = AE_OK;
|
|
|
|
|
|
|
|
/* if we could find DMAR table, then there are DMAR devices */
|
2016-12-14 15:04:39 +08:00
|
|
|
status = acpi_get_table(ACPI_SIG_DMAR, 0, &dmar_tbl);
|
2008-09-04 07:58:32 +08:00
|
|
|
|
|
|
|
if (ACPI_SUCCESS(status) && !dmar_tbl) {
|
2012-06-09 05:13:11 +08:00
|
|
|
pr_warn("Unable to map DMAR\n");
|
2008-09-04 07:58:32 +08:00
|
|
|
status = AE_NOT_FOUND;
|
|
|
|
}
|
|
|
|
|
2017-03-16 22:23:51 +08:00
|
|
|
return ACPI_SUCCESS(status) ? 0 : -ENOENT;
|
2008-09-04 07:58:32 +08:00
|
|
|
}
|
2008-07-11 02:16:38 +08:00
|
|
|
|
2014-11-09 22:47:56 +08:00
|
|
|
static int dmar_walk_remapping_entries(struct acpi_dmar_header *start,
|
|
|
|
size_t len, struct dmar_res_callback *cb)
|
|
|
|
{
|
|
|
|
struct acpi_dmar_header *iter, *next;
|
|
|
|
struct acpi_dmar_header *end = ((void *)start) + len;
|
|
|
|
|
2017-03-16 22:23:52 +08:00
|
|
|
for (iter = start; iter < end; iter = next) {
|
2014-11-09 22:47:56 +08:00
|
|
|
next = (void *)iter + iter->length;
|
|
|
|
if (iter->length == 0) {
|
|
|
|
/* Avoid looping forever on bad ACPI tables */
|
|
|
|
pr_debug(FW_BUG "Invalid 0-length structure\n");
|
|
|
|
break;
|
|
|
|
} else if (next > end) {
|
|
|
|
/* Avoid passing table end */
|
2015-06-12 15:57:06 +08:00
|
|
|
pr_warn(FW_BUG "Record passes table end\n");
|
2017-03-16 22:23:52 +08:00
|
|
|
return -EINVAL;
|
2014-11-09 22:47:56 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
if (cb->print_entry)
|
|
|
|
dmar_table_print_dmar_entry(iter);
|
|
|
|
|
|
|
|
if (iter->type >= ACPI_DMAR_TYPE_RESERVED) {
|
|
|
|
/* continue for forward compatibility */
|
|
|
|
pr_debug("Unknown DMAR structure type %d\n",
|
|
|
|
iter->type);
|
|
|
|
} else if (cb->cb[iter->type]) {
|
2017-03-16 22:23:52 +08:00
|
|
|
int ret;
|
|
|
|
|
2014-11-09 22:47:56 +08:00
|
|
|
ret = cb->cb[iter->type](iter, cb->arg[iter->type]);
|
2017-03-16 22:23:52 +08:00
|
|
|
if (ret)
|
|
|
|
return ret;
|
2014-11-09 22:47:56 +08:00
|
|
|
} else if (!cb->ignore_unhandled) {
|
|
|
|
pr_warn("No handler for DMAR structure type %d\n",
|
|
|
|
iter->type);
|
2017-03-16 22:23:52 +08:00
|
|
|
return -EINVAL;
|
2014-11-09 22:47:56 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-03-16 22:23:52 +08:00
|
|
|
return 0;
|
2014-11-09 22:47:56 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static inline int dmar_walk_dmar_table(struct acpi_table_dmar *dmar,
|
|
|
|
struct dmar_res_callback *cb)
|
|
|
|
{
|
|
|
|
return dmar_walk_remapping_entries((void *)(dmar + 1),
|
|
|
|
dmar->header.length - sizeof(*dmar), cb);
|
|
|
|
}
|
|
|
|
|
2007-10-22 07:41:41 +08:00
|
|
|
/**
|
|
|
|
* parse_dmar_table - parses the DMA reporting table
|
|
|
|
*/
|
|
|
|
static int __init
|
|
|
|
parse_dmar_table(void)
|
|
|
|
{
|
|
|
|
struct acpi_table_dmar *dmar;
|
2013-05-20 15:57:32 +08:00
|
|
|
int drhd_count = 0;
|
2017-03-16 22:23:53 +08:00
|
|
|
int ret;
|
2014-11-09 22:47:56 +08:00
|
|
|
struct dmar_res_callback cb = {
|
|
|
|
.print_entry = true,
|
|
|
|
.ignore_unhandled = true,
|
|
|
|
.arg[ACPI_DMAR_TYPE_HARDWARE_UNIT] = &drhd_count,
|
|
|
|
.cb[ACPI_DMAR_TYPE_HARDWARE_UNIT] = &dmar_parse_one_drhd,
|
|
|
|
.cb[ACPI_DMAR_TYPE_RESERVED_MEMORY] = &dmar_parse_one_rmrr,
|
|
|
|
.cb[ACPI_DMAR_TYPE_ROOT_ATS] = &dmar_parse_one_atsr,
|
|
|
|
.cb[ACPI_DMAR_TYPE_HARDWARE_AFFINITY] = &dmar_parse_one_rhsa,
|
|
|
|
.cb[ACPI_DMAR_TYPE_NAMESPACE] = &dmar_parse_one_andd,
|
|
|
|
};
|
2007-10-22 07:41:41 +08:00
|
|
|
|
2008-09-04 07:58:32 +08:00
|
|
|
/*
|
|
|
|
* Do it again, earlier dmar_tbl mapping could be mapped with
|
|
|
|
* fixed map.
|
|
|
|
*/
|
|
|
|
dmar_table_detect();
|
|
|
|
|
2009-07-01 10:31:10 +08:00
|
|
|
/*
|
|
|
|
* ACPI tables may not be DMA protected by tboot, so use DMAR copy
|
|
|
|
* SINIT saved in SinitMleData in TXT heap (which is DMA protected)
|
|
|
|
*/
|
|
|
|
dmar_tbl = tboot_get_dmar_table(dmar_tbl);
|
|
|
|
|
2007-10-22 07:41:41 +08:00
|
|
|
dmar = (struct acpi_table_dmar *)dmar_tbl;
|
|
|
|
if (!dmar)
|
|
|
|
return -ENODEV;
|
|
|
|
|
2008-10-17 09:02:32 +08:00
|
|
|
if (dmar->width < PAGE_SHIFT - 1) {
|
2012-06-09 05:13:11 +08:00
|
|
|
pr_warn("Invalid DMAR haw\n");
|
2007-10-22 07:41:41 +08:00
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
2012-06-09 05:13:11 +08:00
|
|
|
pr_info("Host address width %d\n", dmar->width + 1);
|
2014-11-09 22:47:56 +08:00
|
|
|
ret = dmar_walk_dmar_table(dmar, &cb);
|
|
|
|
if (ret == 0 && drhd_count == 0)
|
2013-05-20 15:57:32 +08:00
|
|
|
pr_warn(FW_BUG "No DRHD structure found in DMAR table\n");
|
2014-11-09 22:47:56 +08:00
|
|
|
|
2007-10-22 07:41:41 +08:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2014-03-07 23:08:36 +08:00
|
|
|
static int dmar_pci_device_match(struct dmar_dev_scope devices[],
|
|
|
|
int cnt, struct pci_dev *dev)
|
2008-07-11 02:16:35 +08:00
|
|
|
{
|
|
|
|
int index;
|
2014-03-07 23:08:36 +08:00
|
|
|
struct device *tmp;
|
2008-07-11 02:16:35 +08:00
|
|
|
|
|
|
|
while (dev) {
|
2014-02-19 14:07:32 +08:00
|
|
|
for_each_active_dev_scope(devices, cnt, index, tmp)
|
2014-03-07 23:08:36 +08:00
|
|
|
if (dev_is_pci(tmp) && dev == to_pci_dev(tmp))
|
2008-07-11 02:16:35 +08:00
|
|
|
return 1;
|
|
|
|
|
|
|
|
/* Check our parent */
|
|
|
|
dev = dev->bus->self;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
struct dmar_drhd_unit *
|
|
|
|
dmar_find_matched_drhd_unit(struct pci_dev *dev)
|
|
|
|
{
|
2014-02-19 14:07:34 +08:00
|
|
|
struct dmar_drhd_unit *dmaru;
|
2008-12-22 16:54:58 +08:00
|
|
|
struct acpi_dmar_hardware_unit *drhd;
|
|
|
|
|
2010-04-09 08:07:55 +08:00
|
|
|
dev = pci_physfn(dev);
|
|
|
|
|
2014-02-19 14:07:34 +08:00
|
|
|
rcu_read_lock();
|
2013-10-31 17:25:16 +08:00
|
|
|
for_each_drhd_unit(dmaru) {
|
2008-12-22 16:54:58 +08:00
|
|
|
drhd = container_of(dmaru->hdr,
|
|
|
|
struct acpi_dmar_hardware_unit,
|
|
|
|
header);
|
|
|
|
|
|
|
|
if (dmaru->include_all &&
|
|
|
|
drhd->segment == pci_domain_nr(dev->bus))
|
2014-02-19 14:07:34 +08:00
|
|
|
goto out;
|
2008-07-11 02:16:35 +08:00
|
|
|
|
2008-12-22 16:54:58 +08:00
|
|
|
if (dmar_pci_device_match(dmaru->devices,
|
|
|
|
dmaru->devices_cnt, dev))
|
2014-02-19 14:07:34 +08:00
|
|
|
goto out;
|
2008-07-11 02:16:35 +08:00
|
|
|
}
|
2014-02-19 14:07:34 +08:00
|
|
|
dmaru = NULL;
|
|
|
|
out:
|
|
|
|
rcu_read_unlock();
|
2008-07-11 02:16:35 +08:00
|
|
|
|
2014-02-19 14:07:34 +08:00
|
|
|
return dmaru;
|
2008-07-11 02:16:35 +08:00
|
|
|
}
|
|
|
|
|
2014-03-08 07:15:42 +08:00
|
|
|
static void __init dmar_acpi_insert_dev_scope(u8 device_number,
|
|
|
|
struct acpi_device *adev)
|
|
|
|
{
|
|
|
|
struct dmar_drhd_unit *dmaru;
|
|
|
|
struct acpi_dmar_hardware_unit *drhd;
|
|
|
|
struct acpi_dmar_device_scope *scope;
|
|
|
|
struct device *tmp;
|
|
|
|
int i;
|
|
|
|
struct acpi_dmar_pci_path *path;
|
|
|
|
|
|
|
|
for_each_drhd_unit(dmaru) {
|
|
|
|
drhd = container_of(dmaru->hdr,
|
|
|
|
struct acpi_dmar_hardware_unit,
|
|
|
|
header);
|
|
|
|
|
|
|
|
for (scope = (void *)(drhd + 1);
|
|
|
|
(unsigned long)scope < ((unsigned long)drhd) + drhd->header.length;
|
|
|
|
scope = ((void *)scope) + scope->length) {
|
2014-07-30 12:21:00 +08:00
|
|
|
if (scope->entry_type != ACPI_DMAR_SCOPE_TYPE_NAMESPACE)
|
2014-03-08 07:15:42 +08:00
|
|
|
continue;
|
|
|
|
if (scope->enumeration_id != device_number)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
path = (void *)(scope + 1);
|
|
|
|
pr_info("ACPI device \"%s\" under DMAR at %llx as %02x:%02x.%d\n",
|
|
|
|
dev_name(&adev->dev), dmaru->reg_base_addr,
|
|
|
|
scope->bus, path->device, path->function);
|
|
|
|
for_each_dev_scope(dmaru->devices, dmaru->devices_cnt, i, tmp)
|
|
|
|
if (tmp == NULL) {
|
|
|
|
dmaru->devices[i].bus = scope->bus;
|
|
|
|
dmaru->devices[i].devfn = PCI_DEVFN(path->device,
|
|
|
|
path->function);
|
|
|
|
rcu_assign_pointer(dmaru->devices[i].dev,
|
|
|
|
get_device(&adev->dev));
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
BUG_ON(i >= dmaru->devices_cnt);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
pr_warn("No IOMMU scope found for ANDD enumeration ID %d (%s)\n",
|
|
|
|
device_number, dev_name(&adev->dev));
|
|
|
|
}
|
|
|
|
|
|
|
|
static int __init dmar_acpi_dev_scope_init(void)
|
|
|
|
{
|
2014-03-26 03:16:40 +08:00
|
|
|
struct acpi_dmar_andd *andd;
|
|
|
|
|
|
|
|
if (dmar_tbl == NULL)
|
|
|
|
return -ENODEV;
|
|
|
|
|
2014-04-01 21:58:36 +08:00
|
|
|
for (andd = (void *)dmar_tbl + sizeof(struct acpi_table_dmar);
|
|
|
|
((unsigned long)andd) < ((unsigned long)dmar_tbl) + dmar_tbl->length;
|
|
|
|
andd = ((void *)andd) + andd->header.length) {
|
2014-07-30 12:21:00 +08:00
|
|
|
if (andd->header.type == ACPI_DMAR_TYPE_NAMESPACE) {
|
2014-03-08 07:15:42 +08:00
|
|
|
acpi_handle h;
|
|
|
|
struct acpi_device *adev;
|
|
|
|
|
|
|
|
if (!ACPI_SUCCESS(acpi_get_handle(ACPI_ROOT_OBJECT,
|
2014-07-30 12:21:00 +08:00
|
|
|
andd->device_name,
|
2014-03-08 07:15:42 +08:00
|
|
|
&h))) {
|
|
|
|
pr_err("Failed to find handle for ACPI object %s\n",
|
2014-07-30 12:21:00 +08:00
|
|
|
andd->device_name);
|
2014-03-08 07:15:42 +08:00
|
|
|
continue;
|
|
|
|
}
|
2014-08-22 05:06:48 +08:00
|
|
|
if (acpi_bus_get_device(h, &adev)) {
|
2014-03-08 07:15:42 +08:00
|
|
|
pr_err("Failed to get device for ACPI object %s\n",
|
2014-07-30 12:21:00 +08:00
|
|
|
andd->device_name);
|
2014-03-08 07:15:42 +08:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
dmar_acpi_insert_dev_scope(andd->device_number, adev);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2008-07-11 02:16:37 +08:00
|
|
|
int __init dmar_dev_scope_init(void)
|
|
|
|
{
|
2014-02-19 14:07:36 +08:00
|
|
|
struct pci_dev *dev = NULL;
|
|
|
|
struct dmar_pci_notify_info *info;
|
2008-07-11 02:16:37 +08:00
|
|
|
|
2014-02-19 14:07:36 +08:00
|
|
|
if (dmar_dev_scope_status != 1)
|
|
|
|
return dmar_dev_scope_status;
|
2011-08-24 08:05:19 +08:00
|
|
|
|
2014-02-19 14:07:36 +08:00
|
|
|
if (list_empty(&dmar_drhd_units)) {
|
|
|
|
dmar_dev_scope_status = -ENODEV;
|
|
|
|
} else {
|
|
|
|
dmar_dev_scope_status = 0;
|
|
|
|
|
2014-03-28 19:28:40 +08:00
|
|
|
dmar_acpi_dev_scope_init();
|
|
|
|
|
2014-02-19 14:07:36 +08:00
|
|
|
for_each_pci_dev(dev) {
|
|
|
|
if (dev->is_virtfn)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
info = dmar_alloc_pci_notify_info(dev,
|
|
|
|
BUS_NOTIFY_ADD_DEVICE);
|
|
|
|
if (!info) {
|
|
|
|
return dmar_dev_scope_status;
|
|
|
|
} else {
|
|
|
|
dmar_pci_bus_add_dev(info);
|
|
|
|
dmar_free_pci_notify_info(info);
|
|
|
|
}
|
|
|
|
}
|
2008-07-11 02:16:37 +08:00
|
|
|
}
|
|
|
|
|
2014-02-19 14:07:36 +08:00
|
|
|
return dmar_dev_scope_status;
|
2008-07-11 02:16:37 +08:00
|
|
|
}
|
|
|
|
|
2018-02-13 00:48:20 +08:00
|
|
|
void __init dmar_register_bus_notifier(void)
|
2017-10-06 21:00:53 +08:00
|
|
|
{
|
|
|
|
bus_register_notifier(&pci_bus_type, &dmar_pci_bus_nb);
|
|
|
|
}
|
|
|
|
|
2007-10-22 07:41:41 +08:00
|
|
|
|
|
|
|
int __init dmar_table_init(void)
|
|
|
|
{
|
2008-07-11 02:16:37 +08:00
|
|
|
static int dmar_table_initialized;
|
2007-11-22 07:07:14 +08:00
|
|
|
int ret;
|
|
|
|
|
2014-01-06 14:18:24 +08:00
|
|
|
if (dmar_table_initialized == 0) {
|
|
|
|
ret = parse_dmar_table();
|
|
|
|
if (ret < 0) {
|
|
|
|
if (ret != -ENODEV)
|
2015-06-12 15:57:06 +08:00
|
|
|
pr_info("Parse DMAR table failure.\n");
|
2014-01-06 14:18:24 +08:00
|
|
|
} else if (list_empty(&dmar_drhd_units)) {
|
|
|
|
pr_info("No DMAR devices found\n");
|
|
|
|
ret = -ENODEV;
|
|
|
|
}
|
2007-11-22 07:07:14 +08:00
|
|
|
|
2014-01-06 14:18:24 +08:00
|
|
|
if (ret < 0)
|
|
|
|
dmar_table_initialized = ret;
|
|
|
|
else
|
|
|
|
dmar_table_initialized = 1;
|
2007-10-22 07:41:41 +08:00
|
|
|
}
|
2007-11-22 07:07:14 +08:00
|
|
|
|
2014-01-06 14:18:24 +08:00
|
|
|
return dmar_table_initialized < 0 ? dmar_table_initialized : 0;
|
2007-10-22 07:41:41 +08:00
|
|
|
}
|
|
|
|
|
2010-04-04 02:37:23 +08:00
|
|
|
static void warn_invalid_dmar(u64 addr, const char *message)
|
|
|
|
{
|
2010-04-04 02:38:43 +08:00
|
|
|
WARN_TAINT_ONCE(
|
|
|
|
1, TAINT_FIRMWARE_WORKAROUND,
|
|
|
|
"Your BIOS is broken; DMAR reported at address %llx%s!\n"
|
|
|
|
"BIOS vendor: %s; Ver: %s; Product Version: %s\n",
|
|
|
|
addr, message,
|
|
|
|
dmi_get_system_info(DMI_BIOS_VENDOR),
|
|
|
|
dmi_get_system_info(DMI_BIOS_VERSION),
|
|
|
|
dmi_get_system_info(DMI_PRODUCT_VERSION));
|
2010-04-04 02:37:23 +08:00
|
|
|
}
|
2009-12-02 17:20:27 +08:00
|
|
|
|
2014-11-09 22:47:56 +08:00
|
|
|
static int __ref
|
|
|
|
dmar_validate_one_drhd(struct acpi_dmar_header *entry, void *arg)
|
2009-11-10 06:15:15 +08:00
|
|
|
{
|
|
|
|
struct acpi_dmar_hardware_unit *drhd;
|
2014-11-09 22:47:56 +08:00
|
|
|
void __iomem *addr;
|
|
|
|
u64 cap, ecap;
|
2009-11-10 06:15:15 +08:00
|
|
|
|
2014-11-09 22:47:56 +08:00
|
|
|
drhd = (void *)entry;
|
|
|
|
if (!drhd->address) {
|
|
|
|
warn_invalid_dmar(0, "");
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
2009-12-02 17:17:13 +08:00
|
|
|
|
2014-11-09 22:47:58 +08:00
|
|
|
if (arg)
|
|
|
|
addr = ioremap(drhd->address, VTD_PAGE_SIZE);
|
|
|
|
else
|
|
|
|
addr = early_ioremap(drhd->address, VTD_PAGE_SIZE);
|
2014-11-09 22:47:56 +08:00
|
|
|
if (!addr) {
|
2015-06-12 15:57:06 +08:00
|
|
|
pr_warn("Can't validate DRHD address: %llx\n", drhd->address);
|
2014-11-09 22:47:56 +08:00
|
|
|
return -EINVAL;
|
|
|
|
}
|
2014-11-09 22:47:58 +08:00
|
|
|
|
2014-11-09 22:47:56 +08:00
|
|
|
cap = dmar_readq(addr + DMAR_CAP_REG);
|
|
|
|
ecap = dmar_readq(addr + DMAR_ECAP_REG);
|
2014-11-09 22:47:58 +08:00
|
|
|
|
|
|
|
if (arg)
|
|
|
|
iounmap(addr);
|
|
|
|
else
|
|
|
|
early_iounmap(addr, VTD_PAGE_SIZE);
|
2009-11-10 06:15:15 +08:00
|
|
|
|
2014-11-09 22:47:56 +08:00
|
|
|
if (cap == (uint64_t)-1 && ecap == (uint64_t)-1) {
|
|
|
|
warn_invalid_dmar(drhd->address, " returns all ones");
|
|
|
|
return -EINVAL;
|
2009-11-10 06:15:15 +08:00
|
|
|
}
|
2009-12-02 17:17:13 +08:00
|
|
|
|
|
|
|
return 0;
|
2009-11-10 06:15:15 +08:00
|
|
|
}
|
|
|
|
|
2010-08-27 01:57:57 +08:00
|
|
|
int __init detect_intel_iommu(void)
|
2008-07-11 02:16:43 +08:00
|
|
|
{
|
|
|
|
int ret;
|
2014-11-09 22:47:56 +08:00
|
|
|
struct dmar_res_callback validate_drhd_cb = {
|
|
|
|
.cb[ACPI_DMAR_TYPE_HARDWARE_UNIT] = &dmar_validate_one_drhd,
|
|
|
|
.ignore_unhandled = true,
|
|
|
|
};
|
2008-07-11 02:16:43 +08:00
|
|
|
|
iommu/vt-d: Introduce a rwsem to protect global data structures
Introduce a global rwsem dmar_global_lock, which will be used to
protect DMAR related global data structures from DMAR/PCI/memory
device hotplug operations in process context.
DMA and interrupt remapping related data structures are read most,
and only change when memory/PCI/DMAR hotplug event happens.
So a global rwsem solution is adopted for balance between simplicity
and performance.
For interrupt remapping driver, function intel_irq_remapping_supported(),
dmar_table_init(), intel_enable_irq_remapping(), disable_irq_remapping(),
reenable_irq_remapping() and enable_drhd_fault_handling() etc
are called during booting, suspending and resuming with interrupt
disabled, so no need to take the global lock.
For interrupt remapping entry allocation, the locking model is:
down_read(&dmar_global_lock);
/* Find corresponding iommu */
iommu = map_hpet_to_ir(id);
if (iommu)
/*
* Allocate remapping entry and mark entry busy,
* the IOMMU won't be hot-removed until the
* allocated entry has been released.
*/
index = alloc_irte(iommu, irq, 1);
up_read(&dmar_global_lock);
For DMA remmaping driver, we only uses the dmar_global_lock rwsem to
protect functions which are only called in process context. For any
function which may be called in interrupt context, we will use RCU
to protect them in following patches.
Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
Signed-off-by: Joerg Roedel <joro@8bytes.org>
2014-02-19 14:07:33 +08:00
|
|
|
down_write(&dmar_global_lock);
|
2008-09-04 07:58:32 +08:00
|
|
|
ret = dmar_table_detect();
|
2017-03-16 22:23:51 +08:00
|
|
|
if (!ret)
|
|
|
|
ret = dmar_walk_dmar_table((struct acpi_table_dmar *)dmar_tbl,
|
|
|
|
&validate_drhd_cb);
|
|
|
|
if (!ret && !no_iommu && !iommu_detected && !dmar_disabled) {
|
2014-11-09 22:47:56 +08:00
|
|
|
iommu_detected = 1;
|
|
|
|
/* Make sure ACS will be enabled */
|
|
|
|
pci_request_acs();
|
|
|
|
}
|
2011-08-24 08:05:22 +08:00
|
|
|
|
2009-11-10 18:46:16 +08:00
|
|
|
#ifdef CONFIG_X86
|
2017-03-16 22:23:51 +08:00
|
|
|
if (!ret)
|
2014-11-09 22:47:56 +08:00
|
|
|
x86_init.iommu.iommu_init = intel_iommu_init;
|
2008-07-11 02:16:43 +08:00
|
|
|
#endif
|
2014-11-09 22:47:56 +08:00
|
|
|
|
2017-01-05 09:13:31 +08:00
|
|
|
if (dmar_tbl) {
|
|
|
|
acpi_put_table(dmar_tbl);
|
|
|
|
dmar_tbl = NULL;
|
|
|
|
}
|
iommu/vt-d: Introduce a rwsem to protect global data structures
Introduce a global rwsem dmar_global_lock, which will be used to
protect DMAR related global data structures from DMAR/PCI/memory
device hotplug operations in process context.
DMA and interrupt remapping related data structures are read most,
and only change when memory/PCI/DMAR hotplug event happens.
So a global rwsem solution is adopted for balance between simplicity
and performance.
For interrupt remapping driver, function intel_irq_remapping_supported(),
dmar_table_init(), intel_enable_irq_remapping(), disable_irq_remapping(),
reenable_irq_remapping() and enable_drhd_fault_handling() etc
are called during booting, suspending and resuming with interrupt
disabled, so no need to take the global lock.
For interrupt remapping entry allocation, the locking model is:
down_read(&dmar_global_lock);
/* Find corresponding iommu */
iommu = map_hpet_to_ir(id);
if (iommu)
/*
* Allocate remapping entry and mark entry busy,
* the IOMMU won't be hot-removed until the
* allocated entry has been released.
*/
index = alloc_irte(iommu, irq, 1);
up_read(&dmar_global_lock);
For DMA remmaping driver, we only uses the dmar_global_lock rwsem to
protect functions which are only called in process context. For any
function which may be called in interrupt context, we will use RCU
to protect them in following patches.
Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
Signed-off-by: Joerg Roedel <joro@8bytes.org>
2014-02-19 14:07:33 +08:00
|
|
|
up_write(&dmar_global_lock);
|
2010-08-27 01:57:57 +08:00
|
|
|
|
2017-03-16 22:23:51 +08:00
|
|
|
return ret ? ret : 1;
|
2008-07-11 02:16:43 +08:00
|
|
|
}
|
|
|
|
|
2012-06-05 05:29:02 +08:00
|
|
|
static void unmap_iommu(struct intel_iommu *iommu)
|
|
|
|
{
|
|
|
|
iounmap(iommu->reg);
|
|
|
|
release_mem_region(iommu->reg_phys, iommu->reg_size);
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* map_iommu: map the iommu's registers
|
|
|
|
* @iommu: the iommu to map
|
|
|
|
* @phys_addr: the physical address of the base resgister
|
2012-06-09 05:13:11 +08:00
|
|
|
*
|
2012-06-05 05:29:02 +08:00
|
|
|
* Memory map the iommu's registers. Start w/ a single page, and
|
2012-06-09 05:13:11 +08:00
|
|
|
* possibly expand if that turns out to be insufficent.
|
2012-06-05 05:29:02 +08:00
|
|
|
*/
|
|
|
|
static int map_iommu(struct intel_iommu *iommu, u64 phys_addr)
|
|
|
|
{
|
|
|
|
int map_size, err=0;
|
|
|
|
|
|
|
|
iommu->reg_phys = phys_addr;
|
|
|
|
iommu->reg_size = VTD_PAGE_SIZE;
|
|
|
|
|
|
|
|
if (!request_mem_region(iommu->reg_phys, iommu->reg_size, iommu->name)) {
|
2015-06-12 15:57:06 +08:00
|
|
|
pr_err("Can't reserve memory\n");
|
2012-06-05 05:29:02 +08:00
|
|
|
err = -EBUSY;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
iommu->reg = ioremap(iommu->reg_phys, iommu->reg_size);
|
|
|
|
if (!iommu->reg) {
|
2015-06-12 15:57:06 +08:00
|
|
|
pr_err("Can't map the region\n");
|
2012-06-05 05:29:02 +08:00
|
|
|
err = -ENOMEM;
|
|
|
|
goto release;
|
|
|
|
}
|
|
|
|
|
|
|
|
iommu->cap = dmar_readq(iommu->reg + DMAR_CAP_REG);
|
|
|
|
iommu->ecap = dmar_readq(iommu->reg + DMAR_ECAP_REG);
|
|
|
|
|
|
|
|
if (iommu->cap == (uint64_t)-1 && iommu->ecap == (uint64_t)-1) {
|
|
|
|
err = -EINVAL;
|
|
|
|
warn_invalid_dmar(phys_addr, " returns all ones");
|
|
|
|
goto unmap;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* the registers might be more than one page */
|
|
|
|
map_size = max_t(int, ecap_max_iotlb_offset(iommu->ecap),
|
|
|
|
cap_max_fault_reg_offset(iommu->cap));
|
|
|
|
map_size = VTD_PAGE_ALIGN(map_size);
|
|
|
|
if (map_size > iommu->reg_size) {
|
|
|
|
iounmap(iommu->reg);
|
|
|
|
release_mem_region(iommu->reg_phys, iommu->reg_size);
|
|
|
|
iommu->reg_size = map_size;
|
|
|
|
if (!request_mem_region(iommu->reg_phys, iommu->reg_size,
|
|
|
|
iommu->name)) {
|
2015-06-12 15:57:06 +08:00
|
|
|
pr_err("Can't reserve memory\n");
|
2012-06-05 05:29:02 +08:00
|
|
|
err = -EBUSY;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
iommu->reg = ioremap(iommu->reg_phys, iommu->reg_size);
|
|
|
|
if (!iommu->reg) {
|
2015-06-12 15:57:06 +08:00
|
|
|
pr_err("Can't map the region\n");
|
2012-06-05 05:29:02 +08:00
|
|
|
err = -ENOMEM;
|
|
|
|
goto release;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
err = 0;
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
unmap:
|
|
|
|
iounmap(iommu->reg);
|
|
|
|
release:
|
|
|
|
release_mem_region(iommu->reg_phys, iommu->reg_size);
|
|
|
|
out:
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2014-11-09 22:47:57 +08:00
|
|
|
static int dmar_alloc_seq_id(struct intel_iommu *iommu)
|
|
|
|
{
|
|
|
|
iommu->seq_id = find_first_zero_bit(dmar_seq_ids,
|
|
|
|
DMAR_UNITS_SUPPORTED);
|
|
|
|
if (iommu->seq_id >= DMAR_UNITS_SUPPORTED) {
|
|
|
|
iommu->seq_id = -1;
|
|
|
|
} else {
|
|
|
|
set_bit(iommu->seq_id, dmar_seq_ids);
|
|
|
|
sprintf(iommu->name, "dmar%d", iommu->seq_id);
|
|
|
|
}
|
|
|
|
|
|
|
|
return iommu->seq_id;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void dmar_free_seq_id(struct intel_iommu *iommu)
|
|
|
|
{
|
|
|
|
if (iommu->seq_id >= 0) {
|
|
|
|
clear_bit(iommu->seq_id, dmar_seq_ids);
|
|
|
|
iommu->seq_id = -1;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2014-01-06 14:18:16 +08:00
|
|
|
static int alloc_iommu(struct dmar_drhd_unit *drhd)
|
2008-07-11 02:16:35 +08:00
|
|
|
{
|
2008-07-11 02:16:36 +08:00
|
|
|
struct intel_iommu *iommu;
|
2013-04-23 16:35:03 +08:00
|
|
|
u32 ver, sts;
|
2009-01-04 06:56:27 +08:00
|
|
|
int agaw = 0;
|
2009-04-25 08:30:20 +08:00
|
|
|
int msagaw = 0;
|
2012-06-05 05:29:02 +08:00
|
|
|
int err;
|
2008-07-11 02:16:36 +08:00
|
|
|
|
2009-12-02 17:20:27 +08:00
|
|
|
if (!drhd->reg_base_addr) {
|
2010-04-04 02:37:23 +08:00
|
|
|
warn_invalid_dmar(0, "");
|
2009-12-02 17:20:27 +08:00
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
2008-07-11 02:16:36 +08:00
|
|
|
iommu = kzalloc(sizeof(*iommu), GFP_KERNEL);
|
|
|
|
if (!iommu)
|
2008-07-11 02:16:37 +08:00
|
|
|
return -ENOMEM;
|
2008-07-11 02:16:36 +08:00
|
|
|
|
2014-11-09 22:47:57 +08:00
|
|
|
if (dmar_alloc_seq_id(iommu) < 0) {
|
2015-06-12 15:57:06 +08:00
|
|
|
pr_err("Failed to allocate seq_id\n");
|
2014-11-09 22:47:57 +08:00
|
|
|
err = -ENOSPC;
|
|
|
|
goto error;
|
|
|
|
}
|
2008-07-11 02:16:35 +08:00
|
|
|
|
2012-06-05 05:29:02 +08:00
|
|
|
err = map_iommu(iommu, drhd->reg_base_addr);
|
|
|
|
if (err) {
|
2015-06-12 15:57:06 +08:00
|
|
|
pr_err("Failed to map %s\n", iommu->name);
|
2014-11-09 22:47:57 +08:00
|
|
|
goto error_free_seq_id;
|
2008-07-11 02:16:35 +08:00
|
|
|
}
|
2009-08-04 16:17:20 +08:00
|
|
|
|
2012-06-05 05:29:02 +08:00
|
|
|
err = -EINVAL;
|
2008-12-08 15:34:06 +08:00
|
|
|
agaw = iommu_calculate_agaw(iommu);
|
|
|
|
if (agaw < 0) {
|
2012-06-05 05:29:01 +08:00
|
|
|
pr_err("Cannot get a valid agaw for iommu (seq_id = %d)\n",
|
|
|
|
iommu->seq_id);
|
2009-08-04 16:17:20 +08:00
|
|
|
goto err_unmap;
|
2009-04-25 08:30:20 +08:00
|
|
|
}
|
|
|
|
msagaw = iommu_calculate_max_sagaw(iommu);
|
|
|
|
if (msagaw < 0) {
|
2012-06-05 05:29:01 +08:00
|
|
|
pr_err("Cannot get a valid max agaw for iommu (seq_id = %d)\n",
|
2008-12-08 15:34:06 +08:00
|
|
|
iommu->seq_id);
|
2009-08-04 16:17:20 +08:00
|
|
|
goto err_unmap;
|
2008-12-08 15:34:06 +08:00
|
|
|
}
|
|
|
|
iommu->agaw = agaw;
|
2009-04-25 08:30:20 +08:00
|
|
|
iommu->msagaw = msagaw;
|
2014-03-10 04:49:45 +08:00
|
|
|
iommu->segment = drhd->segment;
|
2008-12-08 15:34:06 +08:00
|
|
|
|
2009-10-03 02:01:21 +08:00
|
|
|
iommu->node = -1;
|
|
|
|
|
2008-07-11 02:16:35 +08:00
|
|
|
ver = readl(iommu->reg + DMAR_VER_REG);
|
2015-06-12 15:57:06 +08:00
|
|
|
pr_info("%s: reg_base_addr %llx ver %d:%d cap %llx ecap %llx\n",
|
|
|
|
iommu->name,
|
2008-10-17 09:02:32 +08:00
|
|
|
(unsigned long long)drhd->reg_base_addr,
|
|
|
|
DMAR_VER_MAJOR(ver), DMAR_VER_MINOR(ver),
|
|
|
|
(unsigned long long)iommu->cap,
|
|
|
|
(unsigned long long)iommu->ecap);
|
2008-07-11 02:16:35 +08:00
|
|
|
|
2013-04-23 16:35:03 +08:00
|
|
|
/* Reflect status in gcmd */
|
|
|
|
sts = readl(iommu->reg + DMAR_GSTS_REG);
|
|
|
|
if (sts & DMA_GSTS_IRES)
|
|
|
|
iommu->gcmd |= DMA_GCMD_IRE;
|
|
|
|
if (sts & DMA_GSTS_TES)
|
|
|
|
iommu->gcmd |= DMA_GCMD_TE;
|
|
|
|
if (sts & DMA_GSTS_QIES)
|
|
|
|
iommu->gcmd |= DMA_GCMD_QIE;
|
|
|
|
|
2011-07-19 22:19:51 +08:00
|
|
|
raw_spin_lock_init(&iommu->register_lock);
|
2008-07-11 02:16:35 +08:00
|
|
|
|
2016-01-07 19:16:51 +08:00
|
|
|
if (intel_iommu_enabled) {
|
2017-02-01 23:56:46 +08:00
|
|
|
err = iommu_device_sysfs_add(&iommu->iommu, NULL,
|
|
|
|
intel_iommu_groups,
|
|
|
|
"%s", iommu->name);
|
|
|
|
if (err)
|
2016-01-07 19:16:51 +08:00
|
|
|
goto err_unmap;
|
iommu/vt-d: Make use of IOMMU sysfs support
Register our DRHD IOMMUs, cross link devices, and provide a base set
of attributes for the IOMMU. Note that IRQ remapping support parses
the DMAR table very early in boot, well before the iommu_class can
reasonably be setup, so our registration is split between
intel_iommu_init(), which occurs later, and alloc_iommu(), which
typically occurs much earlier, but may happen at any time later
with IOMMU hot-add support.
On a typical desktop system, this provides the following (pruned):
$ find /sys | grep dmar
/sys/devices/virtual/iommu/dmar0
/sys/devices/virtual/iommu/dmar0/devices
/sys/devices/virtual/iommu/dmar0/devices/0000:00:02.0
/sys/devices/virtual/iommu/dmar0/intel-iommu
/sys/devices/virtual/iommu/dmar0/intel-iommu/cap
/sys/devices/virtual/iommu/dmar0/intel-iommu/ecap
/sys/devices/virtual/iommu/dmar0/intel-iommu/address
/sys/devices/virtual/iommu/dmar0/intel-iommu/version
/sys/devices/virtual/iommu/dmar1
/sys/devices/virtual/iommu/dmar1/devices
/sys/devices/virtual/iommu/dmar1/devices/0000:00:00.0
/sys/devices/virtual/iommu/dmar1/devices/0000:00:01.0
/sys/devices/virtual/iommu/dmar1/devices/0000:00:16.0
/sys/devices/virtual/iommu/dmar1/devices/0000:00:1a.0
/sys/devices/virtual/iommu/dmar1/devices/0000:00:1b.0
/sys/devices/virtual/iommu/dmar1/devices/0000:00:1c.0
...
/sys/devices/virtual/iommu/dmar1/intel-iommu
/sys/devices/virtual/iommu/dmar1/intel-iommu/cap
/sys/devices/virtual/iommu/dmar1/intel-iommu/ecap
/sys/devices/virtual/iommu/dmar1/intel-iommu/address
/sys/devices/virtual/iommu/dmar1/intel-iommu/version
/sys/class/iommu/dmar0
/sys/class/iommu/dmar1
(devices also link back to the dmar units)
This makes address, version, capabilities, and extended capabilities
available, just like printed on boot. I've tried not to duplicate
data that can be found in the DMAR table, with the exception of the
address, which provides an easy way to associate the sysfs device with
a DRHD entry in the DMAR. It's tempting to add scopes and RMRR data
here, but the full DMAR table is already exposed under /sys/firmware/
and therefore already provides a way for userspace to learn such
details.
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
2014-06-13 06:12:31 +08:00
|
|
|
|
2017-02-01 20:23:08 +08:00
|
|
|
iommu_device_set_ops(&iommu->iommu, &intel_iommu_ops);
|
|
|
|
|
|
|
|
err = iommu_device_register(&iommu->iommu);
|
|
|
|
if (err)
|
2016-01-07 19:16:51 +08:00
|
|
|
goto err_unmap;
|
2016-01-05 07:27:57 +08:00
|
|
|
}
|
|
|
|
|
2016-01-07 19:16:51 +08:00
|
|
|
drhd->iommu = iommu;
|
|
|
|
|
2008-07-11 02:16:37 +08:00
|
|
|
return 0;
|
2009-08-04 16:17:20 +08:00
|
|
|
|
2014-11-09 22:47:57 +08:00
|
|
|
err_unmap:
|
2012-06-05 05:29:02 +08:00
|
|
|
unmap_iommu(iommu);
|
2014-11-09 22:47:57 +08:00
|
|
|
error_free_seq_id:
|
|
|
|
dmar_free_seq_id(iommu);
|
|
|
|
error:
|
2008-07-11 02:16:35 +08:00
|
|
|
kfree(iommu);
|
2012-06-05 05:29:02 +08:00
|
|
|
return err;
|
2008-07-11 02:16:35 +08:00
|
|
|
}
|
|
|
|
|
2014-01-06 14:18:20 +08:00
|
|
|
static void free_iommu(struct intel_iommu *iommu)
|
2008-07-11 02:16:35 +08:00
|
|
|
{
|
2017-02-15 22:42:21 +08:00
|
|
|
if (intel_iommu_enabled) {
|
|
|
|
iommu_device_unregister(&iommu->iommu);
|
|
|
|
iommu_device_sysfs_remove(&iommu->iommu);
|
|
|
|
}
|
iommu/vt-d: Make use of IOMMU sysfs support
Register our DRHD IOMMUs, cross link devices, and provide a base set
of attributes for the IOMMU. Note that IRQ remapping support parses
the DMAR table very early in boot, well before the iommu_class can
reasonably be setup, so our registration is split between
intel_iommu_init(), which occurs later, and alloc_iommu(), which
typically occurs much earlier, but may happen at any time later
with IOMMU hot-add support.
On a typical desktop system, this provides the following (pruned):
$ find /sys | grep dmar
/sys/devices/virtual/iommu/dmar0
/sys/devices/virtual/iommu/dmar0/devices
/sys/devices/virtual/iommu/dmar0/devices/0000:00:02.0
/sys/devices/virtual/iommu/dmar0/intel-iommu
/sys/devices/virtual/iommu/dmar0/intel-iommu/cap
/sys/devices/virtual/iommu/dmar0/intel-iommu/ecap
/sys/devices/virtual/iommu/dmar0/intel-iommu/address
/sys/devices/virtual/iommu/dmar0/intel-iommu/version
/sys/devices/virtual/iommu/dmar1
/sys/devices/virtual/iommu/dmar1/devices
/sys/devices/virtual/iommu/dmar1/devices/0000:00:00.0
/sys/devices/virtual/iommu/dmar1/devices/0000:00:01.0
/sys/devices/virtual/iommu/dmar1/devices/0000:00:16.0
/sys/devices/virtual/iommu/dmar1/devices/0000:00:1a.0
/sys/devices/virtual/iommu/dmar1/devices/0000:00:1b.0
/sys/devices/virtual/iommu/dmar1/devices/0000:00:1c.0
...
/sys/devices/virtual/iommu/dmar1/intel-iommu
/sys/devices/virtual/iommu/dmar1/intel-iommu/cap
/sys/devices/virtual/iommu/dmar1/intel-iommu/ecap
/sys/devices/virtual/iommu/dmar1/intel-iommu/address
/sys/devices/virtual/iommu/dmar1/intel-iommu/version
/sys/class/iommu/dmar0
/sys/class/iommu/dmar1
(devices also link back to the dmar units)
This makes address, version, capabilities, and extended capabilities
available, just like printed on boot. I've tried not to duplicate
data that can be found in the DMAR table, with the exception of the
address, which provides an easy way to associate the sysfs device with
a DRHD entry in the DMAR. It's tempting to add scopes and RMRR data
here, but the full DMAR table is already exposed under /sys/firmware/
and therefore already provides a way for userspace to learn such
details.
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
2014-06-13 06:12:31 +08:00
|
|
|
|
2014-01-06 14:18:20 +08:00
|
|
|
if (iommu->irq) {
|
2015-10-07 22:37:03 +08:00
|
|
|
if (iommu->pr_irq) {
|
|
|
|
free_irq(iommu->pr_irq, iommu);
|
|
|
|
dmar_free_hwirq(iommu->pr_irq);
|
|
|
|
iommu->pr_irq = 0;
|
|
|
|
}
|
2014-01-06 14:18:20 +08:00
|
|
|
free_irq(iommu->irq, iommu);
|
2014-05-07 23:44:11 +08:00
|
|
|
dmar_free_hwirq(iommu->irq);
|
2015-04-13 14:11:41 +08:00
|
|
|
iommu->irq = 0;
|
2014-01-06 14:18:20 +08:00
|
|
|
}
|
2008-07-11 02:16:35 +08:00
|
|
|
|
2014-01-06 14:18:23 +08:00
|
|
|
if (iommu->qi) {
|
|
|
|
free_page((unsigned long)iommu->qi->desc);
|
|
|
|
kfree(iommu->qi->desc_status);
|
|
|
|
kfree(iommu->qi);
|
|
|
|
}
|
|
|
|
|
2008-07-11 02:16:35 +08:00
|
|
|
if (iommu->reg)
|
2012-06-05 05:29:02 +08:00
|
|
|
unmap_iommu(iommu);
|
|
|
|
|
2014-11-09 22:47:57 +08:00
|
|
|
dmar_free_seq_id(iommu);
|
2008-07-11 02:16:35 +08:00
|
|
|
kfree(iommu);
|
|
|
|
}
|
2008-07-11 02:16:42 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Reclaim all the submitted descriptors which have completed its work.
|
|
|
|
*/
|
|
|
|
static inline void reclaim_free_desc(struct q_inval *qi)
|
|
|
|
{
|
2009-05-18 13:51:35 +08:00
|
|
|
while (qi->desc_status[qi->free_tail] == QI_DONE ||
|
|
|
|
qi->desc_status[qi->free_tail] == QI_ABORT) {
|
2008-07-11 02:16:42 +08:00
|
|
|
qi->desc_status[qi->free_tail] = QI_FREE;
|
|
|
|
qi->free_tail = (qi->free_tail + 1) % QI_LENGTH;
|
|
|
|
qi->free_cnt++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2009-01-04 16:28:52 +08:00
|
|
|
static int qi_check_fault(struct intel_iommu *iommu, int index)
|
|
|
|
{
|
|
|
|
u32 fault;
|
2009-05-18 13:51:35 +08:00
|
|
|
int head, tail;
|
2009-01-04 16:28:52 +08:00
|
|
|
struct q_inval *qi = iommu->qi;
|
|
|
|
int wait_index = (index + 1) % QI_LENGTH;
|
2018-12-10 09:58:58 +08:00
|
|
|
int shift = qi_shift(iommu);
|
2009-01-04 16:28:52 +08:00
|
|
|
|
2009-05-18 13:51:35 +08:00
|
|
|
if (qi->desc_status[wait_index] == QI_ABORT)
|
|
|
|
return -EAGAIN;
|
|
|
|
|
2009-01-04 16:28:52 +08:00
|
|
|
fault = readl(iommu->reg + DMAR_FSTS_REG);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If IQE happens, the head points to the descriptor associated
|
|
|
|
* with the error. No new descriptors are fetched until the IQE
|
|
|
|
* is cleared.
|
|
|
|
*/
|
|
|
|
if (fault & DMA_FSTS_IQE) {
|
|
|
|
head = readl(iommu->reg + DMAR_IQH_REG);
|
2018-12-10 09:58:58 +08:00
|
|
|
if ((head >> shift) == index) {
|
|
|
|
struct qi_desc *desc = qi->desc + head;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* desc->qw2 and desc->qw3 are either reserved or
|
|
|
|
* used by software as private data. We won't print
|
|
|
|
* out these two qw's for security consideration.
|
|
|
|
*/
|
|
|
|
pr_err("VT-d detected invalid descriptor: qw0 = %llx, qw1 = %llx\n",
|
|
|
|
(unsigned long long)desc->qw0,
|
|
|
|
(unsigned long long)desc->qw1);
|
|
|
|
memcpy(desc, qi->desc + (wait_index << shift),
|
|
|
|
1 << shift);
|
2009-01-04 16:28:52 +08:00
|
|
|
writel(DMA_FSTS_IQE, iommu->reg + DMAR_FSTS_REG);
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2009-05-18 13:51:35 +08:00
|
|
|
/*
|
|
|
|
* If ITE happens, all pending wait_desc commands are aborted.
|
|
|
|
* No new descriptors are fetched until the ITE is cleared.
|
|
|
|
*/
|
|
|
|
if (fault & DMA_FSTS_ITE) {
|
|
|
|
head = readl(iommu->reg + DMAR_IQH_REG);
|
2018-12-10 09:58:58 +08:00
|
|
|
head = ((head >> shift) - 1 + QI_LENGTH) % QI_LENGTH;
|
2009-05-18 13:51:35 +08:00
|
|
|
head |= 1;
|
|
|
|
tail = readl(iommu->reg + DMAR_IQT_REG);
|
2018-12-10 09:58:58 +08:00
|
|
|
tail = ((tail >> shift) - 1 + QI_LENGTH) % QI_LENGTH;
|
2009-05-18 13:51:35 +08:00
|
|
|
|
|
|
|
writel(DMA_FSTS_ITE, iommu->reg + DMAR_FSTS_REG);
|
|
|
|
|
|
|
|
do {
|
|
|
|
if (qi->desc_status[head] == QI_IN_USE)
|
|
|
|
qi->desc_status[head] = QI_ABORT;
|
|
|
|
head = (head - 2 + QI_LENGTH) % QI_LENGTH;
|
|
|
|
} while (head != tail);
|
|
|
|
|
|
|
|
if (qi->desc_status[wait_index] == QI_ABORT)
|
|
|
|
return -EAGAIN;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (fault & DMA_FSTS_ICE)
|
|
|
|
writel(DMA_FSTS_ICE, iommu->reg + DMAR_FSTS_REG);
|
|
|
|
|
2009-01-04 16:28:52 +08:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2008-07-11 02:16:42 +08:00
|
|
|
/*
|
|
|
|
* Submit the queued invalidation descriptor to the remapping
|
|
|
|
* hardware unit and wait for its completion.
|
|
|
|
*/
|
2009-01-04 16:28:52 +08:00
|
|
|
int qi_submit_sync(struct qi_desc *desc, struct intel_iommu *iommu)
|
2008-07-11 02:16:42 +08:00
|
|
|
{
|
2009-05-18 13:51:35 +08:00
|
|
|
int rc;
|
2008-07-11 02:16:42 +08:00
|
|
|
struct q_inval *qi = iommu->qi;
|
2018-12-10 09:58:58 +08:00
|
|
|
int offset, shift, length;
|
|
|
|
struct qi_desc wait_desc;
|
2008-07-11 02:16:42 +08:00
|
|
|
int wait_index, index;
|
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
if (!qi)
|
2009-01-04 16:28:52 +08:00
|
|
|
return 0;
|
2008-07-11 02:16:42 +08:00
|
|
|
|
2009-05-18 13:51:35 +08:00
|
|
|
restart:
|
|
|
|
rc = 0;
|
|
|
|
|
2011-07-19 23:02:07 +08:00
|
|
|
raw_spin_lock_irqsave(&qi->q_lock, flags);
|
2008-07-11 02:16:42 +08:00
|
|
|
while (qi->free_cnt < 3) {
|
2011-07-19 23:02:07 +08:00
|
|
|
raw_spin_unlock_irqrestore(&qi->q_lock, flags);
|
2008-07-11 02:16:42 +08:00
|
|
|
cpu_relax();
|
2011-07-19 23:02:07 +08:00
|
|
|
raw_spin_lock_irqsave(&qi->q_lock, flags);
|
2008-07-11 02:16:42 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
index = qi->free_head;
|
|
|
|
wait_index = (index + 1) % QI_LENGTH;
|
2018-12-10 09:58:58 +08:00
|
|
|
shift = qi_shift(iommu);
|
|
|
|
length = 1 << shift;
|
2008-07-11 02:16:42 +08:00
|
|
|
|
|
|
|
qi->desc_status[index] = qi->desc_status[wait_index] = QI_IN_USE;
|
|
|
|
|
2018-12-10 09:58:58 +08:00
|
|
|
offset = index << shift;
|
|
|
|
memcpy(qi->desc + offset, desc, length);
|
|
|
|
wait_desc.qw0 = QI_IWD_STATUS_DATA(QI_DONE) |
|
2009-01-04 16:28:52 +08:00
|
|
|
QI_IWD_STATUS_WRITE | QI_IWD_TYPE;
|
2018-12-10 09:58:58 +08:00
|
|
|
wait_desc.qw1 = virt_to_phys(&qi->desc_status[wait_index]);
|
|
|
|
wait_desc.qw2 = 0;
|
|
|
|
wait_desc.qw3 = 0;
|
2008-07-11 02:16:42 +08:00
|
|
|
|
2018-12-10 09:58:58 +08:00
|
|
|
offset = wait_index << shift;
|
|
|
|
memcpy(qi->desc + offset, &wait_desc, length);
|
2008-07-11 02:16:42 +08:00
|
|
|
|
|
|
|
qi->free_head = (qi->free_head + 2) % QI_LENGTH;
|
|
|
|
qi->free_cnt -= 2;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* update the HW tail register indicating the presence of
|
|
|
|
* new descriptors.
|
|
|
|
*/
|
2018-12-10 09:58:58 +08:00
|
|
|
writel(qi->free_head << shift, iommu->reg + DMAR_IQT_REG);
|
2008-07-11 02:16:42 +08:00
|
|
|
|
|
|
|
while (qi->desc_status[wait_index] != QI_DONE) {
|
2008-10-17 07:31:54 +08:00
|
|
|
/*
|
|
|
|
* We will leave the interrupts disabled, to prevent interrupt
|
|
|
|
* context to queue another cmd while a cmd is already submitted
|
|
|
|
* and waiting for completion on this cpu. This is to avoid
|
|
|
|
* a deadlock where the interrupt context can wait indefinitely
|
|
|
|
* for free slots in the queue.
|
|
|
|
*/
|
2009-01-04 16:28:52 +08:00
|
|
|
rc = qi_check_fault(iommu, index);
|
|
|
|
if (rc)
|
2009-05-18 13:51:35 +08:00
|
|
|
break;
|
2009-01-04 16:28:52 +08:00
|
|
|
|
2011-07-19 23:02:07 +08:00
|
|
|
raw_spin_unlock(&qi->q_lock);
|
2008-07-11 02:16:42 +08:00
|
|
|
cpu_relax();
|
2011-07-19 23:02:07 +08:00
|
|
|
raw_spin_lock(&qi->q_lock);
|
2008-07-11 02:16:42 +08:00
|
|
|
}
|
2009-05-18 13:51:35 +08:00
|
|
|
|
|
|
|
qi->desc_status[index] = QI_DONE;
|
2008-07-11 02:16:42 +08:00
|
|
|
|
|
|
|
reclaim_free_desc(qi);
|
2011-07-19 23:02:07 +08:00
|
|
|
raw_spin_unlock_irqrestore(&qi->q_lock, flags);
|
2009-01-04 16:28:52 +08:00
|
|
|
|
2009-05-18 13:51:35 +08:00
|
|
|
if (rc == -EAGAIN)
|
|
|
|
goto restart;
|
|
|
|
|
2009-01-04 16:28:52 +08:00
|
|
|
return rc;
|
2008-07-11 02:16:42 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Flush the global interrupt entry cache.
|
|
|
|
*/
|
|
|
|
void qi_global_iec(struct intel_iommu *iommu)
|
|
|
|
{
|
|
|
|
struct qi_desc desc;
|
|
|
|
|
2018-12-10 09:58:58 +08:00
|
|
|
desc.qw0 = QI_IEC_TYPE;
|
|
|
|
desc.qw1 = 0;
|
|
|
|
desc.qw2 = 0;
|
|
|
|
desc.qw3 = 0;
|
2008-07-11 02:16:42 +08:00
|
|
|
|
2009-01-04 16:28:52 +08:00
|
|
|
/* should never fail */
|
2008-07-11 02:16:42 +08:00
|
|
|
qi_submit_sync(&desc, iommu);
|
|
|
|
}
|
|
|
|
|
2009-05-11 00:16:06 +08:00
|
|
|
void qi_flush_context(struct intel_iommu *iommu, u16 did, u16 sid, u8 fm,
|
|
|
|
u64 type)
|
2008-10-17 07:31:55 +08:00
|
|
|
{
|
|
|
|
struct qi_desc desc;
|
|
|
|
|
2018-12-10 09:58:58 +08:00
|
|
|
desc.qw0 = QI_CC_FM(fm) | QI_CC_SID(sid) | QI_CC_DID(did)
|
2008-10-17 07:31:55 +08:00
|
|
|
| QI_CC_GRAN(type) | QI_CC_TYPE;
|
2018-12-10 09:58:58 +08:00
|
|
|
desc.qw1 = 0;
|
|
|
|
desc.qw2 = 0;
|
|
|
|
desc.qw3 = 0;
|
2008-10-17 07:31:55 +08:00
|
|
|
|
2009-05-11 00:16:06 +08:00
|
|
|
qi_submit_sync(&desc, iommu);
|
2008-10-17 07:31:55 +08:00
|
|
|
}
|
|
|
|
|
2009-05-11 02:58:49 +08:00
|
|
|
void qi_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr,
|
|
|
|
unsigned int size_order, u64 type)
|
2008-10-17 07:31:55 +08:00
|
|
|
{
|
|
|
|
u8 dw = 0, dr = 0;
|
|
|
|
|
|
|
|
struct qi_desc desc;
|
|
|
|
int ih = 0;
|
|
|
|
|
|
|
|
if (cap_write_drain(iommu->cap))
|
|
|
|
dw = 1;
|
|
|
|
|
|
|
|
if (cap_read_drain(iommu->cap))
|
|
|
|
dr = 1;
|
|
|
|
|
2018-12-10 09:58:58 +08:00
|
|
|
desc.qw0 = QI_IOTLB_DID(did) | QI_IOTLB_DR(dr) | QI_IOTLB_DW(dw)
|
2008-10-17 07:31:55 +08:00
|
|
|
| QI_IOTLB_GRAN(type) | QI_IOTLB_TYPE;
|
2018-12-10 09:58:58 +08:00
|
|
|
desc.qw1 = QI_IOTLB_ADDR(addr) | QI_IOTLB_IH(ih)
|
2008-10-17 07:31:55 +08:00
|
|
|
| QI_IOTLB_AM(size_order);
|
2018-12-10 09:58:58 +08:00
|
|
|
desc.qw2 = 0;
|
|
|
|
desc.qw3 = 0;
|
2008-10-17 07:31:55 +08:00
|
|
|
|
2009-05-11 02:58:49 +08:00
|
|
|
qi_submit_sync(&desc, iommu);
|
2008-10-17 07:31:55 +08:00
|
|
|
}
|
|
|
|
|
2018-06-08 00:57:00 +08:00
|
|
|
void qi_flush_dev_iotlb(struct intel_iommu *iommu, u16 sid, u16 pfsid,
|
|
|
|
u16 qdep, u64 addr, unsigned mask)
|
2009-05-18 13:51:35 +08:00
|
|
|
{
|
|
|
|
struct qi_desc desc;
|
|
|
|
|
|
|
|
if (mask) {
|
2018-05-03 21:25:17 +08:00
|
|
|
WARN_ON_ONCE(addr & ((1ULL << (VTD_PAGE_SHIFT + mask)) - 1));
|
2017-08-11 17:42:46 +08:00
|
|
|
addr |= (1ULL << (VTD_PAGE_SHIFT + mask - 1)) - 1;
|
2018-12-10 09:58:58 +08:00
|
|
|
desc.qw1 = QI_DEV_IOTLB_ADDR(addr) | QI_DEV_IOTLB_SIZE;
|
2009-05-18 13:51:35 +08:00
|
|
|
} else
|
2018-12-10 09:58:58 +08:00
|
|
|
desc.qw1 = QI_DEV_IOTLB_ADDR(addr);
|
2009-05-18 13:51:35 +08:00
|
|
|
|
|
|
|
if (qdep >= QI_DEV_IOTLB_MAX_INVS)
|
|
|
|
qdep = 0;
|
|
|
|
|
2018-12-10 09:58:58 +08:00
|
|
|
desc.qw0 = QI_DEV_IOTLB_SID(sid) | QI_DEV_IOTLB_QDEP(qdep) |
|
2018-06-08 00:57:00 +08:00
|
|
|
QI_DIOTLB_TYPE | QI_DEV_IOTLB_PFSID(pfsid);
|
2018-12-10 09:58:58 +08:00
|
|
|
desc.qw2 = 0;
|
|
|
|
desc.qw3 = 0;
|
2009-05-18 13:51:35 +08:00
|
|
|
|
|
|
|
qi_submit_sync(&desc, iommu);
|
|
|
|
}
|
|
|
|
|
2009-03-17 08:04:56 +08:00
|
|
|
/*
|
|
|
|
* Disable Queued Invalidation interface.
|
|
|
|
*/
|
|
|
|
void dmar_disable_qi(struct intel_iommu *iommu)
|
|
|
|
{
|
|
|
|
unsigned long flags;
|
|
|
|
u32 sts;
|
|
|
|
cycles_t start_time = get_cycles();
|
|
|
|
|
|
|
|
if (!ecap_qis(iommu->ecap))
|
|
|
|
return;
|
|
|
|
|
2011-07-19 22:19:51 +08:00
|
|
|
raw_spin_lock_irqsave(&iommu->register_lock, flags);
|
2009-03-17 08:04:56 +08:00
|
|
|
|
2016-01-14 05:15:03 +08:00
|
|
|
sts = readl(iommu->reg + DMAR_GSTS_REG);
|
2009-03-17 08:04:56 +08:00
|
|
|
if (!(sts & DMA_GSTS_QIES))
|
|
|
|
goto end;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Give a chance to HW to complete the pending invalidation requests.
|
|
|
|
*/
|
|
|
|
while ((readl(iommu->reg + DMAR_IQT_REG) !=
|
|
|
|
readl(iommu->reg + DMAR_IQH_REG)) &&
|
|
|
|
(DMAR_OPERATION_TIMEOUT > (get_cycles() - start_time)))
|
|
|
|
cpu_relax();
|
|
|
|
|
|
|
|
iommu->gcmd &= ~DMA_GCMD_QIE;
|
|
|
|
writel(iommu->gcmd, iommu->reg + DMAR_GCMD_REG);
|
|
|
|
|
|
|
|
IOMMU_WAIT_OP(iommu, DMAR_GSTS_REG, readl,
|
|
|
|
!(sts & DMA_GSTS_QIES), sts);
|
|
|
|
end:
|
2011-07-19 22:19:51 +08:00
|
|
|
raw_spin_unlock_irqrestore(&iommu->register_lock, flags);
|
2009-03-17 08:04:56 +08:00
|
|
|
}
|
|
|
|
|
2009-03-28 05:22:43 +08:00
|
|
|
/*
|
|
|
|
* Enable queued invalidation.
|
|
|
|
*/
|
|
|
|
static void __dmar_enable_qi(struct intel_iommu *iommu)
|
|
|
|
{
|
2009-05-11 03:30:58 +08:00
|
|
|
u32 sts;
|
2009-03-28 05:22:43 +08:00
|
|
|
unsigned long flags;
|
|
|
|
struct q_inval *qi = iommu->qi;
|
2018-12-10 09:58:58 +08:00
|
|
|
u64 val = virt_to_phys(qi->desc);
|
2009-03-28 05:22:43 +08:00
|
|
|
|
|
|
|
qi->free_head = qi->free_tail = 0;
|
|
|
|
qi->free_cnt = QI_LENGTH;
|
|
|
|
|
2018-12-10 09:58:58 +08:00
|
|
|
/*
|
|
|
|
* Set DW=1 and QS=1 in IQA_REG when Scalable Mode capability
|
|
|
|
* is present.
|
|
|
|
*/
|
|
|
|
if (ecap_smts(iommu->ecap))
|
|
|
|
val |= (1 << 11) | 1;
|
|
|
|
|
2011-07-19 22:19:51 +08:00
|
|
|
raw_spin_lock_irqsave(&iommu->register_lock, flags);
|
2009-03-28 05:22:43 +08:00
|
|
|
|
|
|
|
/* write zero to the tail reg */
|
|
|
|
writel(0, iommu->reg + DMAR_IQT_REG);
|
|
|
|
|
2018-12-10 09:58:58 +08:00
|
|
|
dmar_writeq(iommu->reg + DMAR_IQA_REG, val);
|
2009-03-28 05:22:43 +08:00
|
|
|
|
|
|
|
iommu->gcmd |= DMA_GCMD_QIE;
|
2009-05-11 03:30:58 +08:00
|
|
|
writel(iommu->gcmd, iommu->reg + DMAR_GCMD_REG);
|
2009-03-28 05:22:43 +08:00
|
|
|
|
|
|
|
/* Make sure hardware complete it */
|
|
|
|
IOMMU_WAIT_OP(iommu, DMAR_GSTS_REG, readl, (sts & DMA_GSTS_QIES), sts);
|
|
|
|
|
2011-07-19 22:19:51 +08:00
|
|
|
raw_spin_unlock_irqrestore(&iommu->register_lock, flags);
|
2009-03-28 05:22:43 +08:00
|
|
|
}
|
|
|
|
|
2008-07-11 02:16:42 +08:00
|
|
|
/*
|
|
|
|
* Enable Queued Invalidation interface. This is a must to support
|
|
|
|
* interrupt-remapping. Also used by DMA-remapping, which replaces
|
|
|
|
* register based IOTLB invalidation.
|
|
|
|
*/
|
|
|
|
int dmar_enable_qi(struct intel_iommu *iommu)
|
|
|
|
{
|
|
|
|
struct q_inval *qi;
|
2009-10-03 02:01:22 +08:00
|
|
|
struct page *desc_page;
|
2008-07-11 02:16:42 +08:00
|
|
|
|
|
|
|
if (!ecap_qis(iommu->ecap))
|
|
|
|
return -ENOENT;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* queued invalidation is already setup and enabled.
|
|
|
|
*/
|
|
|
|
if (iommu->qi)
|
|
|
|
return 0;
|
|
|
|
|
2009-03-17 08:05:05 +08:00
|
|
|
iommu->qi = kmalloc(sizeof(*qi), GFP_ATOMIC);
|
2008-07-11 02:16:42 +08:00
|
|
|
if (!iommu->qi)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
qi = iommu->qi;
|
|
|
|
|
2018-12-10 09:58:58 +08:00
|
|
|
/*
|
|
|
|
* Need two pages to accommodate 256 descriptors of 256 bits each
|
|
|
|
* if the remapping hardware supports scalable mode translation.
|
|
|
|
*/
|
|
|
|
desc_page = alloc_pages_node(iommu->node, GFP_ATOMIC | __GFP_ZERO,
|
|
|
|
!!ecap_smts(iommu->ecap));
|
2009-10-03 02:01:22 +08:00
|
|
|
if (!desc_page) {
|
2008-07-11 02:16:42 +08:00
|
|
|
kfree(qi);
|
2014-01-06 14:18:26 +08:00
|
|
|
iommu->qi = NULL;
|
2008-07-11 02:16:42 +08:00
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
|
2009-10-03 02:01:22 +08:00
|
|
|
qi->desc = page_address(desc_page);
|
|
|
|
|
treewide: kzalloc() -> kcalloc()
The kzalloc() function has a 2-factor argument form, kcalloc(). This
patch replaces cases of:
kzalloc(a * b, gfp)
with:
kcalloc(a * b, gfp)
as well as handling cases of:
kzalloc(a * b * c, gfp)
with:
kzalloc(array3_size(a, b, c), gfp)
as it's slightly less ugly than:
kzalloc_array(array_size(a, b), c, gfp)
This does, however, attempt to ignore constant size factors like:
kzalloc(4 * 1024, gfp)
though any constants defined via macros get caught up in the conversion.
Any factors with a sizeof() of "unsigned char", "char", and "u8" were
dropped, since they're redundant.
The Coccinelle script used for this was:
// Fix redundant parens around sizeof().
@@
type TYPE;
expression THING, E;
@@
(
kzalloc(
- (sizeof(TYPE)) * E
+ sizeof(TYPE) * E
, ...)
|
kzalloc(
- (sizeof(THING)) * E
+ sizeof(THING) * E
, ...)
)
// Drop single-byte sizes and redundant parens.
@@
expression COUNT;
typedef u8;
typedef __u8;
@@
(
kzalloc(
- sizeof(u8) * (COUNT)
+ COUNT
, ...)
|
kzalloc(
- sizeof(__u8) * (COUNT)
+ COUNT
, ...)
|
kzalloc(
- sizeof(char) * (COUNT)
+ COUNT
, ...)
|
kzalloc(
- sizeof(unsigned char) * (COUNT)
+ COUNT
, ...)
|
kzalloc(
- sizeof(u8) * COUNT
+ COUNT
, ...)
|
kzalloc(
- sizeof(__u8) * COUNT
+ COUNT
, ...)
|
kzalloc(
- sizeof(char) * COUNT
+ COUNT
, ...)
|
kzalloc(
- sizeof(unsigned char) * COUNT
+ COUNT
, ...)
)
// 2-factor product with sizeof(type/expression) and identifier or constant.
@@
type TYPE;
expression THING;
identifier COUNT_ID;
constant COUNT_CONST;
@@
(
- kzalloc
+ kcalloc
(
- sizeof(TYPE) * (COUNT_ID)
+ COUNT_ID, sizeof(TYPE)
, ...)
|
- kzalloc
+ kcalloc
(
- sizeof(TYPE) * COUNT_ID
+ COUNT_ID, sizeof(TYPE)
, ...)
|
- kzalloc
+ kcalloc
(
- sizeof(TYPE) * (COUNT_CONST)
+ COUNT_CONST, sizeof(TYPE)
, ...)
|
- kzalloc
+ kcalloc
(
- sizeof(TYPE) * COUNT_CONST
+ COUNT_CONST, sizeof(TYPE)
, ...)
|
- kzalloc
+ kcalloc
(
- sizeof(THING) * (COUNT_ID)
+ COUNT_ID, sizeof(THING)
, ...)
|
- kzalloc
+ kcalloc
(
- sizeof(THING) * COUNT_ID
+ COUNT_ID, sizeof(THING)
, ...)
|
- kzalloc
+ kcalloc
(
- sizeof(THING) * (COUNT_CONST)
+ COUNT_CONST, sizeof(THING)
, ...)
|
- kzalloc
+ kcalloc
(
- sizeof(THING) * COUNT_CONST
+ COUNT_CONST, sizeof(THING)
, ...)
)
// 2-factor product, only identifiers.
@@
identifier SIZE, COUNT;
@@
- kzalloc
+ kcalloc
(
- SIZE * COUNT
+ COUNT, SIZE
, ...)
// 3-factor product with 1 sizeof(type) or sizeof(expression), with
// redundant parens removed.
@@
expression THING;
identifier STRIDE, COUNT;
type TYPE;
@@
(
kzalloc(
- sizeof(TYPE) * (COUNT) * (STRIDE)
+ array3_size(COUNT, STRIDE, sizeof(TYPE))
, ...)
|
kzalloc(
- sizeof(TYPE) * (COUNT) * STRIDE
+ array3_size(COUNT, STRIDE, sizeof(TYPE))
, ...)
|
kzalloc(
- sizeof(TYPE) * COUNT * (STRIDE)
+ array3_size(COUNT, STRIDE, sizeof(TYPE))
, ...)
|
kzalloc(
- sizeof(TYPE) * COUNT * STRIDE
+ array3_size(COUNT, STRIDE, sizeof(TYPE))
, ...)
|
kzalloc(
- sizeof(THING) * (COUNT) * (STRIDE)
+ array3_size(COUNT, STRIDE, sizeof(THING))
, ...)
|
kzalloc(
- sizeof(THING) * (COUNT) * STRIDE
+ array3_size(COUNT, STRIDE, sizeof(THING))
, ...)
|
kzalloc(
- sizeof(THING) * COUNT * (STRIDE)
+ array3_size(COUNT, STRIDE, sizeof(THING))
, ...)
|
kzalloc(
- sizeof(THING) * COUNT * STRIDE
+ array3_size(COUNT, STRIDE, sizeof(THING))
, ...)
)
// 3-factor product with 2 sizeof(variable), with redundant parens removed.
@@
expression THING1, THING2;
identifier COUNT;
type TYPE1, TYPE2;
@@
(
kzalloc(
- sizeof(TYPE1) * sizeof(TYPE2) * COUNT
+ array3_size(COUNT, sizeof(TYPE1), sizeof(TYPE2))
, ...)
|
kzalloc(
- sizeof(TYPE1) * sizeof(THING2) * (COUNT)
+ array3_size(COUNT, sizeof(TYPE1), sizeof(TYPE2))
, ...)
|
kzalloc(
- sizeof(THING1) * sizeof(THING2) * COUNT
+ array3_size(COUNT, sizeof(THING1), sizeof(THING2))
, ...)
|
kzalloc(
- sizeof(THING1) * sizeof(THING2) * (COUNT)
+ array3_size(COUNT, sizeof(THING1), sizeof(THING2))
, ...)
|
kzalloc(
- sizeof(TYPE1) * sizeof(THING2) * COUNT
+ array3_size(COUNT, sizeof(TYPE1), sizeof(THING2))
, ...)
|
kzalloc(
- sizeof(TYPE1) * sizeof(THING2) * (COUNT)
+ array3_size(COUNT, sizeof(TYPE1), sizeof(THING2))
, ...)
)
// 3-factor product, only identifiers, with redundant parens removed.
@@
identifier STRIDE, SIZE, COUNT;
@@
(
kzalloc(
- (COUNT) * STRIDE * SIZE
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kzalloc(
- COUNT * (STRIDE) * SIZE
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kzalloc(
- COUNT * STRIDE * (SIZE)
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kzalloc(
- (COUNT) * (STRIDE) * SIZE
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kzalloc(
- COUNT * (STRIDE) * (SIZE)
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kzalloc(
- (COUNT) * STRIDE * (SIZE)
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kzalloc(
- (COUNT) * (STRIDE) * (SIZE)
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kzalloc(
- COUNT * STRIDE * SIZE
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
)
// Any remaining multi-factor products, first at least 3-factor products,
// when they're not all constants...
@@
expression E1, E2, E3;
constant C1, C2, C3;
@@
(
kzalloc(C1 * C2 * C3, ...)
|
kzalloc(
- (E1) * E2 * E3
+ array3_size(E1, E2, E3)
, ...)
|
kzalloc(
- (E1) * (E2) * E3
+ array3_size(E1, E2, E3)
, ...)
|
kzalloc(
- (E1) * (E2) * (E3)
+ array3_size(E1, E2, E3)
, ...)
|
kzalloc(
- E1 * E2 * E3
+ array3_size(E1, E2, E3)
, ...)
)
// And then all remaining 2 factors products when they're not all constants,
// keeping sizeof() as the second factor argument.
@@
expression THING, E1, E2;
type TYPE;
constant C1, C2, C3;
@@
(
kzalloc(sizeof(THING) * C2, ...)
|
kzalloc(sizeof(TYPE) * C2, ...)
|
kzalloc(C1 * C2 * C3, ...)
|
kzalloc(C1 * C2, ...)
|
- kzalloc
+ kcalloc
(
- sizeof(TYPE) * (E2)
+ E2, sizeof(TYPE)
, ...)
|
- kzalloc
+ kcalloc
(
- sizeof(TYPE) * E2
+ E2, sizeof(TYPE)
, ...)
|
- kzalloc
+ kcalloc
(
- sizeof(THING) * (E2)
+ E2, sizeof(THING)
, ...)
|
- kzalloc
+ kcalloc
(
- sizeof(THING) * E2
+ E2, sizeof(THING)
, ...)
|
- kzalloc
+ kcalloc
(
- (E1) * E2
+ E1, E2
, ...)
|
- kzalloc
+ kcalloc
(
- (E1) * (E2)
+ E1, E2
, ...)
|
- kzalloc
+ kcalloc
(
- E1 * E2
+ E1, E2
, ...)
)
Signed-off-by: Kees Cook <keescook@chromium.org>
2018-06-13 05:03:40 +08:00
|
|
|
qi->desc_status = kcalloc(QI_LENGTH, sizeof(int), GFP_ATOMIC);
|
2008-07-11 02:16:42 +08:00
|
|
|
if (!qi->desc_status) {
|
|
|
|
free_page((unsigned long) qi->desc);
|
|
|
|
kfree(qi);
|
2014-01-06 14:18:26 +08:00
|
|
|
iommu->qi = NULL;
|
2008-07-11 02:16:42 +08:00
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
|
2011-07-19 23:02:07 +08:00
|
|
|
raw_spin_lock_init(&qi->q_lock);
|
2008-07-11 02:16:42 +08:00
|
|
|
|
2009-03-28 05:22:43 +08:00
|
|
|
__dmar_enable_qi(iommu);
|
2008-07-11 02:16:42 +08:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
2009-03-17 08:04:54 +08:00
|
|
|
|
|
|
|
/* iommu interrupt handling. Most stuff are MSI-like. */
|
|
|
|
|
2009-03-17 08:04:55 +08:00
|
|
|
enum faulttype {
|
|
|
|
DMA_REMAP,
|
|
|
|
INTR_REMAP,
|
|
|
|
UNKNOWN,
|
|
|
|
};
|
|
|
|
|
|
|
|
static const char *dma_remap_fault_reasons[] =
|
2009-03-17 08:04:54 +08:00
|
|
|
{
|
|
|
|
"Software",
|
|
|
|
"Present bit in root entry is clear",
|
|
|
|
"Present bit in context entry is clear",
|
|
|
|
"Invalid context entry",
|
|
|
|
"Access beyond MGAW",
|
|
|
|
"PTE Write access is not set",
|
|
|
|
"PTE Read access is not set",
|
|
|
|
"Next page table ptr is invalid",
|
|
|
|
"Root table address invalid",
|
|
|
|
"Context table ptr is invalid",
|
|
|
|
"non-zero reserved fields in RTP",
|
|
|
|
"non-zero reserved fields in CTP",
|
|
|
|
"non-zero reserved fields in PTE",
|
2013-03-06 10:43:17 +08:00
|
|
|
"PCE for translation request specifies blocking",
|
2009-03-17 08:04:54 +08:00
|
|
|
};
|
2009-03-17 08:04:55 +08:00
|
|
|
|
2012-03-31 02:47:07 +08:00
|
|
|
static const char *irq_remap_fault_reasons[] =
|
2009-03-17 08:04:55 +08:00
|
|
|
{
|
|
|
|
"Detected reserved fields in the decoded interrupt-remapped request",
|
|
|
|
"Interrupt index exceeded the interrupt-remapping table size",
|
|
|
|
"Present field in the IRTE entry is clear",
|
|
|
|
"Error accessing interrupt-remapping table pointed by IRTA_REG",
|
|
|
|
"Detected reserved fields in the IRTE entry",
|
|
|
|
"Blocked a compatibility format interrupt request",
|
|
|
|
"Blocked an interrupt request due to source-id verification failure",
|
|
|
|
};
|
|
|
|
|
2013-12-18 14:31:46 +08:00
|
|
|
static const char *dmar_get_fault_reason(u8 fault_reason, int *fault_type)
|
2009-03-17 08:04:54 +08:00
|
|
|
{
|
2012-05-14 01:09:38 +08:00
|
|
|
if (fault_reason >= 0x20 && (fault_reason - 0x20 <
|
|
|
|
ARRAY_SIZE(irq_remap_fault_reasons))) {
|
2009-03-17 08:04:55 +08:00
|
|
|
*fault_type = INTR_REMAP;
|
2012-03-31 02:47:07 +08:00
|
|
|
return irq_remap_fault_reasons[fault_reason - 0x20];
|
2009-03-17 08:04:55 +08:00
|
|
|
} else if (fault_reason < ARRAY_SIZE(dma_remap_fault_reasons)) {
|
|
|
|
*fault_type = DMA_REMAP;
|
|
|
|
return dma_remap_fault_reasons[fault_reason];
|
|
|
|
} else {
|
|
|
|
*fault_type = UNKNOWN;
|
2009-03-17 08:04:54 +08:00
|
|
|
return "Unknown";
|
2009-03-17 08:04:55 +08:00
|
|
|
}
|
2009-03-17 08:04:54 +08:00
|
|
|
}
|
|
|
|
|
2015-10-07 22:37:03 +08:00
|
|
|
|
|
|
|
static inline int dmar_msi_reg(struct intel_iommu *iommu, int irq)
|
|
|
|
{
|
|
|
|
if (iommu->irq == irq)
|
|
|
|
return DMAR_FECTL_REG;
|
|
|
|
else if (iommu->pr_irq == irq)
|
|
|
|
return DMAR_PECTL_REG;
|
|
|
|
else
|
|
|
|
BUG();
|
|
|
|
}
|
|
|
|
|
2010-09-28 23:15:11 +08:00
|
|
|
void dmar_msi_unmask(struct irq_data *data)
|
2009-03-17 08:04:54 +08:00
|
|
|
{
|
2011-03-28 23:49:12 +08:00
|
|
|
struct intel_iommu *iommu = irq_data_get_irq_handler_data(data);
|
2015-10-07 22:37:03 +08:00
|
|
|
int reg = dmar_msi_reg(iommu, data->irq);
|
2009-03-17 08:04:54 +08:00
|
|
|
unsigned long flag;
|
|
|
|
|
|
|
|
/* unmask it */
|
2011-07-19 22:19:51 +08:00
|
|
|
raw_spin_lock_irqsave(&iommu->register_lock, flag);
|
2015-10-07 22:37:03 +08:00
|
|
|
writel(0, iommu->reg + reg);
|
2009-03-17 08:04:54 +08:00
|
|
|
/* Read a reg to force flush the post write */
|
2015-10-07 22:37:03 +08:00
|
|
|
readl(iommu->reg + reg);
|
2011-07-19 22:19:51 +08:00
|
|
|
raw_spin_unlock_irqrestore(&iommu->register_lock, flag);
|
2009-03-17 08:04:54 +08:00
|
|
|
}
|
|
|
|
|
2010-09-28 23:15:11 +08:00
|
|
|
void dmar_msi_mask(struct irq_data *data)
|
2009-03-17 08:04:54 +08:00
|
|
|
{
|
2011-03-28 23:49:12 +08:00
|
|
|
struct intel_iommu *iommu = irq_data_get_irq_handler_data(data);
|
2015-10-07 22:37:03 +08:00
|
|
|
int reg = dmar_msi_reg(iommu, data->irq);
|
|
|
|
unsigned long flag;
|
2009-03-17 08:04:54 +08:00
|
|
|
|
|
|
|
/* mask it */
|
2011-07-19 22:19:51 +08:00
|
|
|
raw_spin_lock_irqsave(&iommu->register_lock, flag);
|
2015-10-07 22:37:03 +08:00
|
|
|
writel(DMA_FECTL_IM, iommu->reg + reg);
|
2009-03-17 08:04:54 +08:00
|
|
|
/* Read a reg to force flush the post write */
|
2015-10-07 22:37:03 +08:00
|
|
|
readl(iommu->reg + reg);
|
2011-07-19 22:19:51 +08:00
|
|
|
raw_spin_unlock_irqrestore(&iommu->register_lock, flag);
|
2009-03-17 08:04:54 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
void dmar_msi_write(int irq, struct msi_msg *msg)
|
|
|
|
{
|
2011-03-28 23:49:12 +08:00
|
|
|
struct intel_iommu *iommu = irq_get_handler_data(irq);
|
2015-10-07 22:37:03 +08:00
|
|
|
int reg = dmar_msi_reg(iommu, irq);
|
2009-03-17 08:04:54 +08:00
|
|
|
unsigned long flag;
|
|
|
|
|
2011-07-19 22:19:51 +08:00
|
|
|
raw_spin_lock_irqsave(&iommu->register_lock, flag);
|
2015-10-07 22:37:03 +08:00
|
|
|
writel(msg->data, iommu->reg + reg + 4);
|
|
|
|
writel(msg->address_lo, iommu->reg + reg + 8);
|
|
|
|
writel(msg->address_hi, iommu->reg + reg + 12);
|
2011-07-19 22:19:51 +08:00
|
|
|
raw_spin_unlock_irqrestore(&iommu->register_lock, flag);
|
2009-03-17 08:04:54 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
void dmar_msi_read(int irq, struct msi_msg *msg)
|
|
|
|
{
|
2011-03-28 23:49:12 +08:00
|
|
|
struct intel_iommu *iommu = irq_get_handler_data(irq);
|
2015-10-07 22:37:03 +08:00
|
|
|
int reg = dmar_msi_reg(iommu, irq);
|
2009-03-17 08:04:54 +08:00
|
|
|
unsigned long flag;
|
|
|
|
|
2011-07-19 22:19:51 +08:00
|
|
|
raw_spin_lock_irqsave(&iommu->register_lock, flag);
|
2015-10-07 22:37:03 +08:00
|
|
|
msg->data = readl(iommu->reg + reg + 4);
|
|
|
|
msg->address_lo = readl(iommu->reg + reg + 8);
|
|
|
|
msg->address_hi = readl(iommu->reg + reg + 12);
|
2011-07-19 22:19:51 +08:00
|
|
|
raw_spin_unlock_irqrestore(&iommu->register_lock, flag);
|
2009-03-17 08:04:54 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static int dmar_fault_do_one(struct intel_iommu *iommu, int type,
|
|
|
|
u8 fault_reason, u16 source_id, unsigned long long addr)
|
|
|
|
{
|
|
|
|
const char *reason;
|
2009-03-17 08:04:55 +08:00
|
|
|
int fault_type;
|
2009-03-17 08:04:54 +08:00
|
|
|
|
2009-03-17 08:04:55 +08:00
|
|
|
reason = dmar_get_fault_reason(fault_reason, &fault_type);
|
2009-03-17 08:04:54 +08:00
|
|
|
|
2009-03-17 08:04:55 +08:00
|
|
|
if (fault_type == INTR_REMAP)
|
2016-03-18 04:12:31 +08:00
|
|
|
pr_err("[INTR-REMAP] Request device [%02x:%02x.%d] fault index %llx [fault reason %02d] %s\n",
|
|
|
|
source_id >> 8, PCI_SLOT(source_id & 0xFF),
|
2009-03-17 08:04:55 +08:00
|
|
|
PCI_FUNC(source_id & 0xFF), addr >> 48,
|
|
|
|
fault_reason, reason);
|
|
|
|
else
|
2016-03-18 04:12:31 +08:00
|
|
|
pr_err("[%s] Request device [%02x:%02x.%d] fault addr %llx [fault reason %02d] %s\n",
|
|
|
|
type ? "DMA Read" : "DMA Write",
|
|
|
|
source_id >> 8, PCI_SLOT(source_id & 0xFF),
|
2009-03-17 08:04:55 +08:00
|
|
|
PCI_FUNC(source_id & 0xFF), addr, fault_reason, reason);
|
2009-03-17 08:04:54 +08:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
#define PRIMARY_FAULT_REG_LEN (16)
|
2009-03-17 08:04:57 +08:00
|
|
|
irqreturn_t dmar_fault(int irq, void *dev_id)
|
2009-03-17 08:04:54 +08:00
|
|
|
{
|
|
|
|
struct intel_iommu *iommu = dev_id;
|
|
|
|
int reg, fault_index;
|
|
|
|
u32 fault_status;
|
|
|
|
unsigned long flag;
|
2016-03-18 04:12:25 +08:00
|
|
|
static DEFINE_RATELIMIT_STATE(rs,
|
|
|
|
DEFAULT_RATELIMIT_INTERVAL,
|
|
|
|
DEFAULT_RATELIMIT_BURST);
|
|
|
|
|
2011-07-19 22:19:51 +08:00
|
|
|
raw_spin_lock_irqsave(&iommu->register_lock, flag);
|
2009-03-17 08:04:54 +08:00
|
|
|
fault_status = readl(iommu->reg + DMAR_FSTS_REG);
|
2018-03-31 08:33:11 +08:00
|
|
|
if (fault_status && __ratelimit(&rs))
|
2012-06-05 05:29:01 +08:00
|
|
|
pr_err("DRHD: handling fault status reg %x\n", fault_status);
|
2009-03-17 08:04:54 +08:00
|
|
|
|
|
|
|
/* TBD: ignore advanced fault log currently */
|
|
|
|
if (!(fault_status & DMA_FSTS_PPF))
|
2013-03-25 16:20:52 +08:00
|
|
|
goto unlock_exit;
|
2009-03-17 08:04:54 +08:00
|
|
|
|
|
|
|
fault_index = dma_fsts_fault_record_index(fault_status);
|
|
|
|
reg = cap_fault_reg_offset(iommu->cap);
|
|
|
|
while (1) {
|
2018-03-31 08:33:11 +08:00
|
|
|
/* Disable printing, simply clear the fault when ratelimited */
|
|
|
|
bool ratelimited = !__ratelimit(&rs);
|
2009-03-17 08:04:54 +08:00
|
|
|
u8 fault_reason;
|
|
|
|
u16 source_id;
|
|
|
|
u64 guest_addr;
|
|
|
|
int type;
|
|
|
|
u32 data;
|
|
|
|
|
|
|
|
/* highest 32 bits */
|
|
|
|
data = readl(iommu->reg + reg +
|
|
|
|
fault_index * PRIMARY_FAULT_REG_LEN + 12);
|
|
|
|
if (!(data & DMA_FRCD_F))
|
|
|
|
break;
|
|
|
|
|
2016-03-18 04:12:25 +08:00
|
|
|
if (!ratelimited) {
|
|
|
|
fault_reason = dma_frcd_fault_reason(data);
|
|
|
|
type = dma_frcd_type(data);
|
2009-03-17 08:04:54 +08:00
|
|
|
|
2016-03-18 04:12:25 +08:00
|
|
|
data = readl(iommu->reg + reg +
|
|
|
|
fault_index * PRIMARY_FAULT_REG_LEN + 8);
|
|
|
|
source_id = dma_frcd_source_id(data);
|
|
|
|
|
|
|
|
guest_addr = dmar_readq(iommu->reg + reg +
|
|
|
|
fault_index * PRIMARY_FAULT_REG_LEN);
|
|
|
|
guest_addr = dma_frcd_page_addr(guest_addr);
|
|
|
|
}
|
2009-03-17 08:04:54 +08:00
|
|
|
|
|
|
|
/* clear the fault */
|
|
|
|
writel(DMA_FRCD_F, iommu->reg + reg +
|
|
|
|
fault_index * PRIMARY_FAULT_REG_LEN + 12);
|
|
|
|
|
2011-07-19 22:19:51 +08:00
|
|
|
raw_spin_unlock_irqrestore(&iommu->register_lock, flag);
|
2009-03-17 08:04:54 +08:00
|
|
|
|
2016-03-18 04:12:25 +08:00
|
|
|
if (!ratelimited)
|
|
|
|
dmar_fault_do_one(iommu, type, fault_reason,
|
|
|
|
source_id, guest_addr);
|
2009-03-17 08:04:54 +08:00
|
|
|
|
|
|
|
fault_index++;
|
2009-08-20 05:26:11 +08:00
|
|
|
if (fault_index >= cap_num_fault_regs(iommu->cap))
|
2009-03-17 08:04:54 +08:00
|
|
|
fault_index = 0;
|
2011-07-19 22:19:51 +08:00
|
|
|
raw_spin_lock_irqsave(&iommu->register_lock, flag);
|
2009-03-17 08:04:54 +08:00
|
|
|
}
|
|
|
|
|
2017-11-04 00:51:33 +08:00
|
|
|
writel(DMA_FSTS_PFO | DMA_FSTS_PPF | DMA_FSTS_PRO,
|
|
|
|
iommu->reg + DMAR_FSTS_REG);
|
2013-03-25 16:20:52 +08:00
|
|
|
|
|
|
|
unlock_exit:
|
2011-07-19 22:19:51 +08:00
|
|
|
raw_spin_unlock_irqrestore(&iommu->register_lock, flag);
|
2009-03-17 08:04:54 +08:00
|
|
|
return IRQ_HANDLED;
|
|
|
|
}
|
|
|
|
|
|
|
|
int dmar_set_interrupt(struct intel_iommu *iommu)
|
|
|
|
{
|
|
|
|
int irq, ret;
|
|
|
|
|
2009-03-17 08:04:55 +08:00
|
|
|
/*
|
|
|
|
* Check if the fault interrupt is already initialized.
|
|
|
|
*/
|
|
|
|
if (iommu->irq)
|
|
|
|
return 0;
|
|
|
|
|
2015-04-13 14:11:41 +08:00
|
|
|
irq = dmar_alloc_hwirq(iommu->seq_id, iommu->node, iommu);
|
|
|
|
if (irq > 0) {
|
|
|
|
iommu->irq = irq;
|
|
|
|
} else {
|
2015-06-12 15:57:06 +08:00
|
|
|
pr_err("No free IRQ vectors\n");
|
2009-03-17 08:04:54 +08:00
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
2011-07-19 22:25:42 +08:00
|
|
|
ret = request_irq(irq, dmar_fault, IRQF_NO_THREAD, iommu->name, iommu);
|
2009-03-17 08:04:54 +08:00
|
|
|
if (ret)
|
2015-06-12 15:57:06 +08:00
|
|
|
pr_err("Can't request irq\n");
|
2009-03-17 08:04:54 +08:00
|
|
|
return ret;
|
|
|
|
}
|
2009-03-17 08:04:55 +08:00
|
|
|
|
|
|
|
int __init enable_drhd_fault_handling(void)
|
|
|
|
{
|
|
|
|
struct dmar_drhd_unit *drhd;
|
2014-01-06 14:18:18 +08:00
|
|
|
struct intel_iommu *iommu;
|
2009-03-17 08:04:55 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Enable fault control interrupt.
|
|
|
|
*/
|
2014-01-06 14:18:18 +08:00
|
|
|
for_each_iommu(iommu, drhd) {
|
2013-03-25 16:20:52 +08:00
|
|
|
u32 fault_status;
|
2014-01-06 14:18:18 +08:00
|
|
|
int ret = dmar_set_interrupt(iommu);
|
2009-03-17 08:04:55 +08:00
|
|
|
|
|
|
|
if (ret) {
|
2012-06-09 05:13:11 +08:00
|
|
|
pr_err("DRHD %Lx: failed to enable fault, interrupt, ret %d\n",
|
2009-03-17 08:04:55 +08:00
|
|
|
(unsigned long long)drhd->reg_base_addr, ret);
|
|
|
|
return -1;
|
|
|
|
}
|
2010-12-01 14:22:29 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Clear any previous faults.
|
|
|
|
*/
|
|
|
|
dmar_fault(iommu->irq, iommu);
|
2013-03-25 16:20:52 +08:00
|
|
|
fault_status = readl(iommu->reg + DMAR_FSTS_REG);
|
|
|
|
writel(fault_status, iommu->reg + DMAR_FSTS_REG);
|
2009-03-17 08:04:55 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
2009-03-28 05:22:43 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Re-enable Queued Invalidation interface.
|
|
|
|
*/
|
|
|
|
int dmar_reenable_qi(struct intel_iommu *iommu)
|
|
|
|
{
|
|
|
|
if (!ecap_qis(iommu->ecap))
|
|
|
|
return -ENOENT;
|
|
|
|
|
|
|
|
if (!iommu->qi)
|
|
|
|
return -ENOENT;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* First disable queued invalidation.
|
|
|
|
*/
|
|
|
|
dmar_disable_qi(iommu);
|
|
|
|
/*
|
|
|
|
* Then enable queued invalidation again. Since there is no pending
|
|
|
|
* invalidation requests now, it's safe to re-enable queued
|
|
|
|
* invalidation.
|
|
|
|
*/
|
|
|
|
__dmar_enable_qi(iommu);
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
2009-09-10 00:05:39 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Check interrupt remapping support in DMAR table description.
|
|
|
|
*/
|
2009-12-17 06:59:29 +08:00
|
|
|
int __init dmar_ir_support(void)
|
2009-09-10 00:05:39 +08:00
|
|
|
{
|
|
|
|
struct acpi_table_dmar *dmar;
|
|
|
|
dmar = (struct acpi_table_dmar *)dmar_tbl;
|
2010-03-26 02:02:58 +08:00
|
|
|
if (!dmar)
|
|
|
|
return 0;
|
2009-09-10 00:05:39 +08:00
|
|
|
return dmar->flags & 0x1;
|
|
|
|
}
|
2014-01-06 14:18:16 +08:00
|
|
|
|
2014-11-09 22:47:58 +08:00
|
|
|
/* Check whether DMAR units are in use */
|
|
|
|
static inline bool dmar_in_use(void)
|
|
|
|
{
|
|
|
|
return irq_remapping_enabled || intel_iommu_enabled;
|
|
|
|
}
|
|
|
|
|
2014-01-06 14:18:20 +08:00
|
|
|
static int __init dmar_free_unused_resources(void)
|
|
|
|
{
|
|
|
|
struct dmar_drhd_unit *dmaru, *dmaru_n;
|
|
|
|
|
2014-11-09 22:47:58 +08:00
|
|
|
if (dmar_in_use())
|
2014-01-06 14:18:20 +08:00
|
|
|
return 0;
|
|
|
|
|
2014-02-19 14:07:36 +08:00
|
|
|
if (dmar_dev_scope_status != 1 && !list_empty(&dmar_drhd_units))
|
|
|
|
bus_unregister_notifier(&pci_bus_type, &dmar_pci_bus_nb);
|
2014-02-19 14:07:35 +08:00
|
|
|
|
iommu/vt-d: Introduce a rwsem to protect global data structures
Introduce a global rwsem dmar_global_lock, which will be used to
protect DMAR related global data structures from DMAR/PCI/memory
device hotplug operations in process context.
DMA and interrupt remapping related data structures are read most,
and only change when memory/PCI/DMAR hotplug event happens.
So a global rwsem solution is adopted for balance between simplicity
and performance.
For interrupt remapping driver, function intel_irq_remapping_supported(),
dmar_table_init(), intel_enable_irq_remapping(), disable_irq_remapping(),
reenable_irq_remapping() and enable_drhd_fault_handling() etc
are called during booting, suspending and resuming with interrupt
disabled, so no need to take the global lock.
For interrupt remapping entry allocation, the locking model is:
down_read(&dmar_global_lock);
/* Find corresponding iommu */
iommu = map_hpet_to_ir(id);
if (iommu)
/*
* Allocate remapping entry and mark entry busy,
* the IOMMU won't be hot-removed until the
* allocated entry has been released.
*/
index = alloc_irte(iommu, irq, 1);
up_read(&dmar_global_lock);
For DMA remmaping driver, we only uses the dmar_global_lock rwsem to
protect functions which are only called in process context. For any
function which may be called in interrupt context, we will use RCU
to protect them in following patches.
Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
Signed-off-by: Joerg Roedel <joro@8bytes.org>
2014-02-19 14:07:33 +08:00
|
|
|
down_write(&dmar_global_lock);
|
2014-01-06 14:18:20 +08:00
|
|
|
list_for_each_entry_safe(dmaru, dmaru_n, &dmar_drhd_units, list) {
|
|
|
|
list_del(&dmaru->list);
|
|
|
|
dmar_free_drhd(dmaru);
|
|
|
|
}
|
iommu/vt-d: Introduce a rwsem to protect global data structures
Introduce a global rwsem dmar_global_lock, which will be used to
protect DMAR related global data structures from DMAR/PCI/memory
device hotplug operations in process context.
DMA and interrupt remapping related data structures are read most,
and only change when memory/PCI/DMAR hotplug event happens.
So a global rwsem solution is adopted for balance between simplicity
and performance.
For interrupt remapping driver, function intel_irq_remapping_supported(),
dmar_table_init(), intel_enable_irq_remapping(), disable_irq_remapping(),
reenable_irq_remapping() and enable_drhd_fault_handling() etc
are called during booting, suspending and resuming with interrupt
disabled, so no need to take the global lock.
For interrupt remapping entry allocation, the locking model is:
down_read(&dmar_global_lock);
/* Find corresponding iommu */
iommu = map_hpet_to_ir(id);
if (iommu)
/*
* Allocate remapping entry and mark entry busy,
* the IOMMU won't be hot-removed until the
* allocated entry has been released.
*/
index = alloc_irte(iommu, irq, 1);
up_read(&dmar_global_lock);
For DMA remmaping driver, we only uses the dmar_global_lock rwsem to
protect functions which are only called in process context. For any
function which may be called in interrupt context, we will use RCU
to protect them in following patches.
Signed-off-by: Jiang Liu <jiang.liu@linux.intel.com>
Signed-off-by: Joerg Roedel <joro@8bytes.org>
2014-02-19 14:07:33 +08:00
|
|
|
up_write(&dmar_global_lock);
|
2014-01-06 14:18:20 +08:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
late_initcall(dmar_free_unused_resources);
|
2010-08-27 01:58:04 +08:00
|
|
|
IOMMU_INIT_POST(detect_intel_iommu);
|
2014-11-09 22:47:58 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* DMAR Hotplug Support
|
|
|
|
* For more details, please refer to Intel(R) Virtualization Technology
|
|
|
|
* for Directed-IO Architecture Specifiction, Rev 2.2, Section 8.8
|
|
|
|
* "Remapping Hardware Unit Hot Plug".
|
|
|
|
*/
|
2017-06-06 00:40:46 +08:00
|
|
|
static guid_t dmar_hp_guid =
|
|
|
|
GUID_INIT(0xD8C1A3A6, 0xBE9B, 0x4C9B,
|
|
|
|
0x91, 0xBF, 0xC3, 0xCB, 0x81, 0xFC, 0x5D, 0xAF);
|
2014-11-09 22:47:58 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Currently there's only one revision and BIOS will not check the revision id,
|
|
|
|
* so use 0 for safety.
|
|
|
|
*/
|
|
|
|
#define DMAR_DSM_REV_ID 0
|
|
|
|
#define DMAR_DSM_FUNC_DRHD 1
|
|
|
|
#define DMAR_DSM_FUNC_ATSR 2
|
|
|
|
#define DMAR_DSM_FUNC_RHSA 3
|
|
|
|
|
|
|
|
static inline bool dmar_detect_dsm(acpi_handle handle, int func)
|
|
|
|
{
|
2017-06-06 00:40:46 +08:00
|
|
|
return acpi_check_dsm(handle, &dmar_hp_guid, DMAR_DSM_REV_ID, 1 << func);
|
2014-11-09 22:47:58 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static int dmar_walk_dsm_resource(acpi_handle handle, int func,
|
|
|
|
dmar_res_handler_t handler, void *arg)
|
|
|
|
{
|
|
|
|
int ret = -ENODEV;
|
|
|
|
union acpi_object *obj;
|
|
|
|
struct acpi_dmar_header *start;
|
|
|
|
struct dmar_res_callback callback;
|
|
|
|
static int res_type[] = {
|
|
|
|
[DMAR_DSM_FUNC_DRHD] = ACPI_DMAR_TYPE_HARDWARE_UNIT,
|
|
|
|
[DMAR_DSM_FUNC_ATSR] = ACPI_DMAR_TYPE_ROOT_ATS,
|
|
|
|
[DMAR_DSM_FUNC_RHSA] = ACPI_DMAR_TYPE_HARDWARE_AFFINITY,
|
|
|
|
};
|
|
|
|
|
|
|
|
if (!dmar_detect_dsm(handle, func))
|
|
|
|
return 0;
|
|
|
|
|
2017-06-06 00:40:46 +08:00
|
|
|
obj = acpi_evaluate_dsm_typed(handle, &dmar_hp_guid, DMAR_DSM_REV_ID,
|
2014-11-09 22:47:58 +08:00
|
|
|
func, NULL, ACPI_TYPE_BUFFER);
|
|
|
|
if (!obj)
|
|
|
|
return -ENODEV;
|
|
|
|
|
|
|
|
memset(&callback, 0, sizeof(callback));
|
|
|
|
callback.cb[res_type[func]] = handler;
|
|
|
|
callback.arg[res_type[func]] = arg;
|
|
|
|
start = (struct acpi_dmar_header *)obj->buffer.pointer;
|
|
|
|
ret = dmar_walk_remapping_entries(start, obj->buffer.length, &callback);
|
|
|
|
|
|
|
|
ACPI_FREE(obj);
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int dmar_hp_add_drhd(struct acpi_dmar_header *header, void *arg)
|
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
struct dmar_drhd_unit *dmaru;
|
|
|
|
|
|
|
|
dmaru = dmar_find_dmaru((struct acpi_dmar_hardware_unit *)header);
|
|
|
|
if (!dmaru)
|
|
|
|
return -ENODEV;
|
|
|
|
|
|
|
|
ret = dmar_ir_hotplug(dmaru, true);
|
|
|
|
if (ret == 0)
|
|
|
|
ret = dmar_iommu_hotplug(dmaru, true);
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int dmar_hp_remove_drhd(struct acpi_dmar_header *header, void *arg)
|
|
|
|
{
|
|
|
|
int i, ret;
|
|
|
|
struct device *dev;
|
|
|
|
struct dmar_drhd_unit *dmaru;
|
|
|
|
|
|
|
|
dmaru = dmar_find_dmaru((struct acpi_dmar_hardware_unit *)header);
|
|
|
|
if (!dmaru)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* All PCI devices managed by this unit should have been destroyed.
|
|
|
|
*/
|
2016-07-28 11:03:31 +08:00
|
|
|
if (!dmaru->include_all && dmaru->devices && dmaru->devices_cnt) {
|
2014-11-09 22:47:58 +08:00
|
|
|
for_each_active_dev_scope(dmaru->devices,
|
|
|
|
dmaru->devices_cnt, i, dev)
|
|
|
|
return -EBUSY;
|
2016-07-28 11:03:31 +08:00
|
|
|
}
|
2014-11-09 22:47:58 +08:00
|
|
|
|
|
|
|
ret = dmar_ir_hotplug(dmaru, false);
|
|
|
|
if (ret == 0)
|
|
|
|
ret = dmar_iommu_hotplug(dmaru, false);
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int dmar_hp_release_drhd(struct acpi_dmar_header *header, void *arg)
|
|
|
|
{
|
|
|
|
struct dmar_drhd_unit *dmaru;
|
|
|
|
|
|
|
|
dmaru = dmar_find_dmaru((struct acpi_dmar_hardware_unit *)header);
|
|
|
|
if (dmaru) {
|
|
|
|
list_del_rcu(&dmaru->list);
|
|
|
|
synchronize_rcu();
|
|
|
|
dmar_free_drhd(dmaru);
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int dmar_hotplug_insert(acpi_handle handle)
|
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
int drhd_count = 0;
|
|
|
|
|
|
|
|
ret = dmar_walk_dsm_resource(handle, DMAR_DSM_FUNC_DRHD,
|
|
|
|
&dmar_validate_one_drhd, (void *)1);
|
|
|
|
if (ret)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
ret = dmar_walk_dsm_resource(handle, DMAR_DSM_FUNC_DRHD,
|
|
|
|
&dmar_parse_one_drhd, (void *)&drhd_count);
|
|
|
|
if (ret == 0 && drhd_count == 0) {
|
|
|
|
pr_warn(FW_BUG "No DRHD structures in buffer returned by _DSM method\n");
|
|
|
|
goto out;
|
|
|
|
} else if (ret) {
|
|
|
|
goto release_drhd;
|
|
|
|
}
|
|
|
|
|
|
|
|
ret = dmar_walk_dsm_resource(handle, DMAR_DSM_FUNC_RHSA,
|
|
|
|
&dmar_parse_one_rhsa, NULL);
|
|
|
|
if (ret)
|
|
|
|
goto release_drhd;
|
|
|
|
|
|
|
|
ret = dmar_walk_dsm_resource(handle, DMAR_DSM_FUNC_ATSR,
|
|
|
|
&dmar_parse_one_atsr, NULL);
|
|
|
|
if (ret)
|
|
|
|
goto release_atsr;
|
|
|
|
|
|
|
|
ret = dmar_walk_dsm_resource(handle, DMAR_DSM_FUNC_DRHD,
|
|
|
|
&dmar_hp_add_drhd, NULL);
|
|
|
|
if (!ret)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
dmar_walk_dsm_resource(handle, DMAR_DSM_FUNC_DRHD,
|
|
|
|
&dmar_hp_remove_drhd, NULL);
|
|
|
|
release_atsr:
|
|
|
|
dmar_walk_dsm_resource(handle, DMAR_DSM_FUNC_ATSR,
|
|
|
|
&dmar_release_one_atsr, NULL);
|
|
|
|
release_drhd:
|
|
|
|
dmar_walk_dsm_resource(handle, DMAR_DSM_FUNC_DRHD,
|
|
|
|
&dmar_hp_release_drhd, NULL);
|
|
|
|
out:
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int dmar_hotplug_remove(acpi_handle handle)
|
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
ret = dmar_walk_dsm_resource(handle, DMAR_DSM_FUNC_ATSR,
|
|
|
|
&dmar_check_one_atsr, NULL);
|
|
|
|
if (ret)
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
ret = dmar_walk_dsm_resource(handle, DMAR_DSM_FUNC_DRHD,
|
|
|
|
&dmar_hp_remove_drhd, NULL);
|
|
|
|
if (ret == 0) {
|
|
|
|
WARN_ON(dmar_walk_dsm_resource(handle, DMAR_DSM_FUNC_ATSR,
|
|
|
|
&dmar_release_one_atsr, NULL));
|
|
|
|
WARN_ON(dmar_walk_dsm_resource(handle, DMAR_DSM_FUNC_DRHD,
|
|
|
|
&dmar_hp_release_drhd, NULL));
|
|
|
|
} else {
|
|
|
|
dmar_walk_dsm_resource(handle, DMAR_DSM_FUNC_DRHD,
|
|
|
|
&dmar_hp_add_drhd, NULL);
|
|
|
|
}
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2014-11-09 22:47:59 +08:00
|
|
|
static acpi_status dmar_get_dsm_handle(acpi_handle handle, u32 lvl,
|
|
|
|
void *context, void **retval)
|
|
|
|
{
|
|
|
|
acpi_handle *phdl = retval;
|
|
|
|
|
|
|
|
if (dmar_detect_dsm(handle, DMAR_DSM_FUNC_DRHD)) {
|
|
|
|
*phdl = handle;
|
|
|
|
return AE_CTRL_TERMINATE;
|
|
|
|
}
|
|
|
|
|
|
|
|
return AE_OK;
|
|
|
|
}
|
|
|
|
|
2014-11-09 22:47:58 +08:00
|
|
|
static int dmar_device_hotplug(acpi_handle handle, bool insert)
|
|
|
|
{
|
|
|
|
int ret;
|
2014-11-09 22:47:59 +08:00
|
|
|
acpi_handle tmp = NULL;
|
|
|
|
acpi_status status;
|
2014-11-09 22:47:58 +08:00
|
|
|
|
|
|
|
if (!dmar_in_use())
|
|
|
|
return 0;
|
|
|
|
|
2014-11-09 22:47:59 +08:00
|
|
|
if (dmar_detect_dsm(handle, DMAR_DSM_FUNC_DRHD)) {
|
|
|
|
tmp = handle;
|
|
|
|
} else {
|
|
|
|
status = acpi_walk_namespace(ACPI_TYPE_DEVICE, handle,
|
|
|
|
ACPI_UINT32_MAX,
|
|
|
|
dmar_get_dsm_handle,
|
|
|
|
NULL, NULL, &tmp);
|
|
|
|
if (ACPI_FAILURE(status)) {
|
|
|
|
pr_warn("Failed to locate _DSM method.\n");
|
|
|
|
return -ENXIO;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
if (tmp == NULL)
|
2014-11-09 22:47:58 +08:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
down_write(&dmar_global_lock);
|
|
|
|
if (insert)
|
2014-11-09 22:47:59 +08:00
|
|
|
ret = dmar_hotplug_insert(tmp);
|
2014-11-09 22:47:58 +08:00
|
|
|
else
|
2014-11-09 22:47:59 +08:00
|
|
|
ret = dmar_hotplug_remove(tmp);
|
2014-11-09 22:47:58 +08:00
|
|
|
up_write(&dmar_global_lock);
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
int dmar_device_add(acpi_handle handle)
|
|
|
|
{
|
|
|
|
return dmar_device_hotplug(handle, true);
|
|
|
|
}
|
|
|
|
|
|
|
|
int dmar_device_remove(acpi_handle handle)
|
|
|
|
{
|
|
|
|
return dmar_device_hotplug(handle, false);
|
|
|
|
}
|
2018-10-23 15:45:01 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* dmar_platform_optin - Is %DMA_CTRL_PLATFORM_OPT_IN_FLAG set in DMAR table
|
|
|
|
*
|
|
|
|
* Returns true if the platform has %DMA_CTRL_PLATFORM_OPT_IN_FLAG set in
|
|
|
|
* the ACPI DMAR table. This means that the platform boot firmware has made
|
|
|
|
* sure no device can issue DMA outside of RMRR regions.
|
|
|
|
*/
|
|
|
|
bool dmar_platform_optin(void)
|
|
|
|
{
|
|
|
|
struct acpi_table_dmar *dmar;
|
|
|
|
acpi_status status;
|
|
|
|
bool ret;
|
|
|
|
|
|
|
|
status = acpi_get_table(ACPI_SIG_DMAR, 0,
|
|
|
|
(struct acpi_table_header **)&dmar);
|
|
|
|
if (ACPI_FAILURE(status))
|
|
|
|
return false;
|
|
|
|
|
|
|
|
ret = !!(dmar->flags & DMAR_PLATFORM_OPT_IN);
|
|
|
|
acpi_put_table((struct acpi_table_header *)dmar);
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(dmar_platform_optin);
|