PCI/P2PDMA: Support peer-to-peer memory
Some PCI devices may have memory mapped in a BAR space that's intended for
use in peer-to-peer transactions. To enable such transactions the memory
must be registered with ZONE_DEVICE pages so it can be used by DMA
interfaces in existing drivers.
Add an interface for other subsystems to find and allocate chunks of P2P
memory as necessary to facilitate transfers between two PCI peers:
struct pci_dev *pci_p2pmem_find[_many]();
int pci_p2pdma_distance[_many]();
void *pci_alloc_p2pmem();
The new interface requires a driver to collect a list of client devices
involved in the transaction then call pci_p2pmem_find() to obtain any
suitable P2P memory. Alternatively, if the caller knows a device which
provides P2P memory, they can use pci_p2pdma_distance() to determine if it
is usable. With a suitable p2pmem device, memory can then be allocated
with pci_alloc_p2pmem() for use in DMA transactions.
Depending on hardware, using peer-to-peer memory may reduce the bandwidth
of the transfer but can significantly reduce pressure on system memory.
This may be desirable in many cases: for example a system could be designed
with a small CPU connected to a PCIe switch by a small number of lanes
which would maximize the number of lanes available to connect to NVMe
devices.
The code is designed to only utilize the p2pmem device if all the devices
involved in a transfer are behind the same PCI bridge. This is because we
have no way of knowing whether peer-to-peer routing between PCIe Root Ports
is supported (PCIe r4.0, sec 1.3.1). Additionally, the benefits of P2P
transfers that go through the RC is limited to only reducing DRAM usage
and, in some cases, coding convenience. The PCI-SIG may be exploring
adding a new capability bit to advertise whether this is possible for
future hardware.
This commit includes significant rework and feedback from Christoph
Hellwig.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
[bhelgaas: fold in fix from Keith Busch <keith.busch@intel.com>:
https://lore.kernel.org/linux-pci/20181012155920.15418-1-keith.busch@intel.com,
to address comment from Dan Carpenter <dan.carpenter@oracle.com>, fold in
https://lore.kernel.org/linux-pci/20181017160510.17926-1-logang@deltatee.com]
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
2018-10-05 05:27:35 +08:00
|
|
|
/* SPDX-License-Identifier: GPL-2.0 */
|
|
|
|
|
/*
|
|
|
|
|
* PCI Peer 2 Peer DMA support.
|
|
|
|
|
*
|
|
|
|
|
* Copyright (c) 2016-2018, Logan Gunthorpe
|
|
|
|
|
* Copyright (c) 2016-2017, Microsemi Corporation
|
|
|
|
|
* Copyright (c) 2017, Christoph Hellwig
|
|
|
|
|
* Copyright (c) 2018, Eideticom Inc.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#ifndef _LINUX_PCI_P2PDMA_H
|
|
|
|
|
#define _LINUX_PCI_P2PDMA_H
|
|
|
|
|
|
|
|
|
|
#include <linux/pci.h>
|
|
|
|
|
|
|
|
|
|
struct block_device;
|
|
|
|
|
struct scatterlist;
|
|
|
|
|
|
|
|
|
|
#ifdef CONFIG_PCI_P2PDMA
|
|
|
|
|
int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size,
|
|
|
|
|
u64 offset);
|
|
|
|
|
int pci_p2pdma_distance_many(struct pci_dev *provider, struct device **clients,
|
|
|
|
|
int num_clients, bool verbose);
|
|
|
|
|
bool pci_has_p2pmem(struct pci_dev *pdev);
|
|
|
|
|
struct pci_dev *pci_p2pmem_find_many(struct device **clients, int num_clients);
|
|
|
|
|
void *pci_alloc_p2pmem(struct pci_dev *pdev, size_t size);
|
|
|
|
|
void pci_free_p2pmem(struct pci_dev *pdev, void *addr, size_t size);
|
|
|
|
|
pci_bus_addr_t pci_p2pmem_virt_to_bus(struct pci_dev *pdev, void *addr);
|
|
|
|
|
struct scatterlist *pci_p2pmem_alloc_sgl(struct pci_dev *pdev,
|
|
|
|
|
unsigned int *nents, u32 length);
|
|
|
|
|
void pci_p2pmem_free_sgl(struct pci_dev *pdev, struct scatterlist *sgl);
|
|
|
|
|
void pci_p2pmem_publish(struct pci_dev *pdev, bool publish);
|
2019-08-13 01:30:42 +08:00
|
|
|
int pci_p2pdma_map_sg_attrs(struct device *dev, struct scatterlist *sg,
|
|
|
|
|
int nents, enum dma_data_direction dir, unsigned long attrs);
|
2019-08-13 01:30:43 +08:00
|
|
|
void pci_p2pdma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg,
|
|
|
|
|
int nents, enum dma_data_direction dir, unsigned long attrs);
|
2018-10-05 05:27:38 +08:00
|
|
|
int pci_p2pdma_enable_store(const char *page, struct pci_dev **p2p_dev,
|
|
|
|
|
bool *use_p2pdma);
|
|
|
|
|
ssize_t pci_p2pdma_enable_show(char *page, struct pci_dev *p2p_dev,
|
|
|
|
|
bool use_p2pdma);
|
PCI/P2PDMA: Support peer-to-peer memory
Some PCI devices may have memory mapped in a BAR space that's intended for
use in peer-to-peer transactions. To enable such transactions the memory
must be registered with ZONE_DEVICE pages so it can be used by DMA
interfaces in existing drivers.
Add an interface for other subsystems to find and allocate chunks of P2P
memory as necessary to facilitate transfers between two PCI peers:
struct pci_dev *pci_p2pmem_find[_many]();
int pci_p2pdma_distance[_many]();
void *pci_alloc_p2pmem();
The new interface requires a driver to collect a list of client devices
involved in the transaction then call pci_p2pmem_find() to obtain any
suitable P2P memory. Alternatively, if the caller knows a device which
provides P2P memory, they can use pci_p2pdma_distance() to determine if it
is usable. With a suitable p2pmem device, memory can then be allocated
with pci_alloc_p2pmem() for use in DMA transactions.
Depending on hardware, using peer-to-peer memory may reduce the bandwidth
of the transfer but can significantly reduce pressure on system memory.
This may be desirable in many cases: for example a system could be designed
with a small CPU connected to a PCIe switch by a small number of lanes
which would maximize the number of lanes available to connect to NVMe
devices.
The code is designed to only utilize the p2pmem device if all the devices
involved in a transfer are behind the same PCI bridge. This is because we
have no way of knowing whether peer-to-peer routing between PCIe Root Ports
is supported (PCIe r4.0, sec 1.3.1). Additionally, the benefits of P2P
transfers that go through the RC is limited to only reducing DRAM usage
and, in some cases, coding convenience. The PCI-SIG may be exploring
adding a new capability bit to advertise whether this is possible for
future hardware.
This commit includes significant rework and feedback from Christoph
Hellwig.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
[bhelgaas: fold in fix from Keith Busch <keith.busch@intel.com>:
https://lore.kernel.org/linux-pci/20181012155920.15418-1-keith.busch@intel.com,
to address comment from Dan Carpenter <dan.carpenter@oracle.com>, fold in
https://lore.kernel.org/linux-pci/20181017160510.17926-1-logang@deltatee.com]
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
2018-10-05 05:27:35 +08:00
|
|
|
#else /* CONFIG_PCI_P2PDMA */
|
|
|
|
|
static inline int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar,
|
|
|
|
|
size_t size, u64 offset)
|
|
|
|
|
{
|
|
|
|
|
return -EOPNOTSUPP;
|
|
|
|
|
}
|
|
|
|
|
static inline int pci_p2pdma_distance_many(struct pci_dev *provider,
|
|
|
|
|
struct device **clients, int num_clients, bool verbose)
|
|
|
|
|
{
|
|
|
|
|
return -1;
|
|
|
|
|
}
|
|
|
|
|
static inline bool pci_has_p2pmem(struct pci_dev *pdev)
|
|
|
|
|
{
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
static inline struct pci_dev *pci_p2pmem_find_many(struct device **clients,
|
|
|
|
|
int num_clients)
|
|
|
|
|
{
|
|
|
|
|
return NULL;
|
|
|
|
|
}
|
|
|
|
|
static inline void *pci_alloc_p2pmem(struct pci_dev *pdev, size_t size)
|
|
|
|
|
{
|
|
|
|
|
return NULL;
|
|
|
|
|
}
|
|
|
|
|
static inline void pci_free_p2pmem(struct pci_dev *pdev, void *addr,
|
|
|
|
|
size_t size)
|
|
|
|
|
{
|
|
|
|
|
}
|
|
|
|
|
static inline pci_bus_addr_t pci_p2pmem_virt_to_bus(struct pci_dev *pdev,
|
|
|
|
|
void *addr)
|
|
|
|
|
{
|
|
|
|
|
return 0;
|
|
|
|
|
}
|
|
|
|
|
static inline struct scatterlist *pci_p2pmem_alloc_sgl(struct pci_dev *pdev,
|
|
|
|
|
unsigned int *nents, u32 length)
|
|
|
|
|
{
|
|
|
|
|
return NULL;
|
|
|
|
|
}
|
|
|
|
|
static inline void pci_p2pmem_free_sgl(struct pci_dev *pdev,
|
|
|
|
|
struct scatterlist *sgl)
|
|
|
|
|
{
|
|
|
|
|
}
|
|
|
|
|
static inline void pci_p2pmem_publish(struct pci_dev *pdev, bool publish)
|
|
|
|
|
{
|
|
|
|
|
}
|
2019-08-13 01:30:42 +08:00
|
|
|
static inline int pci_p2pdma_map_sg_attrs(struct device *dev,
|
|
|
|
|
struct scatterlist *sg, int nents, enum dma_data_direction dir,
|
|
|
|
|
unsigned long attrs)
|
2018-10-05 05:27:37 +08:00
|
|
|
{
|
|
|
|
|
return 0;
|
|
|
|
|
}
|
2019-08-13 01:30:43 +08:00
|
|
|
static inline void pci_p2pdma_unmap_sg_attrs(struct device *dev,
|
|
|
|
|
struct scatterlist *sg, int nents, enum dma_data_direction dir,
|
|
|
|
|
unsigned long attrs)
|
|
|
|
|
{
|
|
|
|
|
}
|
2018-10-05 05:27:38 +08:00
|
|
|
static inline int pci_p2pdma_enable_store(const char *page,
|
|
|
|
|
struct pci_dev **p2p_dev, bool *use_p2pdma)
|
|
|
|
|
{
|
|
|
|
|
*use_p2pdma = false;
|
|
|
|
|
return 0;
|
|
|
|
|
}
|
|
|
|
|
static inline ssize_t pci_p2pdma_enable_show(char *page,
|
|
|
|
|
struct pci_dev *p2p_dev, bool use_p2pdma)
|
|
|
|
|
{
|
|
|
|
|
return sprintf(page, "none\n");
|
|
|
|
|
}
|
PCI/P2PDMA: Support peer-to-peer memory
Some PCI devices may have memory mapped in a BAR space that's intended for
use in peer-to-peer transactions. To enable such transactions the memory
must be registered with ZONE_DEVICE pages so it can be used by DMA
interfaces in existing drivers.
Add an interface for other subsystems to find and allocate chunks of P2P
memory as necessary to facilitate transfers between two PCI peers:
struct pci_dev *pci_p2pmem_find[_many]();
int pci_p2pdma_distance[_many]();
void *pci_alloc_p2pmem();
The new interface requires a driver to collect a list of client devices
involved in the transaction then call pci_p2pmem_find() to obtain any
suitable P2P memory. Alternatively, if the caller knows a device which
provides P2P memory, they can use pci_p2pdma_distance() to determine if it
is usable. With a suitable p2pmem device, memory can then be allocated
with pci_alloc_p2pmem() for use in DMA transactions.
Depending on hardware, using peer-to-peer memory may reduce the bandwidth
of the transfer but can significantly reduce pressure on system memory.
This may be desirable in many cases: for example a system could be designed
with a small CPU connected to a PCIe switch by a small number of lanes
which would maximize the number of lanes available to connect to NVMe
devices.
The code is designed to only utilize the p2pmem device if all the devices
involved in a transfer are behind the same PCI bridge. This is because we
have no way of knowing whether peer-to-peer routing between PCIe Root Ports
is supported (PCIe r4.0, sec 1.3.1). Additionally, the benefits of P2P
transfers that go through the RC is limited to only reducing DRAM usage
and, in some cases, coding convenience. The PCI-SIG may be exploring
adding a new capability bit to advertise whether this is possible for
future hardware.
This commit includes significant rework and feedback from Christoph
Hellwig.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
[bhelgaas: fold in fix from Keith Busch <keith.busch@intel.com>:
https://lore.kernel.org/linux-pci/20181012155920.15418-1-keith.busch@intel.com,
to address comment from Dan Carpenter <dan.carpenter@oracle.com>, fold in
https://lore.kernel.org/linux-pci/20181017160510.17926-1-logang@deltatee.com]
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
2018-10-05 05:27:35 +08:00
|
|
|
#endif /* CONFIG_PCI_P2PDMA */
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
static inline int pci_p2pdma_distance(struct pci_dev *provider,
|
|
|
|
|
struct device *client, bool verbose)
|
|
|
|
|
{
|
|
|
|
|
return pci_p2pdma_distance_many(provider, &client, 1, verbose);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static inline struct pci_dev *pci_p2pmem_find(struct device *client)
|
|
|
|
|
{
|
|
|
|
|
return pci_p2pmem_find_many(&client, 1);
|
|
|
|
|
}
|
|
|
|
|
|
2019-08-13 01:30:42 +08:00
|
|
|
static inline int pci_p2pdma_map_sg(struct device *dev, struct scatterlist *sg,
|
|
|
|
|
int nents, enum dma_data_direction dir)
|
|
|
|
|
{
|
|
|
|
|
return pci_p2pdma_map_sg_attrs(dev, sg, nents, dir, 0);
|
|
|
|
|
}
|
|
|
|
|
|
2019-08-13 01:30:43 +08:00
|
|
|
static inline void pci_p2pdma_unmap_sg(struct device *dev,
|
|
|
|
|
struct scatterlist *sg, int nents, enum dma_data_direction dir)
|
|
|
|
|
{
|
|
|
|
|
pci_p2pdma_unmap_sg_attrs(dev, sg, nents, dir, 0);
|
|
|
|
|
}
|
|
|
|
|
|
PCI/P2PDMA: Support peer-to-peer memory
Some PCI devices may have memory mapped in a BAR space that's intended for
use in peer-to-peer transactions. To enable such transactions the memory
must be registered with ZONE_DEVICE pages so it can be used by DMA
interfaces in existing drivers.
Add an interface for other subsystems to find and allocate chunks of P2P
memory as necessary to facilitate transfers between two PCI peers:
struct pci_dev *pci_p2pmem_find[_many]();
int pci_p2pdma_distance[_many]();
void *pci_alloc_p2pmem();
The new interface requires a driver to collect a list of client devices
involved in the transaction then call pci_p2pmem_find() to obtain any
suitable P2P memory. Alternatively, if the caller knows a device which
provides P2P memory, they can use pci_p2pdma_distance() to determine if it
is usable. With a suitable p2pmem device, memory can then be allocated
with pci_alloc_p2pmem() for use in DMA transactions.
Depending on hardware, using peer-to-peer memory may reduce the bandwidth
of the transfer but can significantly reduce pressure on system memory.
This may be desirable in many cases: for example a system could be designed
with a small CPU connected to a PCIe switch by a small number of lanes
which would maximize the number of lanes available to connect to NVMe
devices.
The code is designed to only utilize the p2pmem device if all the devices
involved in a transfer are behind the same PCI bridge. This is because we
have no way of knowing whether peer-to-peer routing between PCIe Root Ports
is supported (PCIe r4.0, sec 1.3.1). Additionally, the benefits of P2P
transfers that go through the RC is limited to only reducing DRAM usage
and, in some cases, coding convenience. The PCI-SIG may be exploring
adding a new capability bit to advertise whether this is possible for
future hardware.
This commit includes significant rework and feedback from Christoph
Hellwig.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
[bhelgaas: fold in fix from Keith Busch <keith.busch@intel.com>:
https://lore.kernel.org/linux-pci/20181012155920.15418-1-keith.busch@intel.com,
to address comment from Dan Carpenter <dan.carpenter@oracle.com>, fold in
https://lore.kernel.org/linux-pci/20181017160510.17926-1-logang@deltatee.com]
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
2018-10-05 05:27:35 +08:00
|
|
|
#endif /* _LINUX_PCI_P2P_H */
|