hollalinux
/
linux-sg2042
mirror of https://github.com/sophgo/linux-riscv.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'octeontx2-pf-Add-network-driver-for-physical-function' 

Sunil Goutham says:

====================
octeontx2-pf: Add network driver for physical function

OcteonTX2 SOC's resource virtualization unit (RVU) supports
multiple physical and virtual functions. Each of the PF's
functionality is determined by what kind of resources are attached
to it. If NPA and NIX blocks are attached to a PF it can function
as a highly capable network device.

This patch series add a network driver for the PF. Initial set of
patches adds mailbox communication with admin function (RVU AF)
and configuration of queues. Followed by Rx and tx pkts NAPI
handler and then support for HW offloads like RSS, TSO, Rxhash etc.
Ethtool support to extract stats, config RSS, queue sizes, queue
count is also added.

Added documentation to give a high level overview of HW and
different drivers which will be upstreamed and how they interact.

Changes from v5:
   * Fixed otx2_atomic64_add() non ARM64 fallback definition.
     - Suggested by David Miller

Changes from v4:
   * Replaced pci_set_dma_mask and pci_set_consistent_dma_mask
     fn()s with dma_set_mask_and_coherent().
   * Some additonal code cleanup.
   * Fixed receive buffer segmnetation logic in otx2_alloc_rbuf()
   * Removed all unused BIG_ENDIAN structure definitions.
   * Removed unnecessary memory barriers
     - Sugested by Jakub Kicinski
   * Fixed mailbox initalization failure handling
   * Removed unused function parameter in otx2_skb_add_frag()
     - Suggested by Maciej Fijalkowski

Changes from v3:
   * Fixed receive side scaling reinitialization during interface
     DOWN and UP to retain user configured settings, if any.
   * Removed driver version from ethtool.
   * Fixed otx2_set_rss_hash_opts() to return error incase RSS is
     not enabled.
     - Sugested by Jakub Kicinski

Changes from v2:
   * Removed frames, bytes, dropped packet stats from ethtool to avoid
     duplication of same stats in netlink and ethtool.
     - Sugested by Jakub Kicinski
   * Removed number of channels and ringparam upper bound checking
     in ethtool support.
   * Fixed RSS hash option setting to reject unsupported config.
     - Suggested by Michal Kubecek

Changes from v1:
   * Made driver dependent on 64bit, to fix build errors related to
     non availability of writeq/readq APIs for 32bit platforms.
     - Reported by kbuild test robot
====================

Signed-off-by: David S. Miller <davem@davemloft.net>
This commit is contained in:
David S. Miller 2020-01-27 14:33:40 +01:00
parent 9e0703a265 688b3e829d
commit 3d4864b30b
17 changed files with 5690 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
Documentation/networking/device_drivers/index.rst
View File

@ -22,6 +22,7 @@ Contents:
intel/iavf
intel/ice
google/gve
marvell/octeontx2
mellanox/mlx5
netronome/nfp
pensando/ionic

159
Documentation/networking/device_drivers/marvell/octeontx2.rst Normal file
View File

@ -0,0 +1,159 @@
.. SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
====================================
Marvell OcteonTx2 RVU Kernel Drivers
====================================
Copyright (c) 2020 Marvell International Ltd.
Contents
========
- `Overview`_
- `Drivers`_
- `Basic packet flow`_
Overview
========
Resource virtualization unit (RVU) on Marvell's OcteonTX2 SOC maps HW
resources from the network, crypto and other functional blocks into
PCI-compatible physical and virtual functions. Each functional block
again has multiple local functions (LFs) for provisioning to PCI devices.
RVU supports multiple PCIe SRIOV physical functions (PFs) and virtual
functions (VFs). PF0 is called the administrative / admin function (AF)
and has privileges to provision RVU functional block's LFs to each of the
PF/VF.
RVU managed networking functional blocks
- Network pool or buffer allocator (NPA)
- Network interface controller (NIX)
- Network parser CAM (NPC)
- Schedule/Synchronize/Order unit (SSO)
- Loopback interface (LBK)
RVU managed non-networking functional blocks
- Crypto accelerator (CPT)
- Scheduled timers unit (TIM)
- Schedule/Synchronize/Order unit (SSO)
Used for both networking and non networking usecases
Resource provisioning examples
- A PF/VF with NIX-LF & NPA-LF resources works as a pure network device
- A PF/VF with CPT-LF resource works as a pure crypto offload device.
RVU functional blocks are highly configurable as per software requirements.
Firmware setups following stuff before kernel boots
- Enables required number of RVU PFs based on number of physical links.
- Number of VFs per PF are either static or configurable at compile time.
Based on config, firmware assigns VFs to each of the PFs.
- Also assigns MSIX vectors to each of PF and VFs.
- These are not changed after kernel boot.
Drivers
=======
Linux kernel will have multiple drivers registering to different PF and VFs
of RVU. Wrt networking there will be 3 flavours of drivers.
Admin Function driver
---------------------
As mentioned above RVU PF0 is called the admin function (AF), this driver
supports resource provisioning and configuration of functional blocks.
Doesn't handle any I/O. It sets up few basic stuff but most of the
funcionality is achieved via configuration requests from PFs and VFs.
PF/VFs communicates with AF via a shared memory region (mailbox). Upon
receiving requests AF does resource provisioning and other HW configuration.
AF is always attached to host kernel, but PFs and their VFs may be used by host
kernel itself, or attached to VMs or to userspace applications like
DPDK etc. So AF has to handle provisioning/configuration requests sent
by any device from any domain.
AF driver also interacts with underlying firmware to
- Manage physical ethernet links ie CGX LMACs.
- Retrieve information like speed, duplex, autoneg etc
- Retrieve PHY EEPROM and stats.
- Configure FEC, PAM modes
- etc
From pure networking side AF driver supports following functionality.
- Map a physical link to a RVU PF to which a netdev is registered.
- Attach NIX and NPA block LFs to RVU PF/VF which provide buffer pools, RQs, SQs
for regular networking functionality.
- Flow control (pause frames) enable/disable/config.
- HW PTP timestamping related config.
- NPC parser profile config, basically how to parse pkt and what info to extract.
- NPC extract profile config, what to extract from the pkt to match data in MCAM entries.
- Manage NPC MCAM entries, upon request can frame and install requested packet forwarding rules.
- Defines receive side scaling (RSS) algorithms.
- Defines segmentation offload algorithms (eg TSO)
- VLAN stripping, capture and insertion config.
- SSO and TIM blocks config which provide packet scheduling support.
- Debugfs support, to check current resource provising, current status of
NPA pools, NIX RQ, SQ and CQs, various stats etc which helps in debugging issues.
- And many more.
Physical Function driver
------------------------
This RVU PF handles IO, is mapped to a physical ethernet link and this
driver registers a netdev. This supports SR-IOV. As said above this driver
communicates with AF with a mailbox. To retrieve information from physical
links this driver talks to AF and AF gets that info from firmware and responds
back ie cannot talk to firmware directly.
Supports ethtool for configuring links, RSS, queue count, queue size,
flow control, ntuple filters, dump PHY EEPROM, config FEC etc.
Virtual Function driver
-----------------------
There are two types VFs, VFs that share the physical link with their parent
SR-IOV PF and the VFs which work in pairs using internal HW loopback channels (LBK).
Type1:
- These VFs and their parent PF share a physical link and used for outside communication.
- VFs cannot communicate with AF directly, they send mbox message to PF and PF
forwards that to AF. AF after processing, responds back to PF and PF forwards
the reply to VF.
- From functionality point of view there is no difference between PF and VF as same type
HW resources are attached to both. But user would be able to configure few stuff only
from PF as PF is treated as owner/admin of the link.
Type2:
- RVU PF0 ie admin function creates these VFs and maps them to loopback block's channels.
- A set of two VFs (VF0 & VF1, VF2 & VF3 .. so on) works as a pair ie pkts sent out of
VF0 will be received by VF1 and viceversa.
- These VFs can be used by applications or virtual machines to communicate between them
without sending traffic outside. There is no switch present in HW, hence the support
for loopback VFs.
- These communicate directly with AF (PF0) via mbox.
Except for the IO channels or links used for packet reception and transmission there is
no other difference between these VF types. AF driver takes care of IO channel mapping,
hence same VF driver works for both types of devices.
Basic packet flow
=================
Ingress
-------
1. CGX LMAC receives packet.
2. Forwards the packet to the NIX block.
3. Then submitted to NPC block for parsing and then MCAM lookup to get the destination RVU device.
4. NIX LF attached to the destination RVU device allocates a buffer from RQ mapped buffer pool of NPA block LF.
5. RQ may be selected by RSS or by configuring MCAM rule with a RQ number.
6. Packet is DMA'ed and driver is notified.
Egress
------
1. Driver prepares a send descriptor and submits to SQ for transmission.
2. The SQ is already configured (by AF) to transmit on a specific link/channel.
3. The SQ descriptor ring is maintained in buffers allocated from SQ mapped pool of NPA block LF.
4. NIX block transmits the pkt on the designated channel.
5. NPC MCAM entries can be installed to divert pkt onto a different channel.

10
MAINTAINERS
View File

@ -10000,6 +10000,16 @@ M: Jerin Jacob <jerinj@marvell.com>
L: netdev@vger.kernel.org
S: Supported
F: drivers/net/ethernet/marvell/octeontx2/af/
F: Documentation/networking/device_drivers/marvell/octeontx2.rst
MARVELL OCTEONTX2 PHYSICAL FUNCTION DRIVER
M: Sunil Goutham <sgoutham@marvell.com>
M: Geetha sowjanya <gakula@marvell.com>
M: Subbaraya Sundeep <sbhatta@marvell.com>
M: hariprasad <hkelam@marvell.com>
L: netdev@vger.kernel.org
S: Supported
F: drivers/net/ethernet/marvell/octeontx2/nic/
MATROX FRAMEBUFFER DRIVER
L: linux-fbdev@vger.kernel.org

8
drivers/net/ethernet/marvell/octeontx2/Kconfig
View File

@ -25,3 +25,11 @@ config NDC_DIS_DYNAMIC_CACHING
This config option disables caching of dynamic entries such as NIX SQEs
, NPA stack pages etc in NDC. Also locks down NIX SQ/CQ/RQ/RSS and
NPA Aura/Pool contexts.
config OCTEONTX2_PF
tristate "Marvell OcteonTX2 NIC Physical Function driver"
select OCTEONTX2_MBOX
depends on (64BIT && COMPILE_TEST) || ARM64
depends on PCI
help
This driver supports Marvell's OcteonTX2 NIC physical function.

2
drivers/net/ethernet/marvell/octeontx2/Makefile
View File

@ -3,4 +3,6 @@
# Makefile for Marvell OcteonTX2 device drivers.
#
obj-$(CONFIG_OCTEONTX2_MBOX) += af/
obj-$(CONFIG_OCTEONTX2_AF) += af/
obj-$(CONFIG_OCTEONTX2_PF) += nic/

9
drivers/net/ethernet/marvell/octeontx2/af/common.h
View File

@ -143,8 +143,13 @@ enum nix_scheduler {
NIX_TXSCH_LVL_CNT = 0x5,
};
#define TXSCH_TL1_DFLT_RR_QTM ((1 << 24) - 1)
#define TXSCH_TL1_DFLT_RR_PRIO (0x1ull)
#define TXSCH_RR_QTM_MAX ((1 << 24) - 1)
#define TXSCH_TL1_DFLT_RR_QTM TXSCH_RR_QTM_MAX
#define TXSCH_TL1_DFLT_RR_PRIO (0x1ull)
#define MAX_SCHED_WEIGHT 0xFF
#define DFLT_RR_WEIGHT 71
#define DFLT_RR_QTM ((DFLT_RR_WEIGHT * TXSCH_RR_QTM_MAX) \
/ MAX_SCHED_WEIGHT)
/* Min/Max packet sizes, excluding FCS */
#define NIC_HW_MIN_FRS 40

8
drivers/net/ethernet/marvell/octeontx2/af/mbox.h
View File

@ -210,7 +210,8 @@ M(NIX_SET_RX_CFG, 0x8010, nix_set_rx_cfg, nix_rx_cfg, msg_rsp) \
M(NIX_LSO_FORMAT_CFG, 0x8011, nix_lso_format_cfg, \
nix_lso_format_cfg, \
nix_lso_format_cfg_rsp) \
M(NIX_RXVLAN_ALLOC, 0x8012, nix_rxvlan_alloc, msg_req, msg_rsp)
M(NIX_RXVLAN_ALLOC, 0x8012, nix_rxvlan_alloc, msg_req, msg_rsp) \
M(NIX_GET_MAC_ADDR, 0x8018, nix_get_mac_addr, msg_req, nix_get_mac_addr_rsp) \
/* Messages initiated by AF (range 0xC00 - 0xDFF) */
#define MBOX_UP_CGX_MESSAGES \
@ -618,6 +619,11 @@ struct nix_set_mac_addr {
u8 mac_addr[ETH_ALEN]; /* MAC address to be set for this pcifunc */
};
struct nix_get_mac_addr_rsp {
struct mbox_msghdr hdr;
u8 mac_addr[ETH_ALEN];
};
struct nix_mark_format_cfg {
struct mbox_msghdr hdr;
u8 offset;

17
drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c
View File

@ -2546,6 +2546,23 @@ int rvu_mbox_handler_nix_set_mac_addr(struct rvu *rvu,
return 0;
}
int rvu_mbox_handler_nix_get_mac_addr(struct rvu *rvu,
struct msg_req *req,
struct nix_get_mac_addr_rsp *rsp)
{
u16 pcifunc = req->hdr.pcifunc;
struct rvu_pfvf *pfvf;
if (!is_nixlf_attached(rvu, pcifunc))
return NIX_AF_ERR_AF_LF_INVALID;
pfvf = rvu_get_pfvf(rvu, pcifunc);
ether_addr_copy(rsp->mac_addr, pfvf->mac_addr);
return 0;
}
int rvu_mbox_handler_nix_set_rx_mode(struct rvu *rvu, struct nix_rx_mode *req,
struct msg_rsp *rsp)
{

10
drivers/net/ethernet/marvell/octeontx2/nic/Makefile Normal file
View File

@ -0,0 +1,10 @@
# SPDX-License-Identifier: GPL-2.0
#
# Makefile for Marvell's OcteonTX2 ethernet device drivers
#
obj-$(CONFIG_OCTEONTX2_PF) += octeontx2_nicpf.o
octeontx2_nicpf-y := otx2_pf.o otx2_common.o otx2_txrx.o otx2_ethtool.o
ccflags-y += -I$(srctree)/drivers/net/ethernet/marvell/octeontx2/af

1410
drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.c Normal file
View File

File diff suppressed because it is too large Load Diff

615
drivers/net/ethernet/marvell/octeontx2/nic/otx2_common.h Normal file
View File

@ -0,0 +1,615 @@
/* SPDX-License-Identifier: GPL-2.0 */
/* Marvell OcteonTx2 RVU Ethernet driver
*
* Copyright (C) 2020 Marvell International Ltd.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*/
#ifndef OTX2_COMMON_H
#define OTX2_COMMON_H
#include <linux/pci.h>
#include <linux/iommu.h>
#include <mbox.h>
#include "otx2_reg.h"
#include "otx2_txrx.h"
/* PCI device IDs */
#define PCI_DEVID_OCTEONTX2_RVU_PF 0xA063
#define PCI_SUBSYS_DEVID_96XX_RVU_PFVF 0xB200
/* PCI BAR nos */
#define PCI_CFG_REG_BAR_NUM 2
#define PCI_MBOX_BAR_NUM 4
#define NAME_SIZE 32
enum arua_mapped_qtypes {
AURA_NIX_RQ,
AURA_NIX_SQ,
};
/* NIX LF interrupts range*/
#define NIX_LF_QINT_VEC_START 0x00
#define NIX_LF_CINT_VEC_START 0x40
#define NIX_LF_GINT_VEC 0x80
#define NIX_LF_ERR_VEC 0x81
#define NIX_LF_POISON_VEC 0x82
/* RSS configuration */
struct otx2_rss_info {
u8 enable;
u32 flowkey_cfg;
u16 rss_size;
u8 ind_tbl[MAX_RSS_INDIR_TBL_SIZE];
#define RSS_HASH_KEY_SIZE 44 /* 352 bit key */
u8 key[RSS_HASH_KEY_SIZE];
};
/* NIX (or NPC) RX errors */
enum otx2_errlvl {
NPC_ERRLVL_RE,
NPC_ERRLVL_LID_LA,
NPC_ERRLVL_LID_LB,
NPC_ERRLVL_LID_LC,
NPC_ERRLVL_LID_LD,
NPC_ERRLVL_LID_LE,
NPC_ERRLVL_LID_LF,
NPC_ERRLVL_LID_LG,
NPC_ERRLVL_LID_LH,
NPC_ERRLVL_NIX = 0x0F,
};
enum otx2_errcodes_re {
/* NPC_ERRLVL_RE errcodes */
ERRCODE_FCS = 0x7,
ERRCODE_FCS_RCV = 0x8,
ERRCODE_UNDERSIZE = 0x10,
ERRCODE_OVERSIZE = 0x11,
ERRCODE_OL2_LEN_MISMATCH = 0x12,
/* NPC_ERRLVL_NIX errcodes */
ERRCODE_OL3_LEN = 0x10,
ERRCODE_OL4_LEN = 0x11,
ERRCODE_OL4_CSUM = 0x12,
ERRCODE_IL3_LEN = 0x20,
ERRCODE_IL4_LEN = 0x21,
ERRCODE_IL4_CSUM = 0x22,
};
/* NIX TX stats */
enum nix_stat_lf_tx {
TX_UCAST = 0x0,
TX_BCAST = 0x1,
TX_MCAST = 0x2,
TX_DROP = 0x3,
TX_OCTS = 0x4,
TX_STATS_ENUM_LAST,
};
/* NIX RX stats */
enum nix_stat_lf_rx {
RX_OCTS = 0x0,
RX_UCAST = 0x1,
RX_BCAST = 0x2,
RX_MCAST = 0x3,
RX_DROP = 0x4,
RX_DROP_OCTS = 0x5,
RX_FCS = 0x6,
RX_ERR = 0x7,
RX_DRP_BCAST = 0x8,
RX_DRP_MCAST = 0x9,
RX_DRP_L3BCAST = 0xa,
RX_DRP_L3MCAST = 0xb,
RX_STATS_ENUM_LAST,
};
struct otx2_dev_stats {
u64 rx_bytes;
u64 rx_frames;
u64 rx_ucast_frames;
u64 rx_bcast_frames;
u64 rx_mcast_frames;
u64 rx_drops;
u64 tx_bytes;
u64 tx_frames;
u64 tx_ucast_frames;
u64 tx_bcast_frames;
u64 tx_mcast_frames;
u64 tx_drops;
};
/* Driver counted stats */
struct otx2_drv_stats {
atomic_t rx_fcs_errs;
atomic_t rx_oversize_errs;
atomic_t rx_undersize_errs;
atomic_t rx_csum_errs;
atomic_t rx_len_errs;
atomic_t rx_other_errs;
};
struct mbox {
struct otx2_mbox mbox;
struct work_struct mbox_wrk;
struct otx2_mbox mbox_up;
struct work_struct mbox_up_wrk;
struct otx2_nic *pfvf;
void *bbuf_base; /* Bounce buffer for mbox memory */
struct mutex lock; /* serialize mailbox access */
int num_msgs; /* mbox number of messages */
int up_num_msgs; /* mbox_up number of messages */
};
struct otx2_hw {
struct pci_dev *pdev;
struct otx2_rss_info rss_info;
u16 rx_queues;
u16 tx_queues;
u16 max_queues;
u16 pool_cnt;
u16 rqpool_cnt;
u16 sqpool_cnt;
/* NPA */
u32 stack_pg_ptrs; /* No of ptrs per stack page */
u32 stack_pg_bytes; /* Size of stack page */
u16 sqb_size;
/* NIX */
u16 txschq_list[NIX_TXSCH_LVL_CNT][MAX_TXSCHQ_PER_FUNC];
/* HW settings, coalescing etc */
u16 rx_chan_base;
u16 tx_chan_base;
u16 cq_qcount_wait;
u16 cq_ecount_wait;
u16 rq_skid;
u8 cq_time_wait;
/* For TSO segmentation */
u8 lso_tsov4_idx;
u8 lso_tsov6_idx;
u8 hw_tso;
/* MSI-X */
u8 cint_cnt; /* CQ interrupt count */
u16 npa_msixoff; /* Offset of NPA vectors */
u16 nix_msixoff; /* Offset of NIX vectors */
char *irq_name;
cpumask_var_t *affinity_mask;
/* Stats */
struct otx2_dev_stats dev_stats;
struct otx2_drv_stats drv_stats;
u64 cgx_rx_stats[CGX_RX_STATS_COUNT];
u64 cgx_tx_stats[CGX_TX_STATS_COUNT];
};
struct refill_work {
struct delayed_work pool_refill_work;
struct otx2_nic *pf;
};
struct otx2_nic {
void __iomem *reg_base;
struct net_device *netdev;
void *iommu_domain;
u16 max_frs;
u16 rbsize; /* Receive buffer size */
#define OTX2_FLAG_INTF_DOWN BIT_ULL(2)
u64 flags;
struct otx2_qset qset;
struct otx2_hw hw;
struct pci_dev *pdev;
struct device *dev;
/* Mbox */
struct mbox mbox;
struct workqueue_struct *mbox_wq;
u16 pcifunc; /* RVU PF_FUNC */
struct cgx_link_user_info linfo;
u64 reset_count;
struct work_struct reset_task;
struct refill_work *refill_wrk;
/* Ethtool stuff */
u32 msg_enable;
/* Block address of NIX either BLKADDR_NIX0 or BLKADDR_NIX1 */
int nix_blkaddr;
};
static inline bool is_96xx_A0(struct pci_dev *pdev)
{
return (pdev->revision == 0x00) &&
(pdev->subsystem_device == PCI_SUBSYS_DEVID_96XX_RVU_PFVF);
}
static inline bool is_96xx_B0(struct pci_dev *pdev)
{
return (pdev->revision == 0x01) &&
(pdev->subsystem_device == PCI_SUBSYS_DEVID_96XX_RVU_PFVF);
}
static inline void otx2_setup_dev_hw_settings(struct otx2_nic *pfvf)
{
struct otx2_hw *hw = &pfvf->hw;
pfvf->hw.cq_time_wait = CQ_TIMER_THRESH_DEFAULT;
pfvf->hw.cq_ecount_wait = CQ_CQE_THRESH_DEFAULT;
pfvf->hw.cq_qcount_wait = CQ_QCOUNT_DEFAULT;
hw->hw_tso = true;
if (is_96xx_A0(pfvf->pdev)) {
hw->hw_tso = false;
/* Time based irq coalescing is not supported */
pfvf->hw.cq_qcount_wait = 0x0;
/* Due to HW issue previous silicons required minimum
* 600 unused CQE to avoid CQ overflow.
*/
pfvf->hw.rq_skid = 600;
pfvf->qset.rqe_cnt = Q_COUNT(Q_SIZE_1K);
}
}
/* Register read/write APIs */
static inline void __iomem *otx2_get_regaddr(struct otx2_nic *nic, u64 offset)
{
u64 blkaddr;
switch ((offset >> RVU_FUNC_BLKADDR_SHIFT) & RVU_FUNC_BLKADDR_MASK) {
case BLKTYPE_NIX:
blkaddr = nic->nix_blkaddr;
break;
case BLKTYPE_NPA:
blkaddr = BLKADDR_NPA;
break;
default:
blkaddr = BLKADDR_RVUM;
break;
};
offset &= ~(RVU_FUNC_BLKADDR_MASK << RVU_FUNC_BLKADDR_SHIFT);
offset |= (blkaddr << RVU_FUNC_BLKADDR_SHIFT);
return nic->reg_base + offset;
}
static inline void otx2_write64(struct otx2_nic *nic, u64 offset, u64 val)
{
void __iomem *addr = otx2_get_regaddr(nic, offset);
writeq(val, addr);
}
static inline u64 otx2_read64(struct otx2_nic *nic, u64 offset)
{
void __iomem *addr = otx2_get_regaddr(nic, offset);
return readq(addr);
}
/* Mbox bounce buffer APIs */
static inline int otx2_mbox_bbuf_init(struct mbox *mbox, struct pci_dev *pdev)
{
struct otx2_mbox *otx2_mbox;
struct otx2_mbox_dev *mdev;
mbox->bbuf_base = devm_kmalloc(&pdev->dev, MBOX_SIZE, GFP_KERNEL);
if (!mbox->bbuf_base)
return -ENOMEM;
/* Overwrite mbox mbase to point to bounce buffer, so that PF/VF
* prepare all mbox messages in bounce buffer instead of directly
* in hw mbox memory.
*/
otx2_mbox = &mbox->mbox;
mdev = &otx2_mbox->dev[0];
mdev->mbase = mbox->bbuf_base;
otx2_mbox = &mbox->mbox_up;
mdev = &otx2_mbox->dev[0];
mdev->mbase = mbox->bbuf_base;
return 0;
}
static inline void otx2_sync_mbox_bbuf(struct otx2_mbox *mbox, int devid)
{
u16 msgs_offset = ALIGN(sizeof(struct mbox_hdr), MBOX_MSG_ALIGN);
void *hw_mbase = mbox->hwbase + (devid * MBOX_SIZE);
struct otx2_mbox_dev *mdev = &mbox->dev[devid];
struct mbox_hdr *hdr;
u64 msg_size;
if (mdev->mbase == hw_mbase)
return;
hdr = hw_mbase + mbox->rx_start;
msg_size = hdr->msg_size;
if (msg_size > mbox->rx_size - msgs_offset)
msg_size = mbox->rx_size - msgs_offset;
/* Copy mbox messages from mbox memory to bounce buffer */
memcpy(mdev->mbase + mbox->rx_start,
hw_mbase + mbox->rx_start, msg_size + msgs_offset);
}
static inline void otx2_mbox_lock_init(struct mbox *mbox)
{
mutex_init(&mbox->lock);
}
static inline void otx2_mbox_lock(struct mbox *mbox)
{
mutex_lock(&mbox->lock);
}
static inline void otx2_mbox_unlock(struct mbox *mbox)
{
mutex_unlock(&mbox->lock);
}
/* With the absence of API for 128-bit IO memory access for arm64,
* implement required operations at place.
*/
#if defined(CONFIG_ARM64)
static inline void otx2_write128(u64 lo, u64 hi, void __iomem *addr)
{
__asm__ volatile("stp %x[x0], %x[x1], [%x[p1],#0]!"
::[x0]"r"(lo), [x1]"r"(hi), [p1]"r"(addr));
}
static inline u64 otx2_atomic64_add(u64 incr, u64 *ptr)
{
u64 result;
__asm__ volatile(".cpu generic+lse\n"
"ldadd %x[i], %x[r], [%[b]]"
: [r]"=r"(result), "+m"(*ptr)
: [i]"r"(incr), [b]"r"(ptr)
: "memory");
return result;
}
static inline u64 otx2_lmt_flush(uint64_t addr)
{
u64 result = 0;
__asm__ volatile(".cpu generic+lse\n"
"ldeor xzr,%x[rf],[%[rs]]"
: [rf]"=r"(result)
: [rs]"r"(addr));
return result;
}
#else
#define otx2_write128(lo, hi, addr)
#define otx2_atomic64_add(incr, ptr) ({ *ptr += incr; })
#define otx2_lmt_flush(addr) ({ 0; })
#endif
/* Alloc pointer from pool/aura */
static inline u64 otx2_aura_allocptr(struct otx2_nic *pfvf, int aura)
{
u64 *ptr = (u64 *)otx2_get_regaddr(pfvf,
NPA_LF_AURA_OP_ALLOCX(0));
u64 incr = (u64)aura | BIT_ULL(63);
return otx2_atomic64_add(incr, ptr);
}
/* Free pointer to a pool/aura */
static inline void otx2_aura_freeptr(struct otx2_nic *pfvf,
int aura, s64 buf)
{
otx2_write128((u64)buf, (u64)aura | BIT_ULL(63),
otx2_get_regaddr(pfvf, NPA_LF_AURA_OP_FREE0));
}
/* Update page ref count */
static inline void otx2_get_page(struct otx2_pool *pool)
{
if (!pool->page)
return;
if (pool->pageref)
page_ref_add(pool->page, pool->pageref);
pool->pageref = 0;
pool->page = NULL;
}
static inline int otx2_get_pool_idx(struct otx2_nic *pfvf, int type, int idx)
{
if (type == AURA_NIX_SQ)
return pfvf->hw.rqpool_cnt + idx;
/* AURA_NIX_RQ */
return idx;
}
/* Mbox APIs */
static inline int otx2_sync_mbox_msg(struct mbox *mbox)
{
int err;
if (!otx2_mbox_nonempty(&mbox->mbox, 0))
return 0;
otx2_mbox_msg_send(&mbox->mbox, 0);
err = otx2_mbox_wait_for_rsp(&mbox->mbox, 0);
if (err)
return err;
return otx2_mbox_check_rsp_msgs(&mbox->mbox, 0);
}
static inline int otx2_sync_mbox_up_msg(struct mbox *mbox, int devid)
{
int err;
if (!otx2_mbox_nonempty(&mbox->mbox_up, devid))
return 0;
otx2_mbox_msg_send(&mbox->mbox_up, devid);
err = otx2_mbox_wait_for_rsp(&mbox->mbox_up, devid);
if (err)
return err;
return otx2_mbox_check_rsp_msgs(&mbox->mbox_up, devid);
}
/* Use this API to send mbox msgs in atomic context
* where sleeping is not allowed
*/
static inline int otx2_sync_mbox_msg_busy_poll(struct mbox *mbox)
{
int err;
if (!otx2_mbox_nonempty(&mbox->mbox, 0))
return 0;
otx2_mbox_msg_send(&mbox->mbox, 0);
err = otx2_mbox_busy_poll_for_rsp(&mbox->mbox, 0);
if (err)
return err;
return otx2_mbox_check_rsp_msgs(&mbox->mbox, 0);
}
#define M(_name, _id, _fn_name, _req_type, _rsp_type) \
static struct _req_type __maybe_unused \
*otx2_mbox_alloc_msg_ ## _fn_name(struct mbox *mbox) \
{ \
struct _req_type *req; \
\
req = (struct _req_type *)otx2_mbox_alloc_msg_rsp( \
&mbox->mbox, 0, sizeof(struct _req_type), \
sizeof(struct _rsp_type)); \
if (!req) \
return NULL; \
req->hdr.sig = OTX2_MBOX_REQ_SIG; \
req->hdr.id = _id; \
return req; \
}
MBOX_MESSAGES
#undef M
#define M(_name, _id, _fn_name, _req_type, _rsp_type) \
int \
otx2_mbox_up_handler_ ## _fn_name(struct otx2_nic *pfvf, \
struct _req_type *req, \
struct _rsp_type *rsp); \
MBOX_UP_CGX_MESSAGES
#undef M
/* Time to wait before watchdog kicks off */
#define OTX2_TX_TIMEOUT (100 * HZ)
#define RVU_PFVF_PF_SHIFT 10
#define RVU_PFVF_PF_MASK 0x3F
#define RVU_PFVF_FUNC_SHIFT 0
#define RVU_PFVF_FUNC_MASK 0x3FF
static inline int rvu_get_pf(u16 pcifunc)
{
return (pcifunc >> RVU_PFVF_PF_SHIFT) & RVU_PFVF_PF_MASK;
}
static inline dma_addr_t otx2_dma_map_page(struct otx2_nic *pfvf,
struct page *page,
size_t offset, size_t size,
enum dma_data_direction dir)
{
dma_addr_t iova;
iova = dma_map_page_attrs(pfvf->dev, page,
offset, size, dir, DMA_ATTR_SKIP_CPU_SYNC);
if (unlikely(dma_mapping_error(pfvf->dev, iova)))
return (dma_addr_t)NULL;
return iova;
}
static inline void otx2_dma_unmap_page(struct otx2_nic *pfvf,
dma_addr_t addr, size_t size,
enum dma_data_direction dir)
{
dma_unmap_page_attrs(pfvf->dev, addr, size,
dir, DMA_ATTR_SKIP_CPU_SYNC);
}
/* MSI-X APIs */
void otx2_free_cints(struct otx2_nic *pfvf, int n);
void otx2_set_cints_affinity(struct otx2_nic *pfvf);
int otx2_set_mac_address(struct net_device *netdev, void *p);
int otx2_hw_set_mtu(struct otx2_nic *pfvf, int mtu);
void otx2_tx_timeout(struct net_device *netdev, unsigned int txq);
void otx2_get_mac_from_af(struct net_device *netdev);
void otx2_config_irq_coalescing(struct otx2_nic *pfvf, int qidx);
/* RVU block related APIs */
int otx2_attach_npa_nix(struct otx2_nic *pfvf);
int otx2_detach_resources(struct mbox *mbox);
int otx2_config_npa(struct otx2_nic *pfvf);
int otx2_sq_aura_pool_init(struct otx2_nic *pfvf);
int otx2_rq_aura_pool_init(struct otx2_nic *pfvf);
void otx2_aura_pool_free(struct otx2_nic *pfvf);
void otx2_free_aura_ptr(struct otx2_nic *pfvf, int type);
void otx2_sq_free_sqbs(struct otx2_nic *pfvf);
int otx2_config_nix(struct otx2_nic *pfvf);
int otx2_config_nix_queues(struct otx2_nic *pfvf);
int otx2_txschq_config(struct otx2_nic *pfvf, int lvl);
int otx2_txsch_alloc(struct otx2_nic *pfvf);
int otx2_txschq_stop(struct otx2_nic *pfvf);
void otx2_sqb_flush(struct otx2_nic *pfvf);
dma_addr_t otx2_alloc_rbuf(struct otx2_nic *pfvf, struct otx2_pool *pool,
gfp_t gfp);
int otx2_rxtx_enable(struct otx2_nic *pfvf, bool enable);
void otx2_ctx_disable(struct mbox *mbox, int type, bool npa);
void otx2_cleanup_rx_cqes(struct otx2_nic *pfvf, struct otx2_cq_queue *cq);
void otx2_cleanup_tx_cqes(struct otx2_nic *pfvf, struct otx2_cq_queue *cq);
/* RSS configuration APIs*/
int otx2_rss_init(struct otx2_nic *pfvf);
int otx2_set_flowkey_cfg(struct otx2_nic *pfvf);
void otx2_set_rss_key(struct otx2_nic *pfvf);
int otx2_set_rss_table(struct otx2_nic *pfvf);
/* Mbox handlers */
void mbox_handler_msix_offset(struct otx2_nic *pfvf,
struct msix_offset_rsp *rsp);
void mbox_handler_npa_lf_alloc(struct otx2_nic *pfvf,
struct npa_lf_alloc_rsp *rsp);
void mbox_handler_nix_lf_alloc(struct otx2_nic *pfvf,
struct nix_lf_alloc_rsp *rsp);
void mbox_handler_nix_txsch_alloc(struct otx2_nic *pf,
struct nix_txsch_alloc_rsp *rsp);
void mbox_handler_cgx_stats(struct otx2_nic *pfvf,
struct cgx_stats_rsp *rsp);
/* Device stats APIs */
void otx2_get_dev_stats(struct otx2_nic *pfvf);
void otx2_get_stats64(struct net_device *netdev,
struct rtnl_link_stats64 *stats);
void otx2_update_lmac_stats(struct otx2_nic *pfvf);
int otx2_update_rq_stats(struct otx2_nic *pfvf, int qidx);
int otx2_update_sq_stats(struct otx2_nic *pfvf, int qidx);
void otx2_set_ethtool_ops(struct net_device *netdev);
int otx2_open(struct net_device *netdev);
int otx2_stop(struct net_device *netdev);
int otx2_set_real_num_queues(struct net_device *netdev,
int tx_queues, int rx_queues);
#endif /* OTX2_COMMON_H */

662
drivers/net/ethernet/marvell/octeontx2/nic/otx2_ethtool.c Normal file
View File

@ -0,0 +1,662 @@
// SPDX-License-Identifier: GPL-2.0
/* Marvell OcteonTx2 RVU Ethernet driver
*
* Copyright (C) 2020 Marvell International Ltd.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*/
#include <linux/pci.h>
#include <linux/ethtool.h>
#include <linux/stddef.h>
#include <linux/etherdevice.h>
#include <linux/log2.h>
#include "otx2_common.h"
#define DRV_NAME "octeontx2-nicpf"
struct otx2_stat {
char name[ETH_GSTRING_LEN];
unsigned int index;
};
/* HW device stats */
#define OTX2_DEV_STAT(stat) { \
.name = #stat, \
.index = offsetof(struct otx2_dev_stats, stat) / sizeof(u64), \
}
static const struct otx2_stat otx2_dev_stats[] = {
OTX2_DEV_STAT(rx_ucast_frames),
OTX2_DEV_STAT(rx_bcast_frames),
OTX2_DEV_STAT(rx_mcast_frames),
OTX2_DEV_STAT(tx_ucast_frames),
OTX2_DEV_STAT(tx_bcast_frames),
OTX2_DEV_STAT(tx_mcast_frames),
};
/* Driver level stats */
#define OTX2_DRV_STAT(stat) { \
.name = #stat, \
.index = offsetof(struct otx2_drv_stats, stat) / sizeof(atomic_t), \
}
static const struct otx2_stat otx2_drv_stats[] = {
OTX2_DRV_STAT(rx_fcs_errs),
OTX2_DRV_STAT(rx_oversize_errs),
OTX2_DRV_STAT(rx_undersize_errs),
OTX2_DRV_STAT(rx_csum_errs),
OTX2_DRV_STAT(rx_len_errs),
OTX2_DRV_STAT(rx_other_errs),
};
static const struct otx2_stat otx2_queue_stats[] = {
{ "bytes", 0 },
{ "frames", 1 },
};
static const unsigned int otx2_n_dev_stats = ARRAY_SIZE(otx2_dev_stats);
static const unsigned int otx2_n_drv_stats = ARRAY_SIZE(otx2_drv_stats);
static const unsigned int otx2_n_queue_stats = ARRAY_SIZE(otx2_queue_stats);
static void otx2_dev_open(struct net_device *netdev)
{
otx2_open(netdev);
}
static void otx2_dev_stop(struct net_device *netdev)
{
otx2_stop(netdev);
}
static void otx2_get_drvinfo(struct net_device *netdev,
struct ethtool_drvinfo *info)
{
struct otx2_nic *pfvf = netdev_priv(netdev);
strlcpy(info->driver, DRV_NAME, sizeof(info->driver));
strlcpy(info->bus_info, pci_name(pfvf->pdev), sizeof(info->bus_info));
}
static void otx2_get_qset_strings(struct otx2_nic *pfvf, u8 **data, int qset)
{
int start_qidx = qset * pfvf->hw.rx_queues;
int qidx, stats;
for (qidx = 0; qidx < pfvf->hw.rx_queues; qidx++) {
for (stats = 0; stats < otx2_n_queue_stats; stats++) {
sprintf(*data, "rxq%d: %s", qidx + start_qidx,
otx2_queue_stats[stats].name);
*data += ETH_GSTRING_LEN;
}
}
for (qidx = 0; qidx < pfvf->hw.tx_queues; qidx++) {
for (stats = 0; stats < otx2_n_queue_stats; stats++) {
sprintf(*data, "txq%d: %s", qidx + start_qidx,
otx2_queue_stats[stats].name);
*data += ETH_GSTRING_LEN;
}
}
}
static void otx2_get_strings(struct net_device *netdev, u32 sset, u8 *data)
{
struct otx2_nic *pfvf = netdev_priv(netdev);
int stats;
if (sset != ETH_SS_STATS)
return;
for (stats = 0; stats < otx2_n_dev_stats; stats++) {
memcpy(data, otx2_dev_stats[stats].name, ETH_GSTRING_LEN);
data += ETH_GSTRING_LEN;
}
for (stats = 0; stats < otx2_n_drv_stats; stats++) {
memcpy(data, otx2_drv_stats[stats].name, ETH_GSTRING_LEN);
data += ETH_GSTRING_LEN;
}
otx2_get_qset_strings(pfvf, &data, 0);
for (stats = 0; stats < CGX_RX_STATS_COUNT; stats++) {
sprintf(data, "cgx_rxstat%d: ", stats);
data += ETH_GSTRING_LEN;
}
for (stats = 0; stats < CGX_TX_STATS_COUNT; stats++) {
sprintf(data, "cgx_txstat%d: ", stats);
data += ETH_GSTRING_LEN;
}
strcpy(data, "reset_count");
data += ETH_GSTRING_LEN;
}
static void otx2_get_qset_stats(struct otx2_nic *pfvf,
struct ethtool_stats *stats, u64 **data)
{
int stat, qidx;
if (!pfvf)
return;
for (qidx = 0; qidx < pfvf->hw.rx_queues; qidx++) {
if (!otx2_update_rq_stats(pfvf, qidx)) {
for (stat = 0; stat < otx2_n_queue_stats; stat++)
*((*data)++) = 0;
continue;
}
for (stat = 0; stat < otx2_n_queue_stats; stat++)
*((*data)++) = ((u64 *)&pfvf->qset.rq[qidx].stats)
[otx2_queue_stats[stat].index];
}
for (qidx = 0; qidx < pfvf->hw.tx_queues; qidx++) {
if (!otx2_update_sq_stats(pfvf, qidx)) {
for (stat = 0; stat < otx2_n_queue_stats; stat++)
*((*data)++) = 0;
continue;
}
for (stat = 0; stat < otx2_n_queue_stats; stat++)
*((*data)++) = ((u64 *)&pfvf->qset.sq[qidx].stats)
[otx2_queue_stats[stat].index];
}
}
/* Get device and per queue statistics */
static void otx2_get_ethtool_stats(struct net_device *netdev,
struct ethtool_stats *stats, u64 *data)
{
struct otx2_nic *pfvf = netdev_priv(netdev);
int stat;
otx2_get_dev_stats(pfvf);
for (stat = 0; stat < otx2_n_dev_stats; stat++)
*(data++) = ((u64 *)&pfvf->hw.dev_stats)
[otx2_dev_stats[stat].index];
for (stat = 0; stat < otx2_n_drv_stats; stat++)
*(data++) = atomic_read(&((atomic_t *)&pfvf->hw.drv_stats)
[otx2_drv_stats[stat].index]);
otx2_get_qset_stats(pfvf, stats, &data);
otx2_update_lmac_stats(pfvf);
for (stat = 0; stat < CGX_RX_STATS_COUNT; stat++)
*(data++) = pfvf->hw.cgx_rx_stats[stat];
for (stat = 0; stat < CGX_TX_STATS_COUNT; stat++)
*(data++) = pfvf->hw.cgx_tx_stats[stat];
*(data++) = pfvf->reset_count;
}
static int otx2_get_sset_count(struct net_device *netdev, int sset)
{
struct otx2_nic *pfvf = netdev_priv(netdev);
int qstats_count;
if (sset != ETH_SS_STATS)
return -EINVAL;
qstats_count = otx2_n_queue_stats *
(pfvf->hw.rx_queues + pfvf->hw.tx_queues);
return otx2_n_dev_stats + otx2_n_drv_stats + qstats_count +
CGX_RX_STATS_COUNT + CGX_TX_STATS_COUNT + 1;
}
/* Get no of queues device supports and current queue count */
static void otx2_get_channels(struct net_device *dev,
struct ethtool_channels *channel)
{
struct otx2_nic *pfvf = netdev_priv(dev);
channel->max_rx = pfvf->hw.max_queues;
channel->max_tx = pfvf->hw.max_queues;
channel->rx_count = pfvf->hw.rx_queues;
channel->tx_count = pfvf->hw.tx_queues;
}
/* Set no of Tx, Rx queues to be used */
static int otx2_set_channels(struct net_device *dev,
struct ethtool_channels *channel)
{
struct otx2_nic *pfvf = netdev_priv(dev);
bool if_up = netif_running(dev);
int err = 0;
if (!channel->rx_count || !channel->tx_count)
return -EINVAL;
if (if_up)
otx2_dev_stop(dev);
err = otx2_set_real_num_queues(dev, channel->tx_count,
channel->rx_count);
if (err)
goto fail;
pfvf->hw.rx_queues = channel->rx_count;
pfvf->hw.tx_queues = channel->tx_count;
pfvf->qset.cq_cnt = pfvf->hw.tx_queues + pfvf->hw.rx_queues;
fail:
if (if_up)
otx2_dev_open(dev);
netdev_info(dev, "Setting num Tx rings to %d, Rx rings to %d success\n",
pfvf->hw.tx_queues, pfvf->hw.rx_queues);
return err;
}
static void otx2_get_ringparam(struct net_device *netdev,
struct ethtool_ringparam *ring)
{
struct otx2_nic *pfvf = netdev_priv(netdev);
struct otx2_qset *qs = &pfvf->qset;
ring->rx_max_pending = Q_COUNT(Q_SIZE_MAX);
ring->rx_pending = qs->rqe_cnt ? qs->rqe_cnt : Q_COUNT(Q_SIZE_256);
ring->tx_max_pending = Q_COUNT(Q_SIZE_MAX);
ring->tx_pending = qs->sqe_cnt ? qs->sqe_cnt : Q_COUNT(Q_SIZE_4K);
}
static int otx2_set_ringparam(struct net_device *netdev,
struct ethtool_ringparam *ring)
{
struct otx2_nic *pfvf = netdev_priv(netdev);
bool if_up = netif_running(netdev);
struct otx2_qset *qs = &pfvf->qset;
u32 rx_count, tx_count;
if (ring->rx_mini_pending || ring->rx_jumbo_pending)
return -EINVAL;
/* Permitted lengths are 16 64 256 1K 4K 16K 64K 256K 1M */
rx_count = ring->rx_pending;
/* On some silicon variants a skid or reserved CQEs are
* needed to avoid CQ overflow.
*/
if (rx_count < pfvf->hw.rq_skid)
rx_count = pfvf->hw.rq_skid;
rx_count = Q_COUNT(Q_SIZE(rx_count, 3));
/* Due pipelining impact minimum 2000 unused SQ CQE's
* need to be maintained to avoid CQ overflow, hence the
* minimum 4K size.
*/
tx_count = clamp_t(u32, ring->tx_pending,
Q_COUNT(Q_SIZE_4K), Q_COUNT(Q_SIZE_MAX));
tx_count = Q_COUNT(Q_SIZE(tx_count, 3));
if (tx_count == qs->sqe_cnt && rx_count == qs->rqe_cnt)
return 0;
if (if_up)
otx2_dev_stop(netdev);
/* Assigned to the nearest possible exponent. */
qs->sqe_cnt = tx_count;
qs->rqe_cnt = rx_count;
if (if_up)
otx2_dev_open(netdev);
return 0;
}
static int otx2_get_coalesce(struct net_device *netdev,
struct ethtool_coalesce *cmd)
{
struct otx2_nic *pfvf = netdev_priv(netdev);
struct otx2_hw *hw = &pfvf->hw;
cmd->rx_coalesce_usecs = hw->cq_time_wait;
cmd->rx_max_coalesced_frames = hw->cq_ecount_wait;
cmd->tx_coalesce_usecs = hw->cq_time_wait;
cmd->tx_max_coalesced_frames = hw->cq_ecount_wait;
return 0;
}
static int otx2_set_coalesce(struct net_device *netdev,
struct ethtool_coalesce *ec)
{
struct otx2_nic *pfvf = netdev_priv(netdev);
struct otx2_hw *hw = &pfvf->hw;
int qidx;
if (ec->use_adaptive_rx_coalesce || ec->use_adaptive_tx_coalesce ||
ec->rx_coalesce_usecs_irq || ec->rx_max_coalesced_frames_irq ||
ec->tx_coalesce_usecs_irq || ec->tx_max_coalesced_frames_irq ||
ec->stats_block_coalesce_usecs || ec->pkt_rate_low ||
ec->rx_coalesce_usecs_low || ec->rx_max_coalesced_frames_low ||
ec->tx_coalesce_usecs_low || ec->tx_max_coalesced_frames_low ||
ec->pkt_rate_high || ec->rx_coalesce_usecs_high ||
ec->rx_max_coalesced_frames_high || ec->tx_coalesce_usecs_high ||
ec->tx_max_coalesced_frames_high || ec->rate_sample_interval)
return -EOPNOTSUPP;
if (!ec->rx_max_coalesced_frames || !ec->tx_max_coalesced_frames)
return 0;
/* 'cq_time_wait' is 8bit and is in multiple of 100ns,
* so clamp the user given value to the range of 1 to 25usec.
*/
ec->rx_coalesce_usecs = clamp_t(u32, ec->rx_coalesce_usecs,
1, CQ_TIMER_THRESH_MAX);
ec->tx_coalesce_usecs = clamp_t(u32, ec->tx_coalesce_usecs,
1, CQ_TIMER_THRESH_MAX);
/* Rx and Tx are mapped to same CQ, check which one
* is changed, if both then choose the min.
*/
if (hw->cq_time_wait == ec->rx_coalesce_usecs)
hw->cq_time_wait = ec->tx_coalesce_usecs;
else if (hw->cq_time_wait == ec->tx_coalesce_usecs)
hw->cq_time_wait = ec->rx_coalesce_usecs;
else
hw->cq_time_wait = min_t(u8, ec->rx_coalesce_usecs,
ec->tx_coalesce_usecs);
/* Max ecount_wait supported is 16bit,
* so clamp the user given value to the range of 1 to 64k.
*/
ec->rx_max_coalesced_frames = clamp_t(u32, ec->rx_max_coalesced_frames,
1, U16_MAX);
ec->tx_max_coalesced_frames = clamp_t(u32, ec->tx_max_coalesced_frames,
1, U16_MAX);
/* Rx and Tx are mapped to same CQ, check which one
* is changed, if both then choose the min.
*/
if (hw->cq_ecount_wait == ec->rx_max_coalesced_frames)
hw->cq_ecount_wait = ec->tx_max_coalesced_frames;
else if (hw->cq_ecount_wait == ec->tx_max_coalesced_frames)
hw->cq_ecount_wait = ec->rx_max_coalesced_frames;
else
hw->cq_ecount_wait = min_t(u16, ec->rx_max_coalesced_frames,
ec->tx_max_coalesced_frames);
if (netif_running(netdev)) {
for (qidx = 0; qidx < pfvf->hw.cint_cnt; qidx++)
otx2_config_irq_coalescing(pfvf, qidx);
}
return 0;
}
static int otx2_get_rss_hash_opts(struct otx2_nic *pfvf,
struct ethtool_rxnfc *nfc)
{
struct otx2_rss_info *rss = &pfvf->hw.rss_info;
if (!(rss->flowkey_cfg &
(NIX_FLOW_KEY_TYPE_IPV4 | NIX_FLOW_KEY_TYPE_IPV6)))
return 0;
/* Mimimum is IPv4 and IPv6, SIP/DIP */
nfc->data = RXH_IP_SRC | RXH_IP_DST;
switch (nfc->flow_type) {
case TCP_V4_FLOW:
case TCP_V6_FLOW:
if (rss->flowkey_cfg & NIX_FLOW_KEY_TYPE_TCP)
nfc->data |= RXH_L4_B_0_1 | RXH_L4_B_2_3;
break;
case UDP_V4_FLOW:
case UDP_V6_FLOW:
if (rss->flowkey_cfg & NIX_FLOW_KEY_TYPE_UDP)
nfc->data |= RXH_L4_B_0_1 | RXH_L4_B_2_3;
break;
case SCTP_V4_FLOW:
case SCTP_V6_FLOW:
if (rss->flowkey_cfg & NIX_FLOW_KEY_TYPE_SCTP)
nfc->data |= RXH_L4_B_0_1 | RXH_L4_B_2_3;
break;
case AH_ESP_V4_FLOW:
case AH_V4_FLOW:
case ESP_V4_FLOW:
case IPV4_FLOW:
case AH_ESP_V6_FLOW:
case AH_V6_FLOW:
case ESP_V6_FLOW:
case IPV6_FLOW:
break;
default:
return -EINVAL;
}
return 0;
}
static int otx2_set_rss_hash_opts(struct otx2_nic *pfvf,
struct ethtool_rxnfc *nfc)
{
struct otx2_rss_info *rss = &pfvf->hw.rss_info;
u32 rxh_l4 = RXH_L4_B_0_1 | RXH_L4_B_2_3;
u32 rss_cfg = rss->flowkey_cfg;
if (!rss->enable) {
netdev_err(pfvf->netdev,
"RSS is disabled, cannot change settings\n");
return -EIO;
}
/* Mimimum is IPv4 and IPv6, SIP/DIP */
if (!(nfc->data & RXH_IP_SRC) || !(nfc->data & RXH_IP_DST))
return -EINVAL;
switch (nfc->flow_type) {
case TCP_V4_FLOW:
case TCP_V6_FLOW:
/* Different config for v4 and v6 is not supported.
* Both of them have to be either 4-tuple or 2-tuple.
*/
switch (nfc->data & rxh_l4) {
case 0:
rss_cfg &= ~NIX_FLOW_KEY_TYPE_TCP;
break;
case (RXH_L4_B_0_1 | RXH_L4_B_2_3):
rss_cfg |= NIX_FLOW_KEY_TYPE_TCP;
break;
default:
return -EINVAL;
}
break;
case UDP_V4_FLOW:
case UDP_V6_FLOW:
switch (nfc->data & rxh_l4) {
case 0:
rss_cfg &= ~NIX_FLOW_KEY_TYPE_UDP;
break;
case (RXH_L4_B_0_1 | RXH_L4_B_2_3):
rss_cfg |= NIX_FLOW_KEY_TYPE_UDP;
break;
default:
return -EINVAL;
}
break;
case SCTP_V4_FLOW:
case SCTP_V6_FLOW:
switch (nfc->data & rxh_l4) {
case 0:
rss_cfg &= ~NIX_FLOW_KEY_TYPE_SCTP;
break;
case (RXH_L4_B_0_1 | RXH_L4_B_2_3):
rss_cfg |= NIX_FLOW_KEY_TYPE_SCTP;
break;
default:
return -EINVAL;
}
break;
case IPV4_FLOW:
case IPV6_FLOW:
rss_cfg = NIX_FLOW_KEY_TYPE_IPV4 | NIX_FLOW_KEY_TYPE_IPV6;
break;
default:
return -EINVAL;
}
rss->flowkey_cfg = rss_cfg;
otx2_set_flowkey_cfg(pfvf);
return 0;
}
static int otx2_get_rxnfc(struct net_device *dev,
struct ethtool_rxnfc *nfc, u32 *rules)
{
struct otx2_nic *pfvf = netdev_priv(dev);
int ret = -EOPNOTSUPP;
switch (nfc->cmd) {
case ETHTOOL_GRXRINGS:
nfc->data = pfvf->hw.rx_queues;
ret = 0;
break;
case ETHTOOL_GRXFH:
return otx2_get_rss_hash_opts(pfvf, nfc);
default:
break;
}
return ret;
}
static int otx2_set_rxnfc(struct net_device *dev, struct ethtool_rxnfc *nfc)
{
struct otx2_nic *pfvf = netdev_priv(dev);
int ret = -EOPNOTSUPP;
switch (nfc->cmd) {
case ETHTOOL_SRXFH:
ret = otx2_set_rss_hash_opts(pfvf, nfc);
break;
default:
break;
}
return ret;
}
static u32 otx2_get_rxfh_key_size(struct net_device *netdev)
{
struct otx2_nic *pfvf = netdev_priv(netdev);
struct otx2_rss_info *rss;
rss = &pfvf->hw.rss_info;
return sizeof(rss->key);
}
static u32 otx2_get_rxfh_indir_size(struct net_device *dev)
{
struct otx2_nic *pfvf = netdev_priv(dev);
return pfvf->hw.rss_info.rss_size;
}
/* Get RSS configuration */
static int otx2_get_rxfh(struct net_device *dev, u32 *indir,
u8 *hkey, u8 *hfunc)
{
struct otx2_nic *pfvf = netdev_priv(dev);
struct otx2_rss_info *rss;
int idx;
rss = &pfvf->hw.rss_info;
if (indir) {
for (idx = 0; idx < rss->rss_size; idx++)
indir[idx] = rss->ind_tbl[idx];
}
if (hkey)
memcpy(hkey, rss->key, sizeof(rss->key));
if (hfunc)
*hfunc = ETH_RSS_HASH_TOP;
return 0;
}
/* Configure RSS table and hash key */
static int otx2_set_rxfh(struct net_device *dev, const u32 *indir,
const u8 *hkey, const u8 hfunc)
{
struct otx2_nic *pfvf = netdev_priv(dev);
struct otx2_rss_info *rss;
int idx;
if (hfunc != ETH_RSS_HASH_NO_CHANGE && hfunc != ETH_RSS_HASH_TOP)
return -EOPNOTSUPP;
rss = &pfvf->hw.rss_info;
if (!rss->enable) {
netdev_err(dev, "RSS is disabled, cannot change settings\n");
return -EIO;
}
if (indir) {
for (idx = 0; idx < rss->rss_size; idx++)
rss->ind_tbl[idx] = indir[idx];
}
if (hkey) {
memcpy(rss->key, hkey, sizeof(rss->key));
otx2_set_rss_key(pfvf);
}
otx2_set_rss_table(pfvf);
return 0;
}
static u32 otx2_get_msglevel(struct net_device *netdev)
{
struct otx2_nic *pfvf = netdev_priv(netdev);
return pfvf->msg_enable;
}
static void otx2_set_msglevel(struct net_device *netdev, u32 val)
{
struct otx2_nic *pfvf = netdev_priv(netdev);
pfvf->msg_enable = val;
}
static u32 otx2_get_link(struct net_device *netdev)
{
struct otx2_nic *pfvf = netdev_priv(netdev);
return pfvf->linfo.link_up;
}
static const struct ethtool_ops otx2_ethtool_ops = {
.get_link = otx2_get_link,
.get_drvinfo = otx2_get_drvinfo,
.get_strings = otx2_get_strings,
.get_ethtool_stats = otx2_get_ethtool_stats,
.get_sset_count = otx2_get_sset_count,
.set_channels = otx2_set_channels,
.get_channels = otx2_get_channels,
.get_ringparam = otx2_get_ringparam,
.set_ringparam = otx2_set_ringparam,
.get_coalesce = otx2_get_coalesce,
.set_coalesce = otx2_set_coalesce,
.get_rxnfc = otx2_get_rxnfc,
.set_rxnfc = otx2_set_rxnfc,
.get_rxfh_key_size = otx2_get_rxfh_key_size,
.get_rxfh_indir_size = otx2_get_rxfh_indir_size,
.get_rxfh = otx2_get_rxfh,
.set_rxfh = otx2_set_rxfh,
.get_msglevel = otx2_get_msglevel,
.set_msglevel = otx2_set_msglevel,
};
void otx2_set_ethtool_ops(struct net_device *netdev)
{
netdev->ethtool_ops = &otx2_ethtool_ops;
}

1349
drivers/net/ethernet/marvell/octeontx2/nic/otx2_pf.c Normal file
View File

File diff suppressed because it is too large Load Diff

147
drivers/net/ethernet/marvell/octeontx2/nic/otx2_reg.h Normal file
View File

@ -0,0 +1,147 @@
/* SPDX-License-Identifier: GPL-2.0 */
/* Marvell OcteonTx2 RVU Ethernet driver
*
* Copyright (C) 2020 Marvell International Ltd.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*/
#ifndef OTX2_REG_H
#define OTX2_REG_H
#include <rvu_struct.h>
/* RVU PF registers */
#define RVU_PF_VFX_PFVF_MBOX0 (0x00000)
#define RVU_PF_VFX_PFVF_MBOX1 (0x00008)
#define RVU_PF_VFX_PFVF_MBOXX(a, b) (0x0 | (a) << 12 | (b) << 3)
#define RVU_PF_VF_BAR4_ADDR (0x10)
#define RVU_PF_BLOCK_ADDRX_DISC(a) (0x200 | (a) << 3)
#define RVU_PF_VFME_STATUSX(a) (0x800 | (a) << 3)
#define RVU_PF_VFTRPENDX(a) (0x820 | (a) << 3)
#define RVU_PF_VFTRPEND_W1SX(a) (0x840 | (a) << 3)
#define RVU_PF_VFPF_MBOX_INTX(a) (0x880 | (a) << 3)
#define RVU_PF_VFPF_MBOX_INT_W1SX(a) (0x8A0 | (a) << 3)
#define RVU_PF_VFPF_MBOX_INT_ENA_W1SX(a) (0x8C0 | (a) << 3)
#define RVU_PF_VFPF_MBOX_INT_ENA_W1CX(a) (0x8E0 | (a) << 3)
#define RVU_PF_VFFLR_INTX(a) (0x900 | (a) << 3)
#define RVU_PF_VFFLR_INT_W1SX(a) (0x920 | (a) << 3)
#define RVU_PF_VFFLR_INT_ENA_W1SX(a) (0x940 | (a) << 3)
#define RVU_PF_VFFLR_INT_ENA_W1CX(a) (0x960 | (a) << 3)
#define RVU_PF_VFME_INTX(a) (0x980 | (a) << 3)
#define RVU_PF_VFME_INT_W1SX(a) (0x9A0 | (a) << 3)
#define RVU_PF_VFME_INT_ENA_W1SX(a) (0x9C0 | (a) << 3)
#define RVU_PF_VFME_INT_ENA_W1CX(a) (0x9E0 | (a) << 3)
#define RVU_PF_PFAF_MBOX0 (0xC00)
#define RVU_PF_PFAF_MBOX1 (0xC08)
#define RVU_PF_PFAF_MBOXX(a) (0xC00 | (a) << 3)
#define RVU_PF_INT (0xc20)
#define RVU_PF_INT_W1S (0xc28)
#define RVU_PF_INT_ENA_W1S (0xc30)
#define RVU_PF_INT_ENA_W1C (0xc38)
#define RVU_PF_MSIX_VECX_ADDR(a) (0x000 | (a) << 4)
#define RVU_PF_MSIX_VECX_CTL(a) (0x008 | (a) << 4)
#define RVU_PF_MSIX_PBAX(a) (0xF0000 | (a) << 3)
#define RVU_FUNC_BLKADDR_SHIFT 20
#define RVU_FUNC_BLKADDR_MASK 0x1FULL
/* NPA LF registers */
#define NPA_LFBASE (BLKTYPE_NPA << RVU_FUNC_BLKADDR_SHIFT)
#define NPA_LF_AURA_OP_ALLOCX(a) (NPA_LFBASE | 0x10 | (a) << 3)
#define NPA_LF_AURA_OP_FREE0 (NPA_LFBASE | 0x20)
#define NPA_LF_AURA_OP_FREE1 (NPA_LFBASE | 0x28)
#define NPA_LF_AURA_OP_CNT (NPA_LFBASE | 0x30)
#define NPA_LF_AURA_OP_LIMIT (NPA_LFBASE | 0x50)
#define NPA_LF_AURA_OP_INT (NPA_LFBASE | 0x60)
#define NPA_LF_AURA_OP_THRESH (NPA_LFBASE | 0x70)
#define NPA_LF_POOL_OP_PC (NPA_LFBASE | 0x100)
#define NPA_LF_POOL_OP_AVAILABLE (NPA_LFBASE | 0x110)
#define NPA_LF_POOL_OP_PTR_START0 (NPA_LFBASE | 0x120)
#define NPA_LF_POOL_OP_PTR_START1 (NPA_LFBASE | 0x128)
#define NPA_LF_POOL_OP_PTR_END0 (NPA_LFBASE | 0x130)
#define NPA_LF_POOL_OP_PTR_END1 (NPA_LFBASE | 0x138)
#define NPA_LF_POOL_OP_INT (NPA_LFBASE | 0x160)
#define NPA_LF_POOL_OP_THRESH (NPA_LFBASE | 0x170)
#define NPA_LF_ERR_INT (NPA_LFBASE | 0x200)
#define NPA_LF_ERR_INT_W1S (NPA_LFBASE | 0x208)
#define NPA_LF_ERR_INT_ENA_W1C (NPA_LFBASE | 0x210)
#define NPA_LF_ERR_INT_ENA_W1S (NPA_LFBASE | 0x218)
#define NPA_LF_RAS (NPA_LFBASE | 0x220)
#define NPA_LF_RAS_W1S (NPA_LFBASE | 0x228)
#define NPA_LF_RAS_ENA_W1C (NPA_LFBASE | 0x230)
#define NPA_LF_RAS_ENA_W1S (NPA_LFBASE | 0x238)
#define NPA_LF_QINTX_CNT(a) (NPA_LFBASE | 0x300 | (a) << 12)
#define NPA_LF_QINTX_INT(a) (NPA_LFBASE | 0x310 | (a) << 12)
#define NPA_LF_QINTX_INT_W1S(a) (NPA_LFBASE | 0x318 | (a) << 12)
#define NPA_LF_QINTX_ENA_W1S(a) (NPA_LFBASE | 0x320 | (a) << 12)
#define NPA_LF_QINTX_ENA_W1C(a) (NPA_LFBASE | 0x330 | (a) << 12)
/* NIX LF registers */
#define NIX_LFBASE (BLKTYPE_NIX << RVU_FUNC_BLKADDR_SHIFT)
#define NIX_LF_RX_SECRETX(a) (NIX_LFBASE | 0x0 | (a) << 3)
#define NIX_LF_CFG (NIX_LFBASE | 0x100)
#define NIX_LF_GINT (NIX_LFBASE | 0x200)
#define NIX_LF_GINT_W1S (NIX_LFBASE | 0x208)
#define NIX_LF_GINT_ENA_W1C (NIX_LFBASE | 0x210)
#define NIX_LF_GINT_ENA_W1S (NIX_LFBASE | 0x218)
#define NIX_LF_ERR_INT (NIX_LFBASE | 0x220)
#define NIX_LF_ERR_INT_W1S (NIX_LFBASE | 0x228)
#define NIX_LF_ERR_INT_ENA_W1C (NIX_LFBASE | 0x230)
#define NIX_LF_ERR_INT_ENA_W1S (NIX_LFBASE | 0x238)
#define NIX_LF_RAS (NIX_LFBASE | 0x240)
#define NIX_LF_RAS_W1S (NIX_LFBASE | 0x248)
#define NIX_LF_RAS_ENA_W1C (NIX_LFBASE | 0x250)
#define NIX_LF_RAS_ENA_W1S (NIX_LFBASE | 0x258)
#define NIX_LF_SQ_OP_ERR_DBG (NIX_LFBASE | 0x260)
#define NIX_LF_MNQ_ERR_DBG (NIX_LFBASE | 0x270)
#define NIX_LF_SEND_ERR_DBG (NIX_LFBASE | 0x280)
#define NIX_LF_TX_STATX(a) (NIX_LFBASE | 0x300 | (a) << 3)
#define NIX_LF_RX_STATX(a) (NIX_LFBASE | 0x400 | (a) << 3)
#define NIX_LF_OP_SENDX(a) (NIX_LFBASE | 0x800 | (a) << 3)
#define NIX_LF_RQ_OP_INT (NIX_LFBASE | 0x900)
#define NIX_LF_RQ_OP_OCTS (NIX_LFBASE | 0x910)
#define NIX_LF_RQ_OP_PKTS (NIX_LFBASE | 0x920)
#define NIX_LF_OP_IPSEC_DYNO_CN (NIX_LFBASE | 0x980)
#define NIX_LF_SQ_OP_INT (NIX_LFBASE | 0xa00)
#define NIX_LF_SQ_OP_OCTS (NIX_LFBASE | 0xa10)
#define NIX_LF_SQ_OP_PKTS (NIX_LFBASE | 0xa20)
#define NIX_LF_SQ_OP_STATUS (NIX_LFBASE | 0xa30)
#define NIX_LF_CQ_OP_INT (NIX_LFBASE | 0xb00)
#define NIX_LF_CQ_OP_DOOR (NIX_LFBASE | 0xb30)
#define NIX_LF_CQ_OP_STATUS (NIX_LFBASE | 0xb40)
#define NIX_LF_QINTX_CNT(a) (NIX_LFBASE | 0xC00 | (a) << 12)
#define NIX_LF_QINTX_INT(a) (NIX_LFBASE | 0xC10 | (a) << 12)
#define NIX_LF_QINTX_INT_W1S(a) (NIX_LFBASE | 0xC18 | (a) << 12)
#define NIX_LF_QINTX_ENA_W1S(a) (NIX_LFBASE | 0xC20 | (a) << 12)
#define NIX_LF_QINTX_ENA_W1C(a) (NIX_LFBASE | 0xC30 | (a) << 12)
#define NIX_LF_CINTX_CNT(a) (NIX_LFBASE | 0xD00 | (a) << 12)
#define NIX_LF_CINTX_WAIT(a) (NIX_LFBASE | 0xD10 | (a) << 12)
#define NIX_LF_CINTX_INT(a) (NIX_LFBASE | 0xD20 | (a) << 12)
#define NIX_LF_CINTX_INT_W1S(a) (NIX_LFBASE | 0xD30 | (a) << 12)
#define NIX_LF_CINTX_ENA_W1S(a) (NIX_LFBASE | 0xD40 | (a) << 12)
#define NIX_LF_CINTX_ENA_W1C(a) (NIX_LFBASE | 0xD50 | (a) << 12)
/* NIX AF transmit scheduler registers */
#define NIX_AF_SMQX_CFG(a) (0x700 | (a) << 16)
#define NIX_AF_TL1X_SCHEDULE(a) (0xC00 | (a) << 16)
#define NIX_AF_TL1X_CIR(a) (0xC20 | (a) << 16)
#define NIX_AF_TL1X_TOPOLOGY(a) (0xC80 | (a) << 16)
#define NIX_AF_TL2X_PARENT(a) (0xE88 | (a) << 16)
#define NIX_AF_TL2X_SCHEDULE(a) (0xE00 | (a) << 16)
#define NIX_AF_TL3X_PARENT(a) (0x1088 | (a) << 16)
#define NIX_AF_TL3X_SCHEDULE(a) (0x1000 | (a) << 16)
#define NIX_AF_TL4X_PARENT(a) (0x1288 | (a) << 16)
#define NIX_AF_TL4X_SCHEDULE(a) (0x1200 | (a) << 16)
#define NIX_AF_MDQX_SCHEDULE(a) (0x1400 | (a) << 16)
#define NIX_AF_MDQX_PARENT(a) (0x1480 | (a) << 16)
#define NIX_AF_TL3_TL2X_LINKX_CFG(a, b) (0x1700 | (a) << 16 | (b) << 3)
/* LMT LF registers */
#define LMT_LFBASE BIT_ULL(RVU_FUNC_BLKADDR_SHIFT)
#define LMT_LF_LMTLINEX(a) (LMT_LFBASE | 0x000 | (a) << 12)
#define LMT_LF_LMTCANCEL (LMT_LFBASE | 0x400)
#endif /* OTX2_REG_H */

276
drivers/net/ethernet/marvell/octeontx2/nic/otx2_struct.h Normal file
View File

@ -0,0 +1,276 @@
/* SPDX-License-Identifier: GPL-2.0 */
/* Marvell OcteonTx2 RVU Ethernet driver
*
* Copyright (C) 2020 Marvell International Ltd.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*/
#ifndef OTX2_STRUCT_H
#define OTX2_STRUCT_H
/* NIX WQE/CQE size 128 byte or 512 byte */
enum nix_cqesz_e {
NIX_XQESZ_W64 = 0x0,
NIX_XQESZ_W16 = 0x1,
};
enum nix_sqes_e {
NIX_SQESZ_W16 = 0x0,
NIX_SQESZ_W8 = 0x1,
};
enum nix_send_ldtype {
NIX_SEND_LDTYPE_LDD = 0x0,
NIX_SEND_LDTYPE_LDT = 0x1,
NIX_SEND_LDTYPE_LDWB = 0x2,
};
/* CSUM offload */
enum nix_sendl3type {
NIX_SENDL3TYPE_NONE = 0x0,
NIX_SENDL3TYPE_IP4 = 0x2,
NIX_SENDL3TYPE_IP4_CKSUM = 0x3,
NIX_SENDL3TYPE_IP6 = 0x4,
};
enum nix_sendl4type {
NIX_SENDL4TYPE_NONE,
NIX_SENDL4TYPE_TCP_CKSUM,
NIX_SENDL4TYPE_SCTP_CKSUM,
NIX_SENDL4TYPE_UDP_CKSUM,
};
/* NIX wqe/cqe types */
enum nix_xqe_type {
NIX_XQE_TYPE_INVALID = 0x0,
NIX_XQE_TYPE_RX = 0x1,
NIX_XQE_TYPE_RX_IPSECS = 0x2,
NIX_XQE_TYPE_RX_IPSECH = 0x3,
NIX_XQE_TYPE_RX_IPSECD = 0x4,
NIX_XQE_TYPE_SEND = 0x8,
};
/* NIX CQE/SQE subdescriptor types */
enum nix_subdc {
NIX_SUBDC_NOP = 0x0,
NIX_SUBDC_EXT = 0x1,
NIX_SUBDC_CRC = 0x2,
NIX_SUBDC_IMM = 0x3,
NIX_SUBDC_SG = 0x4,
NIX_SUBDC_MEM = 0x5,
NIX_SUBDC_JUMP = 0x6,
NIX_SUBDC_WORK = 0x7,
NIX_SUBDC_SOD = 0xf,
};
/* Algorithm for nix_sqe_mem_s header (value of the `alg` field) */
enum nix_sendmemalg {
NIX_SENDMEMALG_E_SET = 0x0,
NIX_SENDMEMALG_E_SETTSTMP = 0x1,
NIX_SENDMEMALG_E_SETRSLT = 0x2,
NIX_SENDMEMALG_E_ADD = 0x8,
NIX_SENDMEMALG_E_SUB = 0x9,
NIX_SENDMEMALG_E_ADDLEN = 0xa,
NIX_SENDMEMALG_E_SUBLEN = 0xb,
NIX_SENDMEMALG_E_ADDMBUF = 0xc,
NIX_SENDMEMALG_E_SUBMBUF = 0xd,
NIX_SENDMEMALG_E_ENUM_LAST = 0xe,
};
/* NIX CQE header structure */
struct nix_cqe_hdr_s {
u64 flow_tag : 32;
u64 q : 20;
u64 reserved_52_57 : 6;
u64 node : 2;
u64 cqe_type : 4;
};
/* NIX CQE RX parse structure */
struct nix_rx_parse_s {
u64 chan : 12;
u64 desc_sizem1 : 5;
u64 rsvd_17 : 1;
u64 express : 1;
u64 wqwd : 1;
u64 errlev : 4;
u64 errcode : 8;
u64 latype : 4;
u64 lbtype : 4;
u64 lctype : 4;
u64 ldtype : 4;
u64 letype : 4;
u64 lftype : 4;
u64 lgtype : 4;
u64 lhtype : 4;
u64 pkt_lenm1 : 16; /* W1 */
u64 l2m : 1;
u64 l2b : 1;
u64 l3m : 1;
u64 l3b : 1;
u64 vtag0_valid : 1;
u64 vtag0_gone : 1;
u64 vtag1_valid : 1;
u64 vtag1_gone : 1;
u64 pkind : 6;
u64 rsvd_95_94 : 2;
u64 vtag0_tci : 16;
u64 vtag1_tci : 16;
u64 laflags : 8; /* W2 */
u64 lbflags : 8;
u64 lcflags : 8;
u64 ldflags : 8;
u64 leflags : 8;
u64 lfflags : 8;
u64 lgflags : 8;
u64 lhflags : 8;
u64 eoh_ptr : 8; /* W3 */
u64 wqe_aura : 20;
u64 pb_aura : 20;
u64 match_id : 16;
u64 laptr : 8; /* W4 */
u64 lbptr : 8;
u64 lcptr : 8;
u64 ldptr : 8;
u64 leptr : 8;
u64 lfptr : 8;
u64 lgptr : 8;
u64 lhptr : 8;
u64 vtag0_ptr : 8; /* W5 */
u64 vtag1_ptr : 8;
u64 flow_key_alg : 5;
u64 rsvd_383_341 : 43;
u64 rsvd_447_384; /* W6 */
};
/* NIX CQE RX scatter/gather subdescriptor structure */
struct nix_rx_sg_s {
u64 seg_size : 16; /* W0 */
u64 seg2_size : 16;
u64 seg3_size : 16;
u64 segs : 2;
u64 rsvd_59_50 : 10;
u64 subdc : 4;
u64 seg_addr;
u64 seg2_addr;
u64 seg3_addr;
};
struct nix_send_comp_s {
u64 status : 8;
u64 sqe_id : 16;
u64 rsvd_24_63 : 40;
};
struct nix_cqe_rx_s {
struct nix_cqe_hdr_s hdr;
struct nix_rx_parse_s parse;
struct nix_rx_sg_s sg;
};
struct nix_cqe_tx_s {
struct nix_cqe_hdr_s hdr;
struct nix_send_comp_s comp;
};
/* NIX SQE header structure */
struct nix_sqe_hdr_s {
u64 total : 18; /* W0 */
u64 reserved_18 : 1;
u64 df : 1;
u64 aura : 20;
u64 sizem1 : 3;
u64 pnc : 1;
u64 sq : 20;
u64 ol3ptr : 8; /* W1 */
u64 ol4ptr : 8;
u64 il3ptr : 8;
u64 il4ptr : 8;
u64 ol3type : 4;
u64 ol4type : 4;
u64 il3type : 4;
u64 il4type : 4;
u64 sqe_id : 16;
};
/* NIX send extended header subdescriptor structure */
struct nix_sqe_ext_s {
u64 lso_mps : 14; /* W0 */
u64 lso : 1;
u64 tstmp : 1;
u64 lso_sb : 8;
u64 lso_format : 5;
u64 rsvd_31_29 : 3;
u64 shp_chg : 9;
u64 shp_dis : 1;
u64 shp_ra : 2;
u64 markptr : 8;
u64 markform : 7;
u64 mark_en : 1;
u64 subdc : 4;
u64 vlan0_ins_ptr : 8; /* W1 */
u64 vlan0_ins_tci : 16;
u64 vlan1_ins_ptr : 8;
u64 vlan1_ins_tci : 16;
u64 vlan0_ins_ena : 1;
u64 vlan1_ins_ena : 1;
u64 rsvd_127_114 : 14;
};
struct nix_sqe_sg_s {
u64 seg1_size : 16;
u64 seg2_size : 16;
u64 seg3_size : 16;
u64 segs : 2;
u64 rsvd_54_50 : 5;
u64 i1 : 1;
u64 i2 : 1;
u64 i3 : 1;
u64 ld_type : 2;
u64 subdc : 4;
};
/* NIX send memory subdescriptor structure */
struct nix_sqe_mem_s {
u64 offset : 16; /* W0 */
u64 rsvd_52_16 : 37;
u64 wmem : 1;
u64 dsz : 2;
u64 alg : 4;
u64 subdc : 4;
u64 addr; /* W1 */
};
enum nix_cqerrint_e {
NIX_CQERRINT_DOOR_ERR = 0,
NIX_CQERRINT_WR_FULL = 1,
NIX_CQERRINT_CQE_FAULT = 2,
};
#define NIX_CQERRINT_BITS (BIT_ULL(NIX_CQERRINT_DOOR_ERR) | \
BIT_ULL(NIX_CQERRINT_CQE_FAULT))
enum nix_rqint_e {
NIX_RQINT_DROP = 0,
NIX_RQINT_RED = 1,
};
#define NIX_RQINT_BITS (BIT_ULL(NIX_RQINT_DROP) | BIT_ULL(NIX_RQINT_RED))
enum nix_sqint_e {
NIX_SQINT_LMT_ERR = 0,
NIX_SQINT_MNQ_ERR = 1,
NIX_SQINT_SEND_ERR = 2,
NIX_SQINT_SQB_ALLOC_FAIL = 3,
};
#define NIX_SQINT_BITS (BIT_ULL(NIX_SQINT_LMT_ERR) | \
BIT_ULL(NIX_SQINT_MNQ_ERR) | \
BIT_ULL(NIX_SQINT_SEND_ERR) | \
BIT_ULL(NIX_SQINT_SQB_ALLOC_FAIL))
#endif /* OTX2_STRUCT_H */

848
drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.c Normal file
View File

@ -0,0 +1,848 @@
// SPDX-License-Identifier: GPL-2.0
/* Marvell OcteonTx2 RVU Ethernet driver
*
* Copyright (C) 2020 Marvell International Ltd.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*/
#include <linux/etherdevice.h>
#include <net/ip.h>
#include <net/tso.h>
#include "otx2_reg.h"
#include "otx2_common.h"
#include "otx2_struct.h"
#include "otx2_txrx.h"
#define CQE_ADDR(CQ, idx) ((CQ)->cqe_base + ((CQ)->cqe_size * (idx)))
static struct nix_cqe_hdr_s *otx2_get_next_cqe(struct otx2_cq_queue *cq)
{
struct nix_cqe_hdr_s *cqe_hdr;
cqe_hdr = (struct nix_cqe_hdr_s *)CQE_ADDR(cq, cq->cq_head);
if (cqe_hdr->cqe_type == NIX_XQE_TYPE_INVALID)
return NULL;
cq->cq_head++;
cq->cq_head &= (cq->cqe_cnt - 1);
return cqe_hdr;
}
static unsigned int frag_num(unsigned int i)
{
#ifdef __BIG_ENDIAN
return (i & ~3) + 3 - (i & 3);
#else
return i;
#endif
}
static dma_addr_t otx2_dma_map_skb_frag(struct otx2_nic *pfvf,
struct sk_buff *skb, int seg, int *len)
{
const skb_frag_t *frag;
struct page *page;
int offset;
/* First segment is always skb->data */
if (!seg) {
page = virt_to_page(skb->data);
offset = offset_in_page(skb->data);
*len = skb_headlen(skb);
} else {
frag = &skb_shinfo(skb)->frags[seg - 1];
page = skb_frag_page(frag);
offset = skb_frag_off(frag);
*len = skb_frag_size(frag);
}
return otx2_dma_map_page(pfvf, page, offset, *len, DMA_TO_DEVICE);
}
static void otx2_dma_unmap_skb_frags(struct otx2_nic *pfvf, struct sg_list *sg)
{
int seg;
for (seg = 0; seg < sg->num_segs; seg++) {
otx2_dma_unmap_page(pfvf, sg->dma_addr[seg],
sg->size[seg], DMA_TO_DEVICE);
}
sg->num_segs = 0;
}
static void otx2_snd_pkt_handler(struct otx2_nic *pfvf,
struct otx2_cq_queue *cq,
struct otx2_snd_queue *sq,
struct nix_cqe_tx_s *cqe,
int budget, int *tx_pkts, int *tx_bytes)
{
struct nix_send_comp_s *snd_comp = &cqe->comp;
struct sk_buff *skb = NULL;
struct sg_list *sg;
if (unlikely(snd_comp->status) && netif_msg_tx_err(pfvf))
net_err_ratelimited("%s: TX%d: Error in send CQ status:%x\n",
pfvf->netdev->name, cq->cint_idx,
snd_comp->status);
sg = &sq->sg[snd_comp->sqe_id];
skb = (struct sk_buff *)sg->skb;
if (unlikely(!skb))
return;
*tx_bytes += skb->len;
(*tx_pkts)++;
otx2_dma_unmap_skb_frags(pfvf, sg);
napi_consume_skb(skb, budget);
sg->skb = (u64)NULL;
}
static void otx2_skb_add_frag(struct otx2_nic *pfvf, struct sk_buff *skb,
u64 iova, int len)
{
struct page *page;
void *va;
va = phys_to_virt(otx2_iova_to_phys(pfvf->iommu_domain, iova));
page = virt_to_page(va);
skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, page,
va - page_address(page), len, pfvf->rbsize);
otx2_dma_unmap_page(pfvf, iova - OTX2_HEAD_ROOM,
pfvf->rbsize, DMA_FROM_DEVICE);
}
static void otx2_set_rxhash(struct otx2_nic *pfvf,
struct nix_cqe_rx_s *cqe, struct sk_buff *skb)
{
enum pkt_hash_types hash_type = PKT_HASH_TYPE_NONE;
struct otx2_rss_info *rss;
u32 hash = 0;
if (!(pfvf->netdev->features & NETIF_F_RXHASH))
return;
rss = &pfvf->hw.rss_info;
if (rss->flowkey_cfg) {
if (rss->flowkey_cfg &
~(NIX_FLOW_KEY_TYPE_IPV4 | NIX_FLOW_KEY_TYPE_IPV6))
hash_type = PKT_HASH_TYPE_L4;
else
hash_type = PKT_HASH_TYPE_L3;
hash = cqe->hdr.flow_tag;
}
skb_set_hash(skb, hash, hash_type);
}
static bool otx2_check_rcv_errors(struct otx2_nic *pfvf,
struct nix_cqe_rx_s *cqe, int qidx)
{
struct otx2_drv_stats *stats = &pfvf->hw.drv_stats;
struct nix_rx_parse_s *parse = &cqe->parse;
if (netif_msg_rx_err(pfvf))
netdev_err(pfvf->netdev,
"RQ%d: Error pkt with errlev:0x%x errcode:0x%x\n",
qidx, parse->errlev, parse->errcode);
if (parse->errlev == NPC_ERRLVL_RE) {
switch (parse->errcode) {
case ERRCODE_FCS:
case ERRCODE_FCS_RCV:
atomic_inc(&stats->rx_fcs_errs);
break;
case ERRCODE_UNDERSIZE:
atomic_inc(&stats->rx_undersize_errs);
break;
case ERRCODE_OVERSIZE:
atomic_inc(&stats->rx_oversize_errs);
break;
case ERRCODE_OL2_LEN_MISMATCH:
atomic_inc(&stats->rx_len_errs);
break;
default:
atomic_inc(&stats->rx_other_errs);
break;
}
} else if (parse->errlev == NPC_ERRLVL_NIX) {
switch (parse->errcode) {
case ERRCODE_OL3_LEN:
case ERRCODE_OL4_LEN:
case ERRCODE_IL3_LEN:
case ERRCODE_IL4_LEN:
atomic_inc(&stats->rx_len_errs);
break;
case ERRCODE_OL4_CSUM:
case ERRCODE_IL4_CSUM:
atomic_inc(&stats->rx_csum_errs);
break;
default:
atomic_inc(&stats->rx_other_errs);
break;
}
} else {
atomic_inc(&stats->rx_other_errs);
/* For now ignore all the NPC parser errors and
* pass the packets to stack.
*/
return false;
}
/* If RXALL is enabled pass on packets to stack. */
if (cqe->sg.segs && (pfvf->netdev->features & NETIF_F_RXALL))
return false;
/* Free buffer back to pool */
if (cqe->sg.segs)
otx2_aura_freeptr(pfvf, qidx, cqe->sg.seg_addr & ~0x07ULL);
return true;
}
static void otx2_rcv_pkt_handler(struct otx2_nic *pfvf,
struct napi_struct *napi,
struct otx2_cq_queue *cq,
struct nix_cqe_rx_s *cqe)
{
struct nix_rx_parse_s *parse = &cqe->parse;
struct sk_buff *skb = NULL;
if (unlikely(parse->errlev || parse->errcode)) {
if (otx2_check_rcv_errors(pfvf, cqe, cq->cq_idx))
return;
}
skb = napi_get_frags(napi);
if (unlikely(!skb))
return;
otx2_skb_add_frag(pfvf, skb, cqe->sg.seg_addr, cqe->sg.seg_size);
cq->pool_ptrs++;
otx2_set_rxhash(pfvf, cqe, skb);
skb_record_rx_queue(skb, cq->cq_idx);
if (pfvf->netdev->features & NETIF_F_RXCSUM)
skb->ip_summed = CHECKSUM_UNNECESSARY;
napi_gro_frags(napi);
}
static int otx2_rx_napi_handler(struct otx2_nic *pfvf,
struct napi_struct *napi,
struct otx2_cq_queue *cq, int budget)
{
struct nix_cqe_rx_s *cqe;
int processed_cqe = 0;
s64 bufptr;
while (likely(processed_cqe < budget)) {
cqe = (struct nix_cqe_rx_s *)CQE_ADDR(cq, cq->cq_head);
if (cqe->hdr.cqe_type == NIX_XQE_TYPE_INVALID ||
!cqe->sg.seg_addr) {
if (!processed_cqe)
return 0;
break;
}
cq->cq_head++;
cq->cq_head &= (cq->cqe_cnt - 1);
otx2_rcv_pkt_handler(pfvf, napi, cq, cqe);
cqe->hdr.cqe_type = NIX_XQE_TYPE_INVALID;
cqe->sg.seg_addr = 0x00;
processed_cqe++;
}
/* Free CQEs to HW */
otx2_write64(pfvf, NIX_LF_CQ_OP_DOOR,
((u64)cq->cq_idx << 32) | processed_cqe);
if (unlikely(!cq->pool_ptrs))
return 0;
/* Refill pool with new buffers */
while (cq->pool_ptrs) {
bufptr = otx2_alloc_rbuf(pfvf, cq->rbpool, GFP_ATOMIC);
if (unlikely(bufptr <= 0)) {
struct refill_work *work;
struct delayed_work *dwork;
work = &pfvf->refill_wrk[cq->cq_idx];
dwork = &work->pool_refill_work;
/* Schedule a task if no other task is running */
if (!cq->refill_task_sched) {
cq->refill_task_sched = true;
schedule_delayed_work(dwork,
msecs_to_jiffies(100));
}
break;
}
otx2_aura_freeptr(pfvf, cq->cq_idx, bufptr + OTX2_HEAD_ROOM);
cq->pool_ptrs--;
}
return processed_cqe;
}
static int otx2_tx_napi_handler(struct otx2_nic *pfvf,
struct otx2_cq_queue *cq, int budget)
{
int tx_pkts = 0, tx_bytes = 0;
struct nix_cqe_tx_s *cqe;
int processed_cqe = 0;
while (likely(processed_cqe < budget)) {
cqe = (struct nix_cqe_tx_s *)otx2_get_next_cqe(cq);
if (unlikely(!cqe)) {
if (!processed_cqe)
return 0;
break;
}
otx2_snd_pkt_handler(pfvf, cq, &pfvf->qset.sq[cq->cint_idx],
cqe, budget, &tx_pkts, &tx_bytes);
cqe->hdr.cqe_type = NIX_XQE_TYPE_INVALID;
processed_cqe++;
}
/* Free CQEs to HW */
otx2_write64(pfvf, NIX_LF_CQ_OP_DOOR,
((u64)cq->cq_idx << 32) | processed_cqe);
if (likely(tx_pkts)) {
struct netdev_queue *txq;
txq = netdev_get_tx_queue(pfvf->netdev, cq->cint_idx);
netdev_tx_completed_queue(txq, tx_pkts, tx_bytes);
/* Check if queue was stopped earlier due to ring full */
smp_mb();
if (netif_tx_queue_stopped(txq) &&
netif_carrier_ok(pfvf->netdev))
netif_tx_wake_queue(txq);
}
return 0;
}
int otx2_napi_handler(struct napi_struct *napi, int budget)
{
struct otx2_cq_poll *cq_poll;
int workdone = 0, cq_idx, i;
struct otx2_cq_queue *cq;
struct otx2_qset *qset;
struct otx2_nic *pfvf;
cq_poll = container_of(napi, struct otx2_cq_poll, napi);
pfvf = (struct otx2_nic *)cq_poll->dev;
qset = &pfvf->qset;
for (i = CQS_PER_CINT - 1; i >= 0; i--) {
cq_idx = cq_poll->cq_ids[i];
if (unlikely(cq_idx == CINT_INVALID_CQ))
continue;
cq = &qset->cq[cq_idx];
if (cq->cq_type == CQ_RX) {
/* If the RQ refill WQ task is running, skip napi
* scheduler for this queue.
*/
if (cq->refill_task_sched)
continue;
workdone += otx2_rx_napi_handler(pfvf, napi,
cq, budget);
} else {
workdone += otx2_tx_napi_handler(pfvf, cq, budget);
}
}
/* Clear the IRQ */
otx2_write64(pfvf, NIX_LF_CINTX_INT(cq_poll->cint_idx), BIT_ULL(0));
if (workdone < budget && napi_complete_done(napi, workdone)) {
/* If interface is going down, don't re-enable IRQ */
if (pfvf->flags & OTX2_FLAG_INTF_DOWN)
return workdone;
/* Re-enable interrupts */
otx2_write64(pfvf, NIX_LF_CINTX_ENA_W1S(cq_poll->cint_idx),
BIT_ULL(0));
}
return workdone;
}
static void otx2_sqe_flush(struct otx2_snd_queue *sq, int size)
{
u64 status;
/* Packet data stores should finish before SQE is flushed to HW */
dma_wmb();
do {
memcpy(sq->lmt_addr, sq->sqe_base, size);
status = otx2_lmt_flush(sq->io_addr);
} while (status == 0);
sq->head++;
sq->head &= (sq->sqe_cnt - 1);
}
#define MAX_SEGS_PER_SG 3
/* Add SQE scatter/gather subdescriptor structure */
static bool otx2_sqe_add_sg(struct otx2_nic *pfvf, struct otx2_snd_queue *sq,
struct sk_buff *skb, int num_segs, int *offset)
{
struct nix_sqe_sg_s *sg = NULL;
u64 dma_addr, *iova = NULL;
u16 *sg_lens = NULL;
int seg, len;
sq->sg[sq->head].num_segs = 0;
for (seg = 0; seg < num_segs; seg++) {
if ((seg % MAX_SEGS_PER_SG) == 0) {
sg = (struct nix_sqe_sg_s *)(sq->sqe_base + *offset);
sg->ld_type = NIX_SEND_LDTYPE_LDD;
sg->subdc = NIX_SUBDC_SG;
sg->segs = 0;
sg_lens = (void *)sg;
iova = (void *)sg + sizeof(*sg);
/* Next subdc always starts at a 16byte boundary.
* So if sg->segs is whether 2 or 3, offset += 16bytes.
*/
if ((num_segs - seg) >= (MAX_SEGS_PER_SG - 1))
*offset += sizeof(*sg) + (3 * sizeof(u64));
else
*offset += sizeof(*sg) + sizeof(u64);
}
dma_addr = otx2_dma_map_skb_frag(pfvf, skb, seg, &len);
if (dma_mapping_error(pfvf->dev, dma_addr))
return false;
sg_lens[frag_num(seg % MAX_SEGS_PER_SG)] = len;
sg->segs++;
*iova++ = dma_addr;
/* Save DMA mapping info for later unmapping */
sq->sg[sq->head].dma_addr[seg] = dma_addr;
sq->sg[sq->head].size[seg] = len;
sq->sg[sq->head].num_segs++;
}
sq->sg[sq->head].skb = (u64)skb;
return true;
}
/* Add SQE extended header subdescriptor */
static void otx2_sqe_add_ext(struct otx2_nic *pfvf, struct otx2_snd_queue *sq,
struct sk_buff *skb, int *offset)
{
struct nix_sqe_ext_s *ext;
ext = (struct nix_sqe_ext_s *)(sq->sqe_base + *offset);
ext->subdc = NIX_SUBDC_EXT;
if (skb_shinfo(skb)->gso_size) {
ext->lso = 1;
ext->lso_sb = skb_transport_offset(skb) + tcp_hdrlen(skb);
ext->lso_mps = skb_shinfo(skb)->gso_size;
/* Only TSOv4 and TSOv6 GSO offloads are supported */
if (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4) {
ext->lso_format = pfvf->hw.lso_tsov4_idx;
/* HW adds payload size to 'ip_hdr->tot_len' while
* sending TSO segment, hence set payload length
* in IP header of the packet to just header length.
*/
ip_hdr(skb)->tot_len =
htons(ext->lso_sb - skb_network_offset(skb));
} else {
ext->lso_format = pfvf->hw.lso_tsov6_idx;
ipv6_hdr(skb)->payload_len =
htons(ext->lso_sb - skb_network_offset(skb));
}
}
*offset += sizeof(*ext);
}
/* Add SQE header subdescriptor structure */
static void otx2_sqe_add_hdr(struct otx2_nic *pfvf, struct otx2_snd_queue *sq,
struct nix_sqe_hdr_s *sqe_hdr,
struct sk_buff *skb, u16 qidx)
{
int proto = 0;
/* Check if SQE was framed before, if yes then no need to
* set these constants again and again.
*/
if (!sqe_hdr->total) {
/* Don't free Tx buffers to Aura */
sqe_hdr->df = 1;
sqe_hdr->aura = sq->aura_id;
/* Post a CQE Tx after pkt transmission */
sqe_hdr->pnc = 1;
sqe_hdr->sq = qidx;
}
sqe_hdr->total = skb->len;
/* Set SQE identifier which will be used later for freeing SKB */
sqe_hdr->sqe_id = sq->head;
/* Offload TCP/UDP checksum to HW */
if (skb->ip_summed == CHECKSUM_PARTIAL) {
sqe_hdr->ol3ptr = skb_network_offset(skb);
sqe_hdr->ol4ptr = skb_transport_offset(skb);
/* get vlan protocol Ethertype */
if (eth_type_vlan(skb->protocol))
skb->protocol = vlan_get_protocol(skb);
if (skb->protocol == htons(ETH_P_IP)) {
proto = ip_hdr(skb)->protocol;
/* In case of TSO, HW needs this to be explicitly set.
* So set this always, instead of adding a check.
*/
sqe_hdr->ol3type = NIX_SENDL3TYPE_IP4_CKSUM;
} else if (skb->protocol == htons(ETH_P_IPV6)) {
proto = ipv6_hdr(skb)->nexthdr;
}
if (proto == IPPROTO_TCP)
sqe_hdr->ol4type = NIX_SENDL4TYPE_TCP_CKSUM;
else if (proto == IPPROTO_UDP)
sqe_hdr->ol4type = NIX_SENDL4TYPE_UDP_CKSUM;
}
}
static int otx2_dma_map_tso_skb(struct otx2_nic *pfvf,
struct otx2_snd_queue *sq,
struct sk_buff *skb, int sqe, int hdr_len)
{
int num_segs = skb_shinfo(skb)->nr_frags + 1;
struct sg_list *sg = &sq->sg[sqe];
u64 dma_addr;
int seg, len;
sg->num_segs = 0;
/* Get payload length at skb->data */
len = skb_headlen(skb) - hdr_len;
for (seg = 0; seg < num_segs; seg++) {
/* Skip skb->data, if there is no payload */
if (!seg && !len)
continue;
dma_addr = otx2_dma_map_skb_frag(pfvf, skb, seg, &len);
if (dma_mapping_error(pfvf->dev, dma_addr))
goto unmap;
/* Save DMA mapping info for later unmapping */
sg->dma_addr[sg->num_segs] = dma_addr;
sg->size[sg->num_segs] = len;
sg->num_segs++;
}
return 0;
unmap:
otx2_dma_unmap_skb_frags(pfvf, sg);
return -EINVAL;
}
static u64 otx2_tso_frag_dma_addr(struct otx2_snd_queue *sq,
struct sk_buff *skb, int seg,
u64 seg_addr, int hdr_len, int sqe)
{
struct sg_list *sg = &sq->sg[sqe];
const skb_frag_t *frag;
int offset;
if (seg < 0)
return sg->dma_addr[0] + (seg_addr - (u64)skb->data);
frag = &skb_shinfo(skb)->frags[seg];
offset = seg_addr - (u64)skb_frag_address(frag);
if (skb_headlen(skb) - hdr_len)
seg++;
return sg->dma_addr[seg] + offset;
}
static void otx2_sqe_tso_add_sg(struct otx2_snd_queue *sq,
struct sg_list *list, int *offset)
{
struct nix_sqe_sg_s *sg = NULL;
u16 *sg_lens = NULL;
u64 *iova = NULL;
int seg;
/* Add SG descriptors with buffer addresses */
for (seg = 0; seg < list->num_segs; seg++) {
if ((seg % MAX_SEGS_PER_SG) == 0) {
sg = (struct nix_sqe_sg_s *)(sq->sqe_base + *offset);
sg->ld_type = NIX_SEND_LDTYPE_LDD;
sg->subdc = NIX_SUBDC_SG;
sg->segs = 0;
sg_lens = (void *)sg;
iova = (void *)sg + sizeof(*sg);
/* Next subdc always starts at a 16byte boundary.
* So if sg->segs is whether 2 or 3, offset += 16bytes.
*/
if ((list->num_segs - seg) >= (MAX_SEGS_PER_SG - 1))
*offset += sizeof(*sg) + (3 * sizeof(u64));
else
*offset += sizeof(*sg) + sizeof(u64);
}
sg_lens[frag_num(seg % MAX_SEGS_PER_SG)] = list->size[seg];
*iova++ = list->dma_addr[seg];
sg->segs++;
}
}
static void otx2_sq_append_tso(struct otx2_nic *pfvf, struct otx2_snd_queue *sq,
struct sk_buff *skb, u16 qidx)
{
struct netdev_queue *txq = netdev_get_tx_queue(pfvf->netdev, qidx);
int hdr_len = skb_transport_offset(skb) + tcp_hdrlen(skb);
int tcp_data, seg_len, pkt_len, offset;
struct nix_sqe_hdr_s *sqe_hdr;
int first_sqe = sq->head;
struct sg_list list;
struct tso_t tso;
/* Map SKB's fragments to DMA.
* It's done here to avoid mapping for every TSO segment's packet.
*/
if (otx2_dma_map_tso_skb(pfvf, sq, skb, first_sqe, hdr_len)) {
dev_kfree_skb_any(skb);
return;
}
netdev_tx_sent_queue(txq, skb->len);
tso_start(skb, &tso);
tcp_data = skb->len - hdr_len;
while (tcp_data > 0) {
char *hdr;
seg_len = min_t(int, skb_shinfo(skb)->gso_size, tcp_data);
tcp_data -= seg_len;
/* Set SQE's SEND_HDR */
memset(sq->sqe_base, 0, sq->sqe_size);
sqe_hdr = (struct nix_sqe_hdr_s *)(sq->sqe_base);
otx2_sqe_add_hdr(pfvf, sq, sqe_hdr, skb, qidx);
offset = sizeof(*sqe_hdr);
/* Add TSO segment's pkt header */
hdr = sq->tso_hdrs->base + (sq->head * TSO_HEADER_SIZE);
tso_build_hdr(skb, hdr, &tso, seg_len, tcp_data == 0);
list.dma_addr[0] =
sq->tso_hdrs->iova + (sq->head * TSO_HEADER_SIZE);
list.size[0] = hdr_len;
list.num_segs = 1;
/* Add TSO segment's payload data fragments */
pkt_len = hdr_len;
while (seg_len > 0) {
int size;
size = min_t(int, tso.size, seg_len);
list.size[list.num_segs] = size;
list.dma_addr[list.num_segs] =
otx2_tso_frag_dma_addr(sq, skb,
tso.next_frag_idx - 1,
(u64)tso.data, hdr_len,
first_sqe);
list.num_segs++;
pkt_len += size;
seg_len -= size;
tso_build_data(skb, &tso, size);
}
sqe_hdr->total = pkt_len;
otx2_sqe_tso_add_sg(sq, &list, &offset);
/* DMA mappings and skb needs to be freed only after last
* TSO segment is transmitted out. So set 'PNC' only for
* last segment. Also point last segment's sqe_id to first
* segment's SQE index where skb address and DMA mappings
* are saved.
*/
if (!tcp_data) {
sqe_hdr->pnc = 1;
sqe_hdr->sqe_id = first_sqe;
sq->sg[first_sqe].skb = (u64)skb;
} else {
sqe_hdr->pnc = 0;
}
sqe_hdr->sizem1 = (offset / 16) - 1;
/* Flush SQE to HW */
otx2_sqe_flush(sq, offset);
}
}
static bool is_hw_tso_supported(struct otx2_nic *pfvf,
struct sk_buff *skb)
{
int payload_len, last_seg_size;
if (!pfvf->hw.hw_tso)
return false;
/* HW has an issue due to which when the payload of the last LSO
* segment is shorter than 16 bytes, some header fields may not
* be correctly modified, hence don't offload such TSO segments.
*/
if (!is_96xx_B0(pfvf->pdev))
return true;
payload_len = skb->len - (skb_transport_offset(skb) + tcp_hdrlen(skb));
last_seg_size = payload_len % skb_shinfo(skb)->gso_size;
if (last_seg_size && last_seg_size < 16)
return false;
return true;
}
static int otx2_get_sqe_count(struct otx2_nic *pfvf, struct sk_buff *skb)
{
if (!skb_shinfo(skb)->gso_size)
return 1;
/* HW TSO */
if (is_hw_tso_supported(pfvf, skb))
return 1;
/* SW TSO */
return skb_shinfo(skb)->gso_segs;
}
bool otx2_sq_append_skb(struct net_device *netdev, struct otx2_snd_queue *sq,
struct sk_buff *skb, u16 qidx)
{
struct netdev_queue *txq = netdev_get_tx_queue(netdev, qidx);
struct otx2_nic *pfvf = netdev_priv(netdev);
int offset, num_segs, free_sqe;
struct nix_sqe_hdr_s *sqe_hdr;
/* Check if there is room for new SQE.
* 'Num of SQBs freed to SQ's pool - SQ's Aura count'
* will give free SQE count.
*/
free_sqe = (sq->num_sqbs - *sq->aura_fc_addr) * sq->sqe_per_sqb;
if (free_sqe < sq->sqe_thresh ||
free_sqe < otx2_get_sqe_count(pfvf, skb))
return false;
num_segs = skb_shinfo(skb)->nr_frags + 1;
/* If SKB doesn't fit in a single SQE, linearize it.
* TODO: Consider adding JUMP descriptor instead.
*/
if (unlikely(num_segs > OTX2_MAX_FRAGS_IN_SQE)) {
if (__skb_linearize(skb)) {
dev_kfree_skb_any(skb);
return true;
}
num_segs = skb_shinfo(skb)->nr_frags + 1;
}
if (skb_shinfo(skb)->gso_size && !is_hw_tso_supported(pfvf, skb)) {
otx2_sq_append_tso(pfvf, sq, skb, qidx);
return true;
}
/* Set SQE's SEND_HDR.
* Do not clear the first 64bit as it contains constant info.
*/
memset(sq->sqe_base + 8, 0, sq->sqe_size - 8);
sqe_hdr = (struct nix_sqe_hdr_s *)(sq->sqe_base);
otx2_sqe_add_hdr(pfvf, sq, sqe_hdr, skb, qidx);
offset = sizeof(*sqe_hdr);
/* Add extended header if needed */
otx2_sqe_add_ext(pfvf, sq, skb, &offset);
/* Add SG subdesc with data frags */
if (!otx2_sqe_add_sg(pfvf, sq, skb, num_segs, &offset)) {
otx2_dma_unmap_skb_frags(pfvf, &sq->sg[sq->head]);
return false;
}
sqe_hdr->sizem1 = (offset / 16) - 1;
netdev_tx_sent_queue(txq, skb->len);
/* Flush SQE to HW */
otx2_sqe_flush(sq, offset);
return true;
}
void otx2_cleanup_rx_cqes(struct otx2_nic *pfvf, struct otx2_cq_queue *cq)
{
struct nix_cqe_rx_s *cqe;
int processed_cqe = 0;
u64 iova, pa;
while ((cqe = (struct nix_cqe_rx_s *)otx2_get_next_cqe(cq))) {
if (!cqe->sg.subdc)
continue;
iova = cqe->sg.seg_addr - OTX2_HEAD_ROOM;
pa = otx2_iova_to_phys(pfvf->iommu_domain, iova);
otx2_dma_unmap_page(pfvf, iova, pfvf->rbsize, DMA_FROM_DEVICE);
put_page(virt_to_page(phys_to_virt(pa)));
processed_cqe++;
}
/* Free CQEs to HW */
otx2_write64(pfvf, NIX_LF_CQ_OP_DOOR,
((u64)cq->cq_idx << 32) | processed_cqe);
}
void otx2_cleanup_tx_cqes(struct otx2_nic *pfvf, struct otx2_cq_queue *cq)
{
struct sk_buff *skb = NULL;
struct otx2_snd_queue *sq;
struct nix_cqe_tx_s *cqe;
int processed_cqe = 0;
struct sg_list *sg;
sq = &pfvf->qset.sq[cq->cint_idx];
while ((cqe = (struct nix_cqe_tx_s *)otx2_get_next_cqe(cq))) {
sg = &sq->sg[cqe->comp.sqe_id];
skb = (struct sk_buff *)sg->skb;
if (skb) {
otx2_dma_unmap_skb_frags(pfvf, sg);
dev_kfree_skb_any(skb);
sg->skb = (u64)NULL;
}
processed_cqe++;
}
/* Free CQEs to HW */
otx2_write64(pfvf, NIX_LF_CQ_OP_DOOR,
((u64)cq->cq_idx << 32) | processed_cqe);
}
int otx2_rxtx_enable(struct otx2_nic *pfvf, bool enable)
{
struct msg_req *msg;
int err;
otx2_mbox_lock(&pfvf->mbox);
if (enable)
msg = otx2_mbox_alloc_msg_nix_lf_start_rx(&pfvf->mbox);
else
msg = otx2_mbox_alloc_msg_nix_lf_stop_rx(&pfvf->mbox);
if (!msg) {
otx2_mbox_unlock(&pfvf->mbox);
return -ENOMEM;
}
err = otx2_sync_mbox_msg(&pfvf->mbox);
otx2_mbox_unlock(&pfvf->mbox);
return err;
}

162
drivers/net/ethernet/marvell/octeontx2/nic/otx2_txrx.h Normal file
View File

@ -0,0 +1,162 @@
/* SPDX-License-Identifier: GPL-2.0 */
/* Marvell OcteonTx2 RVU Ethernet driver
*
* Copyright (C) 2020 Marvell International Ltd.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*/
#ifndef OTX2_TXRX_H
#define OTX2_TXRX_H
#include <linux/etherdevice.h>
#include <linux/iommu.h>
#include <linux/if_vlan.h>
#define LBK_CHAN_BASE 0x000
#define SDP_CHAN_BASE 0x700
#define CGX_CHAN_BASE 0x800
#define OTX2_DATA_ALIGN(X) ALIGN(X, OTX2_ALIGN)
#define OTX2_HEAD_ROOM OTX2_ALIGN
#define OTX2_ETH_HLEN (VLAN_ETH_HLEN + VLAN_HLEN)
#define OTX2_MIN_MTU 64
#define OTX2_MAX_MTU (9212 - OTX2_ETH_HLEN)
#define OTX2_MAX_GSO_SEGS 255
#define OTX2_MAX_FRAGS_IN_SQE 9
/* Rx buffer size should be in multiples of 128bytes */
#define RCV_FRAG_LEN1(x) \
((OTX2_HEAD_ROOM + OTX2_DATA_ALIGN(x)) + \
OTX2_DATA_ALIGN(sizeof(struct skb_shared_info)))
/* Prefer 2048 byte buffers for better last level cache
* utilization or data distribution across regions.
*/
#define RCV_FRAG_LEN(x) \
((RCV_FRAG_LEN1(x) < 2048) ? 2048 : RCV_FRAG_LEN1(x))
#define DMA_BUFFER_LEN(x) \
((x) - OTX2_HEAD_ROOM - \
OTX2_DATA_ALIGN(sizeof(struct skb_shared_info)))
/* IRQ triggered when NIX_LF_CINTX_CNT[ECOUNT]
* is equal to this value.
*/
#define CQ_CQE_THRESH_DEFAULT 10
/* IRQ triggered when NIX_LF_CINTX_CNT[ECOUNT]
* is nonzero and this much time elapses after that.
*/
#define CQ_TIMER_THRESH_DEFAULT 1 /* 1 usec */
#define CQ_TIMER_THRESH_MAX 25 /* 25 usec */
/* Min number of CQs (of the ones mapped to this CINT)
* with valid CQEs.
*/
#define CQ_QCOUNT_DEFAULT 1
struct queue_stats {
u64 bytes;
u64 pkts;
};
struct otx2_rcv_queue {
struct queue_stats stats;
};
struct sg_list {
u16 num_segs;
u64 skb;
u64 size[OTX2_MAX_FRAGS_IN_SQE];
u64 dma_addr[OTX2_MAX_FRAGS_IN_SQE];
};
struct otx2_snd_queue {
u8 aura_id;
u16 head;
u16 sqe_size;
u32 sqe_cnt;
u16 num_sqbs;
u16 sqe_thresh;
u8 sqe_per_sqb;
u64 io_addr;
u64 *aura_fc_addr;
u64 *lmt_addr;
void *sqe_base;
struct qmem *sqe;
struct qmem *tso_hdrs;
struct sg_list *sg;
struct queue_stats stats;
u16 sqb_count;
u64 *sqb_ptrs;
} ____cacheline_aligned_in_smp;
enum cq_type {
CQ_RX,
CQ_TX,
CQS_PER_CINT = 2, /* RQ + SQ */
};
struct otx2_cq_poll {
void *dev;
#define CINT_INVALID_CQ 255
u8 cint_idx;
u8 cq_ids[CQS_PER_CINT];
struct napi_struct napi;
};
struct otx2_pool {
struct qmem *stack;
struct qmem *fc_addr;
u8 rbpage_order;
u16 rbsize;
u32 page_offset;
u16 pageref;
struct page *page;
};
struct otx2_cq_queue {
u8 cq_idx;
u8 cq_type;
u8 cint_idx; /* CQ interrupt id */
u8 refill_task_sched;
u16 cqe_size;
u16 pool_ptrs;
u32 cqe_cnt;
u32 cq_head;
void *cqe_base;
struct qmem *cqe;
struct otx2_pool *rbpool;
} ____cacheline_aligned_in_smp;
struct otx2_qset {
u32 rqe_cnt;
u32 sqe_cnt; /* Keep these two at top */
#define OTX2_MAX_CQ_CNT 64
u16 cq_cnt;
u16 xqe_size;
struct otx2_pool *pool;
struct otx2_cq_poll *napi;
struct otx2_cq_queue *cq;
struct otx2_snd_queue *sq;
struct otx2_rcv_queue *rq;
};
/* Translate IOVA to physical address */
static inline u64 otx2_iova_to_phys(void *iommu_domain, dma_addr_t dma_addr)
{
/* Translation is installed only when IOMMU is present */
if (likely(iommu_domain))
return iommu_iova_to_phys(iommu_domain, dma_addr);
return dma_addr;
}
int otx2_napi_handler(struct napi_struct *napi, int budget);
bool otx2_sq_append_skb(struct net_device *netdev, struct otx2_snd_queue *sq,
struct sk_buff *skb, u16 qidx);
#endif /* OTX2_TXRX_H */