2008-10-23 06:47:49 +08:00
|
|
|
/*
|
|
|
|
* Copyright (c) 2007 Mellanox Technologies. All rights reserved.
|
|
|
|
*
|
|
|
|
* This software is available to you under a choice of one of two
|
|
|
|
* licenses. You may choose to be licensed under the terms of the GNU
|
|
|
|
* General Public License (GPL) Version 2, available from the file
|
|
|
|
* COPYING in the main directory of this source tree, or the
|
|
|
|
* OpenIB.org BSD license below:
|
|
|
|
*
|
|
|
|
* Redistribution and use in source and binary forms, with or
|
|
|
|
* without modification, are permitted provided that the following
|
|
|
|
* conditions are met:
|
|
|
|
*
|
|
|
|
* - Redistributions of source code must retain the above
|
|
|
|
* copyright notice, this list of conditions and the following
|
|
|
|
* disclaimer.
|
|
|
|
*
|
|
|
|
* - Redistributions in binary form must reproduce the above
|
|
|
|
* copyright notice, this list of conditions and the following
|
|
|
|
* disclaimer in the documentation and/or other materials
|
|
|
|
* provided with the distribution.
|
|
|
|
*
|
|
|
|
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
|
|
|
* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
|
|
|
* MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
|
|
|
* NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
|
|
|
|
* BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
|
|
|
|
* ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
|
|
|
|
* CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
|
|
* SOFTWARE.
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include <asm/page.h>
|
|
|
|
#include <linux/mlx4/cq.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 16:04:11 +08:00
|
|
|
#include <linux/slab.h>
|
2008-10-23 06:47:49 +08:00
|
|
|
#include <linux/mlx4/qp.h>
|
|
|
|
#include <linux/skbuff.h>
|
|
|
|
#include <linux/if_vlan.h>
|
2014-10-05 17:35:14 +08:00
|
|
|
#include <linux/prefetch.h>
|
2008-10-23 06:47:49 +08:00
|
|
|
#include <linux/vmalloc.h>
|
2010-08-24 11:46:46 +08:00
|
|
|
#include <linux/tcp.h>
|
2013-12-23 22:09:44 +08:00
|
|
|
#include <linux/ip.h>
|
2011-09-16 07:46:05 +08:00
|
|
|
#include <linux/moduleparam.h>
|
2008-10-23 06:47:49 +08:00
|
|
|
|
|
|
|
#include "mlx4_en.h"
|
|
|
|
|
|
|
|
int mlx4_en_create_tx_ring(struct mlx4_en_priv *priv,
|
net/mlx4: Change QP allocation scheme
When using BF (Blue-Flame), the QPN overrides the VLAN, CV, and SV fields
in the WQE. Thus, BF may only be used for QPNs with bits 6,7 unset.
The current Ethernet driver code reserves a Tx QP range with 256b alignment.
This is wrong because if there are more than 64 Tx QPs in use,
QPNs >= base + 65 will have bits 6/7 set.
This problem is not specific for the Ethernet driver, any entity that
tries to reserve more than 64 BF-enabled QPs should fail. Also, using
ranges is not necessary here and is wasteful.
The new mechanism introduced here will support reservation for
"Eth QPs eligible for BF" for all drivers: bare-metal, multi-PF, and VFs
(when hypervisors support WC in VMs). The flow we use is:
1. In mlx4_en, allocate Tx QPs one by one instead of a range allocation,
and request "BF enabled QPs" if BF is supported for the function
2. In the ALLOC_RES FW command, change param1 to:
a. param1[23:0] - number of QPs
b. param1[31-24] - flags controlling QPs reservation
Bit 31 refers to Eth blueflame supported QPs. Those QPs must have
bits 6 and 7 unset in order to be used in Ethernet.
Bits 24-30 of the flags are currently reserved.
When a function tries to allocate a QP, it states the required attributes
for this QP. Those attributes are considered "best-effort". If an attribute,
such as Ethernet BF enabled QP, is a must-have attribute, the function has
to check that attribute is supported before trying to do the allocation.
In a lower layer of the code, mlx4_qp_reserve_range masks out the bits
which are unsupported. If SRIOV is used, the PF validates those attributes
and masks out unsupported attributes as well. In order to notify VFs which
attributes are supported, the VF uses QUERY_FUNC_CAP command. This command's
mailbox is filled by the PF, which notifies which QP allocation attributes
it supports.
Signed-off-by: Eugenia Emantayev <eugenia@mellanox.co.il>
Signed-off-by: Matan Barak <matanb@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-12-11 16:57:54 +08:00
|
|
|
struct mlx4_en_tx_ring **pring, u32 size,
|
2013-12-20 03:20:14 +08:00
|
|
|
u16 stride, int node, int queue_index)
|
2008-10-23 06:47:49 +08:00
|
|
|
{
|
|
|
|
struct mlx4_en_dev *mdev = priv->mdev;
|
2013-11-07 18:19:52 +08:00
|
|
|
struct mlx4_en_tx_ring *ring;
|
2008-10-23 06:47:49 +08:00
|
|
|
int tmp;
|
|
|
|
int err;
|
|
|
|
|
2013-11-07 18:19:54 +08:00
|
|
|
ring = kzalloc_node(sizeof(*ring), GFP_KERNEL, node);
|
2013-11-07 18:19:52 +08:00
|
|
|
if (!ring) {
|
2013-11-07 18:19:54 +08:00
|
|
|
ring = kzalloc(sizeof(*ring), GFP_KERNEL);
|
|
|
|
if (!ring) {
|
|
|
|
en_err(priv, "Failed allocating TX ring\n");
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
2013-11-07 18:19:52 +08:00
|
|
|
}
|
|
|
|
|
2008-10-23 06:47:49 +08:00
|
|
|
ring->size = size;
|
|
|
|
ring->size_mask = size - 1;
|
|
|
|
ring->stride = stride;
|
2015-06-25 16:29:42 +08:00
|
|
|
ring->full_size = ring->size - HEADROOM - MAX_DESC_TXBBS;
|
2008-10-23 06:47:49 +08:00
|
|
|
|
|
|
|
tmp = size * sizeof(struct mlx4_en_tx_info);
|
2014-10-05 17:35:12 +08:00
|
|
|
ring->tx_info = kmalloc_node(tmp, GFP_KERNEL | __GFP_NOWARN, node);
|
2013-11-07 18:19:52 +08:00
|
|
|
if (!ring->tx_info) {
|
2013-11-07 18:19:54 +08:00
|
|
|
ring->tx_info = vmalloc(tmp);
|
|
|
|
if (!ring->tx_info) {
|
|
|
|
err = -ENOMEM;
|
|
|
|
goto err_ring;
|
|
|
|
}
|
2013-11-07 18:19:52 +08:00
|
|
|
}
|
2012-01-29 20:56:23 +08:00
|
|
|
|
2009-06-02 04:27:13 +08:00
|
|
|
en_dbg(DRV, priv, "Allocated tx_info ring at addr:%p size:%d\n",
|
2008-10-23 06:47:49 +08:00
|
|
|
ring->tx_info, tmp);
|
|
|
|
|
2013-11-07 18:19:54 +08:00
|
|
|
ring->bounce_buf = kmalloc_node(MAX_DESC_SIZE, GFP_KERNEL, node);
|
2008-10-23 06:47:49 +08:00
|
|
|
if (!ring->bounce_buf) {
|
2013-11-07 18:19:54 +08:00
|
|
|
ring->bounce_buf = kmalloc(MAX_DESC_SIZE, GFP_KERNEL);
|
|
|
|
if (!ring->bounce_buf) {
|
|
|
|
err = -ENOMEM;
|
|
|
|
goto err_info;
|
|
|
|
}
|
2008-10-23 06:47:49 +08:00
|
|
|
}
|
|
|
|
ring->buf_size = ALIGN(size * ring->stride, MLX4_EN_PAGE_SIZE);
|
|
|
|
|
2013-11-07 18:19:54 +08:00
|
|
|
/* Allocate HW buffers on provided NUMA node */
|
2015-01-25 22:59:35 +08:00
|
|
|
set_dev_node(&mdev->dev->persist->pdev->dev, node);
|
2008-10-23 06:47:49 +08:00
|
|
|
err = mlx4_alloc_hwq_res(mdev->dev, &ring->wqres, ring->buf_size,
|
|
|
|
2 * PAGE_SIZE);
|
2015-01-25 22:59:35 +08:00
|
|
|
set_dev_node(&mdev->dev->persist->pdev->dev, mdev->dev->numa_node);
|
2008-10-23 06:47:49 +08:00
|
|
|
if (err) {
|
2009-06-02 04:27:13 +08:00
|
|
|
en_err(priv, "Failed allocating hwq resources\n");
|
2008-10-23 06:47:49 +08:00
|
|
|
goto err_bounce;
|
|
|
|
}
|
|
|
|
|
|
|
|
err = mlx4_en_map_buffer(&ring->wqres.buf);
|
|
|
|
if (err) {
|
2009-06-02 04:27:13 +08:00
|
|
|
en_err(priv, "Failed to map TX buffer\n");
|
2008-10-23 06:47:49 +08:00
|
|
|
goto err_hwq_res;
|
|
|
|
}
|
|
|
|
|
|
|
|
ring->buf = ring->wqres.buf.direct.buf;
|
|
|
|
|
2014-05-08 03:52:57 +08:00
|
|
|
en_dbg(DRV, priv, "Allocated TX ring (addr:%p) - buf:%p size:%d buf_size:%d dma:%llx\n",
|
|
|
|
ring, ring->buf, ring->size, ring->buf_size,
|
|
|
|
(unsigned long long) ring->wqres.buf.direct.map);
|
2008-10-23 06:47:49 +08:00
|
|
|
|
net/mlx4: Change QP allocation scheme
When using BF (Blue-Flame), the QPN overrides the VLAN, CV, and SV fields
in the WQE. Thus, BF may only be used for QPNs with bits 6,7 unset.
The current Ethernet driver code reserves a Tx QP range with 256b alignment.
This is wrong because if there are more than 64 Tx QPs in use,
QPNs >= base + 65 will have bits 6/7 set.
This problem is not specific for the Ethernet driver, any entity that
tries to reserve more than 64 BF-enabled QPs should fail. Also, using
ranges is not necessary here and is wasteful.
The new mechanism introduced here will support reservation for
"Eth QPs eligible for BF" for all drivers: bare-metal, multi-PF, and VFs
(when hypervisors support WC in VMs). The flow we use is:
1. In mlx4_en, allocate Tx QPs one by one instead of a range allocation,
and request "BF enabled QPs" if BF is supported for the function
2. In the ALLOC_RES FW command, change param1 to:
a. param1[23:0] - number of QPs
b. param1[31-24] - flags controlling QPs reservation
Bit 31 refers to Eth blueflame supported QPs. Those QPs must have
bits 6 and 7 unset in order to be used in Ethernet.
Bits 24-30 of the flags are currently reserved.
When a function tries to allocate a QP, it states the required attributes
for this QP. Those attributes are considered "best-effort". If an attribute,
such as Ethernet BF enabled QP, is a must-have attribute, the function has
to check that attribute is supported before trying to do the allocation.
In a lower layer of the code, mlx4_qp_reserve_range masks out the bits
which are unsupported. If SRIOV is used, the PF validates those attributes
and masks out unsupported attributes as well. In order to notify VFs which
attributes are supported, the VF uses QUERY_FUNC_CAP command. This command's
mailbox is filled by the PF, which notifies which QP allocation attributes
it supports.
Signed-off-by: Eugenia Emantayev <eugenia@mellanox.co.il>
Signed-off-by: Matan Barak <matanb@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-12-11 16:57:54 +08:00
|
|
|
err = mlx4_qp_reserve_range(mdev->dev, 1, 1, &ring->qpn,
|
|
|
|
MLX4_RESERVE_ETH_BF_QP);
|
|
|
|
if (err) {
|
|
|
|
en_err(priv, "failed reserving qp for TX ring\n");
|
|
|
|
goto err_map;
|
|
|
|
}
|
|
|
|
|
2014-05-11 20:15:12 +08:00
|
|
|
err = mlx4_qp_alloc(mdev->dev, ring->qpn, &ring->qp, GFP_KERNEL);
|
2008-10-23 06:47:49 +08:00
|
|
|
if (err) {
|
2009-06-02 04:27:13 +08:00
|
|
|
en_err(priv, "Failed allocating qp %d\n", ring->qpn);
|
net/mlx4: Change QP allocation scheme
When using BF (Blue-Flame), the QPN overrides the VLAN, CV, and SV fields
in the WQE. Thus, BF may only be used for QPNs with bits 6,7 unset.
The current Ethernet driver code reserves a Tx QP range with 256b alignment.
This is wrong because if there are more than 64 Tx QPs in use,
QPNs >= base + 65 will have bits 6/7 set.
This problem is not specific for the Ethernet driver, any entity that
tries to reserve more than 64 BF-enabled QPs should fail. Also, using
ranges is not necessary here and is wasteful.
The new mechanism introduced here will support reservation for
"Eth QPs eligible for BF" for all drivers: bare-metal, multi-PF, and VFs
(when hypervisors support WC in VMs). The flow we use is:
1. In mlx4_en, allocate Tx QPs one by one instead of a range allocation,
and request "BF enabled QPs" if BF is supported for the function
2. In the ALLOC_RES FW command, change param1 to:
a. param1[23:0] - number of QPs
b. param1[31-24] - flags controlling QPs reservation
Bit 31 refers to Eth blueflame supported QPs. Those QPs must have
bits 6 and 7 unset in order to be used in Ethernet.
Bits 24-30 of the flags are currently reserved.
When a function tries to allocate a QP, it states the required attributes
for this QP. Those attributes are considered "best-effort". If an attribute,
such as Ethernet BF enabled QP, is a must-have attribute, the function has
to check that attribute is supported before trying to do the allocation.
In a lower layer of the code, mlx4_qp_reserve_range masks out the bits
which are unsupported. If SRIOV is used, the PF validates those attributes
and masks out unsupported attributes as well. In order to notify VFs which
attributes are supported, the VF uses QUERY_FUNC_CAP command. This command's
mailbox is filled by the PF, which notifies which QP allocation attributes
it supports.
Signed-off-by: Eugenia Emantayev <eugenia@mellanox.co.il>
Signed-off-by: Matan Barak <matanb@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-12-11 16:57:54 +08:00
|
|
|
goto err_reserve;
|
2008-10-23 06:47:49 +08:00
|
|
|
}
|
2009-04-20 12:30:03 +08:00
|
|
|
ring->qp.event = mlx4_en_sqp_event;
|
2008-10-23 06:47:49 +08:00
|
|
|
|
2013-11-07 18:19:54 +08:00
|
|
|
err = mlx4_bf_alloc(mdev->dev, &ring->bf, node);
|
2011-03-23 06:38:52 +08:00
|
|
|
if (err) {
|
2014-05-08 03:52:57 +08:00
|
|
|
en_dbg(DRV, priv, "working without blueflame (%d)\n", err);
|
2011-03-23 06:38:52 +08:00
|
|
|
ring->bf.uar = &mdev->priv_uar;
|
|
|
|
ring->bf.uar->map = mdev->uar_map;
|
|
|
|
ring->bf_enabled = false;
|
2014-07-22 20:44:10 +08:00
|
|
|
ring->bf_alloced = false;
|
|
|
|
priv->pflags &= ~MLX4_EN_PRIV_FLAGS_BLUEFLAME;
|
|
|
|
} else {
|
|
|
|
ring->bf_alloced = true;
|
|
|
|
ring->bf_enabled = !!(priv->pflags &
|
|
|
|
MLX4_EN_PRIV_FLAGS_BLUEFLAME);
|
|
|
|
}
|
2011-03-23 06:38:52 +08:00
|
|
|
|
2013-04-23 14:06:49 +08:00
|
|
|
ring->hwtstamp_tx_type = priv->hwtstamp_config.tx_type;
|
2013-12-20 03:20:14 +08:00
|
|
|
ring->queue_index = queue_index;
|
|
|
|
|
2015-04-29 05:49:29 +08:00
|
|
|
if (queue_index < priv->num_tx_rings_p_up)
|
cpumask_set_cpu_local_first => cpumask_local_spread, lament
da91309e0a7e (cpumask: Utility function to set n'th cpu...) created a
genuinely weird function. I never saw it before, it went through DaveM.
(He only does this to make us other maintainers feel better about our own
mistakes.)
cpumask_set_cpu_local_first's purpose is say "I need to spread things
across N online cpus, choose the ones on this numa node first"; you call
it in a loop.
It can fail. One of the two callers ignores this, the other aborts and
fails the device open.
It can fail in two ways: allocating the off-stack cpumask, or through a
convoluted codepath which AFAICT can only occur if cpu_online_mask
changes. Which shouldn't happen, because if cpu_online_mask can change
while you call this, it could return a now-offline cpu anyway.
It contains a nonsensical test "!cpumask_of_node(numa_node)". This was
drawn to my attention by Geert, who said this causes a warning on Sparc.
It sets a single bit in a cpumask instead of returning a cpu number,
because that's what the callers want.
It could be made more efficient by passing the previous cpu rather than
an index, but that would be more invasive to the callers.
Fixes: da91309e0a7e8966d916a74cce42ed170fde06bf
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> (then rebased)
Tested-by: Amir Vadai <amirv@mellanox.com>
Acked-by: Amir Vadai <amirv@mellanox.com>
Acked-by: David S. Miller <davem@davemloft.net>
2015-05-09 01:44:13 +08:00
|
|
|
cpumask_set_cpu(cpumask_local_spread(queue_index,
|
|
|
|
priv->mdev->dev->numa_node),
|
|
|
|
&ring->affinity_mask);
|
2013-04-23 14:06:49 +08:00
|
|
|
|
2013-11-07 18:19:52 +08:00
|
|
|
*pring = ring;
|
2008-10-23 06:47:49 +08:00
|
|
|
return 0;
|
|
|
|
|
net/mlx4: Change QP allocation scheme
When using BF (Blue-Flame), the QPN overrides the VLAN, CV, and SV fields
in the WQE. Thus, BF may only be used for QPNs with bits 6,7 unset.
The current Ethernet driver code reserves a Tx QP range with 256b alignment.
This is wrong because if there are more than 64 Tx QPs in use,
QPNs >= base + 65 will have bits 6/7 set.
This problem is not specific for the Ethernet driver, any entity that
tries to reserve more than 64 BF-enabled QPs should fail. Also, using
ranges is not necessary here and is wasteful.
The new mechanism introduced here will support reservation for
"Eth QPs eligible for BF" for all drivers: bare-metal, multi-PF, and VFs
(when hypervisors support WC in VMs). The flow we use is:
1. In mlx4_en, allocate Tx QPs one by one instead of a range allocation,
and request "BF enabled QPs" if BF is supported for the function
2. In the ALLOC_RES FW command, change param1 to:
a. param1[23:0] - number of QPs
b. param1[31-24] - flags controlling QPs reservation
Bit 31 refers to Eth blueflame supported QPs. Those QPs must have
bits 6 and 7 unset in order to be used in Ethernet.
Bits 24-30 of the flags are currently reserved.
When a function tries to allocate a QP, it states the required attributes
for this QP. Those attributes are considered "best-effort". If an attribute,
such as Ethernet BF enabled QP, is a must-have attribute, the function has
to check that attribute is supported before trying to do the allocation.
In a lower layer of the code, mlx4_qp_reserve_range masks out the bits
which are unsupported. If SRIOV is used, the PF validates those attributes
and masks out unsupported attributes as well. In order to notify VFs which
attributes are supported, the VF uses QUERY_FUNC_CAP command. This command's
mailbox is filled by the PF, which notifies which QP allocation attributes
it supports.
Signed-off-by: Eugenia Emantayev <eugenia@mellanox.co.il>
Signed-off-by: Matan Barak <matanb@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-12-11 16:57:54 +08:00
|
|
|
err_reserve:
|
|
|
|
mlx4_qp_release_range(mdev->dev, ring->qpn, 1);
|
2008-10-23 06:47:49 +08:00
|
|
|
err_map:
|
|
|
|
mlx4_en_unmap_buffer(&ring->wqres.buf);
|
|
|
|
err_hwq_res:
|
|
|
|
mlx4_free_hwq_res(mdev->dev, &ring->wqres, ring->buf_size);
|
|
|
|
err_bounce:
|
|
|
|
kfree(ring->bounce_buf);
|
|
|
|
ring->bounce_buf = NULL;
|
2013-11-07 18:19:52 +08:00
|
|
|
err_info:
|
2014-10-05 17:35:12 +08:00
|
|
|
kvfree(ring->tx_info);
|
2008-10-23 06:47:49 +08:00
|
|
|
ring->tx_info = NULL;
|
2013-11-07 18:19:52 +08:00
|
|
|
err_ring:
|
|
|
|
kfree(ring);
|
|
|
|
*pring = NULL;
|
2008-10-23 06:47:49 +08:00
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
void mlx4_en_destroy_tx_ring(struct mlx4_en_priv *priv,
|
2013-11-07 18:19:52 +08:00
|
|
|
struct mlx4_en_tx_ring **pring)
|
2008-10-23 06:47:49 +08:00
|
|
|
{
|
|
|
|
struct mlx4_en_dev *mdev = priv->mdev;
|
2013-11-07 18:19:52 +08:00
|
|
|
struct mlx4_en_tx_ring *ring = *pring;
|
2009-06-02 04:27:13 +08:00
|
|
|
en_dbg(DRV, priv, "Destroying tx ring, qpn: %d\n", ring->qpn);
|
2008-10-23 06:47:49 +08:00
|
|
|
|
2014-07-22 20:44:10 +08:00
|
|
|
if (ring->bf_alloced)
|
2011-03-23 06:38:52 +08:00
|
|
|
mlx4_bf_free(mdev->dev, &ring->bf);
|
2008-10-23 06:47:49 +08:00
|
|
|
mlx4_qp_remove(mdev->dev, &ring->qp);
|
|
|
|
mlx4_qp_free(mdev->dev, &ring->qp);
|
2015-06-25 16:29:41 +08:00
|
|
|
mlx4_qp_release_range(priv->mdev->dev, ring->qpn, 1);
|
2008-10-23 06:47:49 +08:00
|
|
|
mlx4_en_unmap_buffer(&ring->wqres.buf);
|
|
|
|
mlx4_free_hwq_res(mdev->dev, &ring->wqres, ring->buf_size);
|
|
|
|
kfree(ring->bounce_buf);
|
|
|
|
ring->bounce_buf = NULL;
|
2014-10-05 17:35:12 +08:00
|
|
|
kvfree(ring->tx_info);
|
2008-10-23 06:47:49 +08:00
|
|
|
ring->tx_info = NULL;
|
2013-11-07 18:19:52 +08:00
|
|
|
kfree(ring);
|
|
|
|
*pring = NULL;
|
2008-10-23 06:47:49 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
int mlx4_en_activate_tx_ring(struct mlx4_en_priv *priv,
|
|
|
|
struct mlx4_en_tx_ring *ring,
|
2012-04-05 05:33:24 +08:00
|
|
|
int cq, int user_prio)
|
2008-10-23 06:47:49 +08:00
|
|
|
{
|
|
|
|
struct mlx4_en_dev *mdev = priv->mdev;
|
|
|
|
int err;
|
|
|
|
|
|
|
|
ring->cqn = cq;
|
|
|
|
ring->prod = 0;
|
|
|
|
ring->cons = 0xffffffff;
|
|
|
|
ring->last_nr_txbb = 1;
|
|
|
|
memset(ring->tx_info, 0, ring->size * sizeof(struct mlx4_en_tx_info));
|
|
|
|
memset(ring->buf, 0, ring->buf_size);
|
|
|
|
|
|
|
|
ring->qp_state = MLX4_QP_STATE_RST;
|
2014-10-05 17:35:11 +08:00
|
|
|
ring->doorbell_qpn = cpu_to_be32(ring->qp.qpn << 8);
|
|
|
|
ring->mr_key = cpu_to_be32(mdev->mr.key);
|
2008-10-23 06:47:49 +08:00
|
|
|
|
|
|
|
mlx4_en_fill_qp_context(priv, ring->size, ring->stride, 1, 0, ring->qpn,
|
2012-04-05 05:33:24 +08:00
|
|
|
ring->cqn, user_prio, &ring->context);
|
2014-07-22 20:44:10 +08:00
|
|
|
if (ring->bf_alloced)
|
2011-03-23 06:38:52 +08:00
|
|
|
ring->context.usr_page = cpu_to_be32(ring->bf.uar->index);
|
2008-10-23 06:47:49 +08:00
|
|
|
|
|
|
|
err = mlx4_qp_to_ready(mdev->dev, &ring->wqres.mtt, &ring->context,
|
|
|
|
&ring->qp, &ring->qp_state);
|
2015-04-29 05:49:29 +08:00
|
|
|
if (!cpumask_empty(&ring->affinity_mask))
|
2013-12-20 03:20:14 +08:00
|
|
|
netif_set_xps_queue(priv->dev, &ring->affinity_mask,
|
|
|
|
ring->queue_index);
|
2008-10-23 06:47:49 +08:00
|
|
|
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
void mlx4_en_deactivate_tx_ring(struct mlx4_en_priv *priv,
|
|
|
|
struct mlx4_en_tx_ring *ring)
|
|
|
|
{
|
|
|
|
struct mlx4_en_dev *mdev = priv->mdev;
|
|
|
|
|
|
|
|
mlx4_qp_modify(mdev->dev, NULL, ring->qp_state,
|
|
|
|
MLX4_QP_STATE_RST, NULL, 0, 0, &ring->qp);
|
|
|
|
}
|
|
|
|
|
2015-06-25 16:29:42 +08:00
|
|
|
static inline bool mlx4_en_is_tx_ring_full(struct mlx4_en_tx_ring *ring)
|
|
|
|
{
|
|
|
|
return ring->prod - ring->cons > ring->full_size;
|
|
|
|
}
|
|
|
|
|
2013-07-26 00:21:23 +08:00
|
|
|
static void mlx4_en_stamp_wqe(struct mlx4_en_priv *priv,
|
|
|
|
struct mlx4_en_tx_ring *ring, int index,
|
|
|
|
u8 owner)
|
|
|
|
{
|
|
|
|
__be32 stamp = cpu_to_be32(STAMP_VAL | (!!owner << STAMP_SHIFT));
|
|
|
|
struct mlx4_en_tx_desc *tx_desc = ring->buf + index * TXBB_SIZE;
|
|
|
|
struct mlx4_en_tx_info *tx_info = &ring->tx_info[index];
|
|
|
|
void *end = ring->buf + ring->buf_size;
|
|
|
|
__be32 *ptr = (__be32 *)tx_desc;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
/* Optimize the common case when there are no wraparounds */
|
|
|
|
if (likely((void *)tx_desc + tx_info->nr_txbb * TXBB_SIZE <= end)) {
|
|
|
|
/* Stamp the freed descriptor */
|
|
|
|
for (i = 0; i < tx_info->nr_txbb * TXBB_SIZE;
|
|
|
|
i += STAMP_STRIDE) {
|
|
|
|
*ptr = stamp;
|
|
|
|
ptr += STAMP_DWORDS;
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
/* Stamp the freed descriptor */
|
|
|
|
for (i = 0; i < tx_info->nr_txbb * TXBB_SIZE;
|
|
|
|
i += STAMP_STRIDE) {
|
|
|
|
*ptr = stamp;
|
|
|
|
ptr += STAMP_DWORDS;
|
|
|
|
if ((void *)ptr >= end) {
|
|
|
|
ptr = ring->buf;
|
|
|
|
stamp ^= cpu_to_be32(0x80000000);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-10-23 06:47:49 +08:00
|
|
|
|
|
|
|
static u32 mlx4_en_free_tx_desc(struct mlx4_en_priv *priv,
|
|
|
|
struct mlx4_en_tx_ring *ring,
|
2013-04-23 14:06:49 +08:00
|
|
|
int index, u8 owner, u64 timestamp)
|
2008-10-23 06:47:49 +08:00
|
|
|
{
|
|
|
|
struct mlx4_en_tx_info *tx_info = &ring->tx_info[index];
|
|
|
|
struct mlx4_en_tx_desc *tx_desc = ring->buf + index * TXBB_SIZE;
|
|
|
|
struct mlx4_wqe_data_seg *data = (void *) tx_desc + tx_info->data_offset;
|
|
|
|
void *end = ring->buf + ring->buf_size;
|
2014-10-05 17:35:13 +08:00
|
|
|
struct sk_buff *skb = tx_info->skb;
|
|
|
|
int nr_maps = tx_info->nr_maps;
|
2008-10-23 06:47:49 +08:00
|
|
|
int i;
|
2013-04-23 14:06:49 +08:00
|
|
|
|
2014-10-05 17:35:14 +08:00
|
|
|
/* We do not touch skb here, so prefetch skb->users location
|
|
|
|
* to speedup consume_skb()
|
|
|
|
*/
|
|
|
|
prefetchw(&skb->users);
|
|
|
|
|
2014-10-05 17:35:13 +08:00
|
|
|
if (unlikely(timestamp)) {
|
|
|
|
struct skb_shared_hwtstamps hwts;
|
|
|
|
|
|
|
|
mlx4_en_fill_hwtstamps(priv->mdev, &hwts, timestamp);
|
2013-04-23 14:06:49 +08:00
|
|
|
skb_tstamp_tx(skb, &hwts);
|
|
|
|
}
|
2008-10-23 06:47:49 +08:00
|
|
|
|
|
|
|
/* Optimize the common case when there are no wraparounds */
|
|
|
|
if (likely((void *) tx_desc + tx_info->nr_txbb * TXBB_SIZE <= end)) {
|
2009-01-09 02:57:15 +08:00
|
|
|
if (!tx_info->inl) {
|
2014-10-05 17:35:13 +08:00
|
|
|
if (tx_info->linear)
|
2012-03-06 12:03:34 +08:00
|
|
|
dma_unmap_single(priv->ddev,
|
2014-10-05 17:35:13 +08:00
|
|
|
tx_info->map0_dma,
|
|
|
|
tx_info->map0_byte_count,
|
|
|
|
PCI_DMA_TODEVICE);
|
|
|
|
else
|
|
|
|
dma_unmap_page(priv->ddev,
|
|
|
|
tx_info->map0_dma,
|
|
|
|
tx_info->map0_byte_count,
|
|
|
|
PCI_DMA_TODEVICE);
|
|
|
|
for (i = 1; i < nr_maps; i++) {
|
|
|
|
data++;
|
2012-03-06 12:03:34 +08:00
|
|
|
dma_unmap_page(priv->ddev,
|
2014-10-05 17:35:13 +08:00
|
|
|
(dma_addr_t)be64_to_cpu(data->addr),
|
|
|
|
be32_to_cpu(data->byte_count),
|
|
|
|
PCI_DMA_TODEVICE);
|
2009-01-09 02:57:15 +08:00
|
|
|
}
|
2008-10-23 06:47:49 +08:00
|
|
|
}
|
|
|
|
} else {
|
2009-01-09 02:57:15 +08:00
|
|
|
if (!tx_info->inl) {
|
|
|
|
if ((void *) data >= end) {
|
2011-06-17 03:08:06 +08:00
|
|
|
data = ring->buf + ((void *)data - end);
|
2009-01-09 02:57:15 +08:00
|
|
|
}
|
2008-10-23 06:47:49 +08:00
|
|
|
|
2014-10-05 17:35:13 +08:00
|
|
|
if (tx_info->linear)
|
2012-03-06 12:03:34 +08:00
|
|
|
dma_unmap_single(priv->ddev,
|
2014-10-05 17:35:13 +08:00
|
|
|
tx_info->map0_dma,
|
|
|
|
tx_info->map0_byte_count,
|
|
|
|
PCI_DMA_TODEVICE);
|
|
|
|
else
|
|
|
|
dma_unmap_page(priv->ddev,
|
|
|
|
tx_info->map0_dma,
|
|
|
|
tx_info->map0_byte_count,
|
|
|
|
PCI_DMA_TODEVICE);
|
|
|
|
for (i = 1; i < nr_maps; i++) {
|
|
|
|
data++;
|
2009-01-09 02:57:15 +08:00
|
|
|
/* Check for wraparound before unmapping */
|
|
|
|
if ((void *) data >= end)
|
2011-06-17 03:08:06 +08:00
|
|
|
data = ring->buf;
|
2012-03-06 12:03:34 +08:00
|
|
|
dma_unmap_page(priv->ddev,
|
2014-10-05 17:35:13 +08:00
|
|
|
(dma_addr_t)be64_to_cpu(data->addr),
|
|
|
|
be32_to_cpu(data->byte_count),
|
|
|
|
PCI_DMA_TODEVICE);
|
2009-01-09 02:57:15 +08:00
|
|
|
}
|
2008-10-23 06:47:49 +08:00
|
|
|
}
|
|
|
|
}
|
2014-09-04 00:18:00 +08:00
|
|
|
dev_consume_skb_any(skb);
|
2008-10-23 06:47:49 +08:00
|
|
|
return tx_info->nr_txbb;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
int mlx4_en_free_tx_buf(struct net_device *dev, struct mlx4_en_tx_ring *ring)
|
|
|
|
{
|
|
|
|
struct mlx4_en_priv *priv = netdev_priv(dev);
|
|
|
|
int cnt = 0;
|
|
|
|
|
|
|
|
/* Skip last polled descriptor */
|
|
|
|
ring->cons += ring->last_nr_txbb;
|
2009-06-02 04:27:13 +08:00
|
|
|
en_dbg(DRV, priv, "Freeing Tx buf - cons:0x%x prod:0x%x\n",
|
2008-10-23 06:47:49 +08:00
|
|
|
ring->cons, ring->prod);
|
|
|
|
|
|
|
|
if ((u32) (ring->prod - ring->cons) > ring->size) {
|
|
|
|
if (netif_msg_tx_err(priv))
|
2009-06-02 04:27:13 +08:00
|
|
|
en_warn(priv, "Tx consumer passed producer!\n");
|
2008-10-23 06:47:49 +08:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
while (ring->cons != ring->prod) {
|
|
|
|
ring->last_nr_txbb = mlx4_en_free_tx_desc(priv, ring,
|
|
|
|
ring->cons & ring->size_mask,
|
2013-04-23 14:06:49 +08:00
|
|
|
!!(ring->cons & ring->size), 0);
|
2008-10-23 06:47:49 +08:00
|
|
|
ring->cons += ring->last_nr_txbb;
|
|
|
|
cnt++;
|
|
|
|
}
|
|
|
|
|
2013-02-06 15:58:41 +08:00
|
|
|
netdev_tx_reset_queue(ring->tx_queue);
|
|
|
|
|
2008-10-23 06:47:49 +08:00
|
|
|
if (cnt)
|
2009-06-02 04:27:13 +08:00
|
|
|
en_dbg(DRV, priv, "Freed %d uncompleted tx descriptors\n", cnt);
|
2008-10-23 06:47:49 +08:00
|
|
|
|
|
|
|
return cnt;
|
|
|
|
}
|
|
|
|
|
2014-07-08 16:28:12 +08:00
|
|
|
static bool mlx4_en_process_tx_cq(struct net_device *dev,
|
|
|
|
struct mlx4_en_cq *cq)
|
2008-10-23 06:47:49 +08:00
|
|
|
{
|
|
|
|
struct mlx4_en_priv *priv = netdev_priv(dev);
|
|
|
|
struct mlx4_cq *mcq = &cq->mcq;
|
2013-11-07 18:19:52 +08:00
|
|
|
struct mlx4_en_tx_ring *ring = priv->tx_ring[cq->ring];
|
2011-11-27 03:55:10 +08:00
|
|
|
struct mlx4_cqe *cqe;
|
2008-10-23 06:47:49 +08:00
|
|
|
u16 index;
|
2013-07-26 00:21:23 +08:00
|
|
|
u16 new_index, ring_index, stamp_index;
|
2008-10-23 06:47:49 +08:00
|
|
|
u32 txbbs_skipped = 0;
|
2013-07-26 00:21:23 +08:00
|
|
|
u32 txbbs_stamp = 0;
|
2011-11-27 03:55:10 +08:00
|
|
|
u32 cons_index = mcq->cons_index;
|
|
|
|
int size = cq->size;
|
|
|
|
u32 size_mask = ring->size_mask;
|
|
|
|
struct mlx4_cqe *buf = cq->buf;
|
2012-04-23 10:18:50 +08:00
|
|
|
u32 packets = 0;
|
|
|
|
u32 bytes = 0;
|
2012-10-21 22:59:24 +08:00
|
|
|
int factor = priv->cqe_factor;
|
2013-04-23 14:06:49 +08:00
|
|
|
u64 timestamp = 0;
|
2013-12-20 03:20:17 +08:00
|
|
|
int done = 0;
|
2014-07-08 16:28:12 +08:00
|
|
|
int budget = priv->tx_work_limit;
|
2014-10-05 17:35:15 +08:00
|
|
|
u32 last_nr_txbb;
|
|
|
|
u32 ring_cons;
|
2008-10-23 06:47:49 +08:00
|
|
|
|
|
|
|
if (!priv->port_up)
|
2014-07-08 16:28:12 +08:00
|
|
|
return true;
|
2008-10-23 06:47:49 +08:00
|
|
|
|
2014-10-08 23:19:27 +08:00
|
|
|
netdev_txq_bql_complete_prefetchw(ring->tx_queue);
|
|
|
|
|
2011-11-27 03:55:10 +08:00
|
|
|
index = cons_index & size_mask;
|
2014-09-18 16:51:01 +08:00
|
|
|
cqe = mlx4_en_get_cqe(buf, index, priv->cqe_size) + factor;
|
2014-10-05 17:35:15 +08:00
|
|
|
last_nr_txbb = ACCESS_ONCE(ring->last_nr_txbb);
|
|
|
|
ring_cons = ACCESS_ONCE(ring->cons);
|
|
|
|
ring_index = ring_cons & size_mask;
|
2013-07-26 00:21:23 +08:00
|
|
|
stamp_index = ring_index;
|
2011-11-27 03:55:10 +08:00
|
|
|
|
|
|
|
/* Process all completed CQEs */
|
|
|
|
while (XNOR(cqe->owner_sr_opcode & MLX4_CQE_OWNER_MASK,
|
2013-12-20 03:20:17 +08:00
|
|
|
cons_index & size) && (done < budget)) {
|
2011-11-27 03:55:10 +08:00
|
|
|
/*
|
|
|
|
* make sure we read the CQE after we read the
|
|
|
|
* ownership bit
|
|
|
|
*/
|
2015-04-09 09:49:36 +08:00
|
|
|
dma_rmb();
|
2011-11-27 03:55:10 +08:00
|
|
|
|
2013-08-21 15:08:57 +08:00
|
|
|
if (unlikely((cqe->owner_sr_opcode & MLX4_CQE_OPCODE_MASK) ==
|
|
|
|
MLX4_CQE_OPCODE_ERROR)) {
|
|
|
|
struct mlx4_err_cqe *cqe_err = (struct mlx4_err_cqe *)cqe;
|
|
|
|
|
|
|
|
en_err(priv, "CQE error - vendor syndrome: 0x%x syndrome: 0x%x\n",
|
|
|
|
cqe_err->vendor_err_syndrome,
|
|
|
|
cqe_err->syndrome);
|
|
|
|
}
|
|
|
|
|
2011-11-27 03:55:10 +08:00
|
|
|
/* Skip over last polled CQE */
|
|
|
|
new_index = be16_to_cpu(cqe->wqe_index) & size_mask;
|
|
|
|
|
2008-10-23 06:47:49 +08:00
|
|
|
do {
|
2014-10-05 17:35:15 +08:00
|
|
|
txbbs_skipped += last_nr_txbb;
|
|
|
|
ring_index = (ring_index + last_nr_txbb) & size_mask;
|
2013-04-23 14:06:49 +08:00
|
|
|
if (ring->tx_info[ring_index].ts_requested)
|
|
|
|
timestamp = mlx4_en_get_cqe_ts(cqe);
|
|
|
|
|
2011-11-27 03:55:10 +08:00
|
|
|
/* free next descriptor */
|
2014-10-05 17:35:15 +08:00
|
|
|
last_nr_txbb = mlx4_en_free_tx_desc(
|
2011-11-27 03:55:10 +08:00
|
|
|
priv, ring, ring_index,
|
2014-10-05 17:35:15 +08:00
|
|
|
!!((ring_cons + txbbs_skipped) &
|
2013-04-23 14:06:49 +08:00
|
|
|
ring->size), timestamp);
|
2013-07-26 00:21:23 +08:00
|
|
|
|
|
|
|
mlx4_en_stamp_wqe(priv, ring, stamp_index,
|
2014-10-05 17:35:15 +08:00
|
|
|
!!((ring_cons + txbbs_stamp) &
|
2013-07-26 00:21:23 +08:00
|
|
|
ring->size));
|
|
|
|
stamp_index = ring_index;
|
|
|
|
txbbs_stamp = txbbs_skipped;
|
2012-04-23 10:18:50 +08:00
|
|
|
packets++;
|
|
|
|
bytes += ring->tx_info[ring_index].nr_bytes;
|
2013-12-20 03:20:17 +08:00
|
|
|
} while ((++done < budget) && (ring_index != new_index));
|
2011-11-27 03:55:10 +08:00
|
|
|
|
|
|
|
++cons_index;
|
|
|
|
index = cons_index & size_mask;
|
2014-09-18 16:51:01 +08:00
|
|
|
cqe = mlx4_en_get_cqe(buf, index, priv->cqe_size) + factor;
|
2011-11-27 03:55:10 +08:00
|
|
|
}
|
2008-10-23 06:47:49 +08:00
|
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
* To prevent CQ overflow we first update CQ consumer and only then
|
|
|
|
* the ring consumer.
|
|
|
|
*/
|
2011-11-27 03:55:10 +08:00
|
|
|
mcq->cons_index = cons_index;
|
2008-10-23 06:47:49 +08:00
|
|
|
mlx4_cq_set_ci(mcq);
|
|
|
|
wmb();
|
2014-10-05 17:35:15 +08:00
|
|
|
|
|
|
|
/* we want to dirty this cache line once */
|
|
|
|
ACCESS_ONCE(ring->last_nr_txbb) = last_nr_txbb;
|
|
|
|
ACCESS_ONCE(ring->cons) = ring_cons + txbbs_skipped;
|
|
|
|
|
2012-04-23 10:18:50 +08:00
|
|
|
netdev_tx_completed_queue(ring->tx_queue, packets, bytes);
|
2008-10-23 06:47:49 +08:00
|
|
|
|
2015-06-25 16:29:42 +08:00
|
|
|
/* Wakeup Tx queue if this stopped, and ring is not full.
|
2012-08-03 08:38:37 +08:00
|
|
|
*/
|
2015-06-25 16:29:42 +08:00
|
|
|
if (netif_tx_queue_stopped(ring->tx_queue) &&
|
|
|
|
!mlx4_en_is_tx_ring_full(ring)) {
|
2012-08-03 08:38:37 +08:00
|
|
|
netif_tx_wake_queue(ring->tx_queue);
|
2014-03-02 16:25:00 +08:00
|
|
|
ring->wake_queue++;
|
2008-10-23 06:47:49 +08:00
|
|
|
}
|
2014-07-08 16:28:12 +08:00
|
|
|
return done < budget;
|
2008-10-23 06:47:49 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
void mlx4_en_tx_irq(struct mlx4_cq *mcq)
|
|
|
|
{
|
|
|
|
struct mlx4_en_cq *cq = container_of(mcq, struct mlx4_en_cq, mcq);
|
|
|
|
struct mlx4_en_priv *priv = netdev_priv(cq->dev);
|
|
|
|
|
2014-10-30 07:54:45 +08:00
|
|
|
if (likely(priv->port_up))
|
|
|
|
napi_schedule_irqoff(&cq->napi);
|
2013-12-20 03:20:17 +08:00
|
|
|
else
|
|
|
|
mlx4_en_arm_cq(priv, cq);
|
2008-10-23 06:47:49 +08:00
|
|
|
}
|
|
|
|
|
2013-12-20 03:20:17 +08:00
|
|
|
/* TX CQ polling - called by NAPI */
|
|
|
|
int mlx4_en_poll_tx_cq(struct napi_struct *napi, int budget)
|
|
|
|
{
|
|
|
|
struct mlx4_en_cq *cq = container_of(napi, struct mlx4_en_cq, napi);
|
|
|
|
struct net_device *dev = cq->dev;
|
|
|
|
struct mlx4_en_priv *priv = netdev_priv(dev);
|
2014-07-08 16:28:12 +08:00
|
|
|
int clean_complete;
|
2013-12-20 03:20:17 +08:00
|
|
|
|
2014-07-08 16:28:12 +08:00
|
|
|
clean_complete = mlx4_en_process_tx_cq(dev, cq);
|
|
|
|
if (!clean_complete)
|
|
|
|
return budget;
|
2013-12-20 03:20:17 +08:00
|
|
|
|
2014-07-08 16:28:12 +08:00
|
|
|
napi_complete(napi);
|
|
|
|
mlx4_en_arm_cq(priv, cq);
|
|
|
|
|
|
|
|
return 0;
|
2013-12-20 03:20:17 +08:00
|
|
|
}
|
2008-10-23 06:47:49 +08:00
|
|
|
|
|
|
|
static struct mlx4_en_tx_desc *mlx4_en_bounce_to_desc(struct mlx4_en_priv *priv,
|
|
|
|
struct mlx4_en_tx_ring *ring,
|
|
|
|
u32 index,
|
|
|
|
unsigned int desc_size)
|
|
|
|
{
|
|
|
|
u32 copy = (ring->size - index) * TXBB_SIZE;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = desc_size - copy - 4; i >= 0; i -= 4) {
|
|
|
|
if ((i & (TXBB_SIZE - 1)) == 0)
|
|
|
|
wmb();
|
|
|
|
|
|
|
|
*((u32 *) (ring->buf + i)) =
|
|
|
|
*((u32 *) (ring->bounce_buf + copy + i));
|
|
|
|
}
|
|
|
|
|
|
|
|
for (i = copy - 4; i >= 4 ; i -= 4) {
|
|
|
|
if ((i & (TXBB_SIZE - 1)) == 0)
|
|
|
|
wmb();
|
|
|
|
|
|
|
|
*((u32 *) (ring->buf + index * TXBB_SIZE + i)) =
|
|
|
|
*((u32 *) (ring->bounce_buf + i));
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Return real descriptor location */
|
|
|
|
return ring->buf + index * TXBB_SIZE;
|
|
|
|
}
|
|
|
|
|
2014-10-05 17:35:20 +08:00
|
|
|
/* Decide if skb can be inlined in tx descriptor to avoid dma mapping
|
|
|
|
*
|
|
|
|
* It seems strange we do not simply use skb_copy_bits().
|
|
|
|
* This would allow to inline all skbs iff skb->len <= inline_thold
|
|
|
|
*
|
|
|
|
* Note that caller already checked skb was not a gso packet
|
|
|
|
*/
|
2014-10-05 17:35:09 +08:00
|
|
|
static bool is_inline(int inline_thold, const struct sk_buff *skb,
|
2014-10-05 17:35:17 +08:00
|
|
|
const struct skb_shared_info *shinfo,
|
2014-10-05 17:35:09 +08:00
|
|
|
void **pfrag)
|
2008-10-23 06:47:49 +08:00
|
|
|
{
|
|
|
|
void *ptr;
|
|
|
|
|
2014-10-05 17:35:20 +08:00
|
|
|
if (skb->len > inline_thold || !inline_thold)
|
|
|
|
return false;
|
2008-10-23 06:47:49 +08:00
|
|
|
|
2014-10-05 17:35:20 +08:00
|
|
|
if (shinfo->nr_frags == 1) {
|
|
|
|
ptr = skb_frag_address_safe(&shinfo->frags[0]);
|
|
|
|
if (unlikely(!ptr))
|
|
|
|
return false;
|
|
|
|
*pfrag = ptr;
|
|
|
|
return true;
|
2008-10-23 06:47:49 +08:00
|
|
|
}
|
2014-10-05 17:35:20 +08:00
|
|
|
if (shinfo->nr_frags)
|
|
|
|
return false;
|
|
|
|
return true;
|
2008-10-23 06:47:49 +08:00
|
|
|
}
|
|
|
|
|
2014-10-05 17:35:09 +08:00
|
|
|
static int inline_size(const struct sk_buff *skb)
|
2008-10-23 06:47:49 +08:00
|
|
|
{
|
|
|
|
if (skb->len + CTRL_SIZE + sizeof(struct mlx4_wqe_inline_seg)
|
|
|
|
<= MLX4_INLINE_ALIGN)
|
|
|
|
return ALIGN(skb->len + CTRL_SIZE +
|
|
|
|
sizeof(struct mlx4_wqe_inline_seg), 16);
|
|
|
|
else
|
|
|
|
return ALIGN(skb->len + CTRL_SIZE + 2 *
|
|
|
|
sizeof(struct mlx4_wqe_inline_seg), 16);
|
|
|
|
}
|
|
|
|
|
2014-10-05 17:35:09 +08:00
|
|
|
static int get_real_size(const struct sk_buff *skb,
|
2014-10-05 17:35:17 +08:00
|
|
|
const struct skb_shared_info *shinfo,
|
2014-10-05 17:35:09 +08:00
|
|
|
struct net_device *dev,
|
2014-10-05 17:35:20 +08:00
|
|
|
int *lso_header_size,
|
|
|
|
bool *inline_ok,
|
|
|
|
void **pfrag)
|
2008-10-23 06:47:49 +08:00
|
|
|
{
|
|
|
|
struct mlx4_en_priv *priv = netdev_priv(dev);
|
|
|
|
int real_size;
|
|
|
|
|
2014-10-05 17:35:17 +08:00
|
|
|
if (shinfo->gso_size) {
|
2014-10-05 17:35:20 +08:00
|
|
|
*inline_ok = false;
|
2013-12-23 22:09:44 +08:00
|
|
|
if (skb->encapsulation)
|
|
|
|
*lso_header_size = (skb_inner_transport_header(skb) - skb->data) + inner_tcp_hdrlen(skb);
|
|
|
|
else
|
|
|
|
*lso_header_size = skb_transport_offset(skb) + tcp_hdrlen(skb);
|
2014-10-05 17:35:17 +08:00
|
|
|
real_size = CTRL_SIZE + shinfo->nr_frags * DS_SIZE +
|
2008-10-23 06:47:49 +08:00
|
|
|
ALIGN(*lso_header_size + 4, DS_SIZE);
|
|
|
|
if (unlikely(*lso_header_size != skb_headlen(skb))) {
|
|
|
|
/* We add a segment for the skb linear buffer only if
|
|
|
|
* it contains data */
|
|
|
|
if (*lso_header_size < skb_headlen(skb))
|
|
|
|
real_size += DS_SIZE;
|
|
|
|
else {
|
|
|
|
if (netif_msg_tx_err(priv))
|
2009-06-02 04:27:13 +08:00
|
|
|
en_warn(priv, "Non-linear headers\n");
|
2008-10-23 06:47:49 +08:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
*lso_header_size = 0;
|
2014-10-05 17:35:20 +08:00
|
|
|
*inline_ok = is_inline(priv->prof->inline_thold, skb,
|
|
|
|
shinfo, pfrag);
|
|
|
|
|
|
|
|
if (*inline_ok)
|
2008-10-23 06:47:49 +08:00
|
|
|
real_size = inline_size(skb);
|
2014-10-05 17:35:20 +08:00
|
|
|
else
|
|
|
|
real_size = CTRL_SIZE +
|
|
|
|
(shinfo->nr_frags + 1) * DS_SIZE;
|
2008-10-23 06:47:49 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
return real_size;
|
|
|
|
}
|
|
|
|
|
2014-10-05 17:35:09 +08:00
|
|
|
static void build_inline_wqe(struct mlx4_en_tx_desc *tx_desc,
|
|
|
|
const struct sk_buff *skb,
|
2014-10-05 17:35:17 +08:00
|
|
|
const struct skb_shared_info *shinfo,
|
2014-10-05 17:35:09 +08:00
|
|
|
int real_size, u16 *vlan_tag,
|
|
|
|
int tx_ind, void *fragptr)
|
2008-10-23 06:47:49 +08:00
|
|
|
{
|
|
|
|
struct mlx4_wqe_inline_seg *inl = &tx_desc->inl;
|
|
|
|
int spc = MLX4_INLINE_ALIGN - CTRL_SIZE - sizeof *inl;
|
2014-10-05 17:35:18 +08:00
|
|
|
unsigned int hlen = skb_headlen(skb);
|
2008-10-23 06:47:49 +08:00
|
|
|
|
|
|
|
if (skb->len <= spc) {
|
2014-03-02 16:24:59 +08:00
|
|
|
if (likely(skb->len >= MIN_PKT_LEN)) {
|
|
|
|
inl->byte_count = cpu_to_be32(1 << 31 | skb->len);
|
|
|
|
} else {
|
|
|
|
inl->byte_count = cpu_to_be32(1 << 31 | MIN_PKT_LEN);
|
|
|
|
memset(((void *)(inl + 1)) + skb->len, 0,
|
|
|
|
MIN_PKT_LEN - skb->len);
|
|
|
|
}
|
2014-10-05 17:35:18 +08:00
|
|
|
skb_copy_from_linear_data(skb, inl + 1, hlen);
|
2014-10-05 17:35:17 +08:00
|
|
|
if (shinfo->nr_frags)
|
2014-10-05 17:35:18 +08:00
|
|
|
memcpy(((void *)(inl + 1)) + hlen, fragptr,
|
2014-10-05 17:35:17 +08:00
|
|
|
skb_frag_size(&shinfo->frags[0]));
|
2008-10-23 06:47:49 +08:00
|
|
|
|
|
|
|
} else {
|
|
|
|
inl->byte_count = cpu_to_be32(1 << 31 | spc);
|
2014-10-05 17:35:18 +08:00
|
|
|
if (hlen <= spc) {
|
|
|
|
skb_copy_from_linear_data(skb, inl + 1, hlen);
|
|
|
|
if (hlen < spc) {
|
|
|
|
memcpy(((void *)(inl + 1)) + hlen,
|
|
|
|
fragptr, spc - hlen);
|
|
|
|
fragptr += spc - hlen;
|
2008-10-23 06:47:49 +08:00
|
|
|
}
|
|
|
|
inl = (void *) (inl + 1) + spc;
|
|
|
|
memcpy(((void *)(inl + 1)), fragptr, skb->len - spc);
|
|
|
|
} else {
|
|
|
|
skb_copy_from_linear_data(skb, inl + 1, spc);
|
|
|
|
inl = (void *) (inl + 1) + spc;
|
|
|
|
skb_copy_from_linear_data_offset(skb, spc, inl + 1,
|
2014-10-05 17:35:18 +08:00
|
|
|
hlen - spc);
|
2014-10-05 17:35:17 +08:00
|
|
|
if (shinfo->nr_frags)
|
2014-10-05 17:35:18 +08:00
|
|
|
memcpy(((void *)(inl + 1)) + hlen - spc,
|
2014-10-05 17:35:17 +08:00
|
|
|
fragptr,
|
|
|
|
skb_frag_size(&shinfo->frags[0]));
|
2008-10-23 06:47:49 +08:00
|
|
|
}
|
|
|
|
|
2015-04-09 09:49:36 +08:00
|
|
|
dma_wmb();
|
2008-10-23 06:47:49 +08:00
|
|
|
inl->byte_count = cpu_to_be32(1 << 31 | (skb->len - spc));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2014-01-10 16:18:26 +08:00
|
|
|
u16 mlx4_en_select_queue(struct net_device *dev, struct sk_buff *skb,
|
2014-02-16 22:55:20 +08:00
|
|
|
void *accel_priv, select_queue_fallback_t fallback)
|
2008-10-23 06:47:49 +08:00
|
|
|
{
|
2012-05-17 08:58:10 +08:00
|
|
|
struct mlx4_en_priv *priv = netdev_priv(dev);
|
2012-12-02 11:49:23 +08:00
|
|
|
u16 rings_p_up = priv->num_tx_rings_p_up;
|
2012-05-17 08:58:10 +08:00
|
|
|
u8 up = 0;
|
2008-10-23 06:47:49 +08:00
|
|
|
|
2012-05-17 08:58:10 +08:00
|
|
|
if (dev->num_tc)
|
|
|
|
return skb_tx_hash(dev, skb);
|
|
|
|
|
2015-01-14 00:13:44 +08:00
|
|
|
if (skb_vlan_tag_present(skb))
|
|
|
|
up = skb_vlan_tag_get(skb) >> VLAN_PRIO_SHIFT;
|
2009-06-02 07:24:07 +08:00
|
|
|
|
2014-02-16 22:55:20 +08:00
|
|
|
return fallback(dev, skb) % rings_p_up + up * rings_p_up;
|
2008-10-23 06:47:49 +08:00
|
|
|
}
|
|
|
|
|
2014-10-05 17:35:09 +08:00
|
|
|
static void mlx4_bf_copy(void __iomem *dst, const void *src,
|
|
|
|
unsigned int bytecnt)
|
2011-03-23 06:38:52 +08:00
|
|
|
{
|
|
|
|
__iowrite64_copy(dst, src, bytecnt / 8);
|
|
|
|
}
|
|
|
|
|
2009-09-01 03:50:58 +08:00
|
|
|
netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_device *dev)
|
2008-10-23 06:47:49 +08:00
|
|
|
{
|
2014-10-05 17:35:17 +08:00
|
|
|
struct skb_shared_info *shinfo = skb_shinfo(skb);
|
2008-10-23 06:47:49 +08:00
|
|
|
struct mlx4_en_priv *priv = netdev_priv(dev);
|
2013-08-21 15:08:58 +08:00
|
|
|
struct device *ddev = priv->ddev;
|
2008-10-23 06:47:49 +08:00
|
|
|
struct mlx4_en_tx_ring *ring;
|
|
|
|
struct mlx4_en_tx_desc *tx_desc;
|
|
|
|
struct mlx4_wqe_data_seg *data;
|
|
|
|
struct mlx4_en_tx_info *tx_info;
|
|
|
|
int tx_ind = 0;
|
|
|
|
int nr_txbb;
|
|
|
|
int desc_size;
|
|
|
|
int real_size;
|
2011-03-23 06:38:52 +08:00
|
|
|
u32 index, bf_index;
|
2008-10-23 06:47:49 +08:00
|
|
|
__be32 op_own;
|
2009-06-02 07:24:07 +08:00
|
|
|
u16 vlan_tag = 0;
|
2015-07-27 19:46:34 +08:00
|
|
|
u16 vlan_proto = 0;
|
2014-10-05 17:35:17 +08:00
|
|
|
int i_frag;
|
2008-10-23 06:47:49 +08:00
|
|
|
int lso_header_size;
|
2014-10-05 17:35:20 +08:00
|
|
|
void *fragptr = NULL;
|
2011-03-23 06:38:52 +08:00
|
|
|
bool bounce = false;
|
2014-09-25 22:17:49 +08:00
|
|
|
bool send_doorbell;
|
2014-10-07 00:30:35 +08:00
|
|
|
bool stop_queue;
|
2014-10-05 17:35:20 +08:00
|
|
|
bool inline_ok;
|
2014-10-05 17:35:16 +08:00
|
|
|
u32 ring_cons;
|
2008-10-23 06:47:49 +08:00
|
|
|
|
2010-08-24 11:46:07 +08:00
|
|
|
if (!priv->port_up)
|
|
|
|
goto tx_drop;
|
|
|
|
|
2014-10-05 17:35:16 +08:00
|
|
|
tx_ind = skb_get_queue_mapping(skb);
|
|
|
|
ring = priv->tx_ring[tx_ind];
|
|
|
|
|
|
|
|
/* fetch ring->cons far ahead before needing it to avoid stall */
|
|
|
|
ring_cons = ACCESS_ONCE(ring->cons);
|
|
|
|
|
2014-10-05 17:35:20 +08:00
|
|
|
real_size = get_real_size(skb, shinfo, dev, &lso_header_size,
|
|
|
|
&inline_ok, &fragptr);
|
2008-10-23 06:47:49 +08:00
|
|
|
if (unlikely(!real_size))
|
2009-06-21 06:15:31 +08:00
|
|
|
goto tx_drop;
|
2008-10-23 06:47:49 +08:00
|
|
|
|
2011-03-31 09:57:33 +08:00
|
|
|
/* Align descriptor to TXBB size */
|
2008-10-23 06:47:49 +08:00
|
|
|
desc_size = ALIGN(real_size, TXBB_SIZE);
|
|
|
|
nr_txbb = desc_size / TXBB_SIZE;
|
|
|
|
if (unlikely(nr_txbb > MAX_DESC_TXBBS)) {
|
|
|
|
if (netif_msg_tx_err(priv))
|
2009-06-02 04:27:13 +08:00
|
|
|
en_warn(priv, "Oversized header or SG list\n");
|
2009-06-21 06:15:31 +08:00
|
|
|
goto tx_drop;
|
2008-10-23 06:47:49 +08:00
|
|
|
}
|
|
|
|
|
2015-07-27 19:46:34 +08:00
|
|
|
if (skb_vlan_tag_present(skb)) {
|
2015-01-14 00:13:44 +08:00
|
|
|
vlan_tag = skb_vlan_tag_get(skb);
|
2015-07-27 19:46:34 +08:00
|
|
|
vlan_proto = be16_to_cpu(skb->vlan_proto);
|
|
|
|
}
|
2008-10-23 06:47:49 +08:00
|
|
|
|
2014-10-08 23:19:27 +08:00
|
|
|
netdev_txq_bql_enqueue_prefetchw(ring->tx_queue);
|
2014-10-05 17:35:14 +08:00
|
|
|
|
2008-10-23 06:47:49 +08:00
|
|
|
/* Track current inflight packets for performance analysis */
|
|
|
|
AVG_PERF_COUNTER(priv->pstats.inflight_avg,
|
2014-10-05 17:35:16 +08:00
|
|
|
(u32)(ring->prod - ring_cons - 1));
|
2008-10-23 06:47:49 +08:00
|
|
|
|
|
|
|
/* Packet is good - grab an index and transmit it */
|
|
|
|
index = ring->prod & ring->size_mask;
|
2011-03-23 06:38:52 +08:00
|
|
|
bf_index = ring->prod;
|
2008-10-23 06:47:49 +08:00
|
|
|
|
|
|
|
/* See if we have enough space for whole descriptor TXBB for setting
|
|
|
|
* SW ownership on next descriptor; if not, use a bounce buffer. */
|
|
|
|
if (likely(index + nr_txbb <= ring->size))
|
|
|
|
tx_desc = ring->buf + index * TXBB_SIZE;
|
2011-03-23 06:38:52 +08:00
|
|
|
else {
|
2008-10-23 06:47:49 +08:00
|
|
|
tx_desc = (struct mlx4_en_tx_desc *) ring->bounce_buf;
|
2011-03-23 06:38:52 +08:00
|
|
|
bounce = true;
|
|
|
|
}
|
2008-10-23 06:47:49 +08:00
|
|
|
|
|
|
|
/* Save skb in tx_info ring */
|
|
|
|
tx_info = &ring->tx_info[index];
|
|
|
|
tx_info->skb = skb;
|
|
|
|
tx_info->nr_txbb = nr_txbb;
|
|
|
|
|
2014-10-05 17:35:09 +08:00
|
|
|
data = &tx_desc->data;
|
2013-08-21 15:08:58 +08:00
|
|
|
if (lso_header_size)
|
|
|
|
data = ((void *)&tx_desc->lso + ALIGN(lso_header_size + 4,
|
|
|
|
DS_SIZE));
|
|
|
|
|
|
|
|
/* valid only for none inline segments */
|
|
|
|
tx_info->data_offset = (void *)data - (void *)tx_desc;
|
|
|
|
|
2014-10-05 17:35:20 +08:00
|
|
|
tx_info->inl = inline_ok;
|
|
|
|
|
2013-08-21 15:08:58 +08:00
|
|
|
tx_info->linear = (lso_header_size < skb_headlen(skb) &&
|
2014-10-05 17:35:20 +08:00
|
|
|
!inline_ok) ? 1 : 0;
|
2013-08-21 15:08:58 +08:00
|
|
|
|
2014-10-05 17:35:17 +08:00
|
|
|
tx_info->nr_maps = shinfo->nr_frags + tx_info->linear;
|
2014-10-05 17:35:13 +08:00
|
|
|
data += tx_info->nr_maps - 1;
|
2013-08-21 15:08:58 +08:00
|
|
|
|
2014-10-05 17:35:20 +08:00
|
|
|
if (!tx_info->inl) {
|
2014-10-05 17:35:13 +08:00
|
|
|
dma_addr_t dma = 0;
|
|
|
|
u32 byte_count = 0;
|
|
|
|
|
2014-10-05 17:35:09 +08:00
|
|
|
/* Map fragments if any */
|
2014-10-05 17:35:17 +08:00
|
|
|
for (i_frag = shinfo->nr_frags - 1; i_frag >= 0; i_frag--) {
|
2014-10-05 17:35:09 +08:00
|
|
|
const struct skb_frag_struct *frag;
|
2014-10-05 17:35:17 +08:00
|
|
|
|
|
|
|
frag = &shinfo->frags[i_frag];
|
2014-10-05 17:35:13 +08:00
|
|
|
byte_count = skb_frag_size(frag);
|
2013-08-21 15:08:58 +08:00
|
|
|
dma = skb_frag_dma_map(ddev, frag,
|
2014-10-05 17:35:13 +08:00
|
|
|
0, byte_count,
|
2013-08-21 15:08:58 +08:00
|
|
|
DMA_TO_DEVICE);
|
|
|
|
if (dma_mapping_error(ddev, dma))
|
|
|
|
goto tx_drop_unmap;
|
|
|
|
|
|
|
|
data->addr = cpu_to_be64(dma);
|
2014-10-05 17:35:11 +08:00
|
|
|
data->lkey = ring->mr_key;
|
2015-04-09 09:49:36 +08:00
|
|
|
dma_wmb();
|
2014-10-05 17:35:13 +08:00
|
|
|
data->byte_count = cpu_to_be32(byte_count);
|
2013-08-21 15:08:58 +08:00
|
|
|
--data;
|
|
|
|
}
|
|
|
|
|
2014-10-05 17:35:09 +08:00
|
|
|
/* Map linear part if needed */
|
2013-08-21 15:08:58 +08:00
|
|
|
if (tx_info->linear) {
|
2014-10-05 17:35:13 +08:00
|
|
|
byte_count = skb_headlen(skb) - lso_header_size;
|
2013-08-21 15:08:59 +08:00
|
|
|
|
2013-08-21 15:08:58 +08:00
|
|
|
dma = dma_map_single(ddev, skb->data +
|
|
|
|
lso_header_size, byte_count,
|
|
|
|
PCI_DMA_TODEVICE);
|
|
|
|
if (dma_mapping_error(ddev, dma))
|
|
|
|
goto tx_drop_unmap;
|
|
|
|
|
|
|
|
data->addr = cpu_to_be64(dma);
|
2014-10-05 17:35:11 +08:00
|
|
|
data->lkey = ring->mr_key;
|
2015-04-09 09:49:36 +08:00
|
|
|
dma_wmb();
|
2013-08-21 15:08:58 +08:00
|
|
|
data->byte_count = cpu_to_be32(byte_count);
|
|
|
|
}
|
2014-10-05 17:35:13 +08:00
|
|
|
/* tx completion can avoid cache line miss for common cases */
|
|
|
|
tx_info->map0_dma = dma;
|
|
|
|
tx_info->map0_byte_count = byte_count;
|
2013-08-21 15:08:58 +08:00
|
|
|
}
|
|
|
|
|
2013-04-23 14:06:49 +08:00
|
|
|
/*
|
|
|
|
* For timestamping add flag to skb_shinfo and
|
|
|
|
* set flag for further reference
|
|
|
|
*/
|
2014-10-05 17:35:19 +08:00
|
|
|
tx_info->ts_requested = 0;
|
2014-10-05 17:35:09 +08:00
|
|
|
if (unlikely(ring->hwtstamp_tx_type == HWTSTAMP_TX_ON &&
|
|
|
|
shinfo->tx_flags & SKBTX_HW_TSTAMP)) {
|
|
|
|
shinfo->tx_flags |= SKBTX_IN_PROGRESS;
|
2013-04-23 14:06:49 +08:00
|
|
|
tx_info->ts_requested = 1;
|
|
|
|
}
|
|
|
|
|
2008-10-23 06:47:49 +08:00
|
|
|
/* Prepare ctrl segement apart opcode+ownership, which depends on
|
|
|
|
* whether LSO is used */
|
2011-11-27 03:55:19 +08:00
|
|
|
tx_desc->ctrl.srcrb_flags = priv->ctrl_flags;
|
2008-10-23 06:47:49 +08:00
|
|
|
if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) {
|
2014-10-30 21:59:27 +08:00
|
|
|
if (!skb->encapsulation)
|
|
|
|
tx_desc->ctrl.srcrb_flags |= cpu_to_be32(MLX4_WQE_CTRL_IP_CSUM |
|
|
|
|
MLX4_WQE_CTRL_TCP_UDP_CSUM);
|
|
|
|
else
|
|
|
|
tx_desc->ctrl.srcrb_flags |= cpu_to_be32(MLX4_WQE_CTRL_IP_CSUM);
|
2011-10-18 09:50:56 +08:00
|
|
|
ring->tx_csum++;
|
2008-10-23 06:47:49 +08:00
|
|
|
}
|
|
|
|
|
2013-02-07 10:25:19 +08:00
|
|
|
if (priv->flags & MLX4_EN_FLAG_ENABLE_HW_LOOPBACK) {
|
2013-08-21 15:08:59 +08:00
|
|
|
struct ethhdr *ethh;
|
|
|
|
|
2013-01-17 13:30:42 +08:00
|
|
|
/* Copy dst mac address to wqe. This allows loopback in eSwitch,
|
|
|
|
* so that VFs and PF can communicate with each other
|
|
|
|
*/
|
|
|
|
ethh = (struct ethhdr *)skb->data;
|
|
|
|
tx_desc->ctrl.srcrb_flags16[0] = get_unaligned((__be16 *)ethh->h_dest);
|
|
|
|
tx_desc->ctrl.imm = get_unaligned((__be32 *)(ethh->h_dest + 2));
|
|
|
|
}
|
|
|
|
|
2008-10-23 06:47:49 +08:00
|
|
|
/* Handle LSO (TSO) packets */
|
|
|
|
if (lso_header_size) {
|
2014-10-05 17:35:17 +08:00
|
|
|
int i;
|
|
|
|
|
2008-10-23 06:47:49 +08:00
|
|
|
/* Mark opcode as LSO */
|
|
|
|
op_own = cpu_to_be32(MLX4_OPCODE_LSO | (1 << 6)) |
|
|
|
|
((ring->prod & ring->size) ?
|
|
|
|
cpu_to_be32(MLX4_EN_BIT_DESC_OWN) : 0);
|
|
|
|
|
|
|
|
/* Fill in the LSO prefix */
|
|
|
|
tx_desc->lso.mss_hdr_size = cpu_to_be32(
|
2014-10-05 17:35:17 +08:00
|
|
|
shinfo->gso_size << 16 | lso_header_size);
|
2008-10-23 06:47:49 +08:00
|
|
|
|
|
|
|
/* Copy headers;
|
|
|
|
* note that we already verified that it is linear */
|
|
|
|
memcpy(tx_desc->lso.header, skb->data, lso_header_size);
|
|
|
|
|
2014-10-02 23:24:21 +08:00
|
|
|
ring->tso_packets++;
|
2014-10-05 17:35:17 +08:00
|
|
|
|
|
|
|
i = ((skb->len - lso_header_size) / shinfo->gso_size) +
|
|
|
|
!!((skb->len - lso_header_size) % shinfo->gso_size);
|
2012-04-23 10:18:50 +08:00
|
|
|
tx_info->nr_bytes = skb->len + (i - 1) * lso_header_size;
|
2008-10-23 06:47:49 +08:00
|
|
|
ring->packets += i;
|
|
|
|
} else {
|
|
|
|
/* Normal (Non LSO) packet */
|
|
|
|
op_own = cpu_to_be32(MLX4_OPCODE_SEND) |
|
|
|
|
((ring->prod & ring->size) ?
|
|
|
|
cpu_to_be32(MLX4_EN_BIT_DESC_OWN) : 0);
|
2012-04-23 10:18:50 +08:00
|
|
|
tx_info->nr_bytes = max_t(unsigned int, skb->len, ETH_ZLEN);
|
2008-10-23 06:47:49 +08:00
|
|
|
ring->packets++;
|
|
|
|
}
|
2012-04-23 10:18:50 +08:00
|
|
|
ring->bytes += tx_info->nr_bytes;
|
|
|
|
netdev_tx_sent_queue(ring->tx_queue, tx_info->nr_bytes);
|
2008-10-23 06:47:49 +08:00
|
|
|
AVG_PERF_COUNTER(priv->pstats.tx_pktsz_avg, skb->len);
|
|
|
|
|
2014-10-05 17:35:20 +08:00
|
|
|
if (tx_info->inl)
|
2014-10-05 17:35:17 +08:00
|
|
|
build_inline_wqe(tx_desc, skb, shinfo, real_size, &vlan_tag,
|
|
|
|
tx_ind, fragptr);
|
2008-10-23 06:47:49 +08:00
|
|
|
|
2013-12-23 22:09:44 +08:00
|
|
|
if (skb->encapsulation) {
|
|
|
|
struct iphdr *ipv4 = (struct iphdr *)skb_inner_network_header(skb);
|
|
|
|
if (ipv4->protocol == IPPROTO_TCP || ipv4->protocol == IPPROTO_UDP)
|
|
|
|
op_own |= cpu_to_be32(MLX4_WQE_CTRL_IIP | MLX4_WQE_CTRL_ILP);
|
|
|
|
else
|
|
|
|
op_own |= cpu_to_be32(MLX4_WQE_CTRL_IIP);
|
|
|
|
}
|
|
|
|
|
2008-10-23 06:47:49 +08:00
|
|
|
ring->prod += nr_txbb;
|
|
|
|
|
|
|
|
/* If we used a bounce buffer then copy descriptor back into place */
|
2014-10-05 17:35:09 +08:00
|
|
|
if (unlikely(bounce))
|
2008-10-23 06:47:49 +08:00
|
|
|
tx_desc = mlx4_en_bounce_to_desc(priv, ring, index, desc_size);
|
|
|
|
|
2013-04-23 14:06:50 +08:00
|
|
|
skb_tx_timestamp(skb);
|
|
|
|
|
2014-10-07 00:30:35 +08:00
|
|
|
/* Check available TXBBs And 2K spare for prefetch */
|
2015-06-25 16:29:42 +08:00
|
|
|
stop_queue = mlx4_en_is_tx_ring_full(ring);
|
2014-10-07 00:30:35 +08:00
|
|
|
if (unlikely(stop_queue)) {
|
|
|
|
netif_tx_stop_queue(ring->tx_queue);
|
|
|
|
ring->queue_stopped++;
|
|
|
|
}
|
2014-09-25 22:17:49 +08:00
|
|
|
send_doorbell = !skb->xmit_more || netif_xmit_stopped(ring->tx_queue);
|
|
|
|
|
2014-10-05 17:35:11 +08:00
|
|
|
real_size = (real_size / 16) & 0x3f;
|
|
|
|
|
2014-09-25 22:17:49 +08:00
|
|
|
if (ring->bf_enabled && desc_size <= MAX_BF && !bounce &&
|
2015-01-14 00:13:44 +08:00
|
|
|
!skb_vlan_tag_present(skb) && send_doorbell) {
|
2014-10-05 17:35:11 +08:00
|
|
|
tx_desc->ctrl.bf_qpn = ring->doorbell_qpn |
|
|
|
|
cpu_to_be32(real_size);
|
2014-03-02 16:25:04 +08:00
|
|
|
|
2011-03-23 06:38:52 +08:00
|
|
|
op_own |= htonl((bf_index & 0xffff) << 8);
|
2014-09-25 22:17:49 +08:00
|
|
|
/* Ensure new descriptor hits memory
|
|
|
|
* before setting ownership of this descriptor to HW
|
|
|
|
*/
|
2015-04-09 09:49:36 +08:00
|
|
|
dma_wmb();
|
2011-03-23 06:38:52 +08:00
|
|
|
tx_desc->ctrl.owner_opcode = op_own;
|
2008-10-23 06:47:49 +08:00
|
|
|
|
2011-03-23 06:38:52 +08:00
|
|
|
wmb();
|
|
|
|
|
2014-10-05 17:35:09 +08:00
|
|
|
mlx4_bf_copy(ring->bf.reg + ring->bf.offset, &tx_desc->ctrl,
|
|
|
|
desc_size);
|
2011-03-23 06:38:52 +08:00
|
|
|
|
|
|
|
wmb();
|
|
|
|
|
|
|
|
ring->bf.offset ^= ring->bf.buf_size;
|
|
|
|
} else {
|
2014-10-05 17:35:09 +08:00
|
|
|
tx_desc->ctrl.vlan_tag = cpu_to_be16(vlan_tag);
|
2015-07-27 19:46:34 +08:00
|
|
|
if (vlan_proto == ETH_P_8021AD)
|
|
|
|
tx_desc->ctrl.ins_vlan = MLX4_WQE_CTRL_INS_SVLAN;
|
|
|
|
else if (vlan_proto == ETH_P_8021Q)
|
|
|
|
tx_desc->ctrl.ins_vlan = MLX4_WQE_CTRL_INS_CVLAN;
|
2015-10-27 23:36:19 +08:00
|
|
|
else
|
|
|
|
tx_desc->ctrl.ins_vlan = 0;
|
2015-07-27 19:46:34 +08:00
|
|
|
|
2014-10-05 17:35:09 +08:00
|
|
|
tx_desc->ctrl.fence_size = real_size;
|
|
|
|
|
2014-09-25 22:17:49 +08:00
|
|
|
/* Ensure new descriptor hits memory
|
|
|
|
* before setting ownership of this descriptor to HW
|
|
|
|
*/
|
2015-04-09 09:49:36 +08:00
|
|
|
dma_wmb();
|
2011-03-23 06:38:52 +08:00
|
|
|
tx_desc->ctrl.owner_opcode = op_own;
|
2014-09-25 22:17:49 +08:00
|
|
|
if (send_doorbell) {
|
|
|
|
wmb();
|
2014-12-22 16:21:57 +08:00
|
|
|
/* Since there is no iowrite*_native() that writes the
|
|
|
|
* value as is, without byteswapping - using the one
|
|
|
|
* the doesn't do byteswapping in the relevant arch
|
|
|
|
* endianness.
|
|
|
|
*/
|
|
|
|
#if defined(__LITTLE_ENDIAN)
|
|
|
|
iowrite32(
|
|
|
|
#else
|
|
|
|
iowrite32be(
|
|
|
|
#endif
|
|
|
|
ring->doorbell_qpn,
|
2014-10-05 17:35:11 +08:00
|
|
|
ring->bf.uar->map + MLX4_SEND_DOORBELL);
|
2014-10-02 23:24:21 +08:00
|
|
|
} else {
|
|
|
|
ring->xmit_more++;
|
2014-09-25 22:17:49 +08:00
|
|
|
}
|
2011-03-23 06:38:52 +08:00
|
|
|
}
|
2008-10-23 06:47:49 +08:00
|
|
|
|
2014-10-07 00:30:35 +08:00
|
|
|
if (unlikely(stop_queue)) {
|
|
|
|
/* If queue was emptied after the if (stop_queue) , and before
|
|
|
|
* the netif_tx_stop_queue() - need to wake the queue,
|
|
|
|
* or else it will remain stopped forever.
|
|
|
|
* Need a memory barrier to make sure ring->cons was not
|
|
|
|
* updated before queue was stopped.
|
|
|
|
*/
|
|
|
|
smp_rmb();
|
|
|
|
|
|
|
|
ring_cons = ACCESS_ONCE(ring->cons);
|
2015-06-25 16:29:42 +08:00
|
|
|
if (unlikely(!mlx4_en_is_tx_ring_full(ring))) {
|
2014-10-07 00:30:35 +08:00
|
|
|
netif_tx_wake_queue(ring->tx_queue);
|
|
|
|
ring->wake_queue++;
|
|
|
|
}
|
|
|
|
}
|
2009-07-06 10:23:38 +08:00
|
|
|
return NETDEV_TX_OK;
|
2009-06-21 06:15:31 +08:00
|
|
|
|
2013-08-21 15:08:58 +08:00
|
|
|
tx_drop_unmap:
|
|
|
|
en_err(priv, "DMA mapping error\n");
|
|
|
|
|
2014-10-05 17:35:17 +08:00
|
|
|
while (++i_frag < shinfo->nr_frags) {
|
|
|
|
++data;
|
2013-08-21 15:08:58 +08:00
|
|
|
dma_unmap_page(ddev, (dma_addr_t) be64_to_cpu(data->addr),
|
|
|
|
be32_to_cpu(data->byte_count),
|
|
|
|
PCI_DMA_TODEVICE);
|
|
|
|
}
|
|
|
|
|
2009-06-21 06:15:31 +08:00
|
|
|
tx_drop:
|
|
|
|
dev_kfree_skb_any(skb);
|
|
|
|
priv->stats.tx_dropped++;
|
|
|
|
return NETDEV_TX_OK;
|
2008-10-23 06:47:49 +08:00
|
|
|
}
|
|
|
|
|