2019-05-27 14:55:01 +08:00
|
|
|
// SPDX-License-Identifier: GPL-2.0-or-later
|
2005-04-17 06:20:36 +08:00
|
|
|
/*
|
|
|
|
* Routines having to do with the 'struct sk_buff' memory handlers.
|
|
|
|
*
|
2008-10-14 10:01:08 +08:00
|
|
|
* Authors: Alan Cox <alan@lxorguk.ukuu.org.uk>
|
2005-04-17 06:20:36 +08:00
|
|
|
* Florian La Roche <rzsfl@rz.uni-sb.de>
|
|
|
|
*
|
|
|
|
* Fixes:
|
|
|
|
* Alan Cox : Fixed the worst of the load
|
|
|
|
* balancer bugs.
|
|
|
|
* Dave Platt : Interrupt stacking fix.
|
|
|
|
* Richard Kooijman : Timestamp fixes.
|
|
|
|
* Alan Cox : Changed buffer format.
|
|
|
|
* Alan Cox : destructor hook for AF_UNIX etc.
|
|
|
|
* Linus Torvalds : Better skb_clone.
|
|
|
|
* Alan Cox : Added skb_copy.
|
|
|
|
* Alan Cox : Added all the changed routines Linus
|
|
|
|
* only put in the headers
|
|
|
|
* Ray VanTassle : Fixed --skb->lock in free
|
|
|
|
* Alan Cox : skb_copy copy arp field
|
|
|
|
* Andi Kleen : slabified it.
|
|
|
|
* Robert Olsson : Removed skb_head_pool
|
|
|
|
*
|
|
|
|
* NOTE:
|
|
|
|
* The __skb_ routines should be called with interrupts
|
|
|
|
* disabled, or you better be *real* sure that the operation is atomic
|
|
|
|
* with respect to whatever list is being frobbed (e.g. via lock_sock()
|
|
|
|
* or via disabling bottom half handlers, etc).
|
|
|
|
*/
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The functions in this file will not compile correctly with gcc 2.4.x
|
|
|
|
*/
|
|
|
|
|
2012-05-17 03:58:40 +08:00
|
|
|
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
#include <linux/module.h>
|
|
|
|
#include <linux/types.h>
|
|
|
|
#include <linux/kernel.h>
|
|
|
|
#include <linux/mm.h>
|
|
|
|
#include <linux/interrupt.h>
|
|
|
|
#include <linux/in.h>
|
|
|
|
#include <linux/inet.h>
|
|
|
|
#include <linux/slab.h>
|
2014-01-26 17:58:16 +08:00
|
|
|
#include <linux/tcp.h>
|
|
|
|
#include <linux/udp.h>
|
2016-06-03 02:05:43 +08:00
|
|
|
#include <linux/sctp.h>
|
2005-04-17 06:20:36 +08:00
|
|
|
#include <linux/netdevice.h>
|
|
|
|
#ifdef CONFIG_NET_CLS_ACT
|
|
|
|
#include <net/pkt_sched.h>
|
|
|
|
#endif
|
|
|
|
#include <linux/string.h>
|
|
|
|
#include <linux/skbuff.h>
|
2007-11-07 15:30:13 +08:00
|
|
|
#include <linux/splice.h>
|
2005-04-17 06:20:36 +08:00
|
|
|
#include <linux/cache.h>
|
|
|
|
#include <linux/rtnetlink.h>
|
|
|
|
#include <linux/init.h>
|
2007-04-03 11:19:53 +08:00
|
|
|
#include <linux/scatterlist.h>
|
2009-02-12 13:03:37 +08:00
|
|
|
#include <linux/errqueue.h>
|
2011-05-21 03:50:29 +08:00
|
|
|
#include <linux/prefetch.h>
|
net: Always untag vlan-tagged traffic on input.
Currently the functionality to untag traffic on input resides
as part of the vlan module and is build only when VLAN support
is enabled in the kernel. When VLAN is disabled, the function
vlan_untag() turns into a stub and doesn't really untag the
packets. This seems to create an interesting interaction
between VMs supporting checksum offloading and some network drivers.
There are some drivers that do not allow the user to change
tx-vlan-offload feature of the driver. These drivers also seem
to assume that any VLAN-tagged traffic they transmit will
have the vlan information in the vlan_tci and not in the vlan
header already in the skb. When transmitting skbs that already
have tagged data with partial checksum set, the checksum doesn't
appear to be updated correctly by the card thus resulting in a
failure to establish TCP connections.
The following is a packet trace taken on the receiver where a
sender is a VM with a VLAN configued. The host VM is running on
doest not have VLAN support and the outging interface on the
host is tg3:
10:12:43.503055 52:54:00:ae:42:3f > 28:d2:44:7d:c2:de, ethertype 802.1Q
(0x8100), length 78: vlan 100, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 27243,
offset 0, flags [DF], proto TCP (6), length 60)
10.0.100.1.58545 > 10.0.100.10.ircu-2: Flags [S], cksum 0xdc39 (incorrect
-> 0x48d9), seq 1069378582, win 29200, options [mss 1460,sackOK,TS val
4294837885 ecr 0,nop,wscale 7], length 0
10:12:44.505556 52:54:00:ae:42:3f > 28:d2:44:7d:c2:de, ethertype 802.1Q
(0x8100), length 78: vlan 100, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 27244,
offset 0, flags [DF], proto TCP (6), length 60)
10.0.100.1.58545 > 10.0.100.10.ircu-2: Flags [S], cksum 0xdc39 (incorrect
-> 0x44ee), seq 1069378582, win 29200, options [mss 1460,sackOK,TS val
4294838888 ecr 0,nop,wscale 7], length 0
This connection finally times out.
I've only access to the TG3 hardware in this configuration thus have
only tested this with TG3 driver. There are a lot of other drivers
that do not permit user changes to vlan acceleration features, and
I don't know if they all suffere from a similar issue.
The patch attempt to fix this another way. It moves the vlan header
stipping code out of the vlan module and always builds it into the
kernel network core. This way, even if vlan is not supported on
a virtualizatoin host, the virtual machines running on top of such
host will still work with VLANs enabled.
CC: Patrick McHardy <kaber@trash.net>
CC: Nithin Nayak Sujir <nsujir@broadcom.com>
CC: Michael Chan <mchan@broadcom.com>
CC: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Vladislav Yasevich <vyasevic@redhat.com>
Acked-by: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-09 02:42:13 +08:00
|
|
|
#include <linux/if_vlan.h>
|
2019-07-07 22:01:57 +08:00
|
|
|
#include <linux/mpls.h>
|
2021-02-19 01:31:24 +08:00
|
|
|
#include <linux/kcov.h>
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
#include <net/protocol.h>
|
|
|
|
#include <net/dst.h>
|
|
|
|
#include <net/sock.h>
|
|
|
|
#include <net/checksum.h>
|
2014-01-09 18:02:46 +08:00
|
|
|
#include <net/ip6_checksum.h>
|
2005-04-17 06:20:36 +08:00
|
|
|
#include <net/xfrm.h>
|
2019-07-07 22:01:54 +08:00
|
|
|
#include <net/mpls.h>
|
2020-01-09 23:59:19 +08:00
|
|
|
#include <net/mptcp.h>
|
2021-10-29 11:01:44 +08:00
|
|
|
#include <net/mctp.h>
|
2021-06-08 03:02:38 +08:00
|
|
|
#include <net/page_pool.h>
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2016-12-25 03:46:01 +08:00
|
|
|
#include <linux/uaccess.h>
|
2009-04-15 07:39:12 +08:00
|
|
|
#include <trace/events/skb.h>
|
2012-04-05 17:35:15 +08:00
|
|
|
#include <linux/highmem.h>
|
2015-01-31 02:29:32 +08:00
|
|
|
#include <linux/capability.h>
|
|
|
|
#include <linux/user_namespace.h>
|
2019-05-29 23:13:48 +08:00
|
|
|
#include <linux/indirect_call_wrapper.h>
|
2006-10-20 04:08:53 +08:00
|
|
|
|
2019-03-26 00:17:23 +08:00
|
|
|
#include "datagram.h"
|
2021-10-22 18:28:37 +08:00
|
|
|
#include "sock_destructor.h"
|
2019-03-26 00:17:23 +08:00
|
|
|
|
2018-02-25 02:20:33 +08:00
|
|
|
struct kmem_cache *skbuff_head_cache __ro_after_init;
|
|
|
|
static struct kmem_cache *skbuff_fclone_cache __ro_after_init;
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
#ifdef CONFIG_SKB_EXTENSIONS
|
|
|
|
static struct kmem_cache *skbuff_ext_cache __ro_after_init;
|
|
|
|
#endif
|
2016-02-03 16:26:57 +08:00
|
|
|
int sysctl_max_skb_frags __read_mostly = MAX_SKB_FRAGS;
|
|
|
|
EXPORT_SYMBOL(sysctl_max_skb_frags);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
/**
|
2013-02-11 21:30:38 +08:00
|
|
|
* skb_panic - private function for out-of-line support
|
|
|
|
* @skb: buffer
|
|
|
|
* @sz: size
|
|
|
|
* @addr: address
|
2013-02-13 19:20:27 +08:00
|
|
|
* @msg: skb_over_panic or skb_under_panic
|
2005-04-17 06:20:36 +08:00
|
|
|
*
|
2013-02-11 21:30:38 +08:00
|
|
|
* Out-of-line support for skb_put() and skb_push().
|
|
|
|
* Called via the wrapper skb_over_panic() or skb_under_panic().
|
|
|
|
* Keep out of line to prevent kernel bloat.
|
|
|
|
* __builtin_return_address is not used because it is not always reliable.
|
2005-04-17 06:20:36 +08:00
|
|
|
*/
|
2013-02-11 21:30:38 +08:00
|
|
|
static void skb_panic(struct sk_buff *skb, unsigned int sz, void *addr,
|
2013-02-13 19:20:27 +08:00
|
|
|
const char msg[])
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2020-04-28 00:37:43 +08:00
|
|
|
pr_emerg("%s: text:%px len:%d put:%d head:%px data:%px tail:%#lx end:%#lx dev:%s\n",
|
2013-02-13 19:20:27 +08:00
|
|
|
msg, addr, skb->len, sz, skb->head, skb->data,
|
2012-05-17 03:58:40 +08:00
|
|
|
(unsigned long)skb->tail, (unsigned long)skb->end,
|
|
|
|
skb->dev ? skb->dev->name : "<NULL>");
|
2005-04-17 06:20:36 +08:00
|
|
|
BUG();
|
|
|
|
}
|
|
|
|
|
2013-02-11 21:30:38 +08:00
|
|
|
static void skb_over_panic(struct sk_buff *skb, unsigned int sz, void *addr)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2013-02-11 21:30:38 +08:00
|
|
|
skb_panic(skb, sz, addr, __func__);
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
2013-02-11 21:30:38 +08:00
|
|
|
static void skb_under_panic(struct sk_buff *skb, unsigned int sz, void *addr)
|
|
|
|
{
|
|
|
|
skb_panic(skb, sz, addr, __func__);
|
|
|
|
}
|
2012-08-01 07:44:19 +08:00
|
|
|
|
2021-02-13 22:12:13 +08:00
|
|
|
#define NAPI_SKB_CACHE_SIZE 64
|
skbuff: introduce {,__}napi_build_skb() which reuses NAPI cache heads
Instead of just bulk-flushing skbuff_heads queued up through
napi_consume_skb() or __kfree_skb_defer(), try to reuse them
on allocation path.
If the cache is empty on allocation, bulk-allocate the first
16 elements, which is more efficient than per-skb allocation.
If the cache is full on freeing, bulk-wipe the second half of
the cache (32 elements).
This also includes custom KASAN poisoning/unpoisoning to be
double sure there are no use-after-free cases.
To not change current behaviour, introduce a new function,
napi_build_skb(), to optionally use a new approach later
in drivers.
Note on selected bulk size, 16:
- this equals to XDP_BULK_QUEUE_SIZE, DEV_MAP_BULK_SIZE
and especially VETH_XDP_BATCH, which is also used to
bulk-allocate skbuff_heads and was tested on powerful
setups;
- this also showed the best performance in the actual
test series (from the array of {8, 16, 32}).
Suggested-by: Edward Cree <ecree.xilinx@gmail.com> # Divide on two halves
Suggested-by: Eric Dumazet <edumazet@google.com> # KASAN poisoning
Cc: Dmitry Vyukov <dvyukov@google.com> # Help with KASAN
Cc: Paolo Abeni <pabeni@redhat.com> # Reduced batch size
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
2021-02-13 22:12:25 +08:00
|
|
|
#define NAPI_SKB_CACHE_BULK 16
|
|
|
|
#define NAPI_SKB_CACHE_HALF (NAPI_SKB_CACHE_SIZE / 2)
|
2021-02-13 22:12:13 +08:00
|
|
|
|
|
|
|
struct napi_alloc_cache {
|
|
|
|
struct page_frag_cache page;
|
|
|
|
unsigned int skb_count;
|
|
|
|
void *skb_cache[NAPI_SKB_CACHE_SIZE];
|
|
|
|
};
|
|
|
|
|
|
|
|
static DEFINE_PER_CPU(struct page_frag_cache, netdev_alloc_cache);
|
|
|
|
static DEFINE_PER_CPU(struct napi_alloc_cache, napi_alloc_cache);
|
|
|
|
|
2021-09-14 11:49:35 +08:00
|
|
|
void *__napi_alloc_frag_align(unsigned int fragsz, unsigned int align_mask)
|
2021-02-13 22:12:13 +08:00
|
|
|
{
|
|
|
|
struct napi_alloc_cache *nc = this_cpu_ptr(&napi_alloc_cache);
|
|
|
|
|
|
|
|
fragsz = SKB_DATA_ALIGN(fragsz);
|
|
|
|
|
2021-09-14 11:49:35 +08:00
|
|
|
return page_frag_alloc_align(&nc->page, fragsz, GFP_ATOMIC, align_mask);
|
2021-02-13 22:12:13 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(__napi_alloc_frag_align);
|
|
|
|
|
|
|
|
void *__netdev_alloc_frag_align(unsigned int fragsz, unsigned int align_mask)
|
|
|
|
{
|
|
|
|
void *data;
|
|
|
|
|
|
|
|
fragsz = SKB_DATA_ALIGN(fragsz);
|
2021-08-13 22:57:49 +08:00
|
|
|
if (in_hardirq() || irqs_disabled()) {
|
2021-09-14 11:49:35 +08:00
|
|
|
struct page_frag_cache *nc = this_cpu_ptr(&netdev_alloc_cache);
|
|
|
|
|
2021-02-13 22:12:13 +08:00
|
|
|
data = page_frag_alloc_align(nc, fragsz, GFP_ATOMIC, align_mask);
|
|
|
|
} else {
|
2021-09-14 11:49:35 +08:00
|
|
|
struct napi_alloc_cache *nc;
|
|
|
|
|
2021-02-13 22:12:13 +08:00
|
|
|
local_bh_disable();
|
2021-09-14 11:49:35 +08:00
|
|
|
nc = this_cpu_ptr(&napi_alloc_cache);
|
|
|
|
data = page_frag_alloc_align(&nc->page, fragsz, GFP_ATOMIC, align_mask);
|
2021-02-13 22:12:13 +08:00
|
|
|
local_bh_enable();
|
|
|
|
}
|
|
|
|
return data;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(__netdev_alloc_frag_align);
|
|
|
|
|
skbuff: introduce {,__}napi_build_skb() which reuses NAPI cache heads
Instead of just bulk-flushing skbuff_heads queued up through
napi_consume_skb() or __kfree_skb_defer(), try to reuse them
on allocation path.
If the cache is empty on allocation, bulk-allocate the first
16 elements, which is more efficient than per-skb allocation.
If the cache is full on freeing, bulk-wipe the second half of
the cache (32 elements).
This also includes custom KASAN poisoning/unpoisoning to be
double sure there are no use-after-free cases.
To not change current behaviour, introduce a new function,
napi_build_skb(), to optionally use a new approach later
in drivers.
Note on selected bulk size, 16:
- this equals to XDP_BULK_QUEUE_SIZE, DEV_MAP_BULK_SIZE
and especially VETH_XDP_BATCH, which is also used to
bulk-allocate skbuff_heads and was tested on powerful
setups;
- this also showed the best performance in the actual
test series (from the array of {8, 16, 32}).
Suggested-by: Edward Cree <ecree.xilinx@gmail.com> # Divide on two halves
Suggested-by: Eric Dumazet <edumazet@google.com> # KASAN poisoning
Cc: Dmitry Vyukov <dvyukov@google.com> # Help with KASAN
Cc: Paolo Abeni <pabeni@redhat.com> # Reduced batch size
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
2021-02-13 22:12:25 +08:00
|
|
|
static struct sk_buff *napi_skb_cache_get(void)
|
|
|
|
{
|
|
|
|
struct napi_alloc_cache *nc = this_cpu_ptr(&napi_alloc_cache);
|
|
|
|
struct sk_buff *skb;
|
|
|
|
|
|
|
|
if (unlikely(!nc->skb_count))
|
|
|
|
nc->skb_count = kmem_cache_alloc_bulk(skbuff_head_cache,
|
|
|
|
GFP_ATOMIC,
|
|
|
|
NAPI_SKB_CACHE_BULK,
|
|
|
|
nc->skb_cache);
|
|
|
|
if (unlikely(!nc->skb_count))
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
skb = nc->skb_cache[--nc->skb_count];
|
|
|
|
kasan_unpoison_object_data(skbuff_head_cache, skb);
|
|
|
|
|
|
|
|
return skb;
|
|
|
|
}
|
|
|
|
|
2019-04-12 23:07:37 +08:00
|
|
|
/* Caller must provide SKB that is memset cleared */
|
2021-02-13 22:11:26 +08:00
|
|
|
static void __build_skb_around(struct sk_buff *skb, void *data,
|
|
|
|
unsigned int frag_size)
|
2019-04-12 23:07:37 +08:00
|
|
|
{
|
|
|
|
struct skb_shared_info *shinfo;
|
|
|
|
unsigned int size = frag_size ? : ksize(data);
|
|
|
|
|
|
|
|
size -= SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
|
|
|
|
|
|
|
|
/* Assumes caller memset cleared SKB */
|
|
|
|
skb->truesize = SKB_TRUESIZE(size);
|
|
|
|
refcount_set(&skb->users, 1);
|
|
|
|
skb->head = data;
|
|
|
|
skb->data = data;
|
|
|
|
skb_reset_tail_pointer(skb);
|
2022-02-22 11:21:12 +08:00
|
|
|
skb_set_end_offset(skb, size);
|
2019-04-12 23:07:37 +08:00
|
|
|
skb->mac_header = (typeof(skb->mac_header))~0U;
|
|
|
|
skb->transport_header = (typeof(skb->transport_header))~0U;
|
|
|
|
|
|
|
|
/* make sure we initialize shinfo sequentially */
|
|
|
|
shinfo = skb_shinfo(skb);
|
|
|
|
memset(shinfo, 0, offsetof(struct skb_shared_info, dataref));
|
|
|
|
atomic_set(&shinfo->dataref, 1);
|
|
|
|
|
2020-10-30 01:36:19 +08:00
|
|
|
skb_set_kcov_handle(skb, kcov_common_handle());
|
2019-04-12 23:07:37 +08:00
|
|
|
}
|
|
|
|
|
net: introduce build_skb()
One of the thing we discussed during netdev 2011 conference was the idea
to change some network drivers to allocate/populate their skb at RX
completion time, right before feeding the skb to network stack.
In old days, we allocated skbs when populating the RX ring.
This means bringing into cpu cache sk_buff and skb_shared_info cache
lines (since we clear/initialize them), then 'queue' skb->data to NIC.
By the time NIC fills a frame in skb->data buffer and host can process
it, cpu probably threw away the cache lines from its caches, because lot
of things happened between the allocation and final use.
So the deal would be to allocate only the data buffer for the NIC to
populate its RX ring buffer. And use build_skb() at RX completion to
attach a data buffer (now filled with an ethernet frame) to a new skb,
initialize the skb_shared_info portion, and give the hot skb to network
stack.
build_skb() is the function to allocate an skb, caller providing the
data buffer that should be attached to it. Drivers are expected to call
skb_reserve() right after build_skb() to adjust skb->data to the
Ethernet frame (usually skipping NET_SKB_PAD and NET_IP_ALIGN, but some
drivers might add a hardware provided alignment)
Data provided to build_skb() MUST have been allocated by a prior
kmalloc() call, with enough room to add SKB_DATA_ALIGN(sizeof(struct
skb_shared_info)) bytes at the end of the data without corrupting
incoming frame.
data = kmalloc(NET_SKB_PAD + NET_IP_ALIGN + 1536 +
SKB_DATA_ALIGN(sizeof(struct skb_shared_info)),
GFP_ATOMIC);
...
skb = build_skb(data);
if (!skb) {
recycle_data(data);
} else {
skb_reserve(skb, NET_SKB_PAD + NET_IP_ALIGN);
...
}
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
CC: Eilon Greenstein <eilong@broadcom.com>
CC: Ben Hutchings <bhutchings@solarflare.com>
CC: Tom Herbert <therbert@google.com>
CC: Jamal Hadi Salim <hadi@mojatatu.com>
CC: Stephen Hemminger <shemminger@vyatta.com>
CC: Thomas Graf <tgraf@infradead.org>
CC: Herbert Xu <herbert@gondor.apana.org.au>
CC: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2011-11-14 14:03:34 +08:00
|
|
|
/**
|
2015-04-25 07:05:01 +08:00
|
|
|
* __build_skb - build a network buffer
|
net: introduce build_skb()
One of the thing we discussed during netdev 2011 conference was the idea
to change some network drivers to allocate/populate their skb at RX
completion time, right before feeding the skb to network stack.
In old days, we allocated skbs when populating the RX ring.
This means bringing into cpu cache sk_buff and skb_shared_info cache
lines (since we clear/initialize them), then 'queue' skb->data to NIC.
By the time NIC fills a frame in skb->data buffer and host can process
it, cpu probably threw away the cache lines from its caches, because lot
of things happened between the allocation and final use.
So the deal would be to allocate only the data buffer for the NIC to
populate its RX ring buffer. And use build_skb() at RX completion to
attach a data buffer (now filled with an ethernet frame) to a new skb,
initialize the skb_shared_info portion, and give the hot skb to network
stack.
build_skb() is the function to allocate an skb, caller providing the
data buffer that should be attached to it. Drivers are expected to call
skb_reserve() right after build_skb() to adjust skb->data to the
Ethernet frame (usually skipping NET_SKB_PAD and NET_IP_ALIGN, but some
drivers might add a hardware provided alignment)
Data provided to build_skb() MUST have been allocated by a prior
kmalloc() call, with enough room to add SKB_DATA_ALIGN(sizeof(struct
skb_shared_info)) bytes at the end of the data without corrupting
incoming frame.
data = kmalloc(NET_SKB_PAD + NET_IP_ALIGN + 1536 +
SKB_DATA_ALIGN(sizeof(struct skb_shared_info)),
GFP_ATOMIC);
...
skb = build_skb(data);
if (!skb) {
recycle_data(data);
} else {
skb_reserve(skb, NET_SKB_PAD + NET_IP_ALIGN);
...
}
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
CC: Eilon Greenstein <eilong@broadcom.com>
CC: Ben Hutchings <bhutchings@solarflare.com>
CC: Tom Herbert <therbert@google.com>
CC: Jamal Hadi Salim <hadi@mojatatu.com>
CC: Stephen Hemminger <shemminger@vyatta.com>
CC: Thomas Graf <tgraf@infradead.org>
CC: Herbert Xu <herbert@gondor.apana.org.au>
CC: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2011-11-14 14:03:34 +08:00
|
|
|
* @data: data buffer provided by caller
|
2015-04-25 07:05:01 +08:00
|
|
|
* @frag_size: size of data, or 0 if head was kmalloced
|
net: introduce build_skb()
One of the thing we discussed during netdev 2011 conference was the idea
to change some network drivers to allocate/populate their skb at RX
completion time, right before feeding the skb to network stack.
In old days, we allocated skbs when populating the RX ring.
This means bringing into cpu cache sk_buff and skb_shared_info cache
lines (since we clear/initialize them), then 'queue' skb->data to NIC.
By the time NIC fills a frame in skb->data buffer and host can process
it, cpu probably threw away the cache lines from its caches, because lot
of things happened between the allocation and final use.
So the deal would be to allocate only the data buffer for the NIC to
populate its RX ring buffer. And use build_skb() at RX completion to
attach a data buffer (now filled with an ethernet frame) to a new skb,
initialize the skb_shared_info portion, and give the hot skb to network
stack.
build_skb() is the function to allocate an skb, caller providing the
data buffer that should be attached to it. Drivers are expected to call
skb_reserve() right after build_skb() to adjust skb->data to the
Ethernet frame (usually skipping NET_SKB_PAD and NET_IP_ALIGN, but some
drivers might add a hardware provided alignment)
Data provided to build_skb() MUST have been allocated by a prior
kmalloc() call, with enough room to add SKB_DATA_ALIGN(sizeof(struct
skb_shared_info)) bytes at the end of the data without corrupting
incoming frame.
data = kmalloc(NET_SKB_PAD + NET_IP_ALIGN + 1536 +
SKB_DATA_ALIGN(sizeof(struct skb_shared_info)),
GFP_ATOMIC);
...
skb = build_skb(data);
if (!skb) {
recycle_data(data);
} else {
skb_reserve(skb, NET_SKB_PAD + NET_IP_ALIGN);
...
}
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
CC: Eilon Greenstein <eilong@broadcom.com>
CC: Ben Hutchings <bhutchings@solarflare.com>
CC: Tom Herbert <therbert@google.com>
CC: Jamal Hadi Salim <hadi@mojatatu.com>
CC: Stephen Hemminger <shemminger@vyatta.com>
CC: Thomas Graf <tgraf@infradead.org>
CC: Herbert Xu <herbert@gondor.apana.org.au>
CC: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2011-11-14 14:03:34 +08:00
|
|
|
*
|
|
|
|
* Allocate a new &sk_buff. Caller provides space holding head and
|
2013-07-24 03:22:39 +08:00
|
|
|
* skb_shared_info. @data must have been allocated by kmalloc() only if
|
2015-04-25 07:05:01 +08:00
|
|
|
* @frag_size is 0, otherwise data should come from the page allocator
|
|
|
|
* or vmalloc()
|
net: introduce build_skb()
One of the thing we discussed during netdev 2011 conference was the idea
to change some network drivers to allocate/populate their skb at RX
completion time, right before feeding the skb to network stack.
In old days, we allocated skbs when populating the RX ring.
This means bringing into cpu cache sk_buff and skb_shared_info cache
lines (since we clear/initialize them), then 'queue' skb->data to NIC.
By the time NIC fills a frame in skb->data buffer and host can process
it, cpu probably threw away the cache lines from its caches, because lot
of things happened between the allocation and final use.
So the deal would be to allocate only the data buffer for the NIC to
populate its RX ring buffer. And use build_skb() at RX completion to
attach a data buffer (now filled with an ethernet frame) to a new skb,
initialize the skb_shared_info portion, and give the hot skb to network
stack.
build_skb() is the function to allocate an skb, caller providing the
data buffer that should be attached to it. Drivers are expected to call
skb_reserve() right after build_skb() to adjust skb->data to the
Ethernet frame (usually skipping NET_SKB_PAD and NET_IP_ALIGN, but some
drivers might add a hardware provided alignment)
Data provided to build_skb() MUST have been allocated by a prior
kmalloc() call, with enough room to add SKB_DATA_ALIGN(sizeof(struct
skb_shared_info)) bytes at the end of the data without corrupting
incoming frame.
data = kmalloc(NET_SKB_PAD + NET_IP_ALIGN + 1536 +
SKB_DATA_ALIGN(sizeof(struct skb_shared_info)),
GFP_ATOMIC);
...
skb = build_skb(data);
if (!skb) {
recycle_data(data);
} else {
skb_reserve(skb, NET_SKB_PAD + NET_IP_ALIGN);
...
}
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
CC: Eilon Greenstein <eilong@broadcom.com>
CC: Ben Hutchings <bhutchings@solarflare.com>
CC: Tom Herbert <therbert@google.com>
CC: Jamal Hadi Salim <hadi@mojatatu.com>
CC: Stephen Hemminger <shemminger@vyatta.com>
CC: Thomas Graf <tgraf@infradead.org>
CC: Herbert Xu <herbert@gondor.apana.org.au>
CC: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2011-11-14 14:03:34 +08:00
|
|
|
* The return is the new skb buffer.
|
|
|
|
* On a failure the return is %NULL, and @data is not freed.
|
|
|
|
* Notes :
|
|
|
|
* Before IO, driver allocates only data buffer where NIC put incoming frame
|
|
|
|
* Driver should add room at head (NET_SKB_PAD) and
|
|
|
|
* MUST add room at tail (SKB_DATA_ALIGN(skb_shared_info))
|
|
|
|
* After IO, driver calls build_skb(), to allocate sk_buff and populate it
|
|
|
|
* before giving packet to stack.
|
|
|
|
* RX rings only contains data buffers, not full skbs.
|
|
|
|
*/
|
2015-04-25 07:05:01 +08:00
|
|
|
struct sk_buff *__build_skb(void *data, unsigned int frag_size)
|
net: introduce build_skb()
One of the thing we discussed during netdev 2011 conference was the idea
to change some network drivers to allocate/populate their skb at RX
completion time, right before feeding the skb to network stack.
In old days, we allocated skbs when populating the RX ring.
This means bringing into cpu cache sk_buff and skb_shared_info cache
lines (since we clear/initialize them), then 'queue' skb->data to NIC.
By the time NIC fills a frame in skb->data buffer and host can process
it, cpu probably threw away the cache lines from its caches, because lot
of things happened between the allocation and final use.
So the deal would be to allocate only the data buffer for the NIC to
populate its RX ring buffer. And use build_skb() at RX completion to
attach a data buffer (now filled with an ethernet frame) to a new skb,
initialize the skb_shared_info portion, and give the hot skb to network
stack.
build_skb() is the function to allocate an skb, caller providing the
data buffer that should be attached to it. Drivers are expected to call
skb_reserve() right after build_skb() to adjust skb->data to the
Ethernet frame (usually skipping NET_SKB_PAD and NET_IP_ALIGN, but some
drivers might add a hardware provided alignment)
Data provided to build_skb() MUST have been allocated by a prior
kmalloc() call, with enough room to add SKB_DATA_ALIGN(sizeof(struct
skb_shared_info)) bytes at the end of the data without corrupting
incoming frame.
data = kmalloc(NET_SKB_PAD + NET_IP_ALIGN + 1536 +
SKB_DATA_ALIGN(sizeof(struct skb_shared_info)),
GFP_ATOMIC);
...
skb = build_skb(data);
if (!skb) {
recycle_data(data);
} else {
skb_reserve(skb, NET_SKB_PAD + NET_IP_ALIGN);
...
}
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
CC: Eilon Greenstein <eilong@broadcom.com>
CC: Ben Hutchings <bhutchings@solarflare.com>
CC: Tom Herbert <therbert@google.com>
CC: Jamal Hadi Salim <hadi@mojatatu.com>
CC: Stephen Hemminger <shemminger@vyatta.com>
CC: Thomas Graf <tgraf@infradead.org>
CC: Herbert Xu <herbert@gondor.apana.org.au>
CC: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2011-11-14 14:03:34 +08:00
|
|
|
{
|
|
|
|
struct sk_buff *skb;
|
|
|
|
|
|
|
|
skb = kmem_cache_alloc(skbuff_head_cache, GFP_ATOMIC);
|
2019-04-12 23:07:37 +08:00
|
|
|
if (unlikely(!skb))
|
net: introduce build_skb()
One of the thing we discussed during netdev 2011 conference was the idea
to change some network drivers to allocate/populate their skb at RX
completion time, right before feeding the skb to network stack.
In old days, we allocated skbs when populating the RX ring.
This means bringing into cpu cache sk_buff and skb_shared_info cache
lines (since we clear/initialize them), then 'queue' skb->data to NIC.
By the time NIC fills a frame in skb->data buffer and host can process
it, cpu probably threw away the cache lines from its caches, because lot
of things happened between the allocation and final use.
So the deal would be to allocate only the data buffer for the NIC to
populate its RX ring buffer. And use build_skb() at RX completion to
attach a data buffer (now filled with an ethernet frame) to a new skb,
initialize the skb_shared_info portion, and give the hot skb to network
stack.
build_skb() is the function to allocate an skb, caller providing the
data buffer that should be attached to it. Drivers are expected to call
skb_reserve() right after build_skb() to adjust skb->data to the
Ethernet frame (usually skipping NET_SKB_PAD and NET_IP_ALIGN, but some
drivers might add a hardware provided alignment)
Data provided to build_skb() MUST have been allocated by a prior
kmalloc() call, with enough room to add SKB_DATA_ALIGN(sizeof(struct
skb_shared_info)) bytes at the end of the data without corrupting
incoming frame.
data = kmalloc(NET_SKB_PAD + NET_IP_ALIGN + 1536 +
SKB_DATA_ALIGN(sizeof(struct skb_shared_info)),
GFP_ATOMIC);
...
skb = build_skb(data);
if (!skb) {
recycle_data(data);
} else {
skb_reserve(skb, NET_SKB_PAD + NET_IP_ALIGN);
...
}
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
CC: Eilon Greenstein <eilong@broadcom.com>
CC: Ben Hutchings <bhutchings@solarflare.com>
CC: Tom Herbert <therbert@google.com>
CC: Jamal Hadi Salim <hadi@mojatatu.com>
CC: Stephen Hemminger <shemminger@vyatta.com>
CC: Thomas Graf <tgraf@infradead.org>
CC: Herbert Xu <herbert@gondor.apana.org.au>
CC: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2011-11-14 14:03:34 +08:00
|
|
|
return NULL;
|
|
|
|
|
|
|
|
memset(skb, 0, offsetof(struct sk_buff, tail));
|
2021-02-13 22:11:26 +08:00
|
|
|
__build_skb_around(skb, data, frag_size);
|
net: introduce build_skb()
One of the thing we discussed during netdev 2011 conference was the idea
to change some network drivers to allocate/populate their skb at RX
completion time, right before feeding the skb to network stack.
In old days, we allocated skbs when populating the RX ring.
This means bringing into cpu cache sk_buff and skb_shared_info cache
lines (since we clear/initialize them), then 'queue' skb->data to NIC.
By the time NIC fills a frame in skb->data buffer and host can process
it, cpu probably threw away the cache lines from its caches, because lot
of things happened between the allocation and final use.
So the deal would be to allocate only the data buffer for the NIC to
populate its RX ring buffer. And use build_skb() at RX completion to
attach a data buffer (now filled with an ethernet frame) to a new skb,
initialize the skb_shared_info portion, and give the hot skb to network
stack.
build_skb() is the function to allocate an skb, caller providing the
data buffer that should be attached to it. Drivers are expected to call
skb_reserve() right after build_skb() to adjust skb->data to the
Ethernet frame (usually skipping NET_SKB_PAD and NET_IP_ALIGN, but some
drivers might add a hardware provided alignment)
Data provided to build_skb() MUST have been allocated by a prior
kmalloc() call, with enough room to add SKB_DATA_ALIGN(sizeof(struct
skb_shared_info)) bytes at the end of the data without corrupting
incoming frame.
data = kmalloc(NET_SKB_PAD + NET_IP_ALIGN + 1536 +
SKB_DATA_ALIGN(sizeof(struct skb_shared_info)),
GFP_ATOMIC);
...
skb = build_skb(data);
if (!skb) {
recycle_data(data);
} else {
skb_reserve(skb, NET_SKB_PAD + NET_IP_ALIGN);
...
}
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
CC: Eilon Greenstein <eilong@broadcom.com>
CC: Ben Hutchings <bhutchings@solarflare.com>
CC: Tom Herbert <therbert@google.com>
CC: Jamal Hadi Salim <hadi@mojatatu.com>
CC: Stephen Hemminger <shemminger@vyatta.com>
CC: Thomas Graf <tgraf@infradead.org>
CC: Herbert Xu <herbert@gondor.apana.org.au>
CC: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2011-11-14 14:03:34 +08:00
|
|
|
|
2021-02-13 22:11:26 +08:00
|
|
|
return skb;
|
net: introduce build_skb()
One of the thing we discussed during netdev 2011 conference was the idea
to change some network drivers to allocate/populate their skb at RX
completion time, right before feeding the skb to network stack.
In old days, we allocated skbs when populating the RX ring.
This means bringing into cpu cache sk_buff and skb_shared_info cache
lines (since we clear/initialize them), then 'queue' skb->data to NIC.
By the time NIC fills a frame in skb->data buffer and host can process
it, cpu probably threw away the cache lines from its caches, because lot
of things happened between the allocation and final use.
So the deal would be to allocate only the data buffer for the NIC to
populate its RX ring buffer. And use build_skb() at RX completion to
attach a data buffer (now filled with an ethernet frame) to a new skb,
initialize the skb_shared_info portion, and give the hot skb to network
stack.
build_skb() is the function to allocate an skb, caller providing the
data buffer that should be attached to it. Drivers are expected to call
skb_reserve() right after build_skb() to adjust skb->data to the
Ethernet frame (usually skipping NET_SKB_PAD and NET_IP_ALIGN, but some
drivers might add a hardware provided alignment)
Data provided to build_skb() MUST have been allocated by a prior
kmalloc() call, with enough room to add SKB_DATA_ALIGN(sizeof(struct
skb_shared_info)) bytes at the end of the data without corrupting
incoming frame.
data = kmalloc(NET_SKB_PAD + NET_IP_ALIGN + 1536 +
SKB_DATA_ALIGN(sizeof(struct skb_shared_info)),
GFP_ATOMIC);
...
skb = build_skb(data);
if (!skb) {
recycle_data(data);
} else {
skb_reserve(skb, NET_SKB_PAD + NET_IP_ALIGN);
...
}
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
CC: Eilon Greenstein <eilong@broadcom.com>
CC: Ben Hutchings <bhutchings@solarflare.com>
CC: Tom Herbert <therbert@google.com>
CC: Jamal Hadi Salim <hadi@mojatatu.com>
CC: Stephen Hemminger <shemminger@vyatta.com>
CC: Thomas Graf <tgraf@infradead.org>
CC: Herbert Xu <herbert@gondor.apana.org.au>
CC: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2011-11-14 14:03:34 +08:00
|
|
|
}
|
2015-04-25 07:05:01 +08:00
|
|
|
|
|
|
|
/* build_skb() is wrapper over __build_skb(), that specifically
|
|
|
|
* takes care of skb->head and skb->pfmemalloc
|
|
|
|
* This means that if @frag_size is not zero, then @data must be backed
|
|
|
|
* by a page fragment, not kmalloc() or vmalloc()
|
|
|
|
*/
|
|
|
|
struct sk_buff *build_skb(void *data, unsigned int frag_size)
|
|
|
|
{
|
|
|
|
struct sk_buff *skb = __build_skb(data, frag_size);
|
|
|
|
|
|
|
|
if (skb && frag_size) {
|
|
|
|
skb->head_frag = 1;
|
2015-08-22 05:11:51 +08:00
|
|
|
if (page_is_pfmemalloc(virt_to_head_page(data)))
|
2015-04-25 07:05:01 +08:00
|
|
|
skb->pfmemalloc = 1;
|
|
|
|
}
|
|
|
|
return skb;
|
|
|
|
}
|
net: introduce build_skb()
One of the thing we discussed during netdev 2011 conference was the idea
to change some network drivers to allocate/populate their skb at RX
completion time, right before feeding the skb to network stack.
In old days, we allocated skbs when populating the RX ring.
This means bringing into cpu cache sk_buff and skb_shared_info cache
lines (since we clear/initialize them), then 'queue' skb->data to NIC.
By the time NIC fills a frame in skb->data buffer and host can process
it, cpu probably threw away the cache lines from its caches, because lot
of things happened between the allocation and final use.
So the deal would be to allocate only the data buffer for the NIC to
populate its RX ring buffer. And use build_skb() at RX completion to
attach a data buffer (now filled with an ethernet frame) to a new skb,
initialize the skb_shared_info portion, and give the hot skb to network
stack.
build_skb() is the function to allocate an skb, caller providing the
data buffer that should be attached to it. Drivers are expected to call
skb_reserve() right after build_skb() to adjust skb->data to the
Ethernet frame (usually skipping NET_SKB_PAD and NET_IP_ALIGN, but some
drivers might add a hardware provided alignment)
Data provided to build_skb() MUST have been allocated by a prior
kmalloc() call, with enough room to add SKB_DATA_ALIGN(sizeof(struct
skb_shared_info)) bytes at the end of the data without corrupting
incoming frame.
data = kmalloc(NET_SKB_PAD + NET_IP_ALIGN + 1536 +
SKB_DATA_ALIGN(sizeof(struct skb_shared_info)),
GFP_ATOMIC);
...
skb = build_skb(data);
if (!skb) {
recycle_data(data);
} else {
skb_reserve(skb, NET_SKB_PAD + NET_IP_ALIGN);
...
}
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
CC: Eilon Greenstein <eilong@broadcom.com>
CC: Ben Hutchings <bhutchings@solarflare.com>
CC: Tom Herbert <therbert@google.com>
CC: Jamal Hadi Salim <hadi@mojatatu.com>
CC: Stephen Hemminger <shemminger@vyatta.com>
CC: Thomas Graf <tgraf@infradead.org>
CC: Herbert Xu <herbert@gondor.apana.org.au>
CC: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2011-11-14 14:03:34 +08:00
|
|
|
EXPORT_SYMBOL(build_skb);
|
|
|
|
|
2019-04-12 23:07:37 +08:00
|
|
|
/**
|
|
|
|
* build_skb_around - build a network buffer around provided skb
|
|
|
|
* @skb: sk_buff provide by caller, must be memset cleared
|
|
|
|
* @data: data buffer provided by caller
|
|
|
|
* @frag_size: size of data, or 0 if head was kmalloced
|
|
|
|
*/
|
|
|
|
struct sk_buff *build_skb_around(struct sk_buff *skb,
|
|
|
|
void *data, unsigned int frag_size)
|
|
|
|
{
|
|
|
|
if (unlikely(!skb))
|
|
|
|
return NULL;
|
|
|
|
|
2021-02-13 22:11:26 +08:00
|
|
|
__build_skb_around(skb, data, frag_size);
|
2019-04-12 23:07:37 +08:00
|
|
|
|
2021-02-13 22:11:26 +08:00
|
|
|
if (frag_size) {
|
2019-04-12 23:07:37 +08:00
|
|
|
skb->head_frag = 1;
|
|
|
|
if (page_is_pfmemalloc(virt_to_head_page(data)))
|
|
|
|
skb->pfmemalloc = 1;
|
|
|
|
}
|
|
|
|
return skb;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(build_skb_around);
|
|
|
|
|
skbuff: introduce {,__}napi_build_skb() which reuses NAPI cache heads
Instead of just bulk-flushing skbuff_heads queued up through
napi_consume_skb() or __kfree_skb_defer(), try to reuse them
on allocation path.
If the cache is empty on allocation, bulk-allocate the first
16 elements, which is more efficient than per-skb allocation.
If the cache is full on freeing, bulk-wipe the second half of
the cache (32 elements).
This also includes custom KASAN poisoning/unpoisoning to be
double sure there are no use-after-free cases.
To not change current behaviour, introduce a new function,
napi_build_skb(), to optionally use a new approach later
in drivers.
Note on selected bulk size, 16:
- this equals to XDP_BULK_QUEUE_SIZE, DEV_MAP_BULK_SIZE
and especially VETH_XDP_BATCH, which is also used to
bulk-allocate skbuff_heads and was tested on powerful
setups;
- this also showed the best performance in the actual
test series (from the array of {8, 16, 32}).
Suggested-by: Edward Cree <ecree.xilinx@gmail.com> # Divide on two halves
Suggested-by: Eric Dumazet <edumazet@google.com> # KASAN poisoning
Cc: Dmitry Vyukov <dvyukov@google.com> # Help with KASAN
Cc: Paolo Abeni <pabeni@redhat.com> # Reduced batch size
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
2021-02-13 22:12:25 +08:00
|
|
|
/**
|
|
|
|
* __napi_build_skb - build a network buffer
|
|
|
|
* @data: data buffer provided by caller
|
|
|
|
* @frag_size: size of data, or 0 if head was kmalloced
|
|
|
|
*
|
|
|
|
* Version of __build_skb() that uses NAPI percpu caches to obtain
|
|
|
|
* skbuff_head instead of inplace allocation.
|
|
|
|
*
|
|
|
|
* Returns a new &sk_buff on success, %NULL on allocation failure.
|
|
|
|
*/
|
|
|
|
static struct sk_buff *__napi_build_skb(void *data, unsigned int frag_size)
|
|
|
|
{
|
|
|
|
struct sk_buff *skb;
|
|
|
|
|
|
|
|
skb = napi_skb_cache_get();
|
|
|
|
if (unlikely(!skb))
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
memset(skb, 0, offsetof(struct sk_buff, tail));
|
|
|
|
__build_skb_around(skb, data, frag_size);
|
|
|
|
|
|
|
|
return skb;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* napi_build_skb - build a network buffer
|
|
|
|
* @data: data buffer provided by caller
|
|
|
|
* @frag_size: size of data, or 0 if head was kmalloced
|
|
|
|
*
|
|
|
|
* Version of __napi_build_skb() that takes care of skb->head_frag
|
|
|
|
* and skb->pfmemalloc when the data is a page or page fragment.
|
|
|
|
*
|
|
|
|
* Returns a new &sk_buff on success, %NULL on allocation failure.
|
|
|
|
*/
|
|
|
|
struct sk_buff *napi_build_skb(void *data, unsigned int frag_size)
|
|
|
|
{
|
|
|
|
struct sk_buff *skb = __napi_build_skb(data, frag_size);
|
|
|
|
|
|
|
|
if (likely(skb) && frag_size) {
|
|
|
|
skb->head_frag = 1;
|
|
|
|
skb_propagate_pfmemalloc(virt_to_head_page(data), skb);
|
|
|
|
}
|
|
|
|
|
|
|
|
return skb;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(napi_build_skb);
|
|
|
|
|
2021-02-13 22:11:00 +08:00
|
|
|
/*
|
|
|
|
* kmalloc_reserve is a wrapper around kmalloc_node_track_caller that tells
|
|
|
|
* the caller if emergency pfmemalloc reserves are being used. If it is and
|
|
|
|
* the socket is later found to be SOCK_MEMALLOC then PFMEMALLOC reserves
|
|
|
|
* may be used. Otherwise, the packet data may be discarded until enough
|
|
|
|
* memory is free
|
|
|
|
*/
|
2021-02-13 22:11:11 +08:00
|
|
|
static void *kmalloc_reserve(size_t size, gfp_t flags, int node,
|
|
|
|
bool *pfmemalloc)
|
2021-02-13 22:11:00 +08:00
|
|
|
{
|
|
|
|
void *obj;
|
|
|
|
bool ret_pfmemalloc = false;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Try a regular allocation, when that fails and we're not entitled
|
|
|
|
* to the reserves, fail.
|
|
|
|
*/
|
|
|
|
obj = kmalloc_node_track_caller(size,
|
|
|
|
flags | __GFP_NOMEMALLOC | __GFP_NOWARN,
|
|
|
|
node);
|
|
|
|
if (obj || !(gfp_pfmemalloc_allowed(flags)))
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
/* Try again but now we are using pfmemalloc reserves */
|
|
|
|
ret_pfmemalloc = true;
|
|
|
|
obj = kmalloc_node_track_caller(size, flags, node);
|
|
|
|
|
|
|
|
out:
|
|
|
|
if (pfmemalloc)
|
|
|
|
*pfmemalloc = ret_pfmemalloc;
|
|
|
|
|
|
|
|
return obj;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Allocate a new skbuff. We do this ourselves so we can fill in a few
|
|
|
|
* 'private' fields and also do memory statistics to find all the
|
|
|
|
* [BEEP] leaks.
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
|
|
|
|
/**
|
|
|
|
* __alloc_skb - allocate a network buffer
|
|
|
|
* @size: size to allocate
|
|
|
|
* @gfp_mask: allocation mask
|
|
|
|
* @flags: If SKB_ALLOC_FCLONE is set, allocate from fclone cache
|
|
|
|
* instead of head cache and allocate a cloned (child) skb.
|
|
|
|
* If SKB_ALLOC_RX is set, __GFP_MEMALLOC will be used for
|
|
|
|
* allocations in case the data is required for writeback
|
|
|
|
* @node: numa node to allocate memory on
|
|
|
|
*
|
|
|
|
* Allocate a new &sk_buff. The returned buffer has no headroom and a
|
|
|
|
* tail room of at least size bytes. The object has a reference count
|
|
|
|
* of one. The return is the buffer. On a failure the return is %NULL.
|
|
|
|
*
|
|
|
|
* Buffers may only be allocated from interrupts using a @gfp_mask of
|
|
|
|
* %GFP_ATOMIC.
|
|
|
|
*/
|
|
|
|
struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask,
|
|
|
|
int flags, int node)
|
|
|
|
{
|
|
|
|
struct kmem_cache *cache;
|
|
|
|
struct sk_buff *skb;
|
2021-09-22 14:17:19 +08:00
|
|
|
unsigned int osize;
|
2021-02-13 22:11:00 +08:00
|
|
|
bool pfmemalloc;
|
2021-09-22 14:17:19 +08:00
|
|
|
u8 *data;
|
2021-02-13 22:11:00 +08:00
|
|
|
|
|
|
|
cache = (flags & SKB_ALLOC_FCLONE)
|
|
|
|
? skbuff_fclone_cache : skbuff_head_cache;
|
|
|
|
|
|
|
|
if (sk_memalloc_socks() && (flags & SKB_ALLOC_RX))
|
|
|
|
gfp_mask |= __GFP_MEMALLOC;
|
|
|
|
|
|
|
|
/* Get the HEAD */
|
2021-02-13 22:12:38 +08:00
|
|
|
if ((flags & (SKB_ALLOC_FCLONE | SKB_ALLOC_NAPI)) == SKB_ALLOC_NAPI &&
|
|
|
|
likely(node == NUMA_NO_NODE || node == numa_mem_id()))
|
|
|
|
skb = napi_skb_cache_get();
|
|
|
|
else
|
|
|
|
skb = kmem_cache_alloc_node(cache, gfp_mask & ~GFP_DMA, node);
|
2021-02-13 22:11:39 +08:00
|
|
|
if (unlikely(!skb))
|
|
|
|
return NULL;
|
2021-02-13 22:11:00 +08:00
|
|
|
prefetchw(skb);
|
|
|
|
|
|
|
|
/* We do our best to align skb_shared_info on a separate cache
|
|
|
|
* line. It usually works because kmalloc(X > SMP_CACHE_BYTES) gives
|
|
|
|
* aligned memory blocks, unless SLUB/SLAB debug is enabled.
|
|
|
|
* Both skb->head and skb_shared_info are cache line aligned.
|
|
|
|
*/
|
|
|
|
size = SKB_DATA_ALIGN(size);
|
|
|
|
size += SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
|
|
|
|
data = kmalloc_reserve(size, gfp_mask, node, &pfmemalloc);
|
2021-02-13 22:11:39 +08:00
|
|
|
if (unlikely(!data))
|
2021-02-13 22:11:00 +08:00
|
|
|
goto nodata;
|
|
|
|
/* kmalloc(size) might give us more room than requested.
|
|
|
|
* Put skb_shared_info exactly at the end of allocated zone,
|
|
|
|
* to allow max possible filling before reallocation.
|
|
|
|
*/
|
2021-09-22 14:17:19 +08:00
|
|
|
osize = ksize(data);
|
|
|
|
size = SKB_WITH_OVERHEAD(osize);
|
2021-02-13 22:11:00 +08:00
|
|
|
prefetchw(data + size);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Only clear those fields we need to clear, not those that we will
|
|
|
|
* actually initialise below. Hence, don't put any more fields after
|
|
|
|
* the tail pointer in struct sk_buff!
|
|
|
|
*/
|
|
|
|
memset(skb, 0, offsetof(struct sk_buff, tail));
|
2021-09-22 14:17:19 +08:00
|
|
|
__build_skb_around(skb, data, osize);
|
2021-02-13 22:11:00 +08:00
|
|
|
skb->pfmemalloc = pfmemalloc;
|
|
|
|
|
|
|
|
if (flags & SKB_ALLOC_FCLONE) {
|
|
|
|
struct sk_buff_fclones *fclones;
|
|
|
|
|
|
|
|
fclones = container_of(skb, struct sk_buff_fclones, skb1);
|
|
|
|
|
|
|
|
skb->fclone = SKB_FCLONE_ORIG;
|
|
|
|
refcount_set(&fclones->fclone_ref, 1);
|
|
|
|
|
|
|
|
fclones->skb2.fclone = SKB_FCLONE_CLONE;
|
|
|
|
}
|
|
|
|
|
|
|
|
return skb;
|
2021-02-13 22:11:39 +08:00
|
|
|
|
2021-02-13 22:11:00 +08:00
|
|
|
nodata:
|
|
|
|
kmem_cache_free(cache, skb);
|
2021-02-13 22:11:39 +08:00
|
|
|
return NULL;
|
2021-02-13 22:11:00 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(__alloc_skb);
|
|
|
|
|
2014-12-10 11:40:49 +08:00
|
|
|
/**
|
|
|
|
* __netdev_alloc_skb - allocate an skbuff for rx on a specific device
|
|
|
|
* @dev: network device to receive on
|
2015-08-24 21:56:54 +08:00
|
|
|
* @len: length to allocate
|
2014-12-10 11:40:49 +08:00
|
|
|
* @gfp_mask: get_free_pages mask, passed to alloc_skb
|
|
|
|
*
|
|
|
|
* Allocate a new &sk_buff and assign it a usage count of one. The
|
|
|
|
* buffer has NET_SKB_PAD headroom built in. Users should allocate
|
|
|
|
* the headroom they think they need without accounting for the
|
|
|
|
* built in space. The built in space is used for optimisations.
|
|
|
|
*
|
|
|
|
* %NULL is returned if there is no free memory.
|
|
|
|
*/
|
2015-05-07 12:11:40 +08:00
|
|
|
struct sk_buff *__netdev_alloc_skb(struct net_device *dev, unsigned int len,
|
|
|
|
gfp_t gfp_mask)
|
2014-12-10 11:40:49 +08:00
|
|
|
{
|
2015-05-07 12:11:57 +08:00
|
|
|
struct page_frag_cache *nc;
|
2014-12-10 11:40:49 +08:00
|
|
|
struct sk_buff *skb;
|
2015-05-07 12:11:40 +08:00
|
|
|
bool pfmemalloc;
|
|
|
|
void *data;
|
|
|
|
|
|
|
|
len += NET_SKB_PAD;
|
2014-12-10 11:40:49 +08:00
|
|
|
|
2021-01-15 23:04:40 +08:00
|
|
|
/* If requested length is either too small or too big,
|
|
|
|
* we use kmalloc() for skb->head allocation.
|
|
|
|
*/
|
|
|
|
if (len <= SKB_WITH_OVERHEAD(1024) ||
|
|
|
|
len > SKB_WITH_OVERHEAD(PAGE_SIZE) ||
|
2015-11-07 08:28:21 +08:00
|
|
|
(gfp_mask & (__GFP_DIRECT_RECLAIM | GFP_DMA))) {
|
2015-05-14 04:34:13 +08:00
|
|
|
skb = __alloc_skb(len, gfp_mask, SKB_ALLOC_RX, NUMA_NO_NODE);
|
|
|
|
if (!skb)
|
|
|
|
goto skb_fail;
|
|
|
|
goto skb_success;
|
|
|
|
}
|
2014-12-10 11:40:49 +08:00
|
|
|
|
2015-05-07 12:11:40 +08:00
|
|
|
len += SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
|
|
|
|
len = SKB_DATA_ALIGN(len);
|
|
|
|
|
|
|
|
if (sk_memalloc_socks())
|
|
|
|
gfp_mask |= __GFP_MEMALLOC;
|
|
|
|
|
2021-08-13 22:57:49 +08:00
|
|
|
if (in_hardirq() || irqs_disabled()) {
|
2019-06-08 03:20:35 +08:00
|
|
|
nc = this_cpu_ptr(&netdev_alloc_cache);
|
|
|
|
data = page_frag_alloc(nc, len, gfp_mask);
|
|
|
|
pfmemalloc = nc->pfmemalloc;
|
|
|
|
} else {
|
|
|
|
local_bh_disable();
|
|
|
|
nc = this_cpu_ptr(&napi_alloc_cache.page);
|
|
|
|
data = page_frag_alloc(nc, len, gfp_mask);
|
|
|
|
pfmemalloc = nc->pfmemalloc;
|
|
|
|
local_bh_enable();
|
|
|
|
}
|
2015-05-07 12:11:40 +08:00
|
|
|
|
|
|
|
if (unlikely(!data))
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
skb = __build_skb(data, len);
|
|
|
|
if (unlikely(!skb)) {
|
2015-05-07 12:12:03 +08:00
|
|
|
skb_free_frag(data);
|
2015-05-07 12:11:40 +08:00
|
|
|
return NULL;
|
2006-08-08 07:09:04 +08:00
|
|
|
}
|
2014-12-10 11:40:49 +08:00
|
|
|
|
2015-05-07 12:11:40 +08:00
|
|
|
if (pfmemalloc)
|
|
|
|
skb->pfmemalloc = 1;
|
|
|
|
skb->head_frag = 1;
|
|
|
|
|
2015-05-14 04:34:13 +08:00
|
|
|
skb_success:
|
2015-05-07 12:11:40 +08:00
|
|
|
skb_reserve(skb, NET_SKB_PAD);
|
|
|
|
skb->dev = dev;
|
|
|
|
|
2015-05-14 04:34:13 +08:00
|
|
|
skb_fail:
|
2006-08-01 13:35:23 +08:00
|
|
|
return skb;
|
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(__netdev_alloc_skb);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2014-12-10 11:40:49 +08:00
|
|
|
/**
|
|
|
|
* __napi_alloc_skb - allocate skbuff for rx in a specific NAPI instance
|
|
|
|
* @napi: napi instance this buffer was allocated for
|
2015-08-24 21:56:54 +08:00
|
|
|
* @len: length to allocate
|
2014-12-10 11:40:49 +08:00
|
|
|
* @gfp_mask: get_free_pages mask, passed to alloc_skb and alloc_pages
|
|
|
|
*
|
|
|
|
* Allocate a new sk_buff for use in NAPI receive. This buffer will
|
|
|
|
* attempt to allocate the head from a special reserved region used
|
|
|
|
* only for NAPI Rx allocation. By doing this we can save several
|
|
|
|
* CPU cycles by avoiding having to disable and re-enable IRQs.
|
|
|
|
*
|
|
|
|
* %NULL is returned if there is no free memory.
|
|
|
|
*/
|
2015-05-07 12:11:40 +08:00
|
|
|
struct sk_buff *__napi_alloc_skb(struct napi_struct *napi, unsigned int len,
|
|
|
|
gfp_t gfp_mask)
|
2014-12-10 11:40:49 +08:00
|
|
|
{
|
2021-01-14 00:18:19 +08:00
|
|
|
struct napi_alloc_cache *nc;
|
2014-12-10 11:40:49 +08:00
|
|
|
struct sk_buff *skb;
|
2015-05-07 12:11:40 +08:00
|
|
|
void *data;
|
|
|
|
|
|
|
|
len += NET_SKB_PAD + NET_IP_ALIGN;
|
2014-12-10 11:40:49 +08:00
|
|
|
|
2021-01-14 00:18:19 +08:00
|
|
|
/* If requested length is either too small or too big,
|
|
|
|
* we use kmalloc() for skb->head allocation.
|
|
|
|
*/
|
|
|
|
if (len <= SKB_WITH_OVERHEAD(1024) ||
|
|
|
|
len > SKB_WITH_OVERHEAD(PAGE_SIZE) ||
|
2015-11-07 08:28:21 +08:00
|
|
|
(gfp_mask & (__GFP_DIRECT_RECLAIM | GFP_DMA))) {
|
2021-02-13 22:12:49 +08:00
|
|
|
skb = __alloc_skb(len, gfp_mask, SKB_ALLOC_RX | SKB_ALLOC_NAPI,
|
|
|
|
NUMA_NO_NODE);
|
2015-05-14 04:34:13 +08:00
|
|
|
if (!skb)
|
|
|
|
goto skb_fail;
|
|
|
|
goto skb_success;
|
|
|
|
}
|
2015-05-07 12:11:40 +08:00
|
|
|
|
2021-01-14 00:18:19 +08:00
|
|
|
nc = this_cpu_ptr(&napi_alloc_cache);
|
2015-05-07 12:11:40 +08:00
|
|
|
len += SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
|
|
|
|
len = SKB_DATA_ALIGN(len);
|
|
|
|
|
|
|
|
if (sk_memalloc_socks())
|
|
|
|
gfp_mask |= __GFP_MEMALLOC;
|
2014-12-10 11:40:49 +08:00
|
|
|
|
2017-01-11 08:58:06 +08:00
|
|
|
data = page_frag_alloc(&nc->page, len, gfp_mask);
|
2015-05-07 12:11:40 +08:00
|
|
|
if (unlikely(!data))
|
|
|
|
return NULL;
|
|
|
|
|
2021-02-13 22:12:49 +08:00
|
|
|
skb = __napi_build_skb(data, len);
|
2015-05-07 12:11:40 +08:00
|
|
|
if (unlikely(!skb)) {
|
2015-05-07 12:12:03 +08:00
|
|
|
skb_free_frag(data);
|
2015-05-07 12:11:40 +08:00
|
|
|
return NULL;
|
2014-12-10 11:40:49 +08:00
|
|
|
}
|
|
|
|
|
2016-02-08 20:14:59 +08:00
|
|
|
if (nc->page.pfmemalloc)
|
2015-05-07 12:11:40 +08:00
|
|
|
skb->pfmemalloc = 1;
|
|
|
|
skb->head_frag = 1;
|
|
|
|
|
2015-05-14 04:34:13 +08:00
|
|
|
skb_success:
|
2015-05-07 12:11:40 +08:00
|
|
|
skb_reserve(skb, NET_SKB_PAD + NET_IP_ALIGN);
|
|
|
|
skb->dev = napi->dev;
|
|
|
|
|
2015-05-14 04:34:13 +08:00
|
|
|
skb_fail:
|
2014-12-10 11:40:49 +08:00
|
|
|
return skb;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(__napi_alloc_skb);
|
|
|
|
|
2008-10-08 05:22:33 +08:00
|
|
|
void skb_add_rx_frag(struct sk_buff *skb, int i, struct page *page, int off,
|
2012-03-24 07:59:33 +08:00
|
|
|
int size, unsigned int truesize)
|
2008-10-08 05:22:33 +08:00
|
|
|
{
|
|
|
|
skb_fill_page_desc(skb, i, page, off, size);
|
|
|
|
skb->len += size;
|
|
|
|
skb->data_len += size;
|
2012-03-24 07:59:33 +08:00
|
|
|
skb->truesize += truesize;
|
2008-10-08 05:22:33 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_add_rx_frag);
|
|
|
|
|
2013-11-01 14:07:47 +08:00
|
|
|
void skb_coalesce_rx_frag(struct sk_buff *skb, int i, int size,
|
|
|
|
unsigned int truesize)
|
|
|
|
{
|
|
|
|
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
|
|
|
|
|
|
|
|
skb_frag_size_add(frag, size);
|
|
|
|
skb->len += size;
|
|
|
|
skb->data_len += size;
|
|
|
|
skb->truesize += truesize;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_coalesce_rx_frag);
|
|
|
|
|
2006-07-14 10:26:39 +08:00
|
|
|
static void skb_drop_list(struct sk_buff **listp)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2013-06-24 21:26:00 +08:00
|
|
|
kfree_skb_list(*listp);
|
2006-07-14 10:26:39 +08:00
|
|
|
*listp = NULL;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
2006-07-14 10:26:39 +08:00
|
|
|
static inline void skb_drop_fraglist(struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
skb_drop_list(&skb_shinfo(skb)->frag_list);
|
|
|
|
}
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
static void skb_clone_fraglist(struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
struct sk_buff *list;
|
|
|
|
|
2009-06-09 15:18:59 +08:00
|
|
|
skb_walk_frags(skb, list)
|
2005-04-17 06:20:36 +08:00
|
|
|
skb_get(list);
|
|
|
|
}
|
|
|
|
|
net: allow skb->head to be a page fragment
skb->head is currently allocated from kmalloc(). This is convenient but
has the drawback the data cannot be converted to a page fragment if
needed.
We have three spots were it hurts :
1) GRO aggregation
When a linear skb must be appended to another skb, GRO uses the
frag_list fallback, very inefficient since we keep all struct sk_buff
around. So drivers enabling GRO but delivering linear skbs to network
stack aren't enabling full GRO power.
2) splice(socket -> pipe).
We must copy the linear part to a page fragment.
This kind of defeats splice() purpose (zero copy claim)
3) TCP coalescing.
Recently introduced, this permits to group several contiguous segments
into a single skb. This shortens queue lengths and save kernel memory,
and greatly reduce probabilities of TCP collapses. This coalescing
doesnt work on linear skbs (or we would need to copy data, this would be
too slow)
Given all these issues, the following patch introduces the possibility
of having skb->head be a fragment in itself. We use a new skb flag,
skb->head_frag to carry this information.
build_skb() is changed to accept a frag_size argument. Drivers willing
to provide a page fragment instead of kmalloc() data will set a non zero
value, set to the fragment size.
Then, on situations we need to convert the skb head to a frag in itself,
we can check if skb->head_frag is set and avoid the copies or various
fallbacks we have.
This means drivers currently using frags could be updated to avoid the
current skb->head allocation and reduce their memory footprint (aka skb
truesize). (thats 512 or 1024 bytes saved per skb). This also makes
bpf/netfilter faster since the 'first frag' will be part of skb linear
part, no need to copy data.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Maciej Żenczykowski <maze@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Tom Herbert <therbert@google.com>
Cc: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Cc: Ben Hutchings <bhutchings@solarflare.com>
Cc: Matt Carlson <mcarlson@broadcom.com>
Cc: Michael Chan <mchan@broadcom.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2012-04-27 08:33:38 +08:00
|
|
|
static void skb_free_head(struct sk_buff *skb)
|
|
|
|
{
|
2015-05-07 12:12:03 +08:00
|
|
|
unsigned char *head = skb->head;
|
|
|
|
|
2021-06-08 03:02:38 +08:00
|
|
|
if (skb->head_frag) {
|
|
|
|
if (skb_pp_recycle(skb, head))
|
|
|
|
return;
|
2015-05-07 12:12:03 +08:00
|
|
|
skb_free_frag(head);
|
2021-06-08 03:02:38 +08:00
|
|
|
} else {
|
2015-05-07 12:12:03 +08:00
|
|
|
kfree(head);
|
2021-06-08 03:02:38 +08:00
|
|
|
}
|
net: allow skb->head to be a page fragment
skb->head is currently allocated from kmalloc(). This is convenient but
has the drawback the data cannot be converted to a page fragment if
needed.
We have three spots were it hurts :
1) GRO aggregation
When a linear skb must be appended to another skb, GRO uses the
frag_list fallback, very inefficient since we keep all struct sk_buff
around. So drivers enabling GRO but delivering linear skbs to network
stack aren't enabling full GRO power.
2) splice(socket -> pipe).
We must copy the linear part to a page fragment.
This kind of defeats splice() purpose (zero copy claim)
3) TCP coalescing.
Recently introduced, this permits to group several contiguous segments
into a single skb. This shortens queue lengths and save kernel memory,
and greatly reduce probabilities of TCP collapses. This coalescing
doesnt work on linear skbs (or we would need to copy data, this would be
too slow)
Given all these issues, the following patch introduces the possibility
of having skb->head be a fragment in itself. We use a new skb flag,
skb->head_frag to carry this information.
build_skb() is changed to accept a frag_size argument. Drivers willing
to provide a page fragment instead of kmalloc() data will set a non zero
value, set to the fragment size.
Then, on situations we need to convert the skb head to a frag in itself,
we can check if skb->head_frag is set and avoid the copies or various
fallbacks we have.
This means drivers currently using frags could be updated to avoid the
current skb->head allocation and reduce their memory footprint (aka skb
truesize). (thats 512 or 1024 bytes saved per skb). This also makes
bpf/netfilter faster since the 'first frag' will be part of skb linear
part, no need to copy data.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Maciej Żenczykowski <maze@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Tom Herbert <therbert@google.com>
Cc: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Cc: Ben Hutchings <bhutchings@solarflare.com>
Cc: Matt Carlson <mcarlson@broadcom.com>
Cc: Michael Chan <mchan@broadcom.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2012-04-27 08:33:38 +08:00
|
|
|
}
|
|
|
|
|
2006-06-30 04:02:35 +08:00
|
|
|
static void skb_release_data(struct sk_buff *skb)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2014-09-24 09:39:30 +08:00
|
|
|
struct skb_shared_info *shinfo = skb_shinfo(skb);
|
|
|
|
int i;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2014-09-24 09:39:30 +08:00
|
|
|
if (skb->cloned &&
|
|
|
|
atomic_sub_return(skb->nohdr ? (1 << SKB_DATAREF_SHIFT) + 1 : 1,
|
|
|
|
&shinfo->dataref))
|
2021-07-16 15:02:18 +08:00
|
|
|
goto exit;
|
2011-07-06 20:22:12 +08:00
|
|
|
|
2021-01-07 06:18:36 +08:00
|
|
|
skb_zcopy_clear(skb, true);
|
|
|
|
|
2014-09-24 09:39:30 +08:00
|
|
|
for (i = 0; i < shinfo->nr_frags; i++)
|
2021-06-08 03:02:38 +08:00
|
|
|
__skb_frag_unref(&shinfo->frags[i], skb->pp_recycle);
|
2011-07-06 20:22:12 +08:00
|
|
|
|
2014-09-24 09:39:30 +08:00
|
|
|
if (shinfo->frag_list)
|
|
|
|
kfree_skb_list(shinfo->frag_list);
|
|
|
|
|
|
|
|
skb_free_head(skb);
|
2021-07-16 15:02:18 +08:00
|
|
|
exit:
|
|
|
|
/* When we clone an SKB we copy the reycling bit. The pp_recycle
|
|
|
|
* bit is only set on the head though, so in order to avoid races
|
|
|
|
* while trying to recycle fragments on __skb_frag_unref() we need
|
|
|
|
* to make one SKB responsible for triggering the recycle path.
|
|
|
|
* So disable the recycling bit if an SKB is cloned and we have
|
2022-02-09 23:02:42 +08:00
|
|
|
* additional references to the fragmented part of the SKB.
|
2021-07-16 15:02:18 +08:00
|
|
|
* Eventually the last SKB will have the recycling bit set and it's
|
|
|
|
* dataref set to 0, which will trigger the recycling
|
|
|
|
*/
|
|
|
|
skb->pp_recycle = 0;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Free an skbuff by memory without cleaning the state.
|
|
|
|
*/
|
2007-11-26 23:11:19 +08:00
|
|
|
static void kfree_skbmem(struct sk_buff *skb)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2014-09-30 04:29:15 +08:00
|
|
|
struct sk_buff_fclones *fclones;
|
2005-08-18 05:57:30 +08:00
|
|
|
|
|
|
|
switch (skb->fclone) {
|
|
|
|
case SKB_FCLONE_UNAVAILABLE:
|
|
|
|
kmem_cache_free(skbuff_head_cache, skb);
|
2014-12-04 09:04:39 +08:00
|
|
|
return;
|
2005-08-18 05:57:30 +08:00
|
|
|
|
|
|
|
case SKB_FCLONE_ORIG:
|
2014-09-30 04:29:15 +08:00
|
|
|
fclones = container_of(skb, struct sk_buff_fclones, skb1);
|
2005-08-18 05:57:30 +08:00
|
|
|
|
2014-12-04 09:04:39 +08:00
|
|
|
/* We usually free the clone (TX completion) before original skb
|
|
|
|
* This test would have no chance to be true for the clone,
|
|
|
|
* while here, branch prediction will be good.
|
2005-08-18 05:57:30 +08:00
|
|
|
*/
|
2017-06-30 18:07:59 +08:00
|
|
|
if (refcount_read(&fclones->fclone_ref) == 1)
|
2014-12-04 09:04:39 +08:00
|
|
|
goto fastpath;
|
|
|
|
break;
|
2014-11-22 03:47:16 +08:00
|
|
|
|
2014-12-04 09:04:39 +08:00
|
|
|
default: /* SKB_FCLONE_CLONE */
|
|
|
|
fclones = container_of(skb, struct sk_buff_fclones, skb2);
|
2005-08-18 05:57:30 +08:00
|
|
|
break;
|
2007-04-21 08:09:22 +08:00
|
|
|
}
|
2017-06-30 18:07:59 +08:00
|
|
|
if (!refcount_dec_and_test(&fclones->fclone_ref))
|
2014-12-04 09:04:39 +08:00
|
|
|
return;
|
|
|
|
fastpath:
|
|
|
|
kmem_cache_free(skbuff_fclone_cache, fclones);
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
2017-06-12 17:23:42 +08:00
|
|
|
void skb_release_head_state(struct sk_buff *skb)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2009-06-02 13:19:30 +08:00
|
|
|
skb_dst_drop(skb);
|
2005-04-20 13:39:42 +08:00
|
|
|
if (skb->destructor) {
|
2021-08-13 22:57:49 +08:00
|
|
|
WARN_ON(in_hardirq());
|
2005-04-17 06:20:36 +08:00
|
|
|
skb->destructor(skb);
|
|
|
|
}
|
2011-12-12 10:58:22 +08:00
|
|
|
#if IS_ENABLED(CONFIG_NF_CONNTRACK)
|
2017-01-24 01:21:56 +08:00
|
|
|
nf_conntrack_put(skb_nfct(skb));
|
2005-04-17 06:20:36 +08:00
|
|
|
#endif
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
skb_ext_put(skb);
|
2008-10-01 17:33:12 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Free everything but the sk_buff shell. */
|
|
|
|
static void skb_release_all(struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
skb_release_head_state(skb);
|
2017-07-24 01:54:47 +08:00
|
|
|
if (likely(skb->head))
|
|
|
|
skb_release_data(skb);
|
2007-11-26 23:11:19 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* __kfree_skb - private function
|
|
|
|
* @skb: buffer
|
|
|
|
*
|
|
|
|
* Free an sk_buff. Release anything attached to the buffer.
|
|
|
|
* Clean the state. This is an internal helper function. Users should
|
|
|
|
* always call kfree_skb
|
|
|
|
*/
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2007-11-26 23:11:19 +08:00
|
|
|
void __kfree_skb(struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
skb_release_all(skb);
|
2005-04-17 06:20:36 +08:00
|
|
|
kfree_skbmem(skb);
|
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(__kfree_skb);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2006-03-21 13:28:35 +08:00
|
|
|
/**
|
2022-01-09 14:36:26 +08:00
|
|
|
* kfree_skb_reason - free an sk_buff with special reason
|
2006-03-21 13:28:35 +08:00
|
|
|
* @skb: buffer to free
|
2022-01-09 14:36:26 +08:00
|
|
|
* @reason: reason why this skb is dropped
|
2006-03-21 13:28:35 +08:00
|
|
|
*
|
|
|
|
* Drop a reference to the buffer and free it if the usage count has
|
2022-01-09 14:36:26 +08:00
|
|
|
* hit zero. Meanwhile, pass the drop reason to 'kfree_skb'
|
|
|
|
* tracepoint.
|
2006-03-21 13:28:35 +08:00
|
|
|
*/
|
2022-01-09 14:36:26 +08:00
|
|
|
void kfree_skb_reason(struct sk_buff *skb, enum skb_drop_reason reason)
|
2006-03-21 13:28:35 +08:00
|
|
|
{
|
2017-06-12 17:23:41 +08:00
|
|
|
if (!skb_unref(skb))
|
2006-03-21 13:28:35 +08:00
|
|
|
return;
|
2017-06-12 17:23:41 +08:00
|
|
|
|
2022-01-09 14:36:26 +08:00
|
|
|
trace_kfree_skb(skb, __builtin_return_address(0), reason);
|
2006-03-21 13:28:35 +08:00
|
|
|
__kfree_skb(skb);
|
|
|
|
}
|
2022-01-09 14:36:26 +08:00
|
|
|
EXPORT_SYMBOL(kfree_skb_reason);
|
2006-03-21 13:28:35 +08:00
|
|
|
|
2022-03-04 14:00:41 +08:00
|
|
|
void kfree_skb_list_reason(struct sk_buff *segs,
|
|
|
|
enum skb_drop_reason reason)
|
2013-06-24 21:26:00 +08:00
|
|
|
{
|
|
|
|
while (segs) {
|
|
|
|
struct sk_buff *next = segs->next;
|
|
|
|
|
2022-03-04 14:00:41 +08:00
|
|
|
kfree_skb_reason(segs, reason);
|
2013-06-24 21:26:00 +08:00
|
|
|
segs = next;
|
|
|
|
}
|
|
|
|
}
|
2022-03-04 14:00:41 +08:00
|
|
|
EXPORT_SYMBOL(kfree_skb_list_reason);
|
2013-06-24 21:26:00 +08:00
|
|
|
|
2019-07-07 17:51:55 +08:00
|
|
|
/* Dump skb information and contents.
|
|
|
|
*
|
|
|
|
* Must only be called from net_ratelimit()-ed paths.
|
|
|
|
*
|
2020-10-05 22:48:38 +08:00
|
|
|
* Dumps whole packets if full_pkt, only headers otherwise.
|
2019-07-07 17:51:55 +08:00
|
|
|
*/
|
|
|
|
void skb_dump(const char *level, const struct sk_buff *skb, bool full_pkt)
|
|
|
|
{
|
|
|
|
struct skb_shared_info *sh = skb_shinfo(skb);
|
|
|
|
struct net_device *dev = skb->dev;
|
|
|
|
struct sock *sk = skb->sk;
|
|
|
|
struct sk_buff *list_skb;
|
|
|
|
bool has_mac, has_trans;
|
|
|
|
int headroom, tailroom;
|
|
|
|
int i, len, seg_len;
|
|
|
|
|
|
|
|
if (full_pkt)
|
|
|
|
len = skb->len;
|
|
|
|
else
|
|
|
|
len = min_t(int, skb->len, MAX_HEADER + 128);
|
|
|
|
|
|
|
|
headroom = skb_headroom(skb);
|
|
|
|
tailroom = skb_tailroom(skb);
|
|
|
|
|
|
|
|
has_mac = skb_mac_header_was_set(skb);
|
|
|
|
has_trans = skb_transport_header_was_set(skb);
|
|
|
|
|
|
|
|
printk("%sskb len=%u headroom=%u headlen=%u tailroom=%u\n"
|
|
|
|
"mac=(%d,%d) net=(%d,%d) trans=%d\n"
|
|
|
|
"shinfo(txflags=%u nr_frags=%u gso(size=%hu type=%u segs=%hu))\n"
|
|
|
|
"csum(0x%x ip_summed=%u complete_sw=%u valid=%u level=%u)\n"
|
|
|
|
"hash(0x%x sw=%u l4=%u) proto=0x%04x pkttype=%u iif=%d\n",
|
|
|
|
level, skb->len, headroom, skb_headlen(skb), tailroom,
|
|
|
|
has_mac ? skb->mac_header : -1,
|
|
|
|
has_mac ? skb_mac_header_len(skb) : -1,
|
|
|
|
skb->network_header,
|
|
|
|
has_trans ? skb_network_header_len(skb) : -1,
|
|
|
|
has_trans ? skb->transport_header : -1,
|
|
|
|
sh->tx_flags, sh->nr_frags,
|
|
|
|
sh->gso_size, sh->gso_type, sh->gso_segs,
|
|
|
|
skb->csum, skb->ip_summed, skb->csum_complete_sw,
|
|
|
|
skb->csum_valid, skb->csum_level,
|
|
|
|
skb->hash, skb->sw_hash, skb->l4_hash,
|
|
|
|
ntohs(skb->protocol), skb->pkt_type, skb->skb_iif);
|
|
|
|
|
|
|
|
if (dev)
|
2021-12-16 17:28:25 +08:00
|
|
|
printk("%sdev name=%s feat=%pNF\n",
|
2019-07-07 17:51:55 +08:00
|
|
|
level, dev->name, &dev->features);
|
|
|
|
if (sk)
|
2019-07-16 23:43:05 +08:00
|
|
|
printk("%ssk family=%hu type=%u proto=%u\n",
|
2019-07-07 17:51:55 +08:00
|
|
|
level, sk->sk_family, sk->sk_type, sk->sk_protocol);
|
|
|
|
|
|
|
|
if (full_pkt && headroom)
|
|
|
|
print_hex_dump(level, "skb headroom: ", DUMP_PREFIX_OFFSET,
|
|
|
|
16, 1, skb->head, headroom, false);
|
|
|
|
|
|
|
|
seg_len = min_t(int, skb_headlen(skb), len);
|
|
|
|
if (seg_len)
|
|
|
|
print_hex_dump(level, "skb linear: ", DUMP_PREFIX_OFFSET,
|
|
|
|
16, 1, skb->data, seg_len, false);
|
|
|
|
len -= seg_len;
|
|
|
|
|
|
|
|
if (full_pkt && tailroom)
|
|
|
|
print_hex_dump(level, "skb tailroom: ", DUMP_PREFIX_OFFSET,
|
|
|
|
16, 1, skb_tail_pointer(skb), tailroom, false);
|
|
|
|
|
|
|
|
for (i = 0; len && i < skb_shinfo(skb)->nr_frags; i++) {
|
|
|
|
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
|
|
|
|
u32 p_off, p_len, copied;
|
|
|
|
struct page *p;
|
|
|
|
u8 *vaddr;
|
|
|
|
|
2019-07-30 22:40:33 +08:00
|
|
|
skb_frag_foreach_page(frag, skb_frag_off(frag),
|
2019-07-07 17:51:55 +08:00
|
|
|
skb_frag_size(frag), p, p_off, p_len,
|
|
|
|
copied) {
|
|
|
|
seg_len = min_t(int, p_len, len);
|
|
|
|
vaddr = kmap_atomic(p);
|
|
|
|
print_hex_dump(level, "skb frag: ",
|
|
|
|
DUMP_PREFIX_OFFSET,
|
|
|
|
16, 1, vaddr + p_off, seg_len, false);
|
|
|
|
kunmap_atomic(vaddr);
|
|
|
|
len -= seg_len;
|
|
|
|
if (!len)
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (full_pkt && skb_has_frag_list(skb)) {
|
|
|
|
printk("skb fraglist:\n");
|
|
|
|
skb_walk_frags(skb, list_skb)
|
|
|
|
skb_dump(level, list_skb, true);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_dump);
|
|
|
|
|
2012-11-01 17:16:28 +08:00
|
|
|
/**
|
|
|
|
* skb_tx_error - report an sk_buff xmit error
|
|
|
|
* @skb: buffer that triggered an error
|
|
|
|
*
|
|
|
|
* Report xmit error if a device callback is tracking this skb.
|
|
|
|
* skb must be freed afterwards.
|
|
|
|
*/
|
|
|
|
void skb_tx_error(struct sk_buff *skb)
|
|
|
|
{
|
sock: enable MSG_ZEROCOPY
Prepare the datapath for refcounted ubuf_info. Clone ubuf_info with
skb_zerocopy_clone() wherever needed due to skb split, merge, resize
or clone.
Split skb_orphan_frags into two variants. The split, merge, .. paths
support reference counted zerocopy buffers, so do not do a deep copy.
Add skb_orphan_frags_rx for paths that may loop packets to receive
sockets. That is not allowed, as it may cause unbounded latency.
Deep copy all zerocopy copy buffers, ref-counted or not, in this path.
The exact locations to modify were chosen by exhaustively searching
through all code that might modify skb_frag references and/or the
the SKBTX_DEV_ZEROCOPY tx_flags bit.
The changes err on the safe side, in two ways.
(1) legacy ubuf_info paths virtio and tap are not modified. They keep
a 1:1 ubuf_info to sk_buff relationship. Calls to skb_orphan_frags
still call skb_copy_ubufs and thus copy frags in this case.
(2) not all copies deep in the stack are addressed yet. skb_shift,
skb_split and skb_try_coalesce can be refined to avoid copying.
These are not in the hot path and this patch is hairy enough as
is, so that is left for future refinement.
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-04 04:29:41 +08:00
|
|
|
skb_zcopy_clear(skb, true);
|
2012-11-01 17:16:28 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_tx_error);
|
|
|
|
|
2020-08-22 06:23:29 +08:00
|
|
|
#ifdef CONFIG_TRACEPOINTS
|
2009-03-11 17:49:55 +08:00
|
|
|
/**
|
|
|
|
* consume_skb - free an skbuff
|
|
|
|
* @skb: buffer to free
|
|
|
|
*
|
|
|
|
* Drop a ref to the buffer and free it if the usage count has hit zero
|
|
|
|
* Functions identically to kfree_skb, but kfree_skb assumes that the frame
|
|
|
|
* is being dropped after a failure and notes that
|
|
|
|
*/
|
|
|
|
void consume_skb(struct sk_buff *skb)
|
|
|
|
{
|
2017-06-12 17:23:41 +08:00
|
|
|
if (!skb_unref(skb))
|
2009-03-11 17:49:55 +08:00
|
|
|
return;
|
2017-06-12 17:23:41 +08:00
|
|
|
|
2010-08-23 17:46:12 +08:00
|
|
|
trace_consume_skb(skb);
|
2009-03-11 17:49:55 +08:00
|
|
|
__kfree_skb(skb);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(consume_skb);
|
2020-08-22 06:23:29 +08:00
|
|
|
#endif
|
2009-03-11 17:49:55 +08:00
|
|
|
|
2017-06-12 17:23:42 +08:00
|
|
|
/**
|
2020-11-16 18:17:58 +08:00
|
|
|
* __consume_stateless_skb - free an skbuff, assuming it is stateless
|
2017-06-12 17:23:42 +08:00
|
|
|
* @skb: buffer to free
|
|
|
|
*
|
2017-09-06 20:44:36 +08:00
|
|
|
* Alike consume_skb(), but this variant assumes that this is the last
|
|
|
|
* skb reference and all the head states have been already dropped
|
2017-06-12 17:23:42 +08:00
|
|
|
*/
|
2017-09-06 20:44:36 +08:00
|
|
|
void __consume_stateless_skb(struct sk_buff *skb)
|
2017-06-12 17:23:42 +08:00
|
|
|
{
|
|
|
|
trace_consume_skb(skb);
|
2017-07-18 00:56:54 +08:00
|
|
|
skb_release_data(skb);
|
2017-06-12 17:23:42 +08:00
|
|
|
kfree_skbmem(skb);
|
|
|
|
}
|
|
|
|
|
skbuff: introduce {,__}napi_build_skb() which reuses NAPI cache heads
Instead of just bulk-flushing skbuff_heads queued up through
napi_consume_skb() or __kfree_skb_defer(), try to reuse them
on allocation path.
If the cache is empty on allocation, bulk-allocate the first
16 elements, which is more efficient than per-skb allocation.
If the cache is full on freeing, bulk-wipe the second half of
the cache (32 elements).
This also includes custom KASAN poisoning/unpoisoning to be
double sure there are no use-after-free cases.
To not change current behaviour, introduce a new function,
napi_build_skb(), to optionally use a new approach later
in drivers.
Note on selected bulk size, 16:
- this equals to XDP_BULK_QUEUE_SIZE, DEV_MAP_BULK_SIZE
and especially VETH_XDP_BATCH, which is also used to
bulk-allocate skbuff_heads and was tested on powerful
setups;
- this also showed the best performance in the actual
test series (from the array of {8, 16, 32}).
Suggested-by: Edward Cree <ecree.xilinx@gmail.com> # Divide on two halves
Suggested-by: Eric Dumazet <edumazet@google.com> # KASAN poisoning
Cc: Dmitry Vyukov <dvyukov@google.com> # Help with KASAN
Cc: Paolo Abeni <pabeni@redhat.com> # Reduced batch size
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
2021-02-13 22:12:25 +08:00
|
|
|
static void napi_skb_cache_put(struct sk_buff *skb)
|
2016-02-08 20:14:59 +08:00
|
|
|
{
|
|
|
|
struct napi_alloc_cache *nc = this_cpu_ptr(&napi_alloc_cache);
|
skbuff: introduce {,__}napi_build_skb() which reuses NAPI cache heads
Instead of just bulk-flushing skbuff_heads queued up through
napi_consume_skb() or __kfree_skb_defer(), try to reuse them
on allocation path.
If the cache is empty on allocation, bulk-allocate the first
16 elements, which is more efficient than per-skb allocation.
If the cache is full on freeing, bulk-wipe the second half of
the cache (32 elements).
This also includes custom KASAN poisoning/unpoisoning to be
double sure there are no use-after-free cases.
To not change current behaviour, introduce a new function,
napi_build_skb(), to optionally use a new approach later
in drivers.
Note on selected bulk size, 16:
- this equals to XDP_BULK_QUEUE_SIZE, DEV_MAP_BULK_SIZE
and especially VETH_XDP_BATCH, which is also used to
bulk-allocate skbuff_heads and was tested on powerful
setups;
- this also showed the best performance in the actual
test series (from the array of {8, 16, 32}).
Suggested-by: Edward Cree <ecree.xilinx@gmail.com> # Divide on two halves
Suggested-by: Eric Dumazet <edumazet@google.com> # KASAN poisoning
Cc: Dmitry Vyukov <dvyukov@google.com> # Help with KASAN
Cc: Paolo Abeni <pabeni@redhat.com> # Reduced batch size
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
2021-02-13 22:12:25 +08:00
|
|
|
u32 i;
|
2016-02-08 20:14:59 +08:00
|
|
|
|
skbuff: introduce {,__}napi_build_skb() which reuses NAPI cache heads
Instead of just bulk-flushing skbuff_heads queued up through
napi_consume_skb() or __kfree_skb_defer(), try to reuse them
on allocation path.
If the cache is empty on allocation, bulk-allocate the first
16 elements, which is more efficient than per-skb allocation.
If the cache is full on freeing, bulk-wipe the second half of
the cache (32 elements).
This also includes custom KASAN poisoning/unpoisoning to be
double sure there are no use-after-free cases.
To not change current behaviour, introduce a new function,
napi_build_skb(), to optionally use a new approach later
in drivers.
Note on selected bulk size, 16:
- this equals to XDP_BULK_QUEUE_SIZE, DEV_MAP_BULK_SIZE
and especially VETH_XDP_BATCH, which is also used to
bulk-allocate skbuff_heads and was tested on powerful
setups;
- this also showed the best performance in the actual
test series (from the array of {8, 16, 32}).
Suggested-by: Edward Cree <ecree.xilinx@gmail.com> # Divide on two halves
Suggested-by: Eric Dumazet <edumazet@google.com> # KASAN poisoning
Cc: Dmitry Vyukov <dvyukov@google.com> # Help with KASAN
Cc: Paolo Abeni <pabeni@redhat.com> # Reduced batch size
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
2021-02-13 22:12:25 +08:00
|
|
|
kasan_poison_object_data(skbuff_head_cache, skb);
|
2016-02-08 20:14:59 +08:00
|
|
|
nc->skb_cache[nc->skb_count++] = skb;
|
|
|
|
|
|
|
|
if (unlikely(nc->skb_count == NAPI_SKB_CACHE_SIZE)) {
|
skbuff: introduce {,__}napi_build_skb() which reuses NAPI cache heads
Instead of just bulk-flushing skbuff_heads queued up through
napi_consume_skb() or __kfree_skb_defer(), try to reuse them
on allocation path.
If the cache is empty on allocation, bulk-allocate the first
16 elements, which is more efficient than per-skb allocation.
If the cache is full on freeing, bulk-wipe the second half of
the cache (32 elements).
This also includes custom KASAN poisoning/unpoisoning to be
double sure there are no use-after-free cases.
To not change current behaviour, introduce a new function,
napi_build_skb(), to optionally use a new approach later
in drivers.
Note on selected bulk size, 16:
- this equals to XDP_BULK_QUEUE_SIZE, DEV_MAP_BULK_SIZE
and especially VETH_XDP_BATCH, which is also used to
bulk-allocate skbuff_heads and was tested on powerful
setups;
- this also showed the best performance in the actual
test series (from the array of {8, 16, 32}).
Suggested-by: Edward Cree <ecree.xilinx@gmail.com> # Divide on two halves
Suggested-by: Eric Dumazet <edumazet@google.com> # KASAN poisoning
Cc: Dmitry Vyukov <dvyukov@google.com> # Help with KASAN
Cc: Paolo Abeni <pabeni@redhat.com> # Reduced batch size
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
2021-02-13 22:12:25 +08:00
|
|
|
for (i = NAPI_SKB_CACHE_HALF; i < NAPI_SKB_CACHE_SIZE; i++)
|
|
|
|
kasan_unpoison_object_data(skbuff_head_cache,
|
|
|
|
nc->skb_cache[i]);
|
|
|
|
|
|
|
|
kmem_cache_free_bulk(skbuff_head_cache, NAPI_SKB_CACHE_HALF,
|
|
|
|
nc->skb_cache + NAPI_SKB_CACHE_HALF);
|
|
|
|
nc->skb_count = NAPI_SKB_CACHE_HALF;
|
2016-02-08 20:14:59 +08:00
|
|
|
}
|
|
|
|
}
|
skbuff: introduce {,__}napi_build_skb() which reuses NAPI cache heads
Instead of just bulk-flushing skbuff_heads queued up through
napi_consume_skb() or __kfree_skb_defer(), try to reuse them
on allocation path.
If the cache is empty on allocation, bulk-allocate the first
16 elements, which is more efficient than per-skb allocation.
If the cache is full on freeing, bulk-wipe the second half of
the cache (32 elements).
This also includes custom KASAN poisoning/unpoisoning to be
double sure there are no use-after-free cases.
To not change current behaviour, introduce a new function,
napi_build_skb(), to optionally use a new approach later
in drivers.
Note on selected bulk size, 16:
- this equals to XDP_BULK_QUEUE_SIZE, DEV_MAP_BULK_SIZE
and especially VETH_XDP_BATCH, which is also used to
bulk-allocate skbuff_heads and was tested on powerful
setups;
- this also showed the best performance in the actual
test series (from the array of {8, 16, 32}).
Suggested-by: Edward Cree <ecree.xilinx@gmail.com> # Divide on two halves
Suggested-by: Eric Dumazet <edumazet@google.com> # KASAN poisoning
Cc: Dmitry Vyukov <dvyukov@google.com> # Help with KASAN
Cc: Paolo Abeni <pabeni@redhat.com> # Reduced batch size
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
2021-02-13 22:12:25 +08:00
|
|
|
|
2016-02-08 20:15:04 +08:00
|
|
|
void __kfree_skb_defer(struct sk_buff *skb)
|
|
|
|
{
|
2021-02-13 22:13:09 +08:00
|
|
|
skb_release_all(skb);
|
|
|
|
napi_skb_cache_put(skb);
|
|
|
|
}
|
|
|
|
|
|
|
|
void napi_skb_free_stolen_head(struct sk_buff *skb)
|
|
|
|
{
|
2021-07-29 00:24:02 +08:00
|
|
|
if (unlikely(skb->slow_gro)) {
|
|
|
|
nf_reset_ct(skb);
|
|
|
|
skb_dst_drop(skb);
|
|
|
|
skb_ext_put(skb);
|
2021-07-29 00:24:03 +08:00
|
|
|
skb_orphan(skb);
|
2021-07-29 00:24:02 +08:00
|
|
|
skb->slow_gro = 0;
|
|
|
|
}
|
skbuff: introduce {,__}napi_build_skb() which reuses NAPI cache heads
Instead of just bulk-flushing skbuff_heads queued up through
napi_consume_skb() or __kfree_skb_defer(), try to reuse them
on allocation path.
If the cache is empty on allocation, bulk-allocate the first
16 elements, which is more efficient than per-skb allocation.
If the cache is full on freeing, bulk-wipe the second half of
the cache (32 elements).
This also includes custom KASAN poisoning/unpoisoning to be
double sure there are no use-after-free cases.
To not change current behaviour, introduce a new function,
napi_build_skb(), to optionally use a new approach later
in drivers.
Note on selected bulk size, 16:
- this equals to XDP_BULK_QUEUE_SIZE, DEV_MAP_BULK_SIZE
and especially VETH_XDP_BATCH, which is also used to
bulk-allocate skbuff_heads and was tested on powerful
setups;
- this also showed the best performance in the actual
test series (from the array of {8, 16, 32}).
Suggested-by: Edward Cree <ecree.xilinx@gmail.com> # Divide on two halves
Suggested-by: Eric Dumazet <edumazet@google.com> # KASAN poisoning
Cc: Dmitry Vyukov <dvyukov@google.com> # Help with KASAN
Cc: Paolo Abeni <pabeni@redhat.com> # Reduced batch size
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
2021-02-13 22:12:25 +08:00
|
|
|
napi_skb_cache_put(skb);
|
2016-02-08 20:15:04 +08:00
|
|
|
}
|
2016-02-08 20:14:59 +08:00
|
|
|
|
|
|
|
void napi_consume_skb(struct sk_buff *skb, int budget)
|
|
|
|
{
|
2016-03-11 16:43:58 +08:00
|
|
|
/* Zero budget indicate non-NAPI context called us, like netpoll */
|
2016-02-08 20:14:59 +08:00
|
|
|
if (unlikely(!budget)) {
|
2016-03-11 16:43:58 +08:00
|
|
|
dev_consume_skb_any(skb);
|
2016-02-08 20:14:59 +08:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2020-11-24 18:49:29 +08:00
|
|
|
lockdep_assert_in_softirq();
|
|
|
|
|
2017-06-14 17:48:48 +08:00
|
|
|
if (!skb_unref(skb))
|
2016-02-08 20:14:59 +08:00
|
|
|
return;
|
2017-06-14 17:48:48 +08:00
|
|
|
|
2016-02-08 20:14:59 +08:00
|
|
|
/* if reaching here SKB is ready to free */
|
|
|
|
trace_consume_skb(skb);
|
|
|
|
|
|
|
|
/* if SKB is a clone, don't handle this case */
|
2016-03-21 02:27:47 +08:00
|
|
|
if (skb->fclone != SKB_FCLONE_UNAVAILABLE) {
|
2016-02-08 20:14:59 +08:00
|
|
|
__kfree_skb(skb);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2021-02-13 22:13:09 +08:00
|
|
|
skb_release_all(skb);
|
skbuff: introduce {,__}napi_build_skb() which reuses NAPI cache heads
Instead of just bulk-flushing skbuff_heads queued up through
napi_consume_skb() or __kfree_skb_defer(), try to reuse them
on allocation path.
If the cache is empty on allocation, bulk-allocate the first
16 elements, which is more efficient than per-skb allocation.
If the cache is full on freeing, bulk-wipe the second half of
the cache (32 elements).
This also includes custom KASAN poisoning/unpoisoning to be
double sure there are no use-after-free cases.
To not change current behaviour, introduce a new function,
napi_build_skb(), to optionally use a new approach later
in drivers.
Note on selected bulk size, 16:
- this equals to XDP_BULK_QUEUE_SIZE, DEV_MAP_BULK_SIZE
and especially VETH_XDP_BATCH, which is also used to
bulk-allocate skbuff_heads and was tested on powerful
setups;
- this also showed the best performance in the actual
test series (from the array of {8, 16, 32}).
Suggested-by: Edward Cree <ecree.xilinx@gmail.com> # Divide on two halves
Suggested-by: Eric Dumazet <edumazet@google.com> # KASAN poisoning
Cc: Dmitry Vyukov <dvyukov@google.com> # Help with KASAN
Cc: Paolo Abeni <pabeni@redhat.com> # Reduced batch size
Signed-off-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: David S. Miller <davem@davemloft.net>
2021-02-13 22:12:25 +08:00
|
|
|
napi_skb_cache_put(skb);
|
2016-02-08 20:14:59 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(napi_consume_skb);
|
|
|
|
|
2021-11-21 08:31:49 +08:00
|
|
|
/* Make sure a field is contained by headers group */
|
net: reorganize sk_buff for faster __copy_skb_header()
With proliferation of bit fields in sk_buff, __copy_skb_header() became
quite expensive, showing as the most expensive function in a GSO
workload.
__copy_skb_header() performance is also critical for non GSO TCP
operations, as it is used from skb_clone()
This patch carefully moves all the fields that were not copied in a
separate zone : cloned, nohdr, fclone, peeked, head_frag, xmit_more
Then I moved all other fields and all other copied fields in a section
delimited by headers_start[0]/headers_end[0] section so that we
can use a single memcpy() call, inlined by compiler using long
word load/stores.
I also tried to make all copies in the natural orders of sk_buff,
to help hardware prefetching.
I made sure sk_buff size did not change.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-09-29 13:18:47 +08:00
|
|
|
#define CHECK_SKB_FIELD(field) \
|
2021-11-21 08:31:49 +08:00
|
|
|
BUILD_BUG_ON(offsetof(struct sk_buff, field) != \
|
|
|
|
offsetof(struct sk_buff, headers.field)); \
|
net: reorganize sk_buff for faster __copy_skb_header()
With proliferation of bit fields in sk_buff, __copy_skb_header() became
quite expensive, showing as the most expensive function in a GSO
workload.
__copy_skb_header() performance is also critical for non GSO TCP
operations, as it is used from skb_clone()
This patch carefully moves all the fields that were not copied in a
separate zone : cloned, nohdr, fclone, peeked, head_frag, xmit_more
Then I moved all other fields and all other copied fields in a section
delimited by headers_start[0]/headers_end[0] section so that we
can use a single memcpy() call, inlined by compiler using long
word load/stores.
I also tried to make all copies in the natural orders of sk_buff,
to help hardware prefetching.
I made sure sk_buff size did not change.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-09-29 13:18:47 +08:00
|
|
|
|
2007-10-14 15:37:30 +08:00
|
|
|
static void __copy_skb_header(struct sk_buff *new, const struct sk_buff *old)
|
|
|
|
{
|
|
|
|
new->tstamp = old->tstamp;
|
net: reorganize sk_buff for faster __copy_skb_header()
With proliferation of bit fields in sk_buff, __copy_skb_header() became
quite expensive, showing as the most expensive function in a GSO
workload.
__copy_skb_header() performance is also critical for non GSO TCP
operations, as it is used from skb_clone()
This patch carefully moves all the fields that were not copied in a
separate zone : cloned, nohdr, fclone, peeked, head_frag, xmit_more
Then I moved all other fields and all other copied fields in a section
delimited by headers_start[0]/headers_end[0] section so that we
can use a single memcpy() call, inlined by compiler using long
word load/stores.
I also tried to make all copies in the natural orders of sk_buff,
to help hardware prefetching.
I made sure sk_buff size did not change.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-09-29 13:18:47 +08:00
|
|
|
/* We do not copy old->sk */
|
2007-10-14 15:37:30 +08:00
|
|
|
new->dev = old->dev;
|
net: reorganize sk_buff for faster __copy_skb_header()
With proliferation of bit fields in sk_buff, __copy_skb_header() became
quite expensive, showing as the most expensive function in a GSO
workload.
__copy_skb_header() performance is also critical for non GSO TCP
operations, as it is used from skb_clone()
This patch carefully moves all the fields that were not copied in a
separate zone : cloned, nohdr, fclone, peeked, head_frag, xmit_more
Then I moved all other fields and all other copied fields in a section
delimited by headers_start[0]/headers_end[0] section so that we
can use a single memcpy() call, inlined by compiler using long
word load/stores.
I also tried to make all copies in the natural orders of sk_buff,
to help hardware prefetching.
I made sure sk_buff size did not change.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-09-29 13:18:47 +08:00
|
|
|
memcpy(new->cb, old->cb, sizeof(old->cb));
|
2010-05-12 07:19:48 +08:00
|
|
|
skb_dst_copy(new, old);
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
__skb_ext_copy(new, old);
|
net: reorganize sk_buff for faster __copy_skb_header()
With proliferation of bit fields in sk_buff, __copy_skb_header() became
quite expensive, showing as the most expensive function in a GSO
workload.
__copy_skb_header() performance is also critical for non GSO TCP
operations, as it is used from skb_clone()
This patch carefully moves all the fields that were not copied in a
separate zone : cloned, nohdr, fclone, peeked, head_frag, xmit_more
Then I moved all other fields and all other copied fields in a section
delimited by headers_start[0]/headers_end[0] section so that we
can use a single memcpy() call, inlined by compiler using long
word load/stores.
I also tried to make all copies in the natural orders of sk_buff,
to help hardware prefetching.
I made sure sk_buff size did not change.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-09-29 13:18:47 +08:00
|
|
|
__nf_copy(new, old, false);
|
|
|
|
|
2021-11-21 08:31:49 +08:00
|
|
|
/* Note : this field could be in the headers group.
|
net: reorganize sk_buff for faster __copy_skb_header()
With proliferation of bit fields in sk_buff, __copy_skb_header() became
quite expensive, showing as the most expensive function in a GSO
workload.
__copy_skb_header() performance is also critical for non GSO TCP
operations, as it is used from skb_clone()
This patch carefully moves all the fields that were not copied in a
separate zone : cloned, nohdr, fclone, peeked, head_frag, xmit_more
Then I moved all other fields and all other copied fields in a section
delimited by headers_start[0]/headers_end[0] section so that we
can use a single memcpy() call, inlined by compiler using long
word load/stores.
I also tried to make all copies in the natural orders of sk_buff,
to help hardware prefetching.
I made sure sk_buff size did not change.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-09-29 13:18:47 +08:00
|
|
|
* It is not yet because we do not want to have a 16 bit hole
|
|
|
|
*/
|
|
|
|
new->queue_mapping = old->queue_mapping;
|
|
|
|
|
2021-11-21 08:31:49 +08:00
|
|
|
memcpy(&new->headers, &old->headers, sizeof(new->headers));
|
net: reorganize sk_buff for faster __copy_skb_header()
With proliferation of bit fields in sk_buff, __copy_skb_header() became
quite expensive, showing as the most expensive function in a GSO
workload.
__copy_skb_header() performance is also critical for non GSO TCP
operations, as it is used from skb_clone()
This patch carefully moves all the fields that were not copied in a
separate zone : cloned, nohdr, fclone, peeked, head_frag, xmit_more
Then I moved all other fields and all other copied fields in a section
delimited by headers_start[0]/headers_end[0] section so that we
can use a single memcpy() call, inlined by compiler using long
word load/stores.
I also tried to make all copies in the natural orders of sk_buff,
to help hardware prefetching.
I made sure sk_buff size did not change.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-09-29 13:18:47 +08:00
|
|
|
CHECK_SKB_FIELD(protocol);
|
|
|
|
CHECK_SKB_FIELD(csum);
|
|
|
|
CHECK_SKB_FIELD(hash);
|
|
|
|
CHECK_SKB_FIELD(priority);
|
|
|
|
CHECK_SKB_FIELD(skb_iif);
|
|
|
|
CHECK_SKB_FIELD(vlan_proto);
|
|
|
|
CHECK_SKB_FIELD(vlan_tci);
|
|
|
|
CHECK_SKB_FIELD(transport_header);
|
|
|
|
CHECK_SKB_FIELD(network_header);
|
|
|
|
CHECK_SKB_FIELD(mac_header);
|
|
|
|
CHECK_SKB_FIELD(inner_protocol);
|
|
|
|
CHECK_SKB_FIELD(inner_transport_header);
|
|
|
|
CHECK_SKB_FIELD(inner_network_header);
|
|
|
|
CHECK_SKB_FIELD(inner_mac_header);
|
|
|
|
CHECK_SKB_FIELD(mark);
|
|
|
|
#ifdef CONFIG_NETWORK_SECMARK
|
|
|
|
CHECK_SKB_FIELD(secmark);
|
|
|
|
#endif
|
|
|
|
#ifdef CONFIG_NET_RX_BUSY_POLL
|
|
|
|
CHECK_SKB_FIELD(napi_id);
|
2007-10-14 15:37:30 +08:00
|
|
|
#endif
|
xps: fix xps for stacked devices
A typical qdisc setup is the following :
bond0 : bonding device, using HTB hierarchy
eth1/eth2 : slaves, multiqueue NIC, using MQ + FQ qdisc
XPS allows to spread packets on specific tx queues, based on the cpu
doing the send.
Problem is that dequeues from bond0 qdisc can happen on random cpus,
due to the fact that qdisc_run() can dequeue a batch of packets.
CPUA -> queue packet P1 on bond0 qdisc, P1->ooo_okay=1
CPUA -> queue packet P2 on bond0 qdisc, P2->ooo_okay=0
CPUB -> dequeue packet P1 from bond0
enqueue packet on eth1/eth2
CPUC -> dequeue packet P2 from bond0
enqueue packet on eth1/eth2 using sk cache (ooo_okay is 0)
get_xps_queue() then might select wrong queue for P1, since current cpu
might be different than CPUA.
P2 might be sent on the old queue (stored in sk->sk_tx_queue_mapping),
if CPUC runs a bit faster (or CPUB spins a bit on qdisc lock)
Effect of this bug is TCP reorders, and more generally not optimal
TX queue placement. (A victim bulk flow can be migrated to the wrong TX
queue for a while)
To fix this, we have to record sender cpu number the first time
dev_queue_xmit() is called for one tx skb.
We can union napi_id (used on receive path) and sender_cpu,
granted we clear sender_cpu in skb_scrub_packet() (credit to Willem for
this union idea)
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Willem de Bruijn <willemb@google.com>
Cc: Nandita Dukkipati <nanditad@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-02-04 15:48:24 +08:00
|
|
|
#ifdef CONFIG_XPS
|
|
|
|
CHECK_SKB_FIELD(sender_cpu);
|
|
|
|
#endif
|
2007-10-14 15:37:30 +08:00
|
|
|
#ifdef CONFIG_NET_SCHED
|
net: reorganize sk_buff for faster __copy_skb_header()
With proliferation of bit fields in sk_buff, __copy_skb_header() became
quite expensive, showing as the most expensive function in a GSO
workload.
__copy_skb_header() performance is also critical for non GSO TCP
operations, as it is used from skb_clone()
This patch carefully moves all the fields that were not copied in a
separate zone : cloned, nohdr, fclone, peeked, head_frag, xmit_more
Then I moved all other fields and all other copied fields in a section
delimited by headers_start[0]/headers_end[0] section so that we
can use a single memcpy() call, inlined by compiler using long
word load/stores.
I also tried to make all copies in the natural orders of sk_buff,
to help hardware prefetching.
I made sure sk_buff size did not change.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-09-29 13:18:47 +08:00
|
|
|
CHECK_SKB_FIELD(tc_index);
|
2007-10-14 15:37:30 +08:00
|
|
|
#endif
|
2013-06-10 16:39:50 +08:00
|
|
|
|
2007-10-14 15:37:30 +08:00
|
|
|
}
|
|
|
|
|
2009-05-23 06:11:37 +08:00
|
|
|
/*
|
|
|
|
* You should not add any new code to this function. Add it to
|
|
|
|
* __copy_skb_header above instead.
|
|
|
|
*/
|
2007-10-14 15:37:52 +08:00
|
|
|
static struct sk_buff *__skb_clone(struct sk_buff *n, struct sk_buff *skb)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
|
|
|
#define C(x) n->x = skb->x
|
|
|
|
|
|
|
|
n->next = n->prev = NULL;
|
|
|
|
n->sk = NULL;
|
2007-10-14 15:37:30 +08:00
|
|
|
__copy_skb_header(n, skb);
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
C(len);
|
|
|
|
C(data_len);
|
2007-03-17 06:00:46 +08:00
|
|
|
C(mac_len);
|
[SKBUFF]: Keep track of writable header len of headerless clones
Currently NAT (and others) that want to modify cloned skbs copy them,
even if in the vast majority of cases its not necessary because the
skb is a clone made by TCP and the portion NAT wants to modify is
actually writable because TCP release the header reference before
cloning.
The problem is that there is no clean way for NAT to find out how
long the writable header area is, so this patch introduces skb->hdr_len
to hold this length. When a headerless skb is cloned skb->hdr_len
is set to the current headroom, for regular clones it is copied from
the original. A new function skb_clone_writable(skb, len) returns
whether the skb is writable up to len bytes from skb->data. To avoid
enlarging the skb the mac_len field is reduced to 16 bit and the
new hdr_len field is put in the remaining 16 bit.
I've done a few rough benchmarks of NAT (not with this exact patch,
but a very similar one). As expected it saves huge amounts of system
time in case of sendfile, bringing it down to basically the same
amount as without NAT, with sendmsg it only helps on loopback,
probably because of the large MTU.
Transmit a 1GB file using sendfile/sendmsg over eth0/lo with and
without NAT:
- sendfile eth0, no NAT: sys 0m0.388s
- sendfile eth0, NAT: sys 0m1.835s
- sendfile eth0: NAT + path: sys 0m0.370s (~ -80%)
- sendfile lo, no NAT: sys 0m0.258s
- sendfile lo, NAT: sys 0m2.609s
- sendfile lo, NAT + patch: sys 0m0.260s (~ -90%)
- sendmsg eth0, no NAT: sys 0m2.508s
- sendmsg eth0, NAT: sys 0m2.539s
- sendmsg eth0, NAT + patch: sys 0m2.445s (no change)
- sendmsg lo, no NAT: sys 0m2.151s
- sendmsg lo, NAT: sys 0m3.557s
- sendmsg lo, NAT + patch: sys 0m2.159s (~ -40%)
I expect other users can see a similar performance improvement,
packet mangling iptables targets, ipip and ip_gre come to mind ..
Signed-off-by: Patrick McHardy <kaber@trash.net>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-06-25 19:35:20 +08:00
|
|
|
n->hdr_len = skb->nohdr ? skb_headroom(skb) : skb->hdr_len;
|
2008-01-08 13:56:41 +08:00
|
|
|
n->cloned = 1;
|
2005-04-17 06:20:36 +08:00
|
|
|
n->nohdr = 0;
|
2018-04-08 04:42:39 +08:00
|
|
|
n->peeked = 0;
|
2018-07-13 19:21:07 +08:00
|
|
|
C(pfmemalloc);
|
2021-06-08 03:02:38 +08:00
|
|
|
C(pp_recycle);
|
2005-04-17 06:20:36 +08:00
|
|
|
n->destructor = NULL;
|
|
|
|
C(tail);
|
|
|
|
C(end);
|
2008-01-08 13:56:41 +08:00
|
|
|
C(head);
|
net: allow skb->head to be a page fragment
skb->head is currently allocated from kmalloc(). This is convenient but
has the drawback the data cannot be converted to a page fragment if
needed.
We have three spots were it hurts :
1) GRO aggregation
When a linear skb must be appended to another skb, GRO uses the
frag_list fallback, very inefficient since we keep all struct sk_buff
around. So drivers enabling GRO but delivering linear skbs to network
stack aren't enabling full GRO power.
2) splice(socket -> pipe).
We must copy the linear part to a page fragment.
This kind of defeats splice() purpose (zero copy claim)
3) TCP coalescing.
Recently introduced, this permits to group several contiguous segments
into a single skb. This shortens queue lengths and save kernel memory,
and greatly reduce probabilities of TCP collapses. This coalescing
doesnt work on linear skbs (or we would need to copy data, this would be
too slow)
Given all these issues, the following patch introduces the possibility
of having skb->head be a fragment in itself. We use a new skb flag,
skb->head_frag to carry this information.
build_skb() is changed to accept a frag_size argument. Drivers willing
to provide a page fragment instead of kmalloc() data will set a non zero
value, set to the fragment size.
Then, on situations we need to convert the skb head to a frag in itself,
we can check if skb->head_frag is set and avoid the copies or various
fallbacks we have.
This means drivers currently using frags could be updated to avoid the
current skb->head allocation and reduce their memory footprint (aka skb
truesize). (thats 512 or 1024 bytes saved per skb). This also makes
bpf/netfilter faster since the 'first frag' will be part of skb linear
part, no need to copy data.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Maciej Żenczykowski <maze@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Tom Herbert <therbert@google.com>
Cc: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Cc: Ben Hutchings <bhutchings@solarflare.com>
Cc: Matt Carlson <mcarlson@broadcom.com>
Cc: Michael Chan <mchan@broadcom.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2012-04-27 08:33:38 +08:00
|
|
|
C(head_frag);
|
2008-01-08 13:56:41 +08:00
|
|
|
C(data);
|
|
|
|
C(truesize);
|
2017-06-30 18:07:58 +08:00
|
|
|
refcount_set(&n->users, 1);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
atomic_inc(&(skb_shinfo(skb)->dataref));
|
|
|
|
skb->cloned = 1;
|
|
|
|
|
|
|
|
return n;
|
2007-10-14 15:37:52 +08:00
|
|
|
#undef C
|
|
|
|
}
|
|
|
|
|
2019-06-04 06:16:58 +08:00
|
|
|
/**
|
|
|
|
* alloc_skb_for_msg() - allocate sk_buff to wrap frag list forming a msg
|
|
|
|
* @first: first sk_buff of the msg
|
|
|
|
*/
|
|
|
|
struct sk_buff *alloc_skb_for_msg(struct sk_buff *first)
|
|
|
|
{
|
|
|
|
struct sk_buff *n;
|
|
|
|
|
|
|
|
n = alloc_skb(0, GFP_ATOMIC);
|
|
|
|
if (!n)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
n->len = first->len;
|
|
|
|
n->data_len = first->len;
|
|
|
|
n->truesize = first->truesize;
|
|
|
|
|
|
|
|
skb_shinfo(n)->frag_list = first;
|
|
|
|
|
|
|
|
__copy_skb_header(n, first);
|
|
|
|
n->destructor = NULL;
|
|
|
|
|
|
|
|
return n;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(alloc_skb_for_msg);
|
|
|
|
|
2007-10-14 15:37:52 +08:00
|
|
|
/**
|
|
|
|
* skb_morph - morph one skb into another
|
|
|
|
* @dst: the skb to receive the contents
|
|
|
|
* @src: the skb to supply the contents
|
|
|
|
*
|
|
|
|
* This is identical to skb_clone except that the target skb is
|
|
|
|
* supplied by the user.
|
|
|
|
*
|
|
|
|
* The target skb is returned upon exit.
|
|
|
|
*/
|
|
|
|
struct sk_buff *skb_morph(struct sk_buff *dst, struct sk_buff *src)
|
|
|
|
{
|
2007-11-26 23:11:19 +08:00
|
|
|
skb_release_all(dst);
|
2007-10-14 15:37:52 +08:00
|
|
|
return __skb_clone(dst, src);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(skb_morph);
|
|
|
|
|
2018-02-16 02:49:32 +08:00
|
|
|
int mm_account_pinned_pages(struct mmpin *mmp, size_t size)
|
2017-08-04 04:29:43 +08:00
|
|
|
{
|
|
|
|
unsigned long max_pg, num_pg, new_pg, old_pg;
|
|
|
|
struct user_struct *user;
|
|
|
|
|
|
|
|
if (capable(CAP_IPC_LOCK) || !size)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
num_pg = (size >> PAGE_SHIFT) + 2; /* worst case */
|
|
|
|
max_pg = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
|
|
|
|
user = mmp->user ? : current_user();
|
|
|
|
|
|
|
|
do {
|
|
|
|
old_pg = atomic_long_read(&user->locked_vm);
|
|
|
|
new_pg = old_pg + num_pg;
|
|
|
|
if (new_pg > max_pg)
|
|
|
|
return -ENOBUFS;
|
|
|
|
} while (atomic_long_cmpxchg(&user->locked_vm, old_pg, new_pg) !=
|
|
|
|
old_pg);
|
|
|
|
|
|
|
|
if (!mmp->user) {
|
|
|
|
mmp->user = get_uid(user);
|
|
|
|
mmp->num_pg = num_pg;
|
|
|
|
} else {
|
|
|
|
mmp->num_pg += num_pg;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
2018-02-16 02:49:32 +08:00
|
|
|
EXPORT_SYMBOL_GPL(mm_account_pinned_pages);
|
2017-08-04 04:29:43 +08:00
|
|
|
|
2018-02-16 02:49:32 +08:00
|
|
|
void mm_unaccount_pinned_pages(struct mmpin *mmp)
|
2017-08-04 04:29:43 +08:00
|
|
|
{
|
|
|
|
if (mmp->user) {
|
|
|
|
atomic_long_sub(mmp->num_pg, &mmp->user->locked_vm);
|
|
|
|
free_uid(mmp->user);
|
|
|
|
}
|
|
|
|
}
|
2018-02-16 02:49:32 +08:00
|
|
|
EXPORT_SYMBOL_GPL(mm_unaccount_pinned_pages);
|
2017-08-04 04:29:43 +08:00
|
|
|
|
2021-01-07 06:18:37 +08:00
|
|
|
struct ubuf_info *msg_zerocopy_alloc(struct sock *sk, size_t size)
|
2017-08-04 04:29:39 +08:00
|
|
|
{
|
|
|
|
struct ubuf_info *uarg;
|
|
|
|
struct sk_buff *skb;
|
|
|
|
|
|
|
|
WARN_ON_ONCE(!in_task());
|
|
|
|
|
|
|
|
skb = sock_omalloc(sk, 0, GFP_KERNEL);
|
|
|
|
if (!skb)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
BUILD_BUG_ON(sizeof(*uarg) > sizeof(skb->cb));
|
|
|
|
uarg = (void *)skb->cb;
|
2017-08-04 04:29:43 +08:00
|
|
|
uarg->mmp.user = NULL;
|
|
|
|
|
|
|
|
if (mm_account_pinned_pages(&uarg->mmp, size)) {
|
|
|
|
kfree_skb(skb);
|
|
|
|
return NULL;
|
|
|
|
}
|
2017-08-04 04:29:39 +08:00
|
|
|
|
2021-01-07 06:18:37 +08:00
|
|
|
uarg->callback = msg_zerocopy_callback;
|
2017-08-04 04:29:42 +08:00
|
|
|
uarg->id = ((u32)atomic_inc_return(&sk->sk_zckey)) - 1;
|
|
|
|
uarg->len = 1;
|
|
|
|
uarg->bytelen = size;
|
2017-08-04 04:29:39 +08:00
|
|
|
uarg->zerocopy = 1;
|
2021-01-07 06:18:39 +08:00
|
|
|
uarg->flags = SKBFL_ZEROCOPY_FRAG;
|
2017-09-01 07:48:22 +08:00
|
|
|
refcount_set(&uarg->refcnt, 1);
|
2017-08-04 04:29:39 +08:00
|
|
|
sock_hold(sk);
|
|
|
|
|
|
|
|
return uarg;
|
|
|
|
}
|
2021-01-07 06:18:37 +08:00
|
|
|
EXPORT_SYMBOL_GPL(msg_zerocopy_alloc);
|
2017-08-04 04:29:39 +08:00
|
|
|
|
|
|
|
static inline struct sk_buff *skb_from_uarg(struct ubuf_info *uarg)
|
|
|
|
{
|
|
|
|
return container_of((void *)uarg, struct sk_buff, cb);
|
|
|
|
}
|
|
|
|
|
2021-01-07 06:18:37 +08:00
|
|
|
struct ubuf_info *msg_zerocopy_realloc(struct sock *sk, size_t size,
|
|
|
|
struct ubuf_info *uarg)
|
2017-08-04 04:29:42 +08:00
|
|
|
{
|
|
|
|
if (uarg) {
|
|
|
|
const u32 byte_limit = 1 << 19; /* limit to a few TSO */
|
|
|
|
u32 bytelen, next;
|
|
|
|
|
|
|
|
/* realloc only when socket is locked (TCP, UDP cork),
|
|
|
|
* so uarg->len and sk_zckey access is serialized
|
|
|
|
*/
|
|
|
|
if (!sock_owned_by_user(sk)) {
|
|
|
|
WARN_ON_ONCE(1);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
bytelen = uarg->bytelen + size;
|
|
|
|
if (uarg->len == USHRT_MAX - 1 || bytelen > byte_limit) {
|
|
|
|
/* TCP can create new skb to attach new uarg */
|
|
|
|
if (sk->sk_type == SOCK_STREAM)
|
|
|
|
goto new_alloc;
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
next = (u32)atomic_read(&sk->sk_zckey);
|
|
|
|
if ((u32)(uarg->id + uarg->len) == next) {
|
2017-08-04 04:29:43 +08:00
|
|
|
if (mm_account_pinned_pages(&uarg->mmp, size))
|
|
|
|
return NULL;
|
2017-08-04 04:29:42 +08:00
|
|
|
uarg->len++;
|
|
|
|
uarg->bytelen = bytelen;
|
|
|
|
atomic_set(&sk->sk_zckey, ++next);
|
2019-05-31 06:01:21 +08:00
|
|
|
|
|
|
|
/* no extra ref when appending to datagram (MSG_MORE) */
|
|
|
|
if (sk->sk_type == SOCK_STREAM)
|
2021-01-07 06:18:41 +08:00
|
|
|
net_zcopy_get(uarg);
|
2019-05-31 06:01:21 +08:00
|
|
|
|
2017-08-04 04:29:42 +08:00
|
|
|
return uarg;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
new_alloc:
|
2021-01-07 06:18:37 +08:00
|
|
|
return msg_zerocopy_alloc(sk, size);
|
2017-08-04 04:29:42 +08:00
|
|
|
}
|
2021-01-07 06:18:37 +08:00
|
|
|
EXPORT_SYMBOL_GPL(msg_zerocopy_realloc);
|
2017-08-04 04:29:42 +08:00
|
|
|
|
|
|
|
static bool skb_zerocopy_notify_extend(struct sk_buff *skb, u32 lo, u16 len)
|
|
|
|
{
|
|
|
|
struct sock_exterr_skb *serr = SKB_EXT_ERR(skb);
|
|
|
|
u32 old_lo, old_hi;
|
|
|
|
u64 sum_len;
|
|
|
|
|
|
|
|
old_lo = serr->ee.ee_info;
|
|
|
|
old_hi = serr->ee.ee_data;
|
|
|
|
sum_len = old_hi - old_lo + 1ULL + len;
|
|
|
|
|
|
|
|
if (sum_len >= (1ULL << 32))
|
|
|
|
return false;
|
|
|
|
|
|
|
|
if (lo != old_hi + 1)
|
|
|
|
return false;
|
|
|
|
|
|
|
|
serr->ee.ee_data += len;
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
2021-01-07 06:18:37 +08:00
|
|
|
static void __msg_zerocopy_callback(struct ubuf_info *uarg)
|
2017-08-04 04:29:39 +08:00
|
|
|
{
|
2017-08-04 04:29:42 +08:00
|
|
|
struct sk_buff *tail, *skb = skb_from_uarg(uarg);
|
2017-08-04 04:29:39 +08:00
|
|
|
struct sock_exterr_skb *serr;
|
|
|
|
struct sock *sk = skb->sk;
|
2017-08-04 04:29:42 +08:00
|
|
|
struct sk_buff_head *q;
|
|
|
|
unsigned long flags;
|
2021-06-10 06:41:57 +08:00
|
|
|
bool is_zerocopy;
|
2017-08-04 04:29:42 +08:00
|
|
|
u32 lo, hi;
|
|
|
|
u16 len;
|
2017-08-04 04:29:39 +08:00
|
|
|
|
2017-08-10 07:09:43 +08:00
|
|
|
mm_unaccount_pinned_pages(&uarg->mmp);
|
|
|
|
|
2017-08-04 04:29:42 +08:00
|
|
|
/* if !len, there was only 1 call, and it was aborted
|
|
|
|
* so do not queue a completion notification
|
|
|
|
*/
|
|
|
|
if (!uarg->len || sock_flag(sk, SOCK_DEAD))
|
2017-08-04 04:29:39 +08:00
|
|
|
goto release;
|
|
|
|
|
2017-08-04 04:29:42 +08:00
|
|
|
len = uarg->len;
|
|
|
|
lo = uarg->id;
|
|
|
|
hi = uarg->id + len - 1;
|
2021-06-10 06:41:57 +08:00
|
|
|
is_zerocopy = uarg->zerocopy;
|
2017-08-04 04:29:42 +08:00
|
|
|
|
2017-08-04 04:29:39 +08:00
|
|
|
serr = SKB_EXT_ERR(skb);
|
|
|
|
memset(serr, 0, sizeof(*serr));
|
|
|
|
serr->ee.ee_errno = 0;
|
|
|
|
serr->ee.ee_origin = SO_EE_ORIGIN_ZEROCOPY;
|
2017-08-04 04:29:42 +08:00
|
|
|
serr->ee.ee_data = hi;
|
|
|
|
serr->ee.ee_info = lo;
|
2021-06-10 06:41:57 +08:00
|
|
|
if (!is_zerocopy)
|
2017-08-04 04:29:39 +08:00
|
|
|
serr->ee.ee_code |= SO_EE_CODE_ZEROCOPY_COPIED;
|
|
|
|
|
2017-08-04 04:29:42 +08:00
|
|
|
q = &sk->sk_error_queue;
|
|
|
|
spin_lock_irqsave(&q->lock, flags);
|
|
|
|
tail = skb_peek_tail(q);
|
|
|
|
if (!tail || SKB_EXT_ERR(tail)->ee.ee_origin != SO_EE_ORIGIN_ZEROCOPY ||
|
|
|
|
!skb_zerocopy_notify_extend(tail, lo, len)) {
|
|
|
|
__skb_queue_tail(q, skb);
|
|
|
|
skb = NULL;
|
|
|
|
}
|
|
|
|
spin_unlock_irqrestore(&q->lock, flags);
|
2017-08-04 04:29:39 +08:00
|
|
|
|
2021-06-28 06:48:21 +08:00
|
|
|
sk_error_report(sk);
|
2017-08-04 04:29:39 +08:00
|
|
|
|
|
|
|
release:
|
|
|
|
consume_skb(skb);
|
|
|
|
sock_put(sk);
|
|
|
|
}
|
2021-01-07 06:18:31 +08:00
|
|
|
|
2021-01-07 06:18:37 +08:00
|
|
|
void msg_zerocopy_callback(struct sk_buff *skb, struct ubuf_info *uarg,
|
|
|
|
bool success)
|
2021-01-07 06:18:31 +08:00
|
|
|
{
|
|
|
|
uarg->zerocopy = uarg->zerocopy & success;
|
|
|
|
|
|
|
|
if (refcount_dec_and_test(&uarg->refcnt))
|
2021-01-07 06:18:37 +08:00
|
|
|
__msg_zerocopy_callback(uarg);
|
2021-01-07 06:18:31 +08:00
|
|
|
}
|
2021-01-07 06:18:37 +08:00
|
|
|
EXPORT_SYMBOL_GPL(msg_zerocopy_callback);
|
2017-08-04 04:29:39 +08:00
|
|
|
|
2021-01-07 06:18:37 +08:00
|
|
|
void msg_zerocopy_put_abort(struct ubuf_info *uarg, bool have_uref)
|
2017-08-04 04:29:39 +08:00
|
|
|
{
|
2021-01-07 06:18:35 +08:00
|
|
|
struct sock *sk = skb_from_uarg(uarg)->sk;
|
2017-08-04 04:29:39 +08:00
|
|
|
|
2021-01-07 06:18:35 +08:00
|
|
|
atomic_dec(&sk->sk_zckey);
|
|
|
|
uarg->len--;
|
2017-08-04 04:29:39 +08:00
|
|
|
|
2021-01-07 06:18:35 +08:00
|
|
|
if (have_uref)
|
2021-01-07 06:18:37 +08:00
|
|
|
msg_zerocopy_callback(NULL, uarg, true);
|
2017-08-04 04:29:39 +08:00
|
|
|
}
|
2021-01-07 06:18:37 +08:00
|
|
|
EXPORT_SYMBOL_GPL(msg_zerocopy_put_abort);
|
2017-08-04 04:29:39 +08:00
|
|
|
|
2018-12-01 04:32:39 +08:00
|
|
|
int skb_zerocopy_iter_dgram(struct sk_buff *skb, struct msghdr *msg, int len)
|
|
|
|
{
|
|
|
|
return __zerocopy_sg_from_iter(skb->sk, skb, &msg->msg_iter, len);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(skb_zerocopy_iter_dgram);
|
|
|
|
|
2017-08-04 04:29:39 +08:00
|
|
|
int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb,
|
|
|
|
struct msghdr *msg, int len,
|
|
|
|
struct ubuf_info *uarg)
|
|
|
|
{
|
2017-08-04 04:29:42 +08:00
|
|
|
struct ubuf_info *orig_uarg = skb_zcopy(skb);
|
2017-08-04 04:29:39 +08:00
|
|
|
struct iov_iter orig_iter = msg->msg_iter;
|
|
|
|
int err, orig_len = skb->len;
|
|
|
|
|
2017-08-04 04:29:42 +08:00
|
|
|
/* An skb can only point to one uarg. This edge case happens when
|
|
|
|
* TCP appends to an skb, but zerocopy_realloc triggered a new alloc.
|
|
|
|
*/
|
|
|
|
if (orig_uarg && uarg != orig_uarg)
|
|
|
|
return -EEXIST;
|
|
|
|
|
2017-08-04 04:29:39 +08:00
|
|
|
err = __zerocopy_sg_from_iter(sk, skb, &msg->msg_iter, len);
|
|
|
|
if (err == -EFAULT || (err == -EMSGSIZE && skb->len == orig_len)) {
|
2017-10-20 00:40:39 +08:00
|
|
|
struct sock *save_sk = skb->sk;
|
|
|
|
|
2017-08-04 04:29:39 +08:00
|
|
|
/* Streams do not free skb on error. Reset to prev state. */
|
|
|
|
msg->msg_iter = orig_iter;
|
2017-10-20 00:40:39 +08:00
|
|
|
skb->sk = sk;
|
2017-08-04 04:29:39 +08:00
|
|
|
___pskb_trim(skb, orig_len);
|
2017-10-20 00:40:39 +08:00
|
|
|
skb->sk = save_sk;
|
2017-08-04 04:29:39 +08:00
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2018-12-01 04:32:40 +08:00
|
|
|
skb_zcopy_set(skb, uarg, NULL);
|
2017-08-04 04:29:39 +08:00
|
|
|
return skb->len - orig_len;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(skb_zerocopy_iter_stream);
|
|
|
|
|
sock: enable MSG_ZEROCOPY
Prepare the datapath for refcounted ubuf_info. Clone ubuf_info with
skb_zerocopy_clone() wherever needed due to skb split, merge, resize
or clone.
Split skb_orphan_frags into two variants. The split, merge, .. paths
support reference counted zerocopy buffers, so do not do a deep copy.
Add skb_orphan_frags_rx for paths that may loop packets to receive
sockets. That is not allowed, as it may cause unbounded latency.
Deep copy all zerocopy copy buffers, ref-counted or not, in this path.
The exact locations to modify were chosen by exhaustively searching
through all code that might modify skb_frag references and/or the
the SKBTX_DEV_ZEROCOPY tx_flags bit.
The changes err on the safe side, in two ways.
(1) legacy ubuf_info paths virtio and tap are not modified. They keep
a 1:1 ubuf_info to sk_buff relationship. Calls to skb_orphan_frags
still call skb_copy_ubufs and thus copy frags in this case.
(2) not all copies deep in the stack are addressed yet. skb_shift,
skb_split and skb_try_coalesce can be refined to avoid copying.
These are not in the hot path and this patch is hairy enough as
is, so that is left for future refinement.
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-04 04:29:41 +08:00
|
|
|
static int skb_zerocopy_clone(struct sk_buff *nskb, struct sk_buff *orig,
|
2017-08-04 04:29:39 +08:00
|
|
|
gfp_t gfp_mask)
|
|
|
|
{
|
|
|
|
if (skb_zcopy(orig)) {
|
|
|
|
if (skb_zcopy(nskb)) {
|
|
|
|
/* !gfp_mask callers are verified to !skb_zcopy(nskb) */
|
|
|
|
if (!gfp_mask) {
|
|
|
|
WARN_ON_ONCE(1);
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
if (skb_uarg(nskb) == skb_uarg(orig))
|
|
|
|
return 0;
|
|
|
|
if (skb_copy_ubufs(nskb, GFP_ATOMIC))
|
|
|
|
return -EIO;
|
|
|
|
}
|
2018-12-01 04:32:40 +08:00
|
|
|
skb_zcopy_set(nskb, skb_uarg(orig), NULL);
|
2017-08-04 04:29:39 +08:00
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2012-07-10 18:55:09 +08:00
|
|
|
/**
|
|
|
|
* skb_copy_ubufs - copy userspace skb frags buffers to kernel
|
2011-08-31 16:03:29 +08:00
|
|
|
* @skb: the skb to modify
|
|
|
|
* @gfp_mask: allocation priority
|
|
|
|
*
|
2021-01-07 06:18:38 +08:00
|
|
|
* This must be called on skb with SKBFL_ZEROCOPY_ENABLE.
|
2011-08-31 16:03:29 +08:00
|
|
|
* It will copy all frags into kernel and drop the reference
|
|
|
|
* to userspace pages.
|
|
|
|
*
|
|
|
|
* If this function is called from an interrupt gfp_mask() must be
|
|
|
|
* %GFP_ATOMIC.
|
|
|
|
*
|
|
|
|
* Returns 0 on success or a negative error code on failure
|
|
|
|
* to allocate kernel memory to copy to.
|
|
|
|
*/
|
|
|
|
int skb_copy_ubufs(struct sk_buff *skb, gfp_t gfp_mask)
|
2011-07-06 20:22:12 +08:00
|
|
|
{
|
|
|
|
int num_frags = skb_shinfo(skb)->nr_frags;
|
|
|
|
struct page *page, *head = NULL;
|
2017-08-04 04:29:38 +08:00
|
|
|
int i, new_frags;
|
|
|
|
u32 d_off;
|
2011-07-06 20:22:12 +08:00
|
|
|
|
2017-08-04 04:29:38 +08:00
|
|
|
if (skb_shared(skb) || skb_unclone(skb, gfp_mask))
|
|
|
|
return -EINVAL;
|
2011-07-06 20:22:12 +08:00
|
|
|
|
2017-12-29 01:38:13 +08:00
|
|
|
if (!num_frags)
|
|
|
|
goto release;
|
|
|
|
|
2017-08-04 04:29:38 +08:00
|
|
|
new_frags = (__skb_pagelen(skb) + PAGE_SIZE - 1) >> PAGE_SHIFT;
|
|
|
|
for (i = 0; i < new_frags; i++) {
|
2012-07-17 10:05:29 +08:00
|
|
|
page = alloc_page(gfp_mask);
|
2011-07-06 20:22:12 +08:00
|
|
|
if (!page) {
|
|
|
|
while (head) {
|
2013-07-12 15:17:23 +08:00
|
|
|
struct page *next = (struct page *)page_private(head);
|
2011-07-06 20:22:12 +08:00
|
|
|
put_page(head);
|
|
|
|
head = next;
|
|
|
|
}
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
2017-08-04 04:29:38 +08:00
|
|
|
set_page_private(page, (unsigned long)head);
|
|
|
|
head = page;
|
|
|
|
}
|
|
|
|
|
|
|
|
page = head;
|
|
|
|
d_off = 0;
|
|
|
|
for (i = 0; i < num_frags; i++) {
|
|
|
|
skb_frag_t *f = &skb_shinfo(skb)->frags[i];
|
|
|
|
u32 p_off, p_len, copied;
|
|
|
|
struct page *p;
|
|
|
|
u8 *vaddr;
|
2017-07-31 20:15:47 +08:00
|
|
|
|
2019-07-30 22:40:33 +08:00
|
|
|
skb_frag_foreach_page(f, skb_frag_off(f), skb_frag_size(f),
|
2017-07-31 20:15:47 +08:00
|
|
|
p, p_off, p_len, copied) {
|
2017-08-04 04:29:38 +08:00
|
|
|
u32 copy, done = 0;
|
2017-07-31 20:15:47 +08:00
|
|
|
vaddr = kmap_atomic(p);
|
2017-08-04 04:29:38 +08:00
|
|
|
|
|
|
|
while (done < p_len) {
|
|
|
|
if (d_off == PAGE_SIZE) {
|
|
|
|
d_off = 0;
|
|
|
|
page = (struct page *)page_private(page);
|
|
|
|
}
|
|
|
|
copy = min_t(u32, PAGE_SIZE - d_off, p_len - done);
|
|
|
|
memcpy(page_address(page) + d_off,
|
|
|
|
vaddr + p_off + done, copy);
|
|
|
|
done += copy;
|
|
|
|
d_off += copy;
|
|
|
|
}
|
2017-07-31 20:15:47 +08:00
|
|
|
kunmap_atomic(vaddr);
|
|
|
|
}
|
2011-07-06 20:22:12 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/* skb frags release userspace buffers */
|
2012-07-17 10:05:29 +08:00
|
|
|
for (i = 0; i < num_frags; i++)
|
2011-10-20 07:01:49 +08:00
|
|
|
skb_frag_unref(skb, i);
|
2011-07-06 20:22:12 +08:00
|
|
|
|
|
|
|
/* skb frags point to kernel buffers */
|
2017-08-04 04:29:38 +08:00
|
|
|
for (i = 0; i < new_frags - 1; i++) {
|
|
|
|
__skb_fill_page_desc(skb, i, head, 0, PAGE_SIZE);
|
2013-07-12 15:17:23 +08:00
|
|
|
head = (struct page *)page_private(head);
|
2011-07-06 20:22:12 +08:00
|
|
|
}
|
2017-08-04 04:29:38 +08:00
|
|
|
__skb_fill_page_desc(skb, new_frags - 1, head, 0, d_off);
|
|
|
|
skb_shinfo(skb)->nr_frags = new_frags;
|
2011-08-31 16:03:29 +08:00
|
|
|
|
2017-12-21 06:37:50 +08:00
|
|
|
release:
|
sock: enable MSG_ZEROCOPY
Prepare the datapath for refcounted ubuf_info. Clone ubuf_info with
skb_zerocopy_clone() wherever needed due to skb split, merge, resize
or clone.
Split skb_orphan_frags into two variants. The split, merge, .. paths
support reference counted zerocopy buffers, so do not do a deep copy.
Add skb_orphan_frags_rx for paths that may loop packets to receive
sockets. That is not allowed, as it may cause unbounded latency.
Deep copy all zerocopy copy buffers, ref-counted or not, in this path.
The exact locations to modify were chosen by exhaustively searching
through all code that might modify skb_frag references and/or the
the SKBTX_DEV_ZEROCOPY tx_flags bit.
The changes err on the safe side, in two ways.
(1) legacy ubuf_info paths virtio and tap are not modified. They keep
a 1:1 ubuf_info to sk_buff relationship. Calls to skb_orphan_frags
still call skb_copy_ubufs and thus copy frags in this case.
(2) not all copies deep in the stack are addressed yet. skb_shift,
skb_split and skb_try_coalesce can be refined to avoid copying.
These are not in the hot path and this patch is hairy enough as
is, so that is left for future refinement.
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-04 04:29:41 +08:00
|
|
|
skb_zcopy_clear(skb, false);
|
2011-07-06 20:22:12 +08:00
|
|
|
return 0;
|
|
|
|
}
|
2012-07-20 17:23:20 +08:00
|
|
|
EXPORT_SYMBOL_GPL(skb_copy_ubufs);
|
2011-07-06 20:22:12 +08:00
|
|
|
|
2007-10-14 15:37:52 +08:00
|
|
|
/**
|
|
|
|
* skb_clone - duplicate an sk_buff
|
|
|
|
* @skb: buffer to clone
|
|
|
|
* @gfp_mask: allocation priority
|
|
|
|
*
|
|
|
|
* Duplicate an &sk_buff. The new one is not owned by a socket. Both
|
|
|
|
* copies share the same packet data but not structure. The new
|
|
|
|
* buffer has a reference count of 1. If the allocation fails the
|
|
|
|
* function returns %NULL otherwise the new buffer is returned.
|
|
|
|
*
|
|
|
|
* If this function is called from an interrupt gfp_mask() must be
|
|
|
|
* %GFP_ATOMIC.
|
|
|
|
*/
|
|
|
|
|
|
|
|
struct sk_buff *skb_clone(struct sk_buff *skb, gfp_t gfp_mask)
|
|
|
|
{
|
2014-09-30 04:29:15 +08:00
|
|
|
struct sk_buff_fclones *fclones = container_of(skb,
|
|
|
|
struct sk_buff_fclones,
|
|
|
|
skb1);
|
2014-12-04 09:04:39 +08:00
|
|
|
struct sk_buff *n;
|
2007-10-14 15:37:52 +08:00
|
|
|
|
2012-07-20 17:23:10 +08:00
|
|
|
if (skb_orphan_frags(skb, gfp_mask))
|
|
|
|
return NULL;
|
2011-07-06 20:22:12 +08:00
|
|
|
|
2007-10-14 15:37:52 +08:00
|
|
|
if (skb->fclone == SKB_FCLONE_ORIG &&
|
2017-06-30 18:07:59 +08:00
|
|
|
refcount_read(&fclones->fclone_ref) == 1) {
|
2014-12-04 09:04:39 +08:00
|
|
|
n = &fclones->skb2;
|
2017-06-30 18:07:59 +08:00
|
|
|
refcount_set(&fclones->fclone_ref, 2);
|
2007-10-14 15:37:52 +08:00
|
|
|
} else {
|
2012-08-01 07:44:19 +08:00
|
|
|
if (skb_pfmemalloc(skb))
|
|
|
|
gfp_mask |= __GFP_MEMALLOC;
|
|
|
|
|
2007-10-14 15:37:52 +08:00
|
|
|
n = kmem_cache_alloc(skbuff_head_cache, gfp_mask);
|
|
|
|
if (!n)
|
|
|
|
return NULL;
|
2008-08-30 18:16:35 +08:00
|
|
|
|
2007-10-14 15:37:52 +08:00
|
|
|
n->fclone = SKB_FCLONE_UNAVAILABLE;
|
|
|
|
}
|
|
|
|
|
|
|
|
return __skb_clone(n, skb);
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(skb_clone);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2018-08-03 15:58:09 +08:00
|
|
|
void skb_headers_offset_update(struct sk_buff *skb, int off)
|
2013-03-07 21:21:40 +08:00
|
|
|
{
|
2013-10-20 02:42:54 +08:00
|
|
|
/* Only adjust this if it actually is csum_start rather than csum */
|
|
|
|
if (skb->ip_summed == CHECKSUM_PARTIAL)
|
|
|
|
skb->csum_start += off;
|
2013-03-07 21:21:40 +08:00
|
|
|
/* {transport,network,mac}_header and tail are relative to skb->head */
|
|
|
|
skb->transport_header += off;
|
|
|
|
skb->network_header += off;
|
|
|
|
if (skb_mac_header_was_set(skb))
|
|
|
|
skb->mac_header += off;
|
|
|
|
skb->inner_transport_header += off;
|
|
|
|
skb->inner_network_header += off;
|
2013-03-07 21:21:46 +08:00
|
|
|
skb->inner_mac_header += off;
|
2013-03-07 21:21:40 +08:00
|
|
|
}
|
2018-08-03 15:58:09 +08:00
|
|
|
EXPORT_SYMBOL(skb_headers_offset_update);
|
2013-03-07 21:21:40 +08:00
|
|
|
|
2018-04-30 15:16:11 +08:00
|
|
|
void skb_copy_header(struct sk_buff *new, const struct sk_buff *old)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2007-10-14 15:37:30 +08:00
|
|
|
__copy_skb_header(new, old);
|
|
|
|
|
2006-06-22 17:40:14 +08:00
|
|
|
skb_shinfo(new)->gso_size = skb_shinfo(old)->gso_size;
|
|
|
|
skb_shinfo(new)->gso_segs = skb_shinfo(old)->gso_segs;
|
|
|
|
skb_shinfo(new)->gso_type = skb_shinfo(old)->gso_type;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
2018-04-30 15:16:11 +08:00
|
|
|
EXPORT_SYMBOL(skb_copy_header);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2012-08-01 07:44:19 +08:00
|
|
|
static inline int skb_alloc_rx_flag(const struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
if (skb_pfmemalloc(skb))
|
|
|
|
return SKB_ALLOC_RX;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/**
|
|
|
|
* skb_copy - create private copy of an sk_buff
|
|
|
|
* @skb: buffer to copy
|
|
|
|
* @gfp_mask: allocation priority
|
|
|
|
*
|
|
|
|
* Make a copy of both an &sk_buff and its data. This is used when the
|
|
|
|
* caller wishes to modify the data and needs a private copy of the
|
|
|
|
* data to alter. Returns %NULL on failure or the pointer to the buffer
|
|
|
|
* on success. The returned buffer has a reference count of 1.
|
|
|
|
*
|
|
|
|
* As by-product this function converts non-linear &sk_buff to linear
|
|
|
|
* one, so that &sk_buff becomes completely private and caller is allowed
|
|
|
|
* to modify all the data of returned buffer. This means that this
|
|
|
|
* function is not recommended for use in circumstances when only
|
|
|
|
* header is going to be modified. Use pskb_copy() instead.
|
|
|
|
*/
|
|
|
|
|
2005-10-07 14:46:04 +08:00
|
|
|
struct sk_buff *skb_copy(const struct sk_buff *skb, gfp_t gfp_mask)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2010-09-01 13:25:10 +08:00
|
|
|
int headerlen = skb_headroom(skb);
|
2012-05-04 22:26:56 +08:00
|
|
|
unsigned int size = skb_end_offset(skb) + skb->data_len;
|
2012-08-01 07:44:19 +08:00
|
|
|
struct sk_buff *n = __alloc_skb(size, gfp_mask,
|
|
|
|
skb_alloc_rx_flag(skb), NUMA_NO_NODE);
|
2010-09-01 13:25:10 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
if (!n)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
/* Set the data pointer */
|
|
|
|
skb_reserve(n, headerlen);
|
|
|
|
/* Set the tail pointer and length */
|
|
|
|
skb_put(n, skb->len);
|
|
|
|
|
2017-10-09 23:37:59 +08:00
|
|
|
BUG_ON(skb_copy_bits(skb, -headerlen, n->head, headerlen + skb->len));
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2018-04-30 15:16:11 +08:00
|
|
|
skb_copy_header(n, skb);
|
2005-04-17 06:20:36 +08:00
|
|
|
return n;
|
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(skb_copy);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
/**
|
2014-06-12 06:36:26 +08:00
|
|
|
* __pskb_copy_fclone - create copy of an sk_buff with private head.
|
2005-04-17 06:20:36 +08:00
|
|
|
* @skb: buffer to copy
|
2011-12-04 05:39:53 +08:00
|
|
|
* @headroom: headroom of new skb
|
2005-04-17 06:20:36 +08:00
|
|
|
* @gfp_mask: allocation priority
|
2014-06-12 06:36:26 +08:00
|
|
|
* @fclone: if true allocate the copy of the skb from the fclone
|
|
|
|
* cache instead of the head cache; it is recommended to set this
|
|
|
|
* to true for the cases where the copy will likely be cloned
|
2005-04-17 06:20:36 +08:00
|
|
|
*
|
|
|
|
* Make a copy of both an &sk_buff and part of its data, located
|
|
|
|
* in header. Fragmented data remain shared. This is used when
|
|
|
|
* the caller wishes to modify only header of &sk_buff and needs
|
|
|
|
* private copy of the header to alter. Returns %NULL on failure
|
|
|
|
* or the pointer to the buffer on success.
|
|
|
|
* The returned buffer has a reference count of 1.
|
|
|
|
*/
|
|
|
|
|
2014-06-12 06:36:26 +08:00
|
|
|
struct sk_buff *__pskb_copy_fclone(struct sk_buff *skb, int headroom,
|
|
|
|
gfp_t gfp_mask, bool fclone)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2011-12-04 05:39:53 +08:00
|
|
|
unsigned int size = skb_headlen(skb) + headroom;
|
2014-06-12 06:36:26 +08:00
|
|
|
int flags = skb_alloc_rx_flag(skb) | (fclone ? SKB_ALLOC_FCLONE : 0);
|
|
|
|
struct sk_buff *n = __alloc_skb(size, gfp_mask, flags, NUMA_NO_NODE);
|
2010-09-01 13:25:10 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
if (!n)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
/* Set the data pointer */
|
2011-12-04 05:39:53 +08:00
|
|
|
skb_reserve(n, headroom);
|
2005-04-17 06:20:36 +08:00
|
|
|
/* Set the tail pointer and length */
|
|
|
|
skb_put(n, skb_headlen(skb));
|
|
|
|
/* Copy the bytes */
|
2007-03-28 05:55:52 +08:00
|
|
|
skb_copy_from_linear_data(skb, n->data, n->len);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2006-11-08 06:57:15 +08:00
|
|
|
n->truesize += skb->data_len;
|
2005-04-17 06:20:36 +08:00
|
|
|
n->data_len = skb->data_len;
|
|
|
|
n->len = skb->len;
|
|
|
|
|
|
|
|
if (skb_shinfo(skb)->nr_frags) {
|
|
|
|
int i;
|
|
|
|
|
sock: enable MSG_ZEROCOPY
Prepare the datapath for refcounted ubuf_info. Clone ubuf_info with
skb_zerocopy_clone() wherever needed due to skb split, merge, resize
or clone.
Split skb_orphan_frags into two variants. The split, merge, .. paths
support reference counted zerocopy buffers, so do not do a deep copy.
Add skb_orphan_frags_rx for paths that may loop packets to receive
sockets. That is not allowed, as it may cause unbounded latency.
Deep copy all zerocopy copy buffers, ref-counted or not, in this path.
The exact locations to modify were chosen by exhaustively searching
through all code that might modify skb_frag references and/or the
the SKBTX_DEV_ZEROCOPY tx_flags bit.
The changes err on the safe side, in two ways.
(1) legacy ubuf_info paths virtio and tap are not modified. They keep
a 1:1 ubuf_info to sk_buff relationship. Calls to skb_orphan_frags
still call skb_copy_ubufs and thus copy frags in this case.
(2) not all copies deep in the stack are addressed yet. skb_shift,
skb_split and skb_try_coalesce can be refined to avoid copying.
These are not in the hot path and this patch is hairy enough as
is, so that is left for future refinement.
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-04 04:29:41 +08:00
|
|
|
if (skb_orphan_frags(skb, gfp_mask) ||
|
|
|
|
skb_zerocopy_clone(n, skb, gfp_mask)) {
|
2012-07-20 17:23:10 +08:00
|
|
|
kfree_skb(n);
|
|
|
|
n = NULL;
|
|
|
|
goto out;
|
2011-07-06 20:22:12 +08:00
|
|
|
}
|
2005-04-17 06:20:36 +08:00
|
|
|
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
|
|
|
|
skb_shinfo(n)->frags[i] = skb_shinfo(skb)->frags[i];
|
2011-08-23 07:44:58 +08:00
|
|
|
skb_frag_ref(skb, i);
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
skb_shinfo(n)->nr_frags = i;
|
|
|
|
}
|
|
|
|
|
2010-08-23 15:13:46 +08:00
|
|
|
if (skb_has_frag_list(skb)) {
|
2005-04-17 06:20:36 +08:00
|
|
|
skb_shinfo(n)->frag_list = skb_shinfo(skb)->frag_list;
|
|
|
|
skb_clone_fraglist(n);
|
|
|
|
}
|
|
|
|
|
2018-04-30 15:16:11 +08:00
|
|
|
skb_copy_header(n, skb);
|
2005-04-17 06:20:36 +08:00
|
|
|
out:
|
|
|
|
return n;
|
|
|
|
}
|
2014-06-12 06:36:26 +08:00
|
|
|
EXPORT_SYMBOL(__pskb_copy_fclone);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
/**
|
|
|
|
* pskb_expand_head - reallocate header of &sk_buff
|
|
|
|
* @skb: buffer to reallocate
|
|
|
|
* @nhead: room to add at head
|
|
|
|
* @ntail: room to add at tail
|
|
|
|
* @gfp_mask: allocation priority
|
|
|
|
*
|
2013-11-07 21:18:26 +08:00
|
|
|
* Expands (or creates identical copy, if @nhead and @ntail are zero)
|
|
|
|
* header of @skb. &sk_buff itself is not changed. &sk_buff MUST have
|
2005-04-17 06:20:36 +08:00
|
|
|
* reference count of 1. Returns zero in the case of success or error,
|
|
|
|
* if expansion failed. In the last case, &sk_buff is not changed.
|
|
|
|
*
|
|
|
|
* All the pointers pointing into skb header may change and must be
|
|
|
|
* reloaded after call to this function.
|
|
|
|
*/
|
|
|
|
|
2005-07-09 05:57:47 +08:00
|
|
|
int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail,
|
2005-10-07 14:46:04 +08:00
|
|
|
gfp_t gfp_mask)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2017-01-27 23:11:27 +08:00
|
|
|
int i, osize = skb_end_offset(skb);
|
|
|
|
int size = osize + nhead + ntail;
|
2005-04-17 06:20:36 +08:00
|
|
|
long off;
|
2017-01-27 23:11:27 +08:00
|
|
|
u8 *data;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2008-10-01 22:09:38 +08:00
|
|
|
BUG_ON(nhead < 0);
|
|
|
|
|
2017-10-09 23:37:59 +08:00
|
|
|
BUG_ON(skb_shared(skb));
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
size = SKB_DATA_ALIGN(size);
|
|
|
|
|
2012-08-01 07:44:19 +08:00
|
|
|
if (skb_pfmemalloc(skb))
|
|
|
|
gfp_mask |= __GFP_MEMALLOC;
|
|
|
|
data = kmalloc_reserve(size + SKB_DATA_ALIGN(sizeof(struct skb_shared_info)),
|
|
|
|
gfp_mask, NUMA_NO_NODE, NULL);
|
2005-04-17 06:20:36 +08:00
|
|
|
if (!data)
|
|
|
|
goto nodata;
|
2012-04-11 04:08:39 +08:00
|
|
|
size = SKB_WITH_OVERHEAD(ksize(data));
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
/* Copy only real data... and, alas, header. This should be
|
2010-09-01 13:25:10 +08:00
|
|
|
* optimized for the cases when header is void.
|
|
|
|
*/
|
|
|
|
memcpy(data + nhead, skb->head, skb_tail_pointer(skb) - skb->head);
|
|
|
|
|
|
|
|
memcpy((struct skb_shared_info *)(data + size),
|
|
|
|
skb_shinfo(skb),
|
2010-07-23 03:09:08 +08:00
|
|
|
offsetof(struct skb_shared_info, frags[skb_shinfo(skb)->nr_frags]));
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2012-05-04 22:26:51 +08:00
|
|
|
/*
|
|
|
|
* if shinfo is shared we must drop the old head gracefully, but if it
|
|
|
|
* is not we can just drop the old head and let the existing refcount
|
|
|
|
* be since all we did is relocate the values
|
|
|
|
*/
|
|
|
|
if (skb_cloned(skb)) {
|
2012-07-20 17:23:10 +08:00
|
|
|
if (skb_orphan_frags(skb, gfp_mask))
|
|
|
|
goto nofrags;
|
sock: enable MSG_ZEROCOPY
Prepare the datapath for refcounted ubuf_info. Clone ubuf_info with
skb_zerocopy_clone() wherever needed due to skb split, merge, resize
or clone.
Split skb_orphan_frags into two variants. The split, merge, .. paths
support reference counted zerocopy buffers, so do not do a deep copy.
Add skb_orphan_frags_rx for paths that may loop packets to receive
sockets. That is not allowed, as it may cause unbounded latency.
Deep copy all zerocopy copy buffers, ref-counted or not, in this path.
The exact locations to modify were chosen by exhaustively searching
through all code that might modify skb_frag references and/or the
the SKBTX_DEV_ZEROCOPY tx_flags bit.
The changes err on the safe side, in two ways.
(1) legacy ubuf_info paths virtio and tap are not modified. They keep
a 1:1 ubuf_info to sk_buff relationship. Calls to skb_orphan_frags
still call skb_copy_ubufs and thus copy frags in this case.
(2) not all copies deep in the stack are addressed yet. skb_shift,
skb_split and skb_try_coalesce can be refined to avoid copying.
These are not in the hot path and this patch is hairy enough as
is, so that is left for future refinement.
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-04 04:29:41 +08:00
|
|
|
if (skb_zcopy(skb))
|
2017-09-01 07:48:22 +08:00
|
|
|
refcount_inc(&skb_uarg(skb)->refcnt);
|
2010-09-03 07:09:32 +08:00
|
|
|
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++)
|
2011-08-23 07:44:58 +08:00
|
|
|
skb_frag_ref(skb, i);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2010-09-03 07:09:32 +08:00
|
|
|
if (skb_has_frag_list(skb))
|
|
|
|
skb_clone_fraglist(skb);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2010-09-03 07:09:32 +08:00
|
|
|
skb_release_data(skb);
|
2012-05-04 22:26:51 +08:00
|
|
|
} else {
|
|
|
|
skb_free_head(skb);
|
2010-09-03 07:09:32 +08:00
|
|
|
}
|
2005-04-17 06:20:36 +08:00
|
|
|
off = (data + nhead) - skb->head;
|
|
|
|
|
|
|
|
skb->head = data;
|
net: allow skb->head to be a page fragment
skb->head is currently allocated from kmalloc(). This is convenient but
has the drawback the data cannot be converted to a page fragment if
needed.
We have three spots were it hurts :
1) GRO aggregation
When a linear skb must be appended to another skb, GRO uses the
frag_list fallback, very inefficient since we keep all struct sk_buff
around. So drivers enabling GRO but delivering linear skbs to network
stack aren't enabling full GRO power.
2) splice(socket -> pipe).
We must copy the linear part to a page fragment.
This kind of defeats splice() purpose (zero copy claim)
3) TCP coalescing.
Recently introduced, this permits to group several contiguous segments
into a single skb. This shortens queue lengths and save kernel memory,
and greatly reduce probabilities of TCP collapses. This coalescing
doesnt work on linear skbs (or we would need to copy data, this would be
too slow)
Given all these issues, the following patch introduces the possibility
of having skb->head be a fragment in itself. We use a new skb flag,
skb->head_frag to carry this information.
build_skb() is changed to accept a frag_size argument. Drivers willing
to provide a page fragment instead of kmalloc() data will set a non zero
value, set to the fragment size.
Then, on situations we need to convert the skb head to a frag in itself,
we can check if skb->head_frag is set and avoid the copies or various
fallbacks we have.
This means drivers currently using frags could be updated to avoid the
current skb->head allocation and reduce their memory footprint (aka skb
truesize). (thats 512 or 1024 bytes saved per skb). This also makes
bpf/netfilter faster since the 'first frag' will be part of skb linear
part, no need to copy data.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Maciej Żenczykowski <maze@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Tom Herbert <therbert@google.com>
Cc: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Cc: Ben Hutchings <bhutchings@solarflare.com>
Cc: Matt Carlson <mcarlson@broadcom.com>
Cc: Michael Chan <mchan@broadcom.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2012-04-27 08:33:38 +08:00
|
|
|
skb->head_frag = 0;
|
2005-04-17 06:20:36 +08:00
|
|
|
skb->data += off;
|
2022-02-22 11:21:12 +08:00
|
|
|
|
|
|
|
skb_set_end_offset(skb, size);
|
2007-04-20 11:43:29 +08:00
|
|
|
#ifdef NET_SKBUFF_DATA_USES_OFFSET
|
2007-04-10 02:45:04 +08:00
|
|
|
off = nhead;
|
|
|
|
#endif
|
2007-04-20 11:29:13 +08:00
|
|
|
skb->tail += off;
|
2013-06-06 21:27:21 +08:00
|
|
|
skb_headers_offset_update(skb, nhead);
|
2005-04-17 06:20:36 +08:00
|
|
|
skb->cloned = 0;
|
[SKBUFF]: Keep track of writable header len of headerless clones
Currently NAT (and others) that want to modify cloned skbs copy them,
even if in the vast majority of cases its not necessary because the
skb is a clone made by TCP and the portion NAT wants to modify is
actually writable because TCP release the header reference before
cloning.
The problem is that there is no clean way for NAT to find out how
long the writable header area is, so this patch introduces skb->hdr_len
to hold this length. When a headerless skb is cloned skb->hdr_len
is set to the current headroom, for regular clones it is copied from
the original. A new function skb_clone_writable(skb, len) returns
whether the skb is writable up to len bytes from skb->data. To avoid
enlarging the skb the mac_len field is reduced to 16 bit and the
new hdr_len field is put in the remaining 16 bit.
I've done a few rough benchmarks of NAT (not with this exact patch,
but a very similar one). As expected it saves huge amounts of system
time in case of sendfile, bringing it down to basically the same
amount as without NAT, with sendmsg it only helps on loopback,
probably because of the large MTU.
Transmit a 1GB file using sendfile/sendmsg over eth0/lo with and
without NAT:
- sendfile eth0, no NAT: sys 0m0.388s
- sendfile eth0, NAT: sys 0m1.835s
- sendfile eth0: NAT + path: sys 0m0.370s (~ -80%)
- sendfile lo, no NAT: sys 0m0.258s
- sendfile lo, NAT: sys 0m2.609s
- sendfile lo, NAT + patch: sys 0m0.260s (~ -90%)
- sendmsg eth0, no NAT: sys 0m2.508s
- sendmsg eth0, NAT: sys 0m2.539s
- sendmsg eth0, NAT + patch: sys 0m2.445s (no change)
- sendmsg lo, no NAT: sys 0m2.151s
- sendmsg lo, NAT: sys 0m3.557s
- sendmsg lo, NAT + patch: sys 0m2.159s (~ -40%)
I expect other users can see a similar performance improvement,
packet mangling iptables targets, ipip and ip_gre come to mind ..
Signed-off-by: Patrick McHardy <kaber@trash.net>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-06-25 19:35:20 +08:00
|
|
|
skb->hdr_len = 0;
|
2005-04-17 06:20:36 +08:00
|
|
|
skb->nohdr = 0;
|
|
|
|
atomic_set(&skb_shinfo(skb)->dataref, 1);
|
2017-01-27 23:11:27 +08:00
|
|
|
|
bpf: add meta pointer for direct access
This work enables generic transfer of metadata from XDP into skb. The
basic idea is that we can make use of the fact that the resulting skb
must be linear and already comes with a larger headroom for supporting
bpf_xdp_adjust_head(), which mangles xdp->data. Here, we base our work
on a similar principle and introduce a small helper bpf_xdp_adjust_meta()
for adjusting a new pointer called xdp->data_meta. Thus, the packet has
a flexible and programmable room for meta data, followed by the actual
packet data. struct xdp_buff is therefore laid out that we first point
to data_hard_start, then data_meta directly prepended to data followed
by data_end marking the end of packet. bpf_xdp_adjust_head() takes into
account whether we have meta data already prepended and if so, memmove()s
this along with the given offset provided there's enough room.
xdp->data_meta is optional and programs are not required to use it. The
rationale is that when we process the packet in XDP (e.g. as DoS filter),
we can push further meta data along with it for the XDP_PASS case, and
give the guarantee that a clsact ingress BPF program on the same device
can pick this up for further post-processing. Since we work with skb
there, we can also set skb->mark, skb->priority or other skb meta data
out of BPF, thus having this scratch space generic and programmable
allows for more flexibility than defining a direct 1:1 transfer of
potentially new XDP members into skb (it's also more efficient as we
don't need to initialize/handle each of such new members). The facility
also works together with GRO aggregation. The scratch space at the head
of the packet can be multiple of 4 byte up to 32 byte large. Drivers not
yet supporting xdp->data_meta can simply be set up with xdp->data_meta
as xdp->data + 1 as bpf_xdp_adjust_meta() will detect this and bail out,
such that the subsequent match against xdp->data for later access is
guaranteed to fail.
The verifier treats xdp->data_meta/xdp->data the same way as we treat
xdp->data/xdp->data_end pointer comparisons. The requirement for doing
the compare against xdp->data is that it hasn't been modified from it's
original address we got from ctx access. It may have a range marking
already from prior successful xdp->data/xdp->data_end pointer comparisons
though.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: John Fastabend <john.fastabend@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-09-25 08:25:51 +08:00
|
|
|
skb_metadata_clear(skb);
|
|
|
|
|
2017-01-27 23:11:27 +08:00
|
|
|
/* It is not generally safe to change skb->truesize.
|
|
|
|
* For the moment, we really care of rx path, or
|
|
|
|
* when skb is orphaned (not attached to a socket).
|
|
|
|
*/
|
|
|
|
if (!skb->sk || skb->destructor == sock_edemux)
|
|
|
|
skb->truesize += size - osize;
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
return 0;
|
|
|
|
|
2011-07-06 20:22:12 +08:00
|
|
|
nofrags:
|
|
|
|
kfree(data);
|
2005-04-17 06:20:36 +08:00
|
|
|
nodata:
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(pskb_expand_head);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
/* Make private copy of skb with writable head and some headroom */
|
|
|
|
|
|
|
|
struct sk_buff *skb_realloc_headroom(struct sk_buff *skb, unsigned int headroom)
|
|
|
|
{
|
|
|
|
struct sk_buff *skb2;
|
|
|
|
int delta = headroom - skb_headroom(skb);
|
|
|
|
|
|
|
|
if (delta <= 0)
|
|
|
|
skb2 = pskb_copy(skb, GFP_ATOMIC);
|
|
|
|
else {
|
|
|
|
skb2 = skb_clone(skb, GFP_ATOMIC);
|
|
|
|
if (skb2 && pskb_expand_head(skb2, SKB_DATA_ALIGN(delta), 0,
|
|
|
|
GFP_ATOMIC)) {
|
|
|
|
kfree_skb(skb2);
|
|
|
|
skb2 = NULL;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return skb2;
|
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(skb_realloc_headroom);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2022-02-22 11:21:13 +08:00
|
|
|
int __skb_unclone_keeptruesize(struct sk_buff *skb, gfp_t pri)
|
|
|
|
{
|
|
|
|
unsigned int saved_end_offset, saved_truesize;
|
|
|
|
struct skb_shared_info *shinfo;
|
|
|
|
int res;
|
|
|
|
|
|
|
|
saved_end_offset = skb_end_offset(skb);
|
|
|
|
saved_truesize = skb->truesize;
|
|
|
|
|
|
|
|
res = pskb_expand_head(skb, 0, 0, pri);
|
|
|
|
if (res)
|
|
|
|
return res;
|
|
|
|
|
|
|
|
skb->truesize = saved_truesize;
|
|
|
|
|
|
|
|
if (likely(skb_end_offset(skb) == saved_end_offset))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
shinfo = skb_shinfo(skb);
|
|
|
|
|
|
|
|
/* We are about to change back skb->end,
|
|
|
|
* we need to move skb_shinfo() to its new location.
|
|
|
|
*/
|
|
|
|
memmove(skb->head + saved_end_offset,
|
|
|
|
shinfo,
|
|
|
|
offsetof(struct skb_shared_info, frags[shinfo->nr_frags]));
|
|
|
|
|
|
|
|
skb_set_end_offset(skb, saved_end_offset);
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2021-08-02 16:52:15 +08:00
|
|
|
/**
|
|
|
|
* skb_expand_head - reallocate header of &sk_buff
|
|
|
|
* @skb: buffer to reallocate
|
|
|
|
* @headroom: needed headroom
|
|
|
|
*
|
|
|
|
* Unlike skb_realloc_headroom, this one does not allocate a new skb
|
|
|
|
* if possible; copies skb->sk to new skb as needed
|
|
|
|
* and frees original skb in case of failures.
|
|
|
|
*
|
|
|
|
* It expect increased headroom and generates warning otherwise.
|
|
|
|
*/
|
|
|
|
|
|
|
|
struct sk_buff *skb_expand_head(struct sk_buff *skb, unsigned int headroom)
|
|
|
|
{
|
|
|
|
int delta = headroom - skb_headroom(skb);
|
2021-10-22 18:28:37 +08:00
|
|
|
int osize = skb_end_offset(skb);
|
|
|
|
struct sock *sk = skb->sk;
|
2021-08-02 16:52:15 +08:00
|
|
|
|
|
|
|
if (WARN_ONCE(delta <= 0,
|
|
|
|
"%s is expecting an increase in the headroom", __func__))
|
|
|
|
return skb;
|
|
|
|
|
2021-10-22 18:28:37 +08:00
|
|
|
delta = SKB_DATA_ALIGN(delta);
|
|
|
|
/* pskb_expand_head() might crash, if skb is shared. */
|
|
|
|
if (skb_shared(skb) || !is_skb_wmem(skb)) {
|
2021-08-02 16:52:15 +08:00
|
|
|
struct sk_buff *nskb = skb_clone(skb, GFP_ATOMIC);
|
|
|
|
|
2021-10-22 18:28:37 +08:00
|
|
|
if (unlikely(!nskb))
|
|
|
|
goto fail;
|
|
|
|
|
|
|
|
if (sk)
|
|
|
|
skb_set_owner_w(nskb, sk);
|
|
|
|
consume_skb(skb);
|
2021-08-02 16:52:15 +08:00
|
|
|
skb = nskb;
|
|
|
|
}
|
2021-10-22 18:28:37 +08:00
|
|
|
if (pskb_expand_head(skb, delta, 0, GFP_ATOMIC))
|
|
|
|
goto fail;
|
|
|
|
|
|
|
|
if (sk && is_skb_wmem(skb)) {
|
|
|
|
delta = skb_end_offset(skb) - osize;
|
|
|
|
refcount_add(delta, &sk->sk_wmem_alloc);
|
|
|
|
skb->truesize += delta;
|
2021-08-02 16:52:15 +08:00
|
|
|
}
|
|
|
|
return skb;
|
2021-10-22 18:28:37 +08:00
|
|
|
|
|
|
|
fail:
|
|
|
|
kfree_skb(skb);
|
|
|
|
return NULL;
|
2021-08-02 16:52:15 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_expand_head);
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/**
|
|
|
|
* skb_copy_expand - copy and expand sk_buff
|
|
|
|
* @skb: buffer to copy
|
|
|
|
* @newheadroom: new free bytes at head
|
|
|
|
* @newtailroom: new free bytes at tail
|
|
|
|
* @gfp_mask: allocation priority
|
|
|
|
*
|
|
|
|
* Make a copy of both an &sk_buff and its data and while doing so
|
|
|
|
* allocate additional space.
|
|
|
|
*
|
|
|
|
* This is used when the caller wishes to modify the data and needs a
|
|
|
|
* private copy of the data to alter as well as more space for new fields.
|
|
|
|
* Returns %NULL on failure or the pointer to the buffer
|
|
|
|
* on success. The returned buffer has a reference count of 1.
|
|
|
|
*
|
|
|
|
* You must pass %GFP_ATOMIC as the allocation priority if this function
|
|
|
|
* is called from an interrupt.
|
|
|
|
*/
|
|
|
|
struct sk_buff *skb_copy_expand(const struct sk_buff *skb,
|
2005-07-09 05:57:47 +08:00
|
|
|
int newheadroom, int newtailroom,
|
2005-10-07 14:46:04 +08:00
|
|
|
gfp_t gfp_mask)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Allocate the copy buffer
|
|
|
|
*/
|
2012-08-01 07:44:19 +08:00
|
|
|
struct sk_buff *n = __alloc_skb(newheadroom + skb->len + newtailroom,
|
|
|
|
gfp_mask, skb_alloc_rx_flag(skb),
|
|
|
|
NUMA_NO_NODE);
|
2007-04-11 09:30:09 +08:00
|
|
|
int oldheadroom = skb_headroom(skb);
|
2005-04-17 06:20:36 +08:00
|
|
|
int head_copy_len, head_copy_off;
|
|
|
|
|
|
|
|
if (!n)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
skb_reserve(n, newheadroom);
|
|
|
|
|
|
|
|
/* Set the tail pointer and length */
|
|
|
|
skb_put(n, skb->len);
|
|
|
|
|
2007-04-11 09:30:09 +08:00
|
|
|
head_copy_len = oldheadroom;
|
2005-04-17 06:20:36 +08:00
|
|
|
head_copy_off = 0;
|
|
|
|
if (newheadroom <= head_copy_len)
|
|
|
|
head_copy_len = newheadroom;
|
|
|
|
else
|
|
|
|
head_copy_off = newheadroom - head_copy_len;
|
|
|
|
|
|
|
|
/* Copy the linear header and data. */
|
2017-10-09 23:37:59 +08:00
|
|
|
BUG_ON(skb_copy_bits(skb, -head_copy_len, n->head + head_copy_off,
|
|
|
|
skb->len + head_copy_len));
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2018-04-30 15:16:11 +08:00
|
|
|
skb_copy_header(n, skb);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2013-10-20 02:42:54 +08:00
|
|
|
skb_headers_offset_update(n, newheadroom - oldheadroom);
|
2007-04-11 09:30:09 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
return n;
|
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(skb_copy_expand);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
/**
|
2017-08-23 06:12:14 +08:00
|
|
|
* __skb_pad - zero pad the tail of an skb
|
2005-04-17 06:20:36 +08:00
|
|
|
* @skb: buffer to pad
|
|
|
|
* @pad: space to pad
|
2017-08-23 06:12:14 +08:00
|
|
|
* @free_on_error: free buffer on error
|
2005-04-17 06:20:36 +08:00
|
|
|
*
|
|
|
|
* Ensure that a buffer is followed by a padding area that is zero
|
|
|
|
* filled. Used by network drivers which may DMA or transfer data
|
|
|
|
* beyond the buffer end onto the wire.
|
|
|
|
*
|
2017-08-23 06:12:14 +08:00
|
|
|
* May return error in out of memory cases. The skb is freed on error
|
|
|
|
* if @free_on_error is true.
|
2005-04-17 06:20:36 +08:00
|
|
|
*/
|
2007-02-09 22:24:36 +08:00
|
|
|
|
2017-08-23 06:12:14 +08:00
|
|
|
int __skb_pad(struct sk_buff *skb, int pad, bool free_on_error)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2006-06-23 17:06:41 +08:00
|
|
|
int err;
|
|
|
|
int ntail;
|
2007-02-09 22:24:36 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/* If the skbuff is non linear tailroom is always zero.. */
|
2006-06-23 17:06:41 +08:00
|
|
|
if (!skb_cloned(skb) && skb_tailroom(skb) >= pad) {
|
2005-04-17 06:20:36 +08:00
|
|
|
memset(skb->data+skb->len, 0, pad);
|
2006-06-23 17:06:41 +08:00
|
|
|
return 0;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
2006-06-23 17:06:41 +08:00
|
|
|
|
2007-04-20 11:43:29 +08:00
|
|
|
ntail = skb->data_len + pad - (skb->end - skb->tail);
|
2006-06-23 17:06:41 +08:00
|
|
|
if (likely(skb_cloned(skb) || ntail > 0)) {
|
|
|
|
err = pskb_expand_head(skb, 0, ntail, GFP_ATOMIC);
|
|
|
|
if (unlikely(err))
|
|
|
|
goto free_skb;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* FIXME: The use of this function with non-linear skb's really needs
|
|
|
|
* to be audited.
|
|
|
|
*/
|
|
|
|
err = skb_linearize(skb);
|
|
|
|
if (unlikely(err))
|
|
|
|
goto free_skb;
|
|
|
|
|
|
|
|
memset(skb->data + skb->len, 0, pad);
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
free_skb:
|
2017-08-23 06:12:14 +08:00
|
|
|
if (free_on_error)
|
|
|
|
kfree_skb(skb);
|
2006-06-23 17:06:41 +08:00
|
|
|
return err;
|
2007-02-09 22:24:36 +08:00
|
|
|
}
|
2017-08-23 06:12:14 +08:00
|
|
|
EXPORT_SYMBOL(__skb_pad);
|
2007-02-09 22:24:36 +08:00
|
|
|
|
2013-11-07 21:18:24 +08:00
|
|
|
/**
|
|
|
|
* pskb_put - add data to the tail of a potentially fragmented buffer
|
|
|
|
* @skb: start of the buffer to use
|
|
|
|
* @tail: tail fragment of the buffer to use
|
|
|
|
* @len: amount of data to add
|
|
|
|
*
|
|
|
|
* This function extends the used data area of the potentially
|
|
|
|
* fragmented buffer. @tail must be the last fragment of @skb -- or
|
|
|
|
* @skb itself. If this would exceed the total buffer size the kernel
|
|
|
|
* will panic. A pointer to the first byte of the extra data is
|
|
|
|
* returned.
|
|
|
|
*/
|
|
|
|
|
networking: make skb_put & friends return void pointers
It seems like a historic accident that these return unsigned char *,
and in many places that means casts are required, more often than not.
Make these functions (skb_put, __skb_put and pskb_put) return void *
and remove all the casts across the tree, adding a (u8 *) cast only
where the unsigned char pointer was used directly, all done with the
following spatch:
@@
expression SKB, LEN;
typedef u8;
identifier fn = { skb_put, __skb_put };
@@
- *(fn(SKB, LEN))
+ *(u8 *)fn(SKB, LEN)
@@
expression E, SKB, LEN;
identifier fn = { skb_put, __skb_put };
type T;
@@
- E = ((T *)(fn(SKB, LEN)))
+ E = fn(SKB, LEN)
which actually doesn't cover pskb_put since there are only three
users overall.
A handful of stragglers were converted manually, notably a macro in
drivers/isdn/i4l/isdn_bsdcomp.c and, oddly enough, one of the many
instances in net/bluetooth/hci_sock.c. In the former file, I also
had to fix one whitespace problem spatch introduced.
Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-06-16 20:29:21 +08:00
|
|
|
void *pskb_put(struct sk_buff *skb, struct sk_buff *tail, int len)
|
2013-11-07 21:18:24 +08:00
|
|
|
{
|
|
|
|
if (tail != skb) {
|
|
|
|
skb->data_len += len;
|
|
|
|
skb->len += len;
|
|
|
|
}
|
|
|
|
return skb_put(tail, len);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(pskb_put);
|
|
|
|
|
2008-03-28 08:43:41 +08:00
|
|
|
/**
|
|
|
|
* skb_put - add data to a buffer
|
|
|
|
* @skb: buffer to use
|
|
|
|
* @len: amount of data to add
|
|
|
|
*
|
|
|
|
* This function extends the used data area of the buffer. If this would
|
|
|
|
* exceed the total buffer size the kernel will panic. A pointer to the
|
|
|
|
* first byte of the extra data is returned.
|
|
|
|
*/
|
networking: make skb_put & friends return void pointers
It seems like a historic accident that these return unsigned char *,
and in many places that means casts are required, more often than not.
Make these functions (skb_put, __skb_put and pskb_put) return void *
and remove all the casts across the tree, adding a (u8 *) cast only
where the unsigned char pointer was used directly, all done with the
following spatch:
@@
expression SKB, LEN;
typedef u8;
identifier fn = { skb_put, __skb_put };
@@
- *(fn(SKB, LEN))
+ *(u8 *)fn(SKB, LEN)
@@
expression E, SKB, LEN;
identifier fn = { skb_put, __skb_put };
type T;
@@
- E = ((T *)(fn(SKB, LEN)))
+ E = fn(SKB, LEN)
which actually doesn't cover pskb_put since there are only three
users overall.
A handful of stragglers were converted manually, notably a macro in
drivers/isdn/i4l/isdn_bsdcomp.c and, oddly enough, one of the many
instances in net/bluetooth/hci_sock.c. In the former file, I also
had to fix one whitespace problem spatch introduced.
Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-06-16 20:29:21 +08:00
|
|
|
void *skb_put(struct sk_buff *skb, unsigned int len)
|
2008-03-28 08:43:41 +08:00
|
|
|
{
|
networking: make skb_put & friends return void pointers
It seems like a historic accident that these return unsigned char *,
and in many places that means casts are required, more often than not.
Make these functions (skb_put, __skb_put and pskb_put) return void *
and remove all the casts across the tree, adding a (u8 *) cast only
where the unsigned char pointer was used directly, all done with the
following spatch:
@@
expression SKB, LEN;
typedef u8;
identifier fn = { skb_put, __skb_put };
@@
- *(fn(SKB, LEN))
+ *(u8 *)fn(SKB, LEN)
@@
expression E, SKB, LEN;
identifier fn = { skb_put, __skb_put };
type T;
@@
- E = ((T *)(fn(SKB, LEN)))
+ E = fn(SKB, LEN)
which actually doesn't cover pskb_put since there are only three
users overall.
A handful of stragglers were converted manually, notably a macro in
drivers/isdn/i4l/isdn_bsdcomp.c and, oddly enough, one of the many
instances in net/bluetooth/hci_sock.c. In the former file, I also
had to fix one whitespace problem spatch introduced.
Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-06-16 20:29:21 +08:00
|
|
|
void *tmp = skb_tail_pointer(skb);
|
2008-03-28 08:43:41 +08:00
|
|
|
SKB_LINEAR_ASSERT(skb);
|
|
|
|
skb->tail += len;
|
|
|
|
skb->len += len;
|
|
|
|
if (unlikely(skb->tail > skb->end))
|
|
|
|
skb_over_panic(skb, len, __builtin_return_address(0));
|
|
|
|
return tmp;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_put);
|
|
|
|
|
2008-03-28 08:52:40 +08:00
|
|
|
/**
|
|
|
|
* skb_push - add data to the start of a buffer
|
|
|
|
* @skb: buffer to use
|
|
|
|
* @len: amount of data to add
|
|
|
|
*
|
|
|
|
* This function extends the used data area of the buffer at the buffer
|
|
|
|
* start. If this would exceed the total buffer headroom the kernel will
|
|
|
|
* panic. A pointer to the first byte of the extra data is returned.
|
|
|
|
*/
|
networking: make skb_push & __skb_push return void pointers
It seems like a historic accident that these return unsigned char *,
and in many places that means casts are required, more often than not.
Make these functions return void * and remove all the casts across
the tree, adding a (u8 *) cast only where the unsigned char pointer
was used directly, all done with the following spatch:
@@
expression SKB, LEN;
typedef u8;
identifier fn = { skb_push, __skb_push, skb_push_rcsum };
@@
- *(fn(SKB, LEN))
+ *(u8 *)fn(SKB, LEN)
@@
expression E, SKB, LEN;
identifier fn = { skb_push, __skb_push, skb_push_rcsum };
type T;
@@
- E = ((T *)(fn(SKB, LEN)))
+ E = fn(SKB, LEN)
@@
expression SKB, LEN;
identifier fn = { skb_push, __skb_push, skb_push_rcsum };
@@
- fn(SKB, LEN)[0]
+ *(u8 *)fn(SKB, LEN)
Note that the last part there converts from push(...)[0] to the
more idiomatic *(u8 *)push(...).
Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-06-16 20:29:23 +08:00
|
|
|
void *skb_push(struct sk_buff *skb, unsigned int len)
|
2008-03-28 08:52:40 +08:00
|
|
|
{
|
|
|
|
skb->data -= len;
|
|
|
|
skb->len += len;
|
2018-08-02 18:04:52 +08:00
|
|
|
if (unlikely(skb->data < skb->head))
|
2008-03-28 08:52:40 +08:00
|
|
|
skb_under_panic(skb, len, __builtin_return_address(0));
|
|
|
|
return skb->data;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_push);
|
|
|
|
|
2008-03-28 08:47:24 +08:00
|
|
|
/**
|
|
|
|
* skb_pull - remove data from the start of a buffer
|
|
|
|
* @skb: buffer to use
|
|
|
|
* @len: amount of data to remove
|
|
|
|
*
|
|
|
|
* This function removes data from the start of a buffer, returning
|
|
|
|
* the memory to the headroom. A pointer to the next data in the buffer
|
|
|
|
* is returned. Once the data has been pulled future pushes will overwrite
|
|
|
|
* the old data.
|
|
|
|
*/
|
networking: make skb_pull & friends return void pointers
It seems like a historic accident that these return unsigned char *,
and in many places that means casts are required, more often than not.
Make these functions return void * and remove all the casts across
the tree, adding a (u8 *) cast only where the unsigned char pointer
was used directly, all done with the following spatch:
@@
expression SKB, LEN;
typedef u8;
identifier fn = {
skb_pull,
__skb_pull,
skb_pull_inline,
__pskb_pull_tail,
__pskb_pull,
pskb_pull
};
@@
- *(fn(SKB, LEN))
+ *(u8 *)fn(SKB, LEN)
@@
expression E, SKB, LEN;
identifier fn = {
skb_pull,
__skb_pull,
skb_pull_inline,
__pskb_pull_tail,
__pskb_pull,
pskb_pull
};
type T;
@@
- E = ((T *)(fn(SKB, LEN)))
+ E = fn(SKB, LEN)
Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-06-16 20:29:22 +08:00
|
|
|
void *skb_pull(struct sk_buff *skb, unsigned int len)
|
2008-03-28 08:47:24 +08:00
|
|
|
{
|
2010-05-02 17:21:44 +08:00
|
|
|
return skb_pull_inline(skb, len);
|
2008-03-28 08:47:24 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_pull);
|
|
|
|
|
2021-12-02 02:54:52 +08:00
|
|
|
/**
|
|
|
|
* skb_pull_data - remove data from the start of a buffer returning its
|
|
|
|
* original position.
|
|
|
|
* @skb: buffer to use
|
|
|
|
* @len: amount of data to remove
|
|
|
|
*
|
|
|
|
* This function removes data from the start of a buffer, returning
|
|
|
|
* the memory to the headroom. A pointer to the original data in the buffer
|
|
|
|
* is returned after checking if there is enough data to pull. Once the
|
|
|
|
* data has been pulled future pushes will overwrite the old data.
|
|
|
|
*/
|
|
|
|
void *skb_pull_data(struct sk_buff *skb, size_t len)
|
|
|
|
{
|
|
|
|
void *data = skb->data;
|
|
|
|
|
|
|
|
if (skb->len < len)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
skb_pull(skb, len);
|
|
|
|
|
|
|
|
return data;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_pull_data);
|
|
|
|
|
2008-03-28 08:54:01 +08:00
|
|
|
/**
|
|
|
|
* skb_trim - remove end from a buffer
|
|
|
|
* @skb: buffer to alter
|
|
|
|
* @len: new length
|
|
|
|
*
|
|
|
|
* Cut the length of a buffer down by removing data from the tail. If
|
|
|
|
* the buffer is already under the length specified it is not modified.
|
|
|
|
* The skb must be linear.
|
|
|
|
*/
|
|
|
|
void skb_trim(struct sk_buff *skb, unsigned int len)
|
|
|
|
{
|
|
|
|
if (skb->len > len)
|
|
|
|
__skb_trim(skb, len);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_trim);
|
|
|
|
|
2006-06-10 07:13:38 +08:00
|
|
|
/* Trims skb to length len. It can change skb pointers.
|
2005-04-17 06:20:36 +08:00
|
|
|
*/
|
|
|
|
|
2006-06-10 07:13:38 +08:00
|
|
|
int ___pskb_trim(struct sk_buff *skb, unsigned int len)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2006-07-14 10:26:39 +08:00
|
|
|
struct sk_buff **fragp;
|
|
|
|
struct sk_buff *frag;
|
2005-04-17 06:20:36 +08:00
|
|
|
int offset = skb_headlen(skb);
|
|
|
|
int nfrags = skb_shinfo(skb)->nr_frags;
|
|
|
|
int i;
|
2006-07-14 10:26:39 +08:00
|
|
|
int err;
|
|
|
|
|
|
|
|
if (skb_cloned(skb) &&
|
|
|
|
unlikely((err = pskb_expand_head(skb, 0, 0, GFP_ATOMIC))))
|
|
|
|
return err;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2006-07-31 11:20:28 +08:00
|
|
|
i = 0;
|
|
|
|
if (offset >= len)
|
|
|
|
goto drop_pages;
|
|
|
|
|
|
|
|
for (; i < nfrags; i++) {
|
2011-10-19 05:00:24 +08:00
|
|
|
int end = offset + skb_frag_size(&skb_shinfo(skb)->frags[i]);
|
2006-07-14 10:26:39 +08:00
|
|
|
|
|
|
|
if (end < len) {
|
|
|
|
offset = end;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
2011-10-19 05:00:24 +08:00
|
|
|
skb_frag_size_set(&skb_shinfo(skb)->frags[i++], len - offset);
|
2006-07-14 10:26:39 +08:00
|
|
|
|
2006-07-31 11:20:28 +08:00
|
|
|
drop_pages:
|
2006-07-14 10:26:39 +08:00
|
|
|
skb_shinfo(skb)->nr_frags = i;
|
|
|
|
|
|
|
|
for (; i < nfrags; i++)
|
2011-08-23 07:44:58 +08:00
|
|
|
skb_frag_unref(skb, i);
|
2006-07-14 10:26:39 +08:00
|
|
|
|
2010-08-23 15:13:46 +08:00
|
|
|
if (skb_has_frag_list(skb))
|
2006-07-14 10:26:39 +08:00
|
|
|
skb_drop_fraglist(skb);
|
2006-07-31 11:20:28 +08:00
|
|
|
goto done;
|
2006-07-14 10:26:39 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
for (fragp = &skb_shinfo(skb)->frag_list; (frag = *fragp);
|
|
|
|
fragp = &frag->next) {
|
|
|
|
int end = offset + frag->len;
|
|
|
|
|
|
|
|
if (skb_shared(frag)) {
|
|
|
|
struct sk_buff *nfrag;
|
|
|
|
|
|
|
|
nfrag = skb_clone(frag, GFP_ATOMIC);
|
|
|
|
if (unlikely(!nfrag))
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
nfrag->next = frag->next;
|
2012-04-19 10:24:53 +08:00
|
|
|
consume_skb(frag);
|
2006-07-14 10:26:39 +08:00
|
|
|
frag = nfrag;
|
|
|
|
*fragp = frag;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
2006-07-14 10:26:39 +08:00
|
|
|
|
|
|
|
if (end < len) {
|
|
|
|
offset = end;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (end > len &&
|
|
|
|
unlikely((err = pskb_trim(frag, len - offset))))
|
|
|
|
return err;
|
|
|
|
|
|
|
|
if (frag->next)
|
|
|
|
skb_drop_list(&frag->next);
|
|
|
|
break;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
2006-07-31 11:20:28 +08:00
|
|
|
done:
|
2006-07-14 10:26:39 +08:00
|
|
|
if (len > skb_headlen(skb)) {
|
2005-04-17 06:20:36 +08:00
|
|
|
skb->data_len -= skb->len - len;
|
|
|
|
skb->len = len;
|
|
|
|
} else {
|
2006-07-14 10:26:39 +08:00
|
|
|
skb->len = len;
|
|
|
|
skb->data_len = 0;
|
2007-04-20 11:29:13 +08:00
|
|
|
skb_set_tail_pointer(skb, len);
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
2017-04-27 00:07:46 +08:00
|
|
|
if (!skb->sk || skb->destructor == sock_edemux)
|
|
|
|
skb_condense(skb);
|
2005-04-17 06:20:36 +08:00
|
|
|
return 0;
|
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(___pskb_trim);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2018-04-19 02:43:15 +08:00
|
|
|
/* Note : use pskb_trim_rcsum() instead of calling this directly
|
|
|
|
*/
|
|
|
|
int pskb_trim_rcsum_slow(struct sk_buff *skb, unsigned int len)
|
|
|
|
{
|
|
|
|
if (skb->ip_summed == CHECKSUM_COMPLETE) {
|
|
|
|
int delta = skb->len - len;
|
|
|
|
|
2018-10-20 08:07:13 +08:00
|
|
|
skb->csum = csum_block_sub(skb->csum,
|
|
|
|
skb_checksum(skb, len, delta, 0),
|
|
|
|
len);
|
2020-12-15 03:07:39 +08:00
|
|
|
} else if (skb->ip_summed == CHECKSUM_PARTIAL) {
|
|
|
|
int hdlen = (len > skb_headlen(skb)) ? skb_headlen(skb) : len;
|
|
|
|
int offset = skb_checksum_start_offset(skb) + skb->csum_offset;
|
|
|
|
|
|
|
|
if (offset + sizeof(__sum16) > hdlen)
|
|
|
|
return -EINVAL;
|
2018-04-19 02:43:15 +08:00
|
|
|
}
|
|
|
|
return __pskb_trim(skb, len);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(pskb_trim_rcsum_slow);
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/**
|
|
|
|
* __pskb_pull_tail - advance tail of skb header
|
|
|
|
* @skb: buffer to reallocate
|
|
|
|
* @delta: number of bytes to advance tail
|
|
|
|
*
|
|
|
|
* The function makes a sense only on a fragmented &sk_buff,
|
|
|
|
* it expands header moving its tail forward and copying necessary
|
|
|
|
* data from fragmented part.
|
|
|
|
*
|
|
|
|
* &sk_buff MUST have reference count of 1.
|
|
|
|
*
|
|
|
|
* Returns %NULL (and &sk_buff does not change) if pull failed
|
|
|
|
* or value of new tail of skb in the case of success.
|
|
|
|
*
|
|
|
|
* All the pointers pointing into skb header may change and must be
|
|
|
|
* reloaded after call to this function.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/* Moves tail of skb head forward, copying data from fragmented part,
|
|
|
|
* when it is necessary.
|
|
|
|
* 1. It may fail due to malloc failure.
|
|
|
|
* 2. It may change skb pointers.
|
|
|
|
*
|
|
|
|
* It is pretty complicated. Luckily, it is called only in exceptional cases.
|
|
|
|
*/
|
networking: make skb_pull & friends return void pointers
It seems like a historic accident that these return unsigned char *,
and in many places that means casts are required, more often than not.
Make these functions return void * and remove all the casts across
the tree, adding a (u8 *) cast only where the unsigned char pointer
was used directly, all done with the following spatch:
@@
expression SKB, LEN;
typedef u8;
identifier fn = {
skb_pull,
__skb_pull,
skb_pull_inline,
__pskb_pull_tail,
__pskb_pull,
pskb_pull
};
@@
- *(fn(SKB, LEN))
+ *(u8 *)fn(SKB, LEN)
@@
expression E, SKB, LEN;
identifier fn = {
skb_pull,
__skb_pull,
skb_pull_inline,
__pskb_pull_tail,
__pskb_pull,
pskb_pull
};
type T;
@@
- E = ((T *)(fn(SKB, LEN)))
+ E = fn(SKB, LEN)
Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-06-16 20:29:22 +08:00
|
|
|
void *__pskb_pull_tail(struct sk_buff *skb, int delta)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
|
|
|
/* If skb has not enough free space at tail, get new one
|
|
|
|
* plus 128 bytes for future expansions. If we have enough
|
|
|
|
* room at tail, reallocate without expansion only if skb is cloned.
|
|
|
|
*/
|
2007-04-20 11:43:29 +08:00
|
|
|
int i, k, eat = (skb->tail + delta) - skb->end;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
if (eat > 0 || skb_cloned(skb)) {
|
|
|
|
if (pskb_expand_head(skb, 0, eat > 0 ? eat + 128 : 0,
|
|
|
|
GFP_ATOMIC))
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
2017-10-09 23:37:59 +08:00
|
|
|
BUG_ON(skb_copy_bits(skb, skb_headlen(skb),
|
|
|
|
skb_tail_pointer(skb), delta));
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
/* Optimization: no fragments, no reasons to preestimate
|
|
|
|
* size of pulled pages. Superb.
|
|
|
|
*/
|
2010-08-23 15:13:46 +08:00
|
|
|
if (!skb_has_frag_list(skb))
|
2005-04-17 06:20:36 +08:00
|
|
|
goto pull_pages;
|
|
|
|
|
|
|
|
/* Estimate size of pulled pages. */
|
|
|
|
eat = delta;
|
|
|
|
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
|
2011-10-19 05:00:24 +08:00
|
|
|
int size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
|
|
|
|
|
|
|
|
if (size >= eat)
|
2005-04-17 06:20:36 +08:00
|
|
|
goto pull_pages;
|
2011-10-19 05:00:24 +08:00
|
|
|
eat -= size;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/* If we need update frag list, we are in troubles.
|
2017-10-15 00:51:36 +08:00
|
|
|
* Certainly, it is possible to add an offset to skb data,
|
2005-04-17 06:20:36 +08:00
|
|
|
* but taking into account that pulling is expected to
|
|
|
|
* be very rare operation, it is worth to fight against
|
|
|
|
* further bloating skb head and crucify ourselves here instead.
|
|
|
|
* Pure masohism, indeed. 8)8)
|
|
|
|
*/
|
|
|
|
if (eat) {
|
|
|
|
struct sk_buff *list = skb_shinfo(skb)->frag_list;
|
|
|
|
struct sk_buff *clone = NULL;
|
|
|
|
struct sk_buff *insp = NULL;
|
|
|
|
|
|
|
|
do {
|
|
|
|
if (list->len <= eat) {
|
|
|
|
/* Eaten as whole. */
|
|
|
|
eat -= list->len;
|
|
|
|
list = list->next;
|
|
|
|
insp = list;
|
|
|
|
} else {
|
|
|
|
/* Eaten partially. */
|
|
|
|
|
|
|
|
if (skb_shared(list)) {
|
|
|
|
/* Sucks! We need to fork list. :-( */
|
|
|
|
clone = skb_clone(list, GFP_ATOMIC);
|
|
|
|
if (!clone)
|
|
|
|
return NULL;
|
|
|
|
insp = list->next;
|
|
|
|
list = clone;
|
|
|
|
} else {
|
|
|
|
/* This may be pulled without
|
|
|
|
* problems. */
|
|
|
|
insp = list;
|
|
|
|
}
|
|
|
|
if (!pskb_pull(list, eat)) {
|
2009-02-25 08:37:32 +08:00
|
|
|
kfree_skb(clone);
|
2005-04-17 06:20:36 +08:00
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
} while (eat);
|
|
|
|
|
|
|
|
/* Free pulled out fragments. */
|
|
|
|
while ((list = skb_shinfo(skb)->frag_list) != insp) {
|
|
|
|
skb_shinfo(skb)->frag_list = list->next;
|
2022-02-20 23:40:52 +08:00
|
|
|
consume_skb(list);
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
/* And insert new clone at head. */
|
|
|
|
if (clone) {
|
|
|
|
clone->next = list;
|
|
|
|
skb_shinfo(skb)->frag_list = clone;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
/* Success! Now we may commit changes to skb data. */
|
|
|
|
|
|
|
|
pull_pages:
|
|
|
|
eat = delta;
|
|
|
|
k = 0;
|
|
|
|
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
|
2011-10-19 05:00:24 +08:00
|
|
|
int size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
|
|
|
|
|
|
|
|
if (size <= eat) {
|
2011-08-23 07:44:58 +08:00
|
|
|
skb_frag_unref(skb, i);
|
2011-10-19 05:00:24 +08:00
|
|
|
eat -= size;
|
2005-04-17 06:20:36 +08:00
|
|
|
} else {
|
2019-07-30 22:40:33 +08:00
|
|
|
skb_frag_t *frag = &skb_shinfo(skb)->frags[k];
|
|
|
|
|
|
|
|
*frag = skb_shinfo(skb)->frags[i];
|
2005-04-17 06:20:36 +08:00
|
|
|
if (eat) {
|
2019-07-30 22:40:33 +08:00
|
|
|
skb_frag_off_add(frag, eat);
|
|
|
|
skb_frag_size_sub(frag, eat);
|
2017-07-17 17:25:02 +08:00
|
|
|
if (!i)
|
|
|
|
goto end;
|
2005-04-17 06:20:36 +08:00
|
|
|
eat = 0;
|
|
|
|
}
|
|
|
|
k++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
skb_shinfo(skb)->nr_frags = k;
|
|
|
|
|
2017-07-17 17:25:02 +08:00
|
|
|
end:
|
2005-04-17 06:20:36 +08:00
|
|
|
skb->tail += delta;
|
|
|
|
skb->data_len -= delta;
|
|
|
|
|
sock: enable MSG_ZEROCOPY
Prepare the datapath for refcounted ubuf_info. Clone ubuf_info with
skb_zerocopy_clone() wherever needed due to skb split, merge, resize
or clone.
Split skb_orphan_frags into two variants. The split, merge, .. paths
support reference counted zerocopy buffers, so do not do a deep copy.
Add skb_orphan_frags_rx for paths that may loop packets to receive
sockets. That is not allowed, as it may cause unbounded latency.
Deep copy all zerocopy copy buffers, ref-counted or not, in this path.
The exact locations to modify were chosen by exhaustively searching
through all code that might modify skb_frag references and/or the
the SKBTX_DEV_ZEROCOPY tx_flags bit.
The changes err on the safe side, in two ways.
(1) legacy ubuf_info paths virtio and tap are not modified. They keep
a 1:1 ubuf_info to sk_buff relationship. Calls to skb_orphan_frags
still call skb_copy_ubufs and thus copy frags in this case.
(2) not all copies deep in the stack are addressed yet. skb_shift,
skb_split and skb_try_coalesce can be refined to avoid copying.
These are not in the hot path and this patch is hairy enough as
is, so that is left for future refinement.
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-04 04:29:41 +08:00
|
|
|
if (!skb->data_len)
|
|
|
|
skb_zcopy_clear(skb, false);
|
|
|
|
|
2007-04-20 11:29:13 +08:00
|
|
|
return skb_tail_pointer(skb);
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(__pskb_pull_tail);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2011-07-30 02:37:31 +08:00
|
|
|
/**
|
|
|
|
* skb_copy_bits - copy bits from skb to kernel buffer
|
|
|
|
* @skb: source skb
|
|
|
|
* @offset: offset in source
|
|
|
|
* @to: destination buffer
|
|
|
|
* @len: number of bytes to copy
|
|
|
|
*
|
|
|
|
* Copy the specified number of bytes from the source skb to the
|
|
|
|
* destination buffer.
|
|
|
|
*
|
|
|
|
* CAUTION ! :
|
|
|
|
* If its prototype is ever changed,
|
|
|
|
* check arch/{*}/net/{*}.S files,
|
|
|
|
* since it is called from BPF assembly code.
|
|
|
|
*/
|
2005-04-17 06:20:36 +08:00
|
|
|
int skb_copy_bits(const struct sk_buff *skb, int offset, void *to, int len)
|
|
|
|
{
|
2007-04-28 06:21:23 +08:00
|
|
|
int start = skb_headlen(skb);
|
2009-06-09 15:18:59 +08:00
|
|
|
struct sk_buff *frag_iter;
|
|
|
|
int i, copy;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
if (offset > (int)skb->len - len)
|
|
|
|
goto fault;
|
|
|
|
|
|
|
|
/* Copy header. */
|
2007-04-28 06:21:23 +08:00
|
|
|
if ((copy = start - offset) > 0) {
|
2005-04-17 06:20:36 +08:00
|
|
|
if (copy > len)
|
|
|
|
copy = len;
|
2007-03-28 05:55:52 +08:00
|
|
|
skb_copy_from_linear_data_offset(skb, offset, to, copy);
|
2005-04-17 06:20:36 +08:00
|
|
|
if ((len -= copy) == 0)
|
|
|
|
return 0;
|
|
|
|
offset += copy;
|
|
|
|
to += copy;
|
|
|
|
}
|
|
|
|
|
|
|
|
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
|
2007-04-28 06:21:23 +08:00
|
|
|
int end;
|
2012-04-05 17:35:15 +08:00
|
|
|
skb_frag_t *f = &skb_shinfo(skb)->frags[i];
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2008-07-26 12:43:18 +08:00
|
|
|
WARN_ON(start > offset + len);
|
2007-04-28 06:21:23 +08:00
|
|
|
|
2012-04-05 17:35:15 +08:00
|
|
|
end = start + skb_frag_size(f);
|
2005-04-17 06:20:36 +08:00
|
|
|
if ((copy = end - offset) > 0) {
|
2017-07-31 20:15:47 +08:00
|
|
|
u32 p_off, p_len, copied;
|
|
|
|
struct page *p;
|
2005-04-17 06:20:36 +08:00
|
|
|
u8 *vaddr;
|
|
|
|
|
|
|
|
if (copy > len)
|
|
|
|
copy = len;
|
|
|
|
|
2017-07-31 20:15:47 +08:00
|
|
|
skb_frag_foreach_page(f,
|
2019-07-30 22:40:33 +08:00
|
|
|
skb_frag_off(f) + offset - start,
|
2017-07-31 20:15:47 +08:00
|
|
|
copy, p, p_off, p_len, copied) {
|
|
|
|
vaddr = kmap_atomic(p);
|
|
|
|
memcpy(to + copied, vaddr + p_off, p_len);
|
|
|
|
kunmap_atomic(vaddr);
|
|
|
|
}
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
if ((len -= copy) == 0)
|
|
|
|
return 0;
|
|
|
|
offset += copy;
|
|
|
|
to += copy;
|
|
|
|
}
|
2007-04-28 06:21:23 +08:00
|
|
|
start = end;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
2009-06-09 15:18:59 +08:00
|
|
|
skb_walk_frags(skb, frag_iter) {
|
|
|
|
int end;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2009-06-09 15:18:59 +08:00
|
|
|
WARN_ON(start > offset + len);
|
|
|
|
|
|
|
|
end = start + frag_iter->len;
|
|
|
|
if ((copy = end - offset) > 0) {
|
|
|
|
if (copy > len)
|
|
|
|
copy = len;
|
|
|
|
if (skb_copy_bits(frag_iter, offset - start, to, copy))
|
|
|
|
goto fault;
|
|
|
|
if ((len -= copy) == 0)
|
|
|
|
return 0;
|
|
|
|
offset += copy;
|
|
|
|
to += copy;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
2009-06-09 15:18:59 +08:00
|
|
|
start = end;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
2011-07-06 20:22:12 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
if (!len)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
fault:
|
|
|
|
return -EFAULT;
|
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(skb_copy_bits);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2007-11-07 15:30:13 +08:00
|
|
|
/*
|
|
|
|
* Callback from splice_to_pipe(), if we need to release some pages
|
|
|
|
* at the end of the spd in case we error'ed out in filling the pipe.
|
|
|
|
*/
|
|
|
|
static void sock_spd_release(struct splice_pipe_desc *spd, unsigned int i)
|
|
|
|
{
|
2009-01-20 09:03:56 +08:00
|
|
|
put_page(spd->pages[i]);
|
|
|
|
}
|
2007-11-07 15:30:13 +08:00
|
|
|
|
2012-04-24 11:06:11 +08:00
|
|
|
static struct page *linear_to_page(struct page *page, unsigned int *len,
|
|
|
|
unsigned int *offset,
|
2013-01-11 22:46:37 +08:00
|
|
|
struct sock *sk)
|
2009-01-20 09:03:56 +08:00
|
|
|
{
|
net: use a per task frag allocator
We currently use a per socket order-0 page cache for tcp_sendmsg()
operations.
This page is used to build fragments for skbs.
Its done to increase probability of coalescing small write() into
single segments in skbs still in write queue (not yet sent)
But it wastes a lot of memory for applications handling many mostly
idle sockets, since each socket holds one page in sk->sk_sndmsg_page
Its also quite inefficient to build TSO 64KB packets, because we need
about 16 pages per skb on arches where PAGE_SIZE = 4096, so we hit
page allocator more than wanted.
This patch adds a per task frag allocator and uses bigger pages,
if available. An automatic fallback is done in case of memory pressure.
(up to 32768 bytes per frag, thats order-3 pages on x86)
This increases TCP stream performance by 20% on loopback device,
but also benefits on other network devices, since 8x less frags are
mapped on transmit and unmapped on tx completion. Alexander Duyck
mentioned a probable performance win on systems with IOMMU enabled.
Its possible some SG enabled hardware cant cope with bigger fragments,
but their ndo_start_xmit() should already handle this, splitting a
fragment in sub fragments, since some arches have PAGE_SIZE=65536
Successfully tested on various ethernet devices.
(ixgbe, igb, bnx2x, tg3, mellanox mlx4)
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Ben Hutchings <bhutchings@solarflare.com>
Cc: Vijay Subramanian <subramanian.vijay@gmail.com>
Cc: Alexander Duyck <alexander.h.duyck@intel.com>
Tested-by: Vijay Subramanian <subramanian.vijay@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2012-09-24 07:04:42 +08:00
|
|
|
struct page_frag *pfrag = sk_page_frag(sk);
|
2009-02-01 16:41:42 +08:00
|
|
|
|
net: use a per task frag allocator
We currently use a per socket order-0 page cache for tcp_sendmsg()
operations.
This page is used to build fragments for skbs.
Its done to increase probability of coalescing small write() into
single segments in skbs still in write queue (not yet sent)
But it wastes a lot of memory for applications handling many mostly
idle sockets, since each socket holds one page in sk->sk_sndmsg_page
Its also quite inefficient to build TSO 64KB packets, because we need
about 16 pages per skb on arches where PAGE_SIZE = 4096, so we hit
page allocator more than wanted.
This patch adds a per task frag allocator and uses bigger pages,
if available. An automatic fallback is done in case of memory pressure.
(up to 32768 bytes per frag, thats order-3 pages on x86)
This increases TCP stream performance by 20% on loopback device,
but also benefits on other network devices, since 8x less frags are
mapped on transmit and unmapped on tx completion. Alexander Duyck
mentioned a probable performance win on systems with IOMMU enabled.
Its possible some SG enabled hardware cant cope with bigger fragments,
but their ndo_start_xmit() should already handle this, splitting a
fragment in sub fragments, since some arches have PAGE_SIZE=65536
Successfully tested on various ethernet devices.
(ixgbe, igb, bnx2x, tg3, mellanox mlx4)
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Ben Hutchings <bhutchings@solarflare.com>
Cc: Vijay Subramanian <subramanian.vijay@gmail.com>
Cc: Alexander Duyck <alexander.h.duyck@intel.com>
Tested-by: Vijay Subramanian <subramanian.vijay@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2012-09-24 07:04:42 +08:00
|
|
|
if (!sk_page_frag_refill(sk, pfrag))
|
|
|
|
return NULL;
|
2009-02-01 16:41:42 +08:00
|
|
|
|
net: use a per task frag allocator
We currently use a per socket order-0 page cache for tcp_sendmsg()
operations.
This page is used to build fragments for skbs.
Its done to increase probability of coalescing small write() into
single segments in skbs still in write queue (not yet sent)
But it wastes a lot of memory for applications handling many mostly
idle sockets, since each socket holds one page in sk->sk_sndmsg_page
Its also quite inefficient to build TSO 64KB packets, because we need
about 16 pages per skb on arches where PAGE_SIZE = 4096, so we hit
page allocator more than wanted.
This patch adds a per task frag allocator and uses bigger pages,
if available. An automatic fallback is done in case of memory pressure.
(up to 32768 bytes per frag, thats order-3 pages on x86)
This increases TCP stream performance by 20% on loopback device,
but also benefits on other network devices, since 8x less frags are
mapped on transmit and unmapped on tx completion. Alexander Duyck
mentioned a probable performance win on systems with IOMMU enabled.
Its possible some SG enabled hardware cant cope with bigger fragments,
but their ndo_start_xmit() should already handle this, splitting a
fragment in sub fragments, since some arches have PAGE_SIZE=65536
Successfully tested on various ethernet devices.
(ixgbe, igb, bnx2x, tg3, mellanox mlx4)
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Ben Hutchings <bhutchings@solarflare.com>
Cc: Vijay Subramanian <subramanian.vijay@gmail.com>
Cc: Alexander Duyck <alexander.h.duyck@intel.com>
Tested-by: Vijay Subramanian <subramanian.vijay@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2012-09-24 07:04:42 +08:00
|
|
|
*len = min_t(unsigned int, *len, pfrag->size - pfrag->offset);
|
2009-02-01 16:41:42 +08:00
|
|
|
|
net: use a per task frag allocator
We currently use a per socket order-0 page cache for tcp_sendmsg()
operations.
This page is used to build fragments for skbs.
Its done to increase probability of coalescing small write() into
single segments in skbs still in write queue (not yet sent)
But it wastes a lot of memory for applications handling many mostly
idle sockets, since each socket holds one page in sk->sk_sndmsg_page
Its also quite inefficient to build TSO 64KB packets, because we need
about 16 pages per skb on arches where PAGE_SIZE = 4096, so we hit
page allocator more than wanted.
This patch adds a per task frag allocator and uses bigger pages,
if available. An automatic fallback is done in case of memory pressure.
(up to 32768 bytes per frag, thats order-3 pages on x86)
This increases TCP stream performance by 20% on loopback device,
but also benefits on other network devices, since 8x less frags are
mapped on transmit and unmapped on tx completion. Alexander Duyck
mentioned a probable performance win on systems with IOMMU enabled.
Its possible some SG enabled hardware cant cope with bigger fragments,
but their ndo_start_xmit() should already handle this, splitting a
fragment in sub fragments, since some arches have PAGE_SIZE=65536
Successfully tested on various ethernet devices.
(ixgbe, igb, bnx2x, tg3, mellanox mlx4)
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Ben Hutchings <bhutchings@solarflare.com>
Cc: Vijay Subramanian <subramanian.vijay@gmail.com>
Cc: Alexander Duyck <alexander.h.duyck@intel.com>
Tested-by: Vijay Subramanian <subramanian.vijay@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2012-09-24 07:04:42 +08:00
|
|
|
memcpy(page_address(pfrag->page) + pfrag->offset,
|
|
|
|
page_address(page) + *offset, *len);
|
|
|
|
*offset = pfrag->offset;
|
|
|
|
pfrag->offset += *len;
|
2009-01-20 09:03:56 +08:00
|
|
|
|
net: use a per task frag allocator
We currently use a per socket order-0 page cache for tcp_sendmsg()
operations.
This page is used to build fragments for skbs.
Its done to increase probability of coalescing small write() into
single segments in skbs still in write queue (not yet sent)
But it wastes a lot of memory for applications handling many mostly
idle sockets, since each socket holds one page in sk->sk_sndmsg_page
Its also quite inefficient to build TSO 64KB packets, because we need
about 16 pages per skb on arches where PAGE_SIZE = 4096, so we hit
page allocator more than wanted.
This patch adds a per task frag allocator and uses bigger pages,
if available. An automatic fallback is done in case of memory pressure.
(up to 32768 bytes per frag, thats order-3 pages on x86)
This increases TCP stream performance by 20% on loopback device,
but also benefits on other network devices, since 8x less frags are
mapped on transmit and unmapped on tx completion. Alexander Duyck
mentioned a probable performance win on systems with IOMMU enabled.
Its possible some SG enabled hardware cant cope with bigger fragments,
but their ndo_start_xmit() should already handle this, splitting a
fragment in sub fragments, since some arches have PAGE_SIZE=65536
Successfully tested on various ethernet devices.
(ixgbe, igb, bnx2x, tg3, mellanox mlx4)
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Ben Hutchings <bhutchings@solarflare.com>
Cc: Vijay Subramanian <subramanian.vijay@gmail.com>
Cc: Alexander Duyck <alexander.h.duyck@intel.com>
Tested-by: Vijay Subramanian <subramanian.vijay@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2012-09-24 07:04:42 +08:00
|
|
|
return pfrag->page;
|
2007-11-07 15:30:13 +08:00
|
|
|
}
|
|
|
|
|
2012-04-22 20:26:16 +08:00
|
|
|
static bool spd_can_coalesce(const struct splice_pipe_desc *spd,
|
|
|
|
struct page *page,
|
|
|
|
unsigned int offset)
|
|
|
|
{
|
|
|
|
return spd->nr_pages &&
|
|
|
|
spd->pages[spd->nr_pages - 1] == page &&
|
|
|
|
(spd->partial[spd->nr_pages - 1].offset +
|
|
|
|
spd->partial[spd->nr_pages - 1].len == offset);
|
|
|
|
}
|
|
|
|
|
2007-11-07 15:30:13 +08:00
|
|
|
/*
|
|
|
|
* Fill page/offset/length into spd, if it can hold more pages.
|
|
|
|
*/
|
2012-04-24 11:06:11 +08:00
|
|
|
static bool spd_fill_page(struct splice_pipe_desc *spd,
|
|
|
|
struct pipe_inode_info *pipe, struct page *page,
|
|
|
|
unsigned int *len, unsigned int offset,
|
2013-01-11 22:46:37 +08:00
|
|
|
bool linear,
|
2012-04-24 11:06:11 +08:00
|
|
|
struct sock *sk)
|
2007-11-07 15:30:13 +08:00
|
|
|
{
|
2012-04-22 20:26:16 +08:00
|
|
|
if (unlikely(spd->nr_pages == MAX_SKB_FRAGS))
|
2012-04-24 11:06:11 +08:00
|
|
|
return true;
|
2007-11-07 15:30:13 +08:00
|
|
|
|
2009-01-20 09:03:56 +08:00
|
|
|
if (linear) {
|
2013-01-11 22:46:37 +08:00
|
|
|
page = linear_to_page(page, len, &offset, sk);
|
2009-01-20 09:03:56 +08:00
|
|
|
if (!page)
|
2012-04-24 11:06:11 +08:00
|
|
|
return true;
|
2012-04-22 20:26:16 +08:00
|
|
|
}
|
|
|
|
if (spd_can_coalesce(spd, page, offset)) {
|
|
|
|
spd->partial[spd->nr_pages - 1].len += *len;
|
2012-04-24 11:06:11 +08:00
|
|
|
return false;
|
2012-04-22 20:26:16 +08:00
|
|
|
}
|
|
|
|
get_page(page);
|
2007-11-07 15:30:13 +08:00
|
|
|
spd->pages[spd->nr_pages] = page;
|
2009-02-01 16:41:42 +08:00
|
|
|
spd->partial[spd->nr_pages].len = *len;
|
2007-11-07 15:30:13 +08:00
|
|
|
spd->partial[spd->nr_pages].offset = offset;
|
|
|
|
spd->nr_pages++;
|
2009-01-20 09:03:56 +08:00
|
|
|
|
2012-04-24 11:06:11 +08:00
|
|
|
return false;
|
2007-11-07 15:30:13 +08:00
|
|
|
}
|
|
|
|
|
2012-04-24 11:06:11 +08:00
|
|
|
static bool __splice_segment(struct page *page, unsigned int poff,
|
|
|
|
unsigned int plen, unsigned int *off,
|
2013-01-11 22:46:37 +08:00
|
|
|
unsigned int *len,
|
2012-04-24 11:35:04 +08:00
|
|
|
struct splice_pipe_desc *spd, bool linear,
|
2012-04-24 11:06:11 +08:00
|
|
|
struct sock *sk,
|
|
|
|
struct pipe_inode_info *pipe)
|
2007-11-07 15:30:13 +08:00
|
|
|
{
|
2008-07-15 15:49:11 +08:00
|
|
|
if (!*len)
|
2012-04-24 11:06:11 +08:00
|
|
|
return true;
|
2008-07-15 15:49:11 +08:00
|
|
|
|
|
|
|
/* skip this segment if already processed */
|
|
|
|
if (*off >= plen) {
|
|
|
|
*off -= plen;
|
2012-04-24 11:06:11 +08:00
|
|
|
return false;
|
2008-06-28 08:27:21 +08:00
|
|
|
}
|
2007-11-07 15:30:13 +08:00
|
|
|
|
2008-07-15 15:49:11 +08:00
|
|
|
/* ignore any bits we already processed */
|
2013-01-06 05:31:18 +08:00
|
|
|
poff += *off;
|
|
|
|
plen -= *off;
|
|
|
|
*off = 0;
|
2007-11-07 15:30:13 +08:00
|
|
|
|
2013-01-11 22:46:37 +08:00
|
|
|
do {
|
|
|
|
unsigned int flen = min(*len, plen);
|
2008-07-15 15:49:11 +08:00
|
|
|
|
2013-01-11 22:46:37 +08:00
|
|
|
if (spd_fill_page(spd, pipe, page, &flen, poff,
|
|
|
|
linear, sk))
|
|
|
|
return true;
|
|
|
|
poff += flen;
|
|
|
|
plen -= flen;
|
|
|
|
*len -= flen;
|
|
|
|
} while (*len && plen);
|
2008-07-15 15:49:11 +08:00
|
|
|
|
2012-04-24 11:06:11 +08:00
|
|
|
return false;
|
2008-07-15 15:49:11 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2012-04-24 11:06:11 +08:00
|
|
|
* Map linear and fragment data from the skb to spd. It reports true if the
|
2008-07-15 15:49:11 +08:00
|
|
|
* pipe is full or if we already spliced the requested length.
|
|
|
|
*/
|
2012-04-24 11:06:11 +08:00
|
|
|
static bool __skb_splice_bits(struct sk_buff *skb, struct pipe_inode_info *pipe,
|
|
|
|
unsigned int *offset, unsigned int *len,
|
|
|
|
struct splice_pipe_desc *spd, struct sock *sk)
|
2008-07-15 15:49:11 +08:00
|
|
|
{
|
|
|
|
int seg;
|
2016-03-08 06:11:04 +08:00
|
|
|
struct sk_buff *iter;
|
2008-07-15 15:49:11 +08:00
|
|
|
|
2012-04-27 10:10:03 +08:00
|
|
|
/* map the linear part :
|
2012-05-03 02:18:42 +08:00
|
|
|
* If skb->head_frag is set, this 'linear' part is backed by a
|
|
|
|
* fragment, and if the head is not shared with any clones then
|
|
|
|
* we can avoid a copy since we own the head portion of this page.
|
2008-07-15 15:49:11 +08:00
|
|
|
*/
|
|
|
|
if (__splice_segment(virt_to_page(skb->data),
|
|
|
|
(unsigned long) skb->data & (PAGE_SIZE - 1),
|
|
|
|
skb_headlen(skb),
|
2013-01-11 22:46:37 +08:00
|
|
|
offset, len, spd,
|
2012-05-03 09:09:42 +08:00
|
|
|
skb_head_is_locked(skb),
|
2012-04-27 10:10:03 +08:00
|
|
|
sk, pipe))
|
2012-04-24 11:06:11 +08:00
|
|
|
return true;
|
2007-11-07 15:30:13 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* then map the fragments
|
|
|
|
*/
|
|
|
|
for (seg = 0; seg < skb_shinfo(skb)->nr_frags; seg++) {
|
|
|
|
const skb_frag_t *f = &skb_shinfo(skb)->frags[seg];
|
|
|
|
|
2011-08-23 07:44:58 +08:00
|
|
|
if (__splice_segment(skb_frag_page(f),
|
2019-07-30 22:40:33 +08:00
|
|
|
skb_frag_off(f), skb_frag_size(f),
|
2013-01-11 22:46:37 +08:00
|
|
|
offset, len, spd, false, sk, pipe))
|
2012-04-24 11:06:11 +08:00
|
|
|
return true;
|
2007-11-07 15:30:13 +08:00
|
|
|
}
|
|
|
|
|
2016-03-08 06:11:04 +08:00
|
|
|
skb_walk_frags(skb, iter) {
|
|
|
|
if (*offset >= iter->len) {
|
|
|
|
*offset -= iter->len;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
/* __skb_splice_bits() only fails if the output has no room
|
|
|
|
* left, so no point in going over the frag_list for the error
|
|
|
|
* case.
|
|
|
|
*/
|
|
|
|
if (__skb_splice_bits(iter, pipe, offset, len, spd, sk))
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
2012-04-24 11:06:11 +08:00
|
|
|
return false;
|
2007-11-07 15:30:13 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Map data from the skb to a pipe. Should handle both the linear part,
|
2016-03-08 06:11:04 +08:00
|
|
|
* the fragments, and the frag list.
|
2007-11-07 15:30:13 +08:00
|
|
|
*/
|
2015-05-21 23:00:00 +08:00
|
|
|
int skb_splice_bits(struct sk_buff *skb, struct sock *sk, unsigned int offset,
|
2007-11-07 15:30:13 +08:00
|
|
|
struct pipe_inode_info *pipe, unsigned int tlen,
|
2016-09-18 09:02:10 +08:00
|
|
|
unsigned int flags)
|
2007-11-07 15:30:13 +08:00
|
|
|
{
|
2012-04-22 20:26:16 +08:00
|
|
|
struct partial_page partial[MAX_SKB_FRAGS];
|
|
|
|
struct page *pages[MAX_SKB_FRAGS];
|
2007-11-07 15:30:13 +08:00
|
|
|
struct splice_pipe_desc spd = {
|
|
|
|
.pages = pages,
|
|
|
|
.partial = partial,
|
2012-06-12 21:24:40 +08:00
|
|
|
.nr_pages_max = MAX_SKB_FRAGS,
|
2014-01-23 02:36:57 +08:00
|
|
|
.ops = &nosteal_pipe_buf_ops,
|
2007-11-07 15:30:13 +08:00
|
|
|
.spd_release = sock_spd_release,
|
|
|
|
};
|
2010-05-20 16:43:18 +08:00
|
|
|
int ret = 0;
|
|
|
|
|
2016-03-08 06:11:04 +08:00
|
|
|
__skb_splice_bits(skb, pipe, &offset, &tlen, &spd, sk);
|
2007-11-07 15:30:13 +08:00
|
|
|
|
2015-05-21 23:00:00 +08:00
|
|
|
if (spd.nr_pages)
|
2016-09-18 09:02:10 +08:00
|
|
|
ret = splice_to_pipe(pipe, &spd);
|
2007-11-07 15:30:13 +08:00
|
|
|
|
2010-05-20 16:43:18 +08:00
|
|
|
return ret;
|
2007-11-07 15:30:13 +08:00
|
|
|
}
|
2015-05-21 23:00:01 +08:00
|
|
|
EXPORT_SYMBOL_GPL(skb_splice_bits);
|
2007-11-07 15:30:13 +08:00
|
|
|
|
2021-03-31 10:32:24 +08:00
|
|
|
static int sendmsg_unlocked(struct sock *sk, struct msghdr *msg,
|
|
|
|
struct kvec *vec, size_t num, size_t size)
|
|
|
|
{
|
|
|
|
struct socket *sock = sk->sk_socket;
|
|
|
|
|
|
|
|
if (!sock)
|
|
|
|
return -EINVAL;
|
|
|
|
return kernel_sendmsg(sock, msg, vec, num, size);
|
|
|
|
}
|
|
|
|
|
|
|
|
static int sendpage_unlocked(struct sock *sk, struct page *page, int offset,
|
|
|
|
size_t size, int flags)
|
|
|
|
{
|
|
|
|
struct socket *sock = sk->sk_socket;
|
|
|
|
|
|
|
|
if (!sock)
|
|
|
|
return -EINVAL;
|
|
|
|
return kernel_sendpage(sock, page, offset, size, flags);
|
|
|
|
}
|
|
|
|
|
|
|
|
typedef int (*sendmsg_func)(struct sock *sk, struct msghdr *msg,
|
|
|
|
struct kvec *vec, size_t num, size_t size);
|
|
|
|
typedef int (*sendpage_func)(struct sock *sk, struct page *page, int offset,
|
|
|
|
size_t size, int flags);
|
|
|
|
static int __skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset,
|
|
|
|
int len, sendmsg_func sendmsg, sendpage_func sendpage)
|
2017-07-29 07:22:42 +08:00
|
|
|
{
|
|
|
|
unsigned int orig_len = len;
|
|
|
|
struct sk_buff *head = skb;
|
|
|
|
unsigned short fragidx;
|
|
|
|
int slen, ret;
|
|
|
|
|
|
|
|
do_frag_list:
|
|
|
|
|
|
|
|
/* Deal with head data */
|
|
|
|
while (offset < skb_headlen(skb) && len) {
|
|
|
|
struct kvec kv;
|
|
|
|
struct msghdr msg;
|
|
|
|
|
|
|
|
slen = min_t(int, len, skb_headlen(skb) - offset);
|
|
|
|
kv.iov_base = skb->data + offset;
|
2017-08-16 13:31:34 +08:00
|
|
|
kv.iov_len = slen;
|
2017-07-29 07:22:42 +08:00
|
|
|
memset(&msg, 0, sizeof(msg));
|
2019-05-24 23:01:00 +08:00
|
|
|
msg.msg_flags = MSG_DONTWAIT;
|
2017-07-29 07:22:42 +08:00
|
|
|
|
2021-03-31 10:32:24 +08:00
|
|
|
ret = INDIRECT_CALL_2(sendmsg, kernel_sendmsg_locked,
|
|
|
|
sendmsg_unlocked, sk, &msg, &kv, 1, slen);
|
2017-07-29 07:22:42 +08:00
|
|
|
if (ret <= 0)
|
|
|
|
goto error;
|
|
|
|
|
|
|
|
offset += ret;
|
|
|
|
len -= ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* All the data was skb head? */
|
|
|
|
if (!len)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
/* Make offset relative to start of frags */
|
|
|
|
offset -= skb_headlen(skb);
|
|
|
|
|
|
|
|
/* Find where we are in frag list */
|
|
|
|
for (fragidx = 0; fragidx < skb_shinfo(skb)->nr_frags; fragidx++) {
|
|
|
|
skb_frag_t *frag = &skb_shinfo(skb)->frags[fragidx];
|
|
|
|
|
2019-07-23 11:08:26 +08:00
|
|
|
if (offset < skb_frag_size(frag))
|
2017-07-29 07:22:42 +08:00
|
|
|
break;
|
|
|
|
|
2019-07-23 11:08:26 +08:00
|
|
|
offset -= skb_frag_size(frag);
|
2017-07-29 07:22:42 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
for (; len && fragidx < skb_shinfo(skb)->nr_frags; fragidx++) {
|
|
|
|
skb_frag_t *frag = &skb_shinfo(skb)->frags[fragidx];
|
|
|
|
|
2019-07-23 11:08:26 +08:00
|
|
|
slen = min_t(size_t, len, skb_frag_size(frag) - offset);
|
2017-07-29 07:22:42 +08:00
|
|
|
|
|
|
|
while (slen) {
|
2021-03-31 10:32:24 +08:00
|
|
|
ret = INDIRECT_CALL_2(sendpage, kernel_sendpage_locked,
|
|
|
|
sendpage_unlocked, sk,
|
|
|
|
skb_frag_page(frag),
|
|
|
|
skb_frag_off(frag) + offset,
|
|
|
|
slen, MSG_DONTWAIT);
|
2017-07-29 07:22:42 +08:00
|
|
|
if (ret <= 0)
|
|
|
|
goto error;
|
|
|
|
|
|
|
|
len -= ret;
|
|
|
|
offset += ret;
|
|
|
|
slen -= ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
offset = 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (len) {
|
|
|
|
/* Process any frag lists */
|
|
|
|
|
|
|
|
if (skb == head) {
|
|
|
|
if (skb_has_frag_list(skb)) {
|
|
|
|
skb = skb_shinfo(skb)->frag_list;
|
|
|
|
goto do_frag_list;
|
|
|
|
}
|
|
|
|
} else if (skb->next) {
|
|
|
|
skb = skb->next;
|
|
|
|
goto do_frag_list;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
out:
|
|
|
|
return orig_len - len;
|
|
|
|
|
|
|
|
error:
|
|
|
|
return orig_len == len ? ret : orig_len - len;
|
|
|
|
}
|
2021-03-31 10:32:24 +08:00
|
|
|
|
|
|
|
/* Send skb data on a socket. Socket must be locked. */
|
|
|
|
int skb_send_sock_locked(struct sock *sk, struct sk_buff *skb, int offset,
|
|
|
|
int len)
|
|
|
|
{
|
|
|
|
return __skb_send_sock(sk, skb, offset, len, kernel_sendmsg_locked,
|
|
|
|
kernel_sendpage_locked);
|
|
|
|
}
|
2017-07-29 07:22:42 +08:00
|
|
|
EXPORT_SYMBOL_GPL(skb_send_sock_locked);
|
|
|
|
|
2021-03-31 10:32:24 +08:00
|
|
|
/* Send skb data on a socket. Socket must be unlocked. */
|
|
|
|
int skb_send_sock(struct sock *sk, struct sk_buff *skb, int offset, int len)
|
|
|
|
{
|
|
|
|
return __skb_send_sock(sk, skb, offset, len, sendmsg_unlocked,
|
|
|
|
sendpage_unlocked);
|
|
|
|
}
|
|
|
|
|
2005-04-20 13:30:14 +08:00
|
|
|
/**
|
|
|
|
* skb_store_bits - store bits from kernel buffer to skb
|
|
|
|
* @skb: destination buffer
|
|
|
|
* @offset: offset in destination
|
|
|
|
* @from: source buffer
|
|
|
|
* @len: number of bytes to copy
|
|
|
|
*
|
|
|
|
* Copy the specified number of bytes from the source buffer to the
|
|
|
|
* destination skb. This function handles all the messy bits of
|
|
|
|
* traversing fragment lists and such.
|
|
|
|
*/
|
|
|
|
|
2007-04-21 07:40:01 +08:00
|
|
|
int skb_store_bits(struct sk_buff *skb, int offset, const void *from, int len)
|
2005-04-20 13:30:14 +08:00
|
|
|
{
|
2007-04-28 06:21:23 +08:00
|
|
|
int start = skb_headlen(skb);
|
2009-06-09 15:18:59 +08:00
|
|
|
struct sk_buff *frag_iter;
|
|
|
|
int i, copy;
|
2005-04-20 13:30:14 +08:00
|
|
|
|
|
|
|
if (offset > (int)skb->len - len)
|
|
|
|
goto fault;
|
|
|
|
|
2007-04-28 06:21:23 +08:00
|
|
|
if ((copy = start - offset) > 0) {
|
2005-04-20 13:30:14 +08:00
|
|
|
if (copy > len)
|
|
|
|
copy = len;
|
2007-03-31 22:55:19 +08:00
|
|
|
skb_copy_to_linear_data_offset(skb, offset, from, copy);
|
2005-04-20 13:30:14 +08:00
|
|
|
if ((len -= copy) == 0)
|
|
|
|
return 0;
|
|
|
|
offset += copy;
|
|
|
|
from += copy;
|
|
|
|
}
|
|
|
|
|
|
|
|
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
|
|
|
|
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
|
2007-04-28 06:21:23 +08:00
|
|
|
int end;
|
|
|
|
|
2008-07-26 12:43:18 +08:00
|
|
|
WARN_ON(start > offset + len);
|
2005-04-20 13:30:14 +08:00
|
|
|
|
2011-10-19 05:00:24 +08:00
|
|
|
end = start + skb_frag_size(frag);
|
2005-04-20 13:30:14 +08:00
|
|
|
if ((copy = end - offset) > 0) {
|
2017-07-31 20:15:47 +08:00
|
|
|
u32 p_off, p_len, copied;
|
|
|
|
struct page *p;
|
2005-04-20 13:30:14 +08:00
|
|
|
u8 *vaddr;
|
|
|
|
|
|
|
|
if (copy > len)
|
|
|
|
copy = len;
|
|
|
|
|
2017-07-31 20:15:47 +08:00
|
|
|
skb_frag_foreach_page(frag,
|
2019-07-30 22:40:33 +08:00
|
|
|
skb_frag_off(frag) + offset - start,
|
2017-07-31 20:15:47 +08:00
|
|
|
copy, p, p_off, p_len, copied) {
|
|
|
|
vaddr = kmap_atomic(p);
|
|
|
|
memcpy(vaddr + p_off, from + copied, p_len);
|
|
|
|
kunmap_atomic(vaddr);
|
|
|
|
}
|
2005-04-20 13:30:14 +08:00
|
|
|
|
|
|
|
if ((len -= copy) == 0)
|
|
|
|
return 0;
|
|
|
|
offset += copy;
|
|
|
|
from += copy;
|
|
|
|
}
|
2007-04-28 06:21:23 +08:00
|
|
|
start = end;
|
2005-04-20 13:30:14 +08:00
|
|
|
}
|
|
|
|
|
2009-06-09 15:18:59 +08:00
|
|
|
skb_walk_frags(skb, frag_iter) {
|
|
|
|
int end;
|
2005-04-20 13:30:14 +08:00
|
|
|
|
2009-06-09 15:18:59 +08:00
|
|
|
WARN_ON(start > offset + len);
|
|
|
|
|
|
|
|
end = start + frag_iter->len;
|
|
|
|
if ((copy = end - offset) > 0) {
|
|
|
|
if (copy > len)
|
|
|
|
copy = len;
|
|
|
|
if (skb_store_bits(frag_iter, offset - start,
|
|
|
|
from, copy))
|
|
|
|
goto fault;
|
|
|
|
if ((len -= copy) == 0)
|
|
|
|
return 0;
|
|
|
|
offset += copy;
|
|
|
|
from += copy;
|
2005-04-20 13:30:14 +08:00
|
|
|
}
|
2009-06-09 15:18:59 +08:00
|
|
|
start = end;
|
2005-04-20 13:30:14 +08:00
|
|
|
}
|
|
|
|
if (!len)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
fault:
|
|
|
|
return -EFAULT;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_store_bits);
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/* Checksum skb data. */
|
2013-10-30 18:50:51 +08:00
|
|
|
__wsum __skb_checksum(const struct sk_buff *skb, int offset, int len,
|
|
|
|
__wsum csum, const struct skb_checksum_ops *ops)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2007-04-28 06:21:23 +08:00
|
|
|
int start = skb_headlen(skb);
|
|
|
|
int i, copy = start - offset;
|
2009-06-09 15:18:59 +08:00
|
|
|
struct sk_buff *frag_iter;
|
2005-04-17 06:20:36 +08:00
|
|
|
int pos = 0;
|
|
|
|
|
|
|
|
/* Checksum header. */
|
|
|
|
if (copy > 0) {
|
|
|
|
if (copy > len)
|
|
|
|
copy = len;
|
2019-05-29 23:13:48 +08:00
|
|
|
csum = INDIRECT_CALL_1(ops->update, csum_partial_ext,
|
|
|
|
skb->data + offset, copy, csum);
|
2005-04-17 06:20:36 +08:00
|
|
|
if ((len -= copy) == 0)
|
|
|
|
return csum;
|
|
|
|
offset += copy;
|
|
|
|
pos = copy;
|
|
|
|
}
|
|
|
|
|
|
|
|
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
|
2007-04-28 06:21:23 +08:00
|
|
|
int end;
|
2012-04-05 17:35:15 +08:00
|
|
|
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
|
2007-04-28 06:21:23 +08:00
|
|
|
|
2008-07-26 12:43:18 +08:00
|
|
|
WARN_ON(start > offset + len);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2012-04-05 17:35:15 +08:00
|
|
|
end = start + skb_frag_size(frag);
|
2005-04-17 06:20:36 +08:00
|
|
|
if ((copy = end - offset) > 0) {
|
2017-07-31 20:15:47 +08:00
|
|
|
u32 p_off, p_len, copied;
|
|
|
|
struct page *p;
|
2006-11-15 13:36:14 +08:00
|
|
|
__wsum csum2;
|
2005-04-17 06:20:36 +08:00
|
|
|
u8 *vaddr;
|
|
|
|
|
|
|
|
if (copy > len)
|
|
|
|
copy = len;
|
2017-07-31 20:15:47 +08:00
|
|
|
|
|
|
|
skb_frag_foreach_page(frag,
|
2019-07-30 22:40:33 +08:00
|
|
|
skb_frag_off(frag) + offset - start,
|
2017-07-31 20:15:47 +08:00
|
|
|
copy, p, p_off, p_len, copied) {
|
|
|
|
vaddr = kmap_atomic(p);
|
2019-05-29 23:13:48 +08:00
|
|
|
csum2 = INDIRECT_CALL_1(ops->update,
|
|
|
|
csum_partial_ext,
|
|
|
|
vaddr + p_off, p_len, 0);
|
2017-07-31 20:15:47 +08:00
|
|
|
kunmap_atomic(vaddr);
|
2019-05-29 23:13:48 +08:00
|
|
|
csum = INDIRECT_CALL_1(ops->combine,
|
|
|
|
csum_block_add_ext, csum,
|
|
|
|
csum2, pos, p_len);
|
2017-07-31 20:15:47 +08:00
|
|
|
pos += p_len;
|
|
|
|
}
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
if (!(len -= copy))
|
|
|
|
return csum;
|
|
|
|
offset += copy;
|
|
|
|
}
|
2007-04-28 06:21:23 +08:00
|
|
|
start = end;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
2009-06-09 15:18:59 +08:00
|
|
|
skb_walk_frags(skb, frag_iter) {
|
|
|
|
int end;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2009-06-09 15:18:59 +08:00
|
|
|
WARN_ON(start > offset + len);
|
|
|
|
|
|
|
|
end = start + frag_iter->len;
|
|
|
|
if ((copy = end - offset) > 0) {
|
|
|
|
__wsum csum2;
|
|
|
|
if (copy > len)
|
|
|
|
copy = len;
|
2013-10-30 18:50:51 +08:00
|
|
|
csum2 = __skb_checksum(frag_iter, offset - start,
|
|
|
|
copy, 0, ops);
|
2019-05-29 23:13:48 +08:00
|
|
|
csum = INDIRECT_CALL_1(ops->combine, csum_block_add_ext,
|
|
|
|
csum, csum2, pos, copy);
|
2009-06-09 15:18:59 +08:00
|
|
|
if ((len -= copy) == 0)
|
|
|
|
return csum;
|
|
|
|
offset += copy;
|
|
|
|
pos += copy;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
2009-06-09 15:18:59 +08:00
|
|
|
start = end;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
2006-01-09 14:24:28 +08:00
|
|
|
BUG_ON(len);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
return csum;
|
|
|
|
}
|
2013-10-30 18:50:51 +08:00
|
|
|
EXPORT_SYMBOL(__skb_checksum);
|
|
|
|
|
|
|
|
__wsum skb_checksum(const struct sk_buff *skb, int offset,
|
|
|
|
int len, __wsum csum)
|
|
|
|
{
|
|
|
|
const struct skb_checksum_ops ops = {
|
2013-11-05 00:10:25 +08:00
|
|
|
.update = csum_partial_ext,
|
2013-10-30 18:50:51 +08:00
|
|
|
.combine = csum_block_add_ext,
|
|
|
|
};
|
|
|
|
|
|
|
|
return __skb_checksum(skb, offset, len, csum, &ops);
|
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(skb_checksum);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
/* Both of above in one bottle. */
|
|
|
|
|
2006-11-15 13:37:33 +08:00
|
|
|
__wsum skb_copy_and_csum_bits(const struct sk_buff *skb, int offset,
|
2020-07-11 08:07:10 +08:00
|
|
|
u8 *to, int len)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2007-04-28 06:21:23 +08:00
|
|
|
int start = skb_headlen(skb);
|
|
|
|
int i, copy = start - offset;
|
2009-06-09 15:18:59 +08:00
|
|
|
struct sk_buff *frag_iter;
|
2005-04-17 06:20:36 +08:00
|
|
|
int pos = 0;
|
2020-07-11 08:07:10 +08:00
|
|
|
__wsum csum = 0;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
/* Copy header. */
|
|
|
|
if (copy > 0) {
|
|
|
|
if (copy > len)
|
|
|
|
copy = len;
|
|
|
|
csum = csum_partial_copy_nocheck(skb->data + offset, to,
|
2020-07-11 12:12:07 +08:00
|
|
|
copy);
|
2005-04-17 06:20:36 +08:00
|
|
|
if ((len -= copy) == 0)
|
|
|
|
return csum;
|
|
|
|
offset += copy;
|
|
|
|
to += copy;
|
|
|
|
pos = copy;
|
|
|
|
}
|
|
|
|
|
|
|
|
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
|
2007-04-28 06:21:23 +08:00
|
|
|
int end;
|
|
|
|
|
2008-07-26 12:43:18 +08:00
|
|
|
WARN_ON(start > offset + len);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2011-10-19 05:00:24 +08:00
|
|
|
end = start + skb_frag_size(&skb_shinfo(skb)->frags[i]);
|
2005-04-17 06:20:36 +08:00
|
|
|
if ((copy = end - offset) > 0) {
|
2017-07-31 20:15:47 +08:00
|
|
|
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
|
|
|
|
u32 p_off, p_len, copied;
|
|
|
|
struct page *p;
|
2006-11-15 13:36:34 +08:00
|
|
|
__wsum csum2;
|
2005-04-17 06:20:36 +08:00
|
|
|
u8 *vaddr;
|
|
|
|
|
|
|
|
if (copy > len)
|
|
|
|
copy = len;
|
2017-07-31 20:15:47 +08:00
|
|
|
|
|
|
|
skb_frag_foreach_page(frag,
|
2019-07-30 22:40:33 +08:00
|
|
|
skb_frag_off(frag) + offset - start,
|
2017-07-31 20:15:47 +08:00
|
|
|
copy, p, p_off, p_len, copied) {
|
|
|
|
vaddr = kmap_atomic(p);
|
|
|
|
csum2 = csum_partial_copy_nocheck(vaddr + p_off,
|
|
|
|
to + copied,
|
2020-07-11 12:12:07 +08:00
|
|
|
p_len);
|
2017-07-31 20:15:47 +08:00
|
|
|
kunmap_atomic(vaddr);
|
|
|
|
csum = csum_block_add(csum, csum2, pos);
|
|
|
|
pos += p_len;
|
|
|
|
}
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
if (!(len -= copy))
|
|
|
|
return csum;
|
|
|
|
offset += copy;
|
|
|
|
to += copy;
|
|
|
|
}
|
2007-04-28 06:21:23 +08:00
|
|
|
start = end;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
2009-06-09 15:18:59 +08:00
|
|
|
skb_walk_frags(skb, frag_iter) {
|
|
|
|
__wsum csum2;
|
|
|
|
int end;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2009-06-09 15:18:59 +08:00
|
|
|
WARN_ON(start > offset + len);
|
|
|
|
|
|
|
|
end = start + frag_iter->len;
|
|
|
|
if ((copy = end - offset) > 0) {
|
|
|
|
if (copy > len)
|
|
|
|
copy = len;
|
|
|
|
csum2 = skb_copy_and_csum_bits(frag_iter,
|
|
|
|
offset - start,
|
2020-07-11 08:07:10 +08:00
|
|
|
to, copy);
|
2009-06-09 15:18:59 +08:00
|
|
|
csum = csum_block_add(csum, csum2, pos);
|
|
|
|
if ((len -= copy) == 0)
|
|
|
|
return csum;
|
|
|
|
offset += copy;
|
|
|
|
to += copy;
|
|
|
|
pos += copy;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
2009-06-09 15:18:59 +08:00
|
|
|
start = end;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
2006-01-09 14:24:28 +08:00
|
|
|
BUG_ON(len);
|
2005-04-17 06:20:36 +08:00
|
|
|
return csum;
|
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(skb_copy_and_csum_bits);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2018-11-09 06:05:42 +08:00
|
|
|
__sum16 __skb_checksum_complete_head(struct sk_buff *skb, int len)
|
|
|
|
{
|
|
|
|
__sum16 sum;
|
|
|
|
|
|
|
|
sum = csum_fold(skb_checksum(skb, 0, len, skb->csum));
|
2018-11-27 01:31:26 +08:00
|
|
|
/* See comments in __skb_checksum_complete(). */
|
2018-11-09 06:05:42 +08:00
|
|
|
if (likely(!sum)) {
|
|
|
|
if (unlikely(skb->ip_summed == CHECKSUM_COMPLETE) &&
|
|
|
|
!skb->csum_complete_sw)
|
2018-11-13 06:47:18 +08:00
|
|
|
netdev_rx_csum_fault(skb->dev, skb);
|
2018-11-09 06:05:42 +08:00
|
|
|
}
|
|
|
|
if (!skb_shared(skb))
|
|
|
|
skb->csum_valid = !sum;
|
|
|
|
return sum;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(__skb_checksum_complete_head);
|
|
|
|
|
2018-11-27 01:31:26 +08:00
|
|
|
/* This function assumes skb->csum already holds pseudo header's checksum,
|
|
|
|
* which has been changed from the hardware checksum, for example, by
|
|
|
|
* __skb_checksum_validate_complete(). And, the original skb->csum must
|
|
|
|
* have been validated unsuccessfully for CHECKSUM_COMPLETE case.
|
|
|
|
*
|
|
|
|
* It returns non-zero if the recomputed checksum is still invalid, otherwise
|
|
|
|
* zero. The new checksum is stored back into skb->csum unless the skb is
|
|
|
|
* shared.
|
|
|
|
*/
|
2018-11-09 06:05:42 +08:00
|
|
|
__sum16 __skb_checksum_complete(struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
__wsum csum;
|
|
|
|
__sum16 sum;
|
|
|
|
|
|
|
|
csum = skb_checksum(skb, 0, skb->len, 0);
|
|
|
|
|
|
|
|
sum = csum_fold(csum_add(skb->csum, csum));
|
2018-11-27 01:31:26 +08:00
|
|
|
/* This check is inverted, because we already knew the hardware
|
|
|
|
* checksum is invalid before calling this function. So, if the
|
|
|
|
* re-computed checksum is valid instead, then we have a mismatch
|
|
|
|
* between the original skb->csum and skb_checksum(). This means either
|
|
|
|
* the original hardware checksum is incorrect or we screw up skb->csum
|
|
|
|
* when moving skb->data around.
|
|
|
|
*/
|
2018-11-09 06:05:42 +08:00
|
|
|
if (likely(!sum)) {
|
|
|
|
if (unlikely(skb->ip_summed == CHECKSUM_COMPLETE) &&
|
|
|
|
!skb->csum_complete_sw)
|
2018-11-13 06:47:18 +08:00
|
|
|
netdev_rx_csum_fault(skb->dev, skb);
|
2018-11-09 06:05:42 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
if (!skb_shared(skb)) {
|
|
|
|
/* Save full packet checksum */
|
|
|
|
skb->csum = csum;
|
|
|
|
skb->ip_summed = CHECKSUM_COMPLETE;
|
|
|
|
skb->csum_complete_sw = 1;
|
|
|
|
skb->csum_valid = !sum;
|
|
|
|
}
|
|
|
|
|
|
|
|
return sum;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(__skb_checksum_complete);
|
|
|
|
|
2017-05-18 21:44:37 +08:00
|
|
|
static __wsum warn_crc32c_csum_update(const void *buff, int len, __wsum sum)
|
|
|
|
{
|
|
|
|
net_warn_ratelimited(
|
|
|
|
"%s: attempt to compute crc32c without libcrc32c.ko\n",
|
|
|
|
__func__);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static __wsum warn_crc32c_csum_combine(__wsum csum, __wsum csum2,
|
|
|
|
int offset, int len)
|
|
|
|
{
|
|
|
|
net_warn_ratelimited(
|
|
|
|
"%s: attempt to compute crc32c without libcrc32c.ko\n",
|
|
|
|
__func__);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static const struct skb_checksum_ops default_crc32c_ops = {
|
|
|
|
.update = warn_crc32c_csum_update,
|
|
|
|
.combine = warn_crc32c_csum_combine,
|
|
|
|
};
|
|
|
|
|
|
|
|
const struct skb_checksum_ops *crc32c_csum_stub __read_mostly =
|
|
|
|
&default_crc32c_ops;
|
|
|
|
EXPORT_SYMBOL(crc32c_csum_stub);
|
|
|
|
|
2013-12-13 22:22:17 +08:00
|
|
|
/**
|
|
|
|
* skb_zerocopy_headlen - Calculate headroom needed for skb_zerocopy()
|
|
|
|
* @from: source buffer
|
|
|
|
*
|
|
|
|
* Calculates the amount of linear headroom needed in the 'to' skb passed
|
|
|
|
* into skb_zerocopy().
|
|
|
|
*/
|
|
|
|
unsigned int
|
|
|
|
skb_zerocopy_headlen(const struct sk_buff *from)
|
|
|
|
{
|
|
|
|
unsigned int hlen = 0;
|
|
|
|
|
|
|
|
if (!from->head_frag ||
|
|
|
|
skb_headlen(from) < L1_CACHE_BYTES ||
|
2021-07-16 07:59:00 +08:00
|
|
|
skb_shinfo(from)->nr_frags >= MAX_SKB_FRAGS) {
|
2013-12-13 22:22:17 +08:00
|
|
|
hlen = skb_headlen(from);
|
2021-07-16 07:59:00 +08:00
|
|
|
if (!hlen)
|
|
|
|
hlen = from->len;
|
|
|
|
}
|
2013-12-13 22:22:17 +08:00
|
|
|
|
|
|
|
if (skb_has_frag_list(from))
|
|
|
|
hlen = from->len;
|
|
|
|
|
|
|
|
return hlen;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(skb_zerocopy_headlen);
|
|
|
|
|
|
|
|
/**
|
|
|
|
* skb_zerocopy - Zero copy skb to skb
|
|
|
|
* @to: destination buffer
|
2014-01-29 00:05:28 +08:00
|
|
|
* @from: source buffer
|
2013-12-13 22:22:17 +08:00
|
|
|
* @len: number of bytes to copy from source buffer
|
|
|
|
* @hlen: size of linear headroom in destination buffer
|
|
|
|
*
|
|
|
|
* Copies up to `len` bytes from `from` to `to` by creating references
|
|
|
|
* to the frags in the source buffer.
|
|
|
|
*
|
|
|
|
* The `hlen` as calculated by skb_zerocopy_headlen() specifies the
|
|
|
|
* headroom in the `to` buffer.
|
2014-03-27 06:37:45 +08:00
|
|
|
*
|
|
|
|
* Return value:
|
|
|
|
* 0: everything is OK
|
|
|
|
* -ENOMEM: couldn't orphan frags of @from due to lack of memory
|
|
|
|
* -EFAULT: skb_copy_bits() found some problem with skb geometry
|
2013-12-13 22:22:17 +08:00
|
|
|
*/
|
2014-03-27 06:37:45 +08:00
|
|
|
int
|
|
|
|
skb_zerocopy(struct sk_buff *to, struct sk_buff *from, int len, int hlen)
|
2013-12-13 22:22:17 +08:00
|
|
|
{
|
|
|
|
int i, j = 0;
|
|
|
|
int plen = 0; /* length of skb->head fragment */
|
2014-03-27 06:37:45 +08:00
|
|
|
int ret;
|
2013-12-13 22:22:17 +08:00
|
|
|
struct page *page;
|
|
|
|
unsigned int offset;
|
|
|
|
|
|
|
|
BUG_ON(!from->head_frag && !hlen);
|
|
|
|
|
|
|
|
/* dont bother with small payloads */
|
2014-03-27 06:37:45 +08:00
|
|
|
if (len <= skb_tailroom(to))
|
|
|
|
return skb_copy_bits(from, 0, skb_put(to, len), len);
|
2013-12-13 22:22:17 +08:00
|
|
|
|
|
|
|
if (hlen) {
|
2014-03-27 06:37:45 +08:00
|
|
|
ret = skb_copy_bits(from, 0, skb_put(to, hlen), hlen);
|
|
|
|
if (unlikely(ret))
|
|
|
|
return ret;
|
2013-12-13 22:22:17 +08:00
|
|
|
len -= hlen;
|
|
|
|
} else {
|
|
|
|
plen = min_t(int, skb_headlen(from), len);
|
|
|
|
if (plen) {
|
|
|
|
page = virt_to_head_page(from->head);
|
|
|
|
offset = from->data - (unsigned char *)page_address(page);
|
|
|
|
__skb_fill_page_desc(to, 0, page, offset, plen);
|
|
|
|
get_page(page);
|
|
|
|
j = 1;
|
|
|
|
len -= plen;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
to->truesize += len + plen;
|
|
|
|
to->len += len + plen;
|
|
|
|
to->data_len += len + plen;
|
|
|
|
|
2014-03-27 06:37:45 +08:00
|
|
|
if (unlikely(skb_orphan_frags(from, GFP_ATOMIC))) {
|
|
|
|
skb_tx_error(from);
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
sock: enable MSG_ZEROCOPY
Prepare the datapath for refcounted ubuf_info. Clone ubuf_info with
skb_zerocopy_clone() wherever needed due to skb split, merge, resize
or clone.
Split skb_orphan_frags into two variants. The split, merge, .. paths
support reference counted zerocopy buffers, so do not do a deep copy.
Add skb_orphan_frags_rx for paths that may loop packets to receive
sockets. That is not allowed, as it may cause unbounded latency.
Deep copy all zerocopy copy buffers, ref-counted or not, in this path.
The exact locations to modify were chosen by exhaustively searching
through all code that might modify skb_frag references and/or the
the SKBTX_DEV_ZEROCOPY tx_flags bit.
The changes err on the safe side, in two ways.
(1) legacy ubuf_info paths virtio and tap are not modified. They keep
a 1:1 ubuf_info to sk_buff relationship. Calls to skb_orphan_frags
still call skb_copy_ubufs and thus copy frags in this case.
(2) not all copies deep in the stack are addressed yet. skb_shift,
skb_split and skb_try_coalesce can be refined to avoid copying.
These are not in the hot path and this patch is hairy enough as
is, so that is left for future refinement.
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-04 04:29:41 +08:00
|
|
|
skb_zerocopy_clone(to, from, GFP_ATOMIC);
|
2014-03-27 06:37:45 +08:00
|
|
|
|
2013-12-13 22:22:17 +08:00
|
|
|
for (i = 0; i < skb_shinfo(from)->nr_frags; i++) {
|
2019-07-23 11:08:26 +08:00
|
|
|
int size;
|
|
|
|
|
2013-12-13 22:22:17 +08:00
|
|
|
if (!len)
|
|
|
|
break;
|
|
|
|
skb_shinfo(to)->frags[j] = skb_shinfo(from)->frags[i];
|
2019-07-23 11:08:26 +08:00
|
|
|
size = min_t(int, skb_frag_size(&skb_shinfo(to)->frags[j]),
|
|
|
|
len);
|
|
|
|
skb_frag_size_set(&skb_shinfo(to)->frags[j], size);
|
|
|
|
len -= size;
|
2013-12-13 22:22:17 +08:00
|
|
|
skb_frag_ref(to, j);
|
|
|
|
j++;
|
|
|
|
}
|
|
|
|
skb_shinfo(to)->nr_frags = j;
|
2014-03-27 06:37:45 +08:00
|
|
|
|
|
|
|
return 0;
|
2013-12-13 22:22:17 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(skb_zerocopy);
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
void skb_copy_and_csum_dev(const struct sk_buff *skb, u8 *to)
|
|
|
|
{
|
2006-11-15 13:24:49 +08:00
|
|
|
__wsum csum;
|
2005-04-17 06:20:36 +08:00
|
|
|
long csstart;
|
|
|
|
|
2006-08-30 07:44:56 +08:00
|
|
|
if (skb->ip_summed == CHECKSUM_PARTIAL)
|
2010-12-14 23:24:08 +08:00
|
|
|
csstart = skb_checksum_start_offset(skb);
|
2005-04-17 06:20:36 +08:00
|
|
|
else
|
|
|
|
csstart = skb_headlen(skb);
|
|
|
|
|
2006-01-09 14:24:28 +08:00
|
|
|
BUG_ON(csstart > skb_headlen(skb));
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2007-03-28 05:55:52 +08:00
|
|
|
skb_copy_from_linear_data(skb, to, csstart);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
csum = 0;
|
|
|
|
if (csstart != skb->len)
|
|
|
|
csum = skb_copy_and_csum_bits(skb, csstart, to + csstart,
|
2020-07-11 08:07:10 +08:00
|
|
|
skb->len - csstart);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2006-08-30 07:44:56 +08:00
|
|
|
if (skb->ip_summed == CHECKSUM_PARTIAL) {
|
2006-11-21 10:07:29 +08:00
|
|
|
long csstuff = csstart + skb->csum_offset;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2006-11-15 13:24:49 +08:00
|
|
|
*((__sum16 *)(to + csstuff)) = csum_fold(csum);
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(skb_copy_and_csum_dev);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
/**
|
|
|
|
* skb_dequeue - remove from the head of the queue
|
|
|
|
* @list: list to dequeue from
|
|
|
|
*
|
|
|
|
* Remove the head of the list. The list lock is taken so the function
|
|
|
|
* may be used safely with other locking list functions. The head item is
|
|
|
|
* returned or %NULL if the list is empty.
|
|
|
|
*/
|
|
|
|
|
|
|
|
struct sk_buff *skb_dequeue(struct sk_buff_head *list)
|
|
|
|
{
|
|
|
|
unsigned long flags;
|
|
|
|
struct sk_buff *result;
|
|
|
|
|
|
|
|
spin_lock_irqsave(&list->lock, flags);
|
|
|
|
result = __skb_dequeue(list);
|
|
|
|
spin_unlock_irqrestore(&list->lock, flags);
|
|
|
|
return result;
|
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(skb_dequeue);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
/**
|
|
|
|
* skb_dequeue_tail - remove from the tail of the queue
|
|
|
|
* @list: list to dequeue from
|
|
|
|
*
|
|
|
|
* Remove the tail of the list. The list lock is taken so the function
|
|
|
|
* may be used safely with other locking list functions. The tail item is
|
|
|
|
* returned or %NULL if the list is empty.
|
|
|
|
*/
|
|
|
|
struct sk_buff *skb_dequeue_tail(struct sk_buff_head *list)
|
|
|
|
{
|
|
|
|
unsigned long flags;
|
|
|
|
struct sk_buff *result;
|
|
|
|
|
|
|
|
spin_lock_irqsave(&list->lock, flags);
|
|
|
|
result = __skb_dequeue_tail(list);
|
|
|
|
spin_unlock_irqrestore(&list->lock, flags);
|
|
|
|
return result;
|
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(skb_dequeue_tail);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
/**
|
|
|
|
* skb_queue_purge - empty a list
|
|
|
|
* @list: list to empty
|
|
|
|
*
|
|
|
|
* Delete all buffers on an &sk_buff list. Each buffer is removed from
|
|
|
|
* the list and one reference dropped. This function takes the list
|
|
|
|
* lock and is atomic with respect to other list locking functions.
|
|
|
|
*/
|
|
|
|
void skb_queue_purge(struct sk_buff_head *list)
|
|
|
|
{
|
|
|
|
struct sk_buff *skb;
|
|
|
|
while ((skb = skb_dequeue(list)) != NULL)
|
|
|
|
kfree_skb(skb);
|
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(skb_queue_purge);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-08 05:49:28 +08:00
|
|
|
/**
|
|
|
|
* skb_rbtree_purge - empty a skb rbtree
|
|
|
|
* @root: root of the rbtree to empty
|
2018-08-03 07:34:38 +08:00
|
|
|
* Return value: the sum of truesizes of all purged skbs.
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-08 05:49:28 +08:00
|
|
|
*
|
|
|
|
* Delete all buffers on an &sk_buff rbtree. Each buffer is removed from
|
|
|
|
* the list and one reference dropped. This function does not take
|
|
|
|
* any lock. Synchronization should be handled by the caller (e.g., TCP
|
|
|
|
* out-of-order queue is protected by the socket lock).
|
|
|
|
*/
|
2018-08-03 07:34:38 +08:00
|
|
|
unsigned int skb_rbtree_purge(struct rb_root *root)
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-08 05:49:28 +08:00
|
|
|
{
|
2017-09-24 03:39:12 +08:00
|
|
|
struct rb_node *p = rb_first(root);
|
2018-08-03 07:34:38 +08:00
|
|
|
unsigned int sum = 0;
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-08 05:49:28 +08:00
|
|
|
|
2017-09-24 03:39:12 +08:00
|
|
|
while (p) {
|
|
|
|
struct sk_buff *skb = rb_entry(p, struct sk_buff, rbnode);
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-08 05:49:28 +08:00
|
|
|
|
2017-09-24 03:39:12 +08:00
|
|
|
p = rb_next(p);
|
|
|
|
rb_erase(&skb->rbnode, root);
|
2018-08-03 07:34:38 +08:00
|
|
|
sum += skb->truesize;
|
2017-09-24 03:39:12 +08:00
|
|
|
kfree_skb(skb);
|
|
|
|
}
|
2018-08-03 07:34:38 +08:00
|
|
|
return sum;
|
tcp: use an RB tree for ooo receive queue
Over the years, TCP BDP has increased by several orders of magnitude,
and some people are considering to reach the 2 Gbytes limit.
Even with current window scale limit of 14, ~1 Gbytes maps to ~740,000
MSS.
In presence of packet losses (or reorders), TCP stores incoming packets
into an out of order queue, and number of skbs sitting there waiting for
the missing packets to be received can be in the 10^5 range.
Most packets are appended to the tail of this queue, and when
packets can finally be transferred to receive queue, we scan the queue
from its head.
However, in presence of heavy losses, we might have to find an arbitrary
point in this queue, involving a linear scan for every incoming packet,
throwing away cpu caches.
This patch converts it to a RB tree, to get bounded latencies.
Yaogong wrote a preliminary patch about 2 years ago.
Eric did the rebase, added ofo_last_skb cache, polishing and tests.
Tested with network dropping between 1 and 10 % packets, with good
success (about 30 % increase of throughput in stress tests)
Next step would be to also use an RB tree for the write queue at sender
side ;)
Signed-off-by: Yaogong Wang <wygivan@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-By: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-09-08 05:49:28 +08:00
|
|
|
}
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/**
|
|
|
|
* skb_queue_head - queue a buffer at the list head
|
|
|
|
* @list: list to use
|
|
|
|
* @newsk: buffer to queue
|
|
|
|
*
|
|
|
|
* Queue a buffer at the start of the list. This function takes the
|
|
|
|
* list lock and can be used safely with other locking &sk_buff functions
|
|
|
|
* safely.
|
|
|
|
*
|
|
|
|
* A buffer cannot be placed on two lists at the same time.
|
|
|
|
*/
|
|
|
|
void skb_queue_head(struct sk_buff_head *list, struct sk_buff *newsk)
|
|
|
|
{
|
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
spin_lock_irqsave(&list->lock, flags);
|
|
|
|
__skb_queue_head(list, newsk);
|
|
|
|
spin_unlock_irqrestore(&list->lock, flags);
|
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(skb_queue_head);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
/**
|
|
|
|
* skb_queue_tail - queue a buffer at the list tail
|
|
|
|
* @list: list to use
|
|
|
|
* @newsk: buffer to queue
|
|
|
|
*
|
|
|
|
* Queue a buffer at the tail of the list. This function takes the
|
|
|
|
* list lock and can be used safely with other locking &sk_buff functions
|
|
|
|
* safely.
|
|
|
|
*
|
|
|
|
* A buffer cannot be placed on two lists at the same time.
|
|
|
|
*/
|
|
|
|
void skb_queue_tail(struct sk_buff_head *list, struct sk_buff *newsk)
|
|
|
|
{
|
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
spin_lock_irqsave(&list->lock, flags);
|
|
|
|
__skb_queue_tail(list, newsk);
|
|
|
|
spin_unlock_irqrestore(&list->lock, flags);
|
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(skb_queue_tail);
|
2005-08-10 10:25:21 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/**
|
|
|
|
* skb_unlink - remove a buffer from a list
|
|
|
|
* @skb: buffer to remove
|
2005-08-10 10:25:21 +08:00
|
|
|
* @list: list to use
|
2005-04-17 06:20:36 +08:00
|
|
|
*
|
2005-08-10 10:25:21 +08:00
|
|
|
* Remove a packet from a list. The list locks are taken and this
|
|
|
|
* function is atomic with respect to other list locked calls
|
2005-04-17 06:20:36 +08:00
|
|
|
*
|
2005-08-10 10:25:21 +08:00
|
|
|
* You must know what list the SKB is on.
|
2005-04-17 06:20:36 +08:00
|
|
|
*/
|
2005-08-10 10:25:21 +08:00
|
|
|
void skb_unlink(struct sk_buff *skb, struct sk_buff_head *list)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2005-08-10 10:25:21 +08:00
|
|
|
unsigned long flags;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2005-08-10 10:25:21 +08:00
|
|
|
spin_lock_irqsave(&list->lock, flags);
|
|
|
|
__skb_unlink(skb, list);
|
|
|
|
spin_unlock_irqrestore(&list->lock, flags);
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(skb_unlink);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
/**
|
|
|
|
* skb_append - append a buffer
|
|
|
|
* @old: buffer to insert after
|
|
|
|
* @newsk: buffer to insert
|
2005-08-10 10:25:21 +08:00
|
|
|
* @list: list to use
|
2005-04-17 06:20:36 +08:00
|
|
|
*
|
|
|
|
* Place a packet after a given packet in a list. The list locks are taken
|
|
|
|
* and this function is atomic with respect to other list locked calls.
|
|
|
|
* A buffer cannot be placed on two lists at the same time.
|
|
|
|
*/
|
2005-08-10 10:25:21 +08:00
|
|
|
void skb_append(struct sk_buff *old, struct sk_buff *newsk, struct sk_buff_head *list)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
|
|
|
unsigned long flags;
|
|
|
|
|
2005-08-10 10:25:21 +08:00
|
|
|
spin_lock_irqsave(&list->lock, flags);
|
2008-04-14 15:05:09 +08:00
|
|
|
__skb_queue_after(list, old, newsk);
|
2005-08-10 10:25:21 +08:00
|
|
|
spin_unlock_irqrestore(&list->lock, flags);
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(skb_append);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
static inline void skb_split_inside_header(struct sk_buff *skb,
|
|
|
|
struct sk_buff* skb1,
|
|
|
|
const u32 len, const int pos)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
2007-03-28 05:55:52 +08:00
|
|
|
skb_copy_from_linear_data_offset(skb, len, skb_put(skb1, pos - len),
|
|
|
|
pos - len);
|
2005-04-17 06:20:36 +08:00
|
|
|
/* And move data appendix as is. */
|
|
|
|
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++)
|
|
|
|
skb_shinfo(skb1)->frags[i] = skb_shinfo(skb)->frags[i];
|
|
|
|
|
|
|
|
skb_shinfo(skb1)->nr_frags = skb_shinfo(skb)->nr_frags;
|
|
|
|
skb_shinfo(skb)->nr_frags = 0;
|
|
|
|
skb1->data_len = skb->data_len;
|
|
|
|
skb1->len += skb1->data_len;
|
|
|
|
skb->data_len = 0;
|
|
|
|
skb->len = len;
|
2007-04-20 11:29:13 +08:00
|
|
|
skb_set_tail_pointer(skb, len);
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
static inline void skb_split_no_header(struct sk_buff *skb,
|
|
|
|
struct sk_buff* skb1,
|
|
|
|
const u32 len, int pos)
|
|
|
|
{
|
|
|
|
int i, k = 0;
|
|
|
|
const int nfrags = skb_shinfo(skb)->nr_frags;
|
|
|
|
|
|
|
|
skb_shinfo(skb)->nr_frags = 0;
|
|
|
|
skb1->len = skb1->data_len = skb->len - len;
|
|
|
|
skb->len = len;
|
|
|
|
skb->data_len = len - pos;
|
|
|
|
|
|
|
|
for (i = 0; i < nfrags; i++) {
|
2011-10-19 05:00:24 +08:00
|
|
|
int size = skb_frag_size(&skb_shinfo(skb)->frags[i]);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
if (pos + size > len) {
|
|
|
|
skb_shinfo(skb1)->frags[k] = skb_shinfo(skb)->frags[i];
|
|
|
|
|
|
|
|
if (pos < len) {
|
|
|
|
/* Split frag.
|
|
|
|
* We have two variants in this case:
|
|
|
|
* 1. Move all the frag to the second
|
|
|
|
* part, if it is possible. F.e.
|
|
|
|
* this approach is mandatory for TUX,
|
|
|
|
* where splitting is expensive.
|
|
|
|
* 2. Split is accurately. We make this.
|
|
|
|
*/
|
2011-08-23 07:44:58 +08:00
|
|
|
skb_frag_ref(skb, i);
|
2019-07-30 22:40:33 +08:00
|
|
|
skb_frag_off_add(&skb_shinfo(skb1)->frags[0], len - pos);
|
2011-10-19 05:00:24 +08:00
|
|
|
skb_frag_size_sub(&skb_shinfo(skb1)->frags[0], len - pos);
|
|
|
|
skb_frag_size_set(&skb_shinfo(skb)->frags[i], len - pos);
|
2005-04-17 06:20:36 +08:00
|
|
|
skb_shinfo(skb)->nr_frags++;
|
|
|
|
}
|
|
|
|
k++;
|
|
|
|
} else
|
|
|
|
skb_shinfo(skb)->nr_frags++;
|
|
|
|
pos += size;
|
|
|
|
}
|
|
|
|
skb_shinfo(skb1)->nr_frags = k;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* skb_split - Split fragmented skb to two parts at length len.
|
|
|
|
* @skb: the buffer to split
|
|
|
|
* @skb1: the buffer to receive the second part
|
|
|
|
* @len: new length for skb
|
|
|
|
*/
|
|
|
|
void skb_split(struct sk_buff *skb, struct sk_buff *skb1, const u32 len)
|
|
|
|
{
|
|
|
|
int pos = skb_headlen(skb);
|
2021-11-03 10:58:44 +08:00
|
|
|
const int zc_flags = SKBFL_SHARED_FRAG | SKBFL_PURE_ZEROCOPY;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2021-11-03 10:58:44 +08:00
|
|
|
skb_shinfo(skb1)->flags |= skb_shinfo(skb)->flags & zc_flags;
|
sock: enable MSG_ZEROCOPY
Prepare the datapath for refcounted ubuf_info. Clone ubuf_info with
skb_zerocopy_clone() wherever needed due to skb split, merge, resize
or clone.
Split skb_orphan_frags into two variants. The split, merge, .. paths
support reference counted zerocopy buffers, so do not do a deep copy.
Add skb_orphan_frags_rx for paths that may loop packets to receive
sockets. That is not allowed, as it may cause unbounded latency.
Deep copy all zerocopy copy buffers, ref-counted or not, in this path.
The exact locations to modify were chosen by exhaustively searching
through all code that might modify skb_frag references and/or the
the SKBTX_DEV_ZEROCOPY tx_flags bit.
The changes err on the safe side, in two ways.
(1) legacy ubuf_info paths virtio and tap are not modified. They keep
a 1:1 ubuf_info to sk_buff relationship. Calls to skb_orphan_frags
still call skb_copy_ubufs and thus copy frags in this case.
(2) not all copies deep in the stack are addressed yet. skb_shift,
skb_split and skb_try_coalesce can be refined to avoid copying.
These are not in the hot path and this patch is hairy enough as
is, so that is left for future refinement.
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-04 04:29:41 +08:00
|
|
|
skb_zerocopy_clone(skb1, skb, 0);
|
2005-04-17 06:20:36 +08:00
|
|
|
if (len < pos) /* Split line is inside header. */
|
|
|
|
skb_split_inside_header(skb, skb1, len, pos);
|
|
|
|
else /* Second chunk has no header, nothing to copy. */
|
|
|
|
skb_split_no_header(skb, skb1, len, pos);
|
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(skb_split);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2008-11-26 05:57:01 +08:00
|
|
|
/* Shifting from/to a cloned skb is a no-go.
|
|
|
|
*
|
|
|
|
* Caller cannot keep skb_shinfo related pointers past calling here!
|
|
|
|
*/
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 13:20:15 +08:00
|
|
|
static int skb_prepare_for_shift(struct sk_buff *skb)
|
|
|
|
{
|
2021-11-02 08:45:55 +08:00
|
|
|
return skb_unclone_keeptruesize(skb, GFP_ATOMIC);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 13:20:15 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* skb_shift - Shifts paged data partially from skb to another
|
|
|
|
* @tgt: buffer into which tail data gets added
|
|
|
|
* @skb: buffer from which the paged data comes from
|
|
|
|
* @shiftlen: shift up to this many bytes
|
|
|
|
*
|
|
|
|
* Attempts to shift up to shiftlen worth of bytes, which may be less than
|
2011-11-21 09:47:11 +08:00
|
|
|
* the length of the skb, from skb to tgt. Returns number bytes shifted.
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 13:20:15 +08:00
|
|
|
* It's up to caller to free skb if everything was shifted.
|
|
|
|
*
|
|
|
|
* If @tgt runs out of frags, the whole operation is aborted.
|
|
|
|
*
|
|
|
|
* Skb cannot include anything else but paged data while tgt is allowed
|
|
|
|
* to have non-paged data as well.
|
|
|
|
*
|
|
|
|
* TODO: full sized shift could be optimized but that would need
|
|
|
|
* specialized skb free'er to handle frags without up-to-date nr_frags.
|
|
|
|
*/
|
|
|
|
int skb_shift(struct sk_buff *tgt, struct sk_buff *skb, int shiftlen)
|
|
|
|
{
|
|
|
|
int from, to, merge, todo;
|
2019-07-23 11:08:26 +08:00
|
|
|
skb_frag_t *fragfrom, *fragto;
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 13:20:15 +08:00
|
|
|
|
|
|
|
BUG_ON(shiftlen > skb->len);
|
tcp: enhance tcp_collapse_retrans() with skb_shift()
In commit 2331ccc5b323 ("tcp: enhance tcp collapsing"),
we made a first step allowing copying right skb to left skb head.
Since all skbs in socket write queue are headless (but possibly the very
first one), this strategy often does not work.
This patch extends tcp_collapse_retrans() to perform frag shifting,
thanks to skb_shift() helper.
This helper needs to not BUG on non headless skbs, as callers are ok
with that.
Tested:
Following packetdrill test now passes :
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
+0 < S 0:0(0) win 32792 <mss 1460,sackOK,nop,nop,nop,wscale 8>
+0 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 8>
+.100 < . 1:1(0) ack 1 win 257
+0 accept(3, ..., ...) = 4
+0 setsockopt(4, SOL_TCP, TCP_NODELAY, [1], 4) = 0
+0 write(4, ..., 200) = 200
+0 > P. 1:201(200) ack 1
+.001 write(4, ..., 200) = 200
+0 > P. 201:401(200) ack 1
+.001 write(4, ..., 200) = 200
+0 > P. 401:601(200) ack 1
+.001 write(4, ..., 200) = 200
+0 > P. 601:801(200) ack 1
+.001 write(4, ..., 200) = 200
+0 > P. 801:1001(200) ack 1
+.001 write(4, ..., 100) = 100
+0 > P. 1001:1101(100) ack 1
+.001 write(4, ..., 100) = 100
+0 > P. 1101:1201(100) ack 1
+.001 write(4, ..., 100) = 100
+0 > P. 1201:1301(100) ack 1
+.001 write(4, ..., 100) = 100
+0 > P. 1301:1401(100) ack 1
+.099 < . 1:1(0) ack 201 win 257
+.001 < . 1:1(0) ack 201 win 257 <nop,nop,sack 1001:1401>
+0 > P. 201:1001(800) ack 1
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-11-16 04:51:50 +08:00
|
|
|
|
|
|
|
if (skb_headlen(skb))
|
|
|
|
return 0;
|
sock: enable MSG_ZEROCOPY
Prepare the datapath for refcounted ubuf_info. Clone ubuf_info with
skb_zerocopy_clone() wherever needed due to skb split, merge, resize
or clone.
Split skb_orphan_frags into two variants. The split, merge, .. paths
support reference counted zerocopy buffers, so do not do a deep copy.
Add skb_orphan_frags_rx for paths that may loop packets to receive
sockets. That is not allowed, as it may cause unbounded latency.
Deep copy all zerocopy copy buffers, ref-counted or not, in this path.
The exact locations to modify were chosen by exhaustively searching
through all code that might modify skb_frag references and/or the
the SKBTX_DEV_ZEROCOPY tx_flags bit.
The changes err on the safe side, in two ways.
(1) legacy ubuf_info paths virtio and tap are not modified. They keep
a 1:1 ubuf_info to sk_buff relationship. Calls to skb_orphan_frags
still call skb_copy_ubufs and thus copy frags in this case.
(2) not all copies deep in the stack are addressed yet. skb_shift,
skb_split and skb_try_coalesce can be refined to avoid copying.
These are not in the hot path and this patch is hairy enough as
is, so that is left for future refinement.
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-04 04:29:41 +08:00
|
|
|
if (skb_zcopy(tgt) || skb_zcopy(skb))
|
|
|
|
return 0;
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 13:20:15 +08:00
|
|
|
|
|
|
|
todo = shiftlen;
|
|
|
|
from = 0;
|
|
|
|
to = skb_shinfo(tgt)->nr_frags;
|
|
|
|
fragfrom = &skb_shinfo(skb)->frags[from];
|
|
|
|
|
|
|
|
/* Actual merge is delayed until the point when we know we can
|
|
|
|
* commit all, so that we don't have to undo partial changes
|
|
|
|
*/
|
|
|
|
if (!to ||
|
2011-08-23 07:44:58 +08:00
|
|
|
!skb_can_coalesce(tgt, to, skb_frag_page(fragfrom),
|
2019-07-30 22:40:33 +08:00
|
|
|
skb_frag_off(fragfrom))) {
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 13:20:15 +08:00
|
|
|
merge = -1;
|
|
|
|
} else {
|
|
|
|
merge = to - 1;
|
|
|
|
|
2011-10-19 05:00:24 +08:00
|
|
|
todo -= skb_frag_size(fragfrom);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 13:20:15 +08:00
|
|
|
if (todo < 0) {
|
|
|
|
if (skb_prepare_for_shift(skb) ||
|
|
|
|
skb_prepare_for_shift(tgt))
|
|
|
|
return 0;
|
|
|
|
|
2008-11-26 05:57:01 +08:00
|
|
|
/* All previous frag pointers might be stale! */
|
|
|
|
fragfrom = &skb_shinfo(skb)->frags[from];
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 13:20:15 +08:00
|
|
|
fragto = &skb_shinfo(tgt)->frags[merge];
|
|
|
|
|
2011-10-19 05:00:24 +08:00
|
|
|
skb_frag_size_add(fragto, shiftlen);
|
|
|
|
skb_frag_size_sub(fragfrom, shiftlen);
|
2019-07-30 22:40:33 +08:00
|
|
|
skb_frag_off_add(fragfrom, shiftlen);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 13:20:15 +08:00
|
|
|
|
|
|
|
goto onlymerged;
|
|
|
|
}
|
|
|
|
|
|
|
|
from++;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Skip full, not-fitting skb to avoid expensive operations */
|
|
|
|
if ((shiftlen == skb->len) &&
|
|
|
|
(skb_shinfo(skb)->nr_frags - from) > (MAX_SKB_FRAGS - to))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
if (skb_prepare_for_shift(skb) || skb_prepare_for_shift(tgt))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
while ((todo > 0) && (from < skb_shinfo(skb)->nr_frags)) {
|
|
|
|
if (to == MAX_SKB_FRAGS)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
fragfrom = &skb_shinfo(skb)->frags[from];
|
|
|
|
fragto = &skb_shinfo(tgt)->frags[to];
|
|
|
|
|
2011-10-19 05:00:24 +08:00
|
|
|
if (todo >= skb_frag_size(fragfrom)) {
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 13:20:15 +08:00
|
|
|
*fragto = *fragfrom;
|
2011-10-19 05:00:24 +08:00
|
|
|
todo -= skb_frag_size(fragfrom);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 13:20:15 +08:00
|
|
|
from++;
|
|
|
|
to++;
|
|
|
|
|
|
|
|
} else {
|
2011-08-23 07:44:58 +08:00
|
|
|
__skb_frag_ref(fragfrom);
|
2019-07-30 22:40:33 +08:00
|
|
|
skb_frag_page_copy(fragto, fragfrom);
|
|
|
|
skb_frag_off_copy(fragto, fragfrom);
|
2011-10-19 05:00:24 +08:00
|
|
|
skb_frag_size_set(fragto, todo);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 13:20:15 +08:00
|
|
|
|
2019-07-30 22:40:33 +08:00
|
|
|
skb_frag_off_add(fragfrom, todo);
|
2011-10-19 05:00:24 +08:00
|
|
|
skb_frag_size_sub(fragfrom, todo);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 13:20:15 +08:00
|
|
|
todo = 0;
|
|
|
|
|
|
|
|
to++;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Ready to "commit" this state change to tgt */
|
|
|
|
skb_shinfo(tgt)->nr_frags = to;
|
|
|
|
|
|
|
|
if (merge >= 0) {
|
|
|
|
fragfrom = &skb_shinfo(skb)->frags[0];
|
|
|
|
fragto = &skb_shinfo(tgt)->frags[merge];
|
|
|
|
|
2011-10-19 05:00:24 +08:00
|
|
|
skb_frag_size_add(fragto, skb_frag_size(fragfrom));
|
2021-06-08 03:02:38 +08:00
|
|
|
__skb_frag_unref(fragfrom, skb->pp_recycle);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 13:20:15 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Reposition in the original skb */
|
|
|
|
to = 0;
|
|
|
|
while (from < skb_shinfo(skb)->nr_frags)
|
|
|
|
skb_shinfo(skb)->frags[to++] = skb_shinfo(skb)->frags[from++];
|
|
|
|
skb_shinfo(skb)->nr_frags = to;
|
|
|
|
|
|
|
|
BUG_ON(todo > 0 && !skb_shinfo(skb)->nr_frags);
|
|
|
|
|
|
|
|
onlymerged:
|
|
|
|
/* Most likely the tgt won't ever need its checksum anymore, skb on
|
|
|
|
* the other hand might need it if it needs to be resent
|
|
|
|
*/
|
|
|
|
tgt->ip_summed = CHECKSUM_PARTIAL;
|
|
|
|
skb->ip_summed = CHECKSUM_PARTIAL;
|
|
|
|
|
|
|
|
/* Yak, is it really working this way? Some helper please? */
|
|
|
|
skb->len -= shiftlen;
|
|
|
|
skb->data_len -= shiftlen;
|
|
|
|
skb->truesize -= shiftlen;
|
|
|
|
tgt->len += shiftlen;
|
|
|
|
tgt->data_len += shiftlen;
|
|
|
|
tgt->truesize += shiftlen;
|
|
|
|
|
|
|
|
return shiftlen;
|
|
|
|
}
|
|
|
|
|
2005-06-24 11:59:51 +08:00
|
|
|
/**
|
|
|
|
* skb_prepare_seq_read - Prepare a sequential read of skb data
|
|
|
|
* @skb: the buffer to read
|
|
|
|
* @from: lower offset of data to be read
|
|
|
|
* @to: upper offset of data to be read
|
|
|
|
* @st: state variable
|
|
|
|
*
|
|
|
|
* Initializes the specified state variable. Must be called before
|
|
|
|
* invoking skb_seq_read() for the first time.
|
|
|
|
*/
|
|
|
|
void skb_prepare_seq_read(struct sk_buff *skb, unsigned int from,
|
|
|
|
unsigned int to, struct skb_seq_state *st)
|
|
|
|
{
|
|
|
|
st->lower_offset = from;
|
|
|
|
st->upper_offset = to;
|
|
|
|
st->root_skb = st->cur_skb = skb;
|
|
|
|
st->frag_idx = st->stepped_offset = 0;
|
|
|
|
st->frag_data = NULL;
|
2021-01-10 06:18:33 +08:00
|
|
|
st->frag_off = 0;
|
2005-06-24 11:59:51 +08:00
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(skb_prepare_seq_read);
|
2005-06-24 11:59:51 +08:00
|
|
|
|
|
|
|
/**
|
|
|
|
* skb_seq_read - Sequentially read skb data
|
|
|
|
* @consumed: number of bytes consumed by the caller so far
|
|
|
|
* @data: destination pointer for data to be returned
|
|
|
|
* @st: state variable
|
|
|
|
*
|
2013-11-07 21:18:26 +08:00
|
|
|
* Reads a block of skb data at @consumed relative to the
|
2005-06-24 11:59:51 +08:00
|
|
|
* lower offset specified to skb_prepare_seq_read(). Assigns
|
2013-11-07 21:18:26 +08:00
|
|
|
* the head of the data block to @data and returns the length
|
2005-06-24 11:59:51 +08:00
|
|
|
* of the block or 0 if the end of the skb data or the upper
|
|
|
|
* offset has been reached.
|
|
|
|
*
|
|
|
|
* The caller is not required to consume all of the data
|
2013-11-07 21:18:26 +08:00
|
|
|
* returned, i.e. @consumed is typically set to the number
|
2005-06-24 11:59:51 +08:00
|
|
|
* of bytes already consumed and the next call to
|
|
|
|
* skb_seq_read() will return the remaining part of the block.
|
|
|
|
*
|
2011-03-31 09:57:33 +08:00
|
|
|
* Note 1: The size of each block of data returned can be arbitrary,
|
2014-09-04 22:44:36 +08:00
|
|
|
* this limitation is the cost for zerocopy sequential
|
2005-06-24 11:59:51 +08:00
|
|
|
* reads of potentially non linear data.
|
|
|
|
*
|
2008-02-14 07:03:25 +08:00
|
|
|
* Note 2: Fragment lists within fragments are not implemented
|
2005-06-24 11:59:51 +08:00
|
|
|
* at the moment, state->root_skb could be replaced with
|
|
|
|
* a stack for this purpose.
|
|
|
|
*/
|
|
|
|
unsigned int skb_seq_read(unsigned int consumed, const u8 **data,
|
|
|
|
struct skb_seq_state *st)
|
|
|
|
{
|
|
|
|
unsigned int block_limit, abs_offset = consumed + st->lower_offset;
|
|
|
|
skb_frag_t *frag;
|
|
|
|
|
2013-06-24 14:33:48 +08:00
|
|
|
if (unlikely(abs_offset >= st->upper_offset)) {
|
|
|
|
if (st->frag_data) {
|
|
|
|
kunmap_atomic(st->frag_data);
|
|
|
|
st->frag_data = NULL;
|
|
|
|
}
|
2005-06-24 11:59:51 +08:00
|
|
|
return 0;
|
2013-06-24 14:33:48 +08:00
|
|
|
}
|
2005-06-24 11:59:51 +08:00
|
|
|
|
|
|
|
next_skb:
|
2009-01-30 08:07:52 +08:00
|
|
|
block_limit = skb_headlen(st->cur_skb) + st->stepped_offset;
|
2005-06-24 11:59:51 +08:00
|
|
|
|
2009-05-19 12:43:27 +08:00
|
|
|
if (abs_offset < block_limit && !st->frag_data) {
|
2009-01-30 08:07:52 +08:00
|
|
|
*data = st->cur_skb->data + (abs_offset - st->stepped_offset);
|
2005-06-24 11:59:51 +08:00
|
|
|
return block_limit - abs_offset;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (st->frag_idx == 0 && !st->frag_data)
|
|
|
|
st->stepped_offset += skb_headlen(st->cur_skb);
|
|
|
|
|
|
|
|
while (st->frag_idx < skb_shinfo(st->cur_skb)->nr_frags) {
|
2021-01-10 06:18:33 +08:00
|
|
|
unsigned int pg_idx, pg_off, pg_sz;
|
|
|
|
|
2005-06-24 11:59:51 +08:00
|
|
|
frag = &skb_shinfo(st->cur_skb)->frags[st->frag_idx];
|
|
|
|
|
2021-01-10 06:18:33 +08:00
|
|
|
pg_idx = 0;
|
|
|
|
pg_off = skb_frag_off(frag);
|
|
|
|
pg_sz = skb_frag_size(frag);
|
|
|
|
|
|
|
|
if (skb_frag_must_loop(skb_frag_page(frag))) {
|
|
|
|
pg_idx = (pg_off + st->frag_off) >> PAGE_SHIFT;
|
|
|
|
pg_off = offset_in_page(pg_off + st->frag_off);
|
|
|
|
pg_sz = min_t(unsigned int, pg_sz - st->frag_off,
|
|
|
|
PAGE_SIZE - pg_off);
|
|
|
|
}
|
|
|
|
|
|
|
|
block_limit = pg_sz + st->stepped_offset;
|
2005-06-24 11:59:51 +08:00
|
|
|
if (abs_offset < block_limit) {
|
|
|
|
if (!st->frag_data)
|
2021-01-10 06:18:33 +08:00
|
|
|
st->frag_data = kmap_atomic(skb_frag_page(frag) + pg_idx);
|
2005-06-24 11:59:51 +08:00
|
|
|
|
2021-01-10 06:18:33 +08:00
|
|
|
*data = (u8 *)st->frag_data + pg_off +
|
2005-06-24 11:59:51 +08:00
|
|
|
(abs_offset - st->stepped_offset);
|
|
|
|
|
|
|
|
return block_limit - abs_offset;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (st->frag_data) {
|
2012-04-05 17:35:15 +08:00
|
|
|
kunmap_atomic(st->frag_data);
|
2005-06-24 11:59:51 +08:00
|
|
|
st->frag_data = NULL;
|
|
|
|
}
|
|
|
|
|
2021-01-10 06:18:33 +08:00
|
|
|
st->stepped_offset += pg_sz;
|
|
|
|
st->frag_off += pg_sz;
|
|
|
|
if (st->frag_off == skb_frag_size(frag)) {
|
|
|
|
st->frag_off = 0;
|
|
|
|
st->frag_idx++;
|
|
|
|
}
|
2005-06-24 11:59:51 +08:00
|
|
|
}
|
|
|
|
|
2007-06-24 14:11:52 +08:00
|
|
|
if (st->frag_data) {
|
2012-04-05 17:35:15 +08:00
|
|
|
kunmap_atomic(st->frag_data);
|
2007-06-24 14:11:52 +08:00
|
|
|
st->frag_data = NULL;
|
|
|
|
}
|
|
|
|
|
2010-08-23 15:13:46 +08:00
|
|
|
if (st->root_skb == st->cur_skb && skb_has_frag_list(st->root_skb)) {
|
2009-01-30 08:12:42 +08:00
|
|
|
st->cur_skb = skb_shinfo(st->root_skb)->frag_list;
|
2005-06-24 11:59:51 +08:00
|
|
|
st->frag_idx = 0;
|
|
|
|
goto next_skb;
|
2009-01-30 08:12:42 +08:00
|
|
|
} else if (st->cur_skb->next) {
|
|
|
|
st->cur_skb = st->cur_skb->next;
|
2009-01-30 08:07:52 +08:00
|
|
|
st->frag_idx = 0;
|
2005-06-24 11:59:51 +08:00
|
|
|
goto next_skb;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(skb_seq_read);
|
2005-06-24 11:59:51 +08:00
|
|
|
|
|
|
|
/**
|
|
|
|
* skb_abort_seq_read - Abort a sequential read of skb data
|
|
|
|
* @st: state variable
|
|
|
|
*
|
|
|
|
* Must be called if skb_seq_read() was not called until it
|
|
|
|
* returned 0.
|
|
|
|
*/
|
|
|
|
void skb_abort_seq_read(struct skb_seq_state *st)
|
|
|
|
{
|
|
|
|
if (st->frag_data)
|
2012-04-05 17:35:15 +08:00
|
|
|
kunmap_atomic(st->frag_data);
|
2005-06-24 11:59:51 +08:00
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(skb_abort_seq_read);
|
2005-06-24 11:59:51 +08:00
|
|
|
|
2005-06-24 12:00:17 +08:00
|
|
|
#define TS_SKB_CB(state) ((struct skb_seq_state *) &((state)->cb))
|
|
|
|
|
|
|
|
static unsigned int skb_ts_get_next_block(unsigned int offset, const u8 **text,
|
|
|
|
struct ts_config *conf,
|
|
|
|
struct ts_state *state)
|
|
|
|
{
|
|
|
|
return skb_seq_read(offset, text, TS_SKB_CB(state));
|
|
|
|
}
|
|
|
|
|
|
|
|
static void skb_ts_finish(struct ts_config *conf, struct ts_state *state)
|
|
|
|
{
|
|
|
|
skb_abort_seq_read(TS_SKB_CB(state));
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* skb_find_text - Find a text pattern in skb data
|
|
|
|
* @skb: the buffer to look in
|
|
|
|
* @from: search offset
|
|
|
|
* @to: search limit
|
|
|
|
* @config: textsearch configuration
|
|
|
|
*
|
|
|
|
* Finds a pattern in the skb data according to the specified
|
|
|
|
* textsearch configuration. Use textsearch_next() to retrieve
|
|
|
|
* subsequent occurrences of the pattern. Returns the offset
|
|
|
|
* to the first occurrence or UINT_MAX if no match was found.
|
|
|
|
*/
|
|
|
|
unsigned int skb_find_text(struct sk_buff *skb, unsigned int from,
|
2015-02-22 18:46:35 +08:00
|
|
|
unsigned int to, struct ts_config *config)
|
2005-06-24 12:00:17 +08:00
|
|
|
{
|
2015-02-22 18:46:35 +08:00
|
|
|
struct ts_state state;
|
2006-06-26 15:00:57 +08:00
|
|
|
unsigned int ret;
|
|
|
|
|
2021-03-01 23:09:44 +08:00
|
|
|
BUILD_BUG_ON(sizeof(struct skb_seq_state) > sizeof(state.cb));
|
|
|
|
|
2005-06-24 12:00:17 +08:00
|
|
|
config->get_next_block = skb_ts_get_next_block;
|
|
|
|
config->finish = skb_ts_finish;
|
|
|
|
|
2015-02-22 18:46:35 +08:00
|
|
|
skb_prepare_seq_read(skb, from, to, TS_SKB_CB(&state));
|
2005-06-24 12:00:17 +08:00
|
|
|
|
2015-02-22 18:46:35 +08:00
|
|
|
ret = textsearch_find(config, &state);
|
2006-06-26 15:00:57 +08:00
|
|
|
return (ret <= to - from ? ret : UINT_MAX);
|
2005-06-24 12:00:17 +08:00
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL(skb_find_text);
|
2005-06-24 12:00:17 +08:00
|
|
|
|
2015-05-21 22:59:58 +08:00
|
|
|
int skb_append_pagefrags(struct sk_buff *skb, struct page *page,
|
|
|
|
int offset, size_t size)
|
|
|
|
{
|
|
|
|
int i = skb_shinfo(skb)->nr_frags;
|
|
|
|
|
|
|
|
if (skb_can_coalesce(skb, i, page, offset)) {
|
|
|
|
skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], size);
|
|
|
|
} else if (i < MAX_SKB_FRAGS) {
|
|
|
|
get_page(page);
|
|
|
|
skb_fill_page_desc(skb, i, page, offset, size);
|
|
|
|
} else {
|
|
|
|
return -EMSGSIZE;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(skb_append_pagefrags);
|
|
|
|
|
2006-03-21 14:43:56 +08:00
|
|
|
/**
|
|
|
|
* skb_pull_rcsum - pull skb and update receive checksum
|
|
|
|
* @skb: buffer to update
|
|
|
|
* @len: length of data pulled
|
|
|
|
*
|
|
|
|
* This function performs an skb_pull on the packet and updates
|
2008-02-13 14:03:25 +08:00
|
|
|
* the CHECKSUM_COMPLETE checksum. It should be used on
|
2006-08-30 07:44:56 +08:00
|
|
|
* receive path processing instead of skb_pull unless you know
|
|
|
|
* that the checksum difference is zero (e.g., a valid IP header)
|
|
|
|
* or you are setting ip_summed to CHECKSUM_NONE.
|
2006-03-21 14:43:56 +08:00
|
|
|
*/
|
networking: make skb_pull & friends return void pointers
It seems like a historic accident that these return unsigned char *,
and in many places that means casts are required, more often than not.
Make these functions return void * and remove all the casts across
the tree, adding a (u8 *) cast only where the unsigned char pointer
was used directly, all done with the following spatch:
@@
expression SKB, LEN;
typedef u8;
identifier fn = {
skb_pull,
__skb_pull,
skb_pull_inline,
__pskb_pull_tail,
__pskb_pull,
pskb_pull
};
@@
- *(fn(SKB, LEN))
+ *(u8 *)fn(SKB, LEN)
@@
expression E, SKB, LEN;
identifier fn = {
skb_pull,
__skb_pull,
skb_pull_inline,
__pskb_pull_tail,
__pskb_pull,
pskb_pull
};
type T;
@@
- E = ((T *)(fn(SKB, LEN)))
+ E = fn(SKB, LEN)
Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-06-16 20:29:22 +08:00
|
|
|
void *skb_pull_rcsum(struct sk_buff *skb, unsigned int len)
|
2006-03-21 14:43:56 +08:00
|
|
|
{
|
2015-09-29 08:24:25 +08:00
|
|
|
unsigned char *data = skb->data;
|
|
|
|
|
2006-03-21 14:43:56 +08:00
|
|
|
BUG_ON(len > skb->len);
|
2015-09-29 08:24:25 +08:00
|
|
|
__skb_pull(skb, len);
|
|
|
|
skb_postpull_rcsum(skb, data, len);
|
|
|
|
return skb->data;
|
2006-03-21 14:43:56 +08:00
|
|
|
}
|
2006-03-21 14:47:55 +08:00
|
|
|
EXPORT_SYMBOL_GPL(skb_pull_rcsum);
|
|
|
|
|
2018-03-22 07:31:03 +08:00
|
|
|
static inline skb_frag_t skb_head_frag_to_page_desc(struct sk_buff *frag_skb)
|
|
|
|
{
|
|
|
|
skb_frag_t head_frag;
|
|
|
|
struct page *page;
|
|
|
|
|
|
|
|
page = virt_to_head_page(frag_skb->head);
|
2019-07-23 11:08:26 +08:00
|
|
|
__skb_frag_set_page(&head_frag, page);
|
2019-07-30 22:40:33 +08:00
|
|
|
skb_frag_off_set(&head_frag, frag_skb->data -
|
|
|
|
(unsigned char *)page_address(page));
|
2019-07-23 11:08:26 +08:00
|
|
|
skb_frag_size_set(&head_frag, skb_headlen(frag_skb));
|
2018-03-22 07:31:03 +08:00
|
|
|
return head_frag;
|
|
|
|
}
|
|
|
|
|
2020-01-25 18:26:44 +08:00
|
|
|
struct sk_buff *skb_segment_list(struct sk_buff *skb,
|
|
|
|
netdev_features_t features,
|
|
|
|
unsigned int offset)
|
|
|
|
{
|
|
|
|
struct sk_buff *list_skb = skb_shinfo(skb)->frag_list;
|
|
|
|
unsigned int tnl_hlen = skb_tnl_header_len(skb);
|
|
|
|
unsigned int delta_truesize = 0;
|
|
|
|
unsigned int delta_len = 0;
|
|
|
|
struct sk_buff *tail = NULL;
|
2021-01-08 10:28:38 +08:00
|
|
|
struct sk_buff *nskb, *tmp;
|
|
|
|
int err;
|
2020-01-25 18:26:44 +08:00
|
|
|
|
|
|
|
skb_push(skb, -skb_network_offset(skb) + offset);
|
|
|
|
|
|
|
|
skb_shinfo(skb)->frag_list = NULL;
|
|
|
|
|
|
|
|
do {
|
|
|
|
nskb = list_skb;
|
|
|
|
list_skb = list_skb->next;
|
|
|
|
|
2021-01-08 10:28:38 +08:00
|
|
|
err = 0;
|
2022-03-01 19:17:09 +08:00
|
|
|
delta_truesize += nskb->truesize;
|
2021-01-08 10:28:38 +08:00
|
|
|
if (skb_shared(nskb)) {
|
|
|
|
tmp = skb_clone(nskb, GFP_ATOMIC);
|
|
|
|
if (tmp) {
|
|
|
|
consume_skb(nskb);
|
|
|
|
nskb = tmp;
|
|
|
|
err = skb_unclone(nskb, GFP_ATOMIC);
|
|
|
|
} else {
|
|
|
|
err = -ENOMEM;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2020-01-25 18:26:44 +08:00
|
|
|
if (!tail)
|
|
|
|
skb->next = nskb;
|
|
|
|
else
|
|
|
|
tail->next = nskb;
|
|
|
|
|
2021-01-08 10:28:38 +08:00
|
|
|
if (unlikely(err)) {
|
|
|
|
nskb->next = list_skb;
|
|
|
|
goto err_linearize;
|
|
|
|
}
|
|
|
|
|
2020-01-25 18:26:44 +08:00
|
|
|
tail = nskb;
|
|
|
|
|
|
|
|
delta_len += nskb->len;
|
|
|
|
|
|
|
|
skb_push(nskb, -skb_network_offset(nskb) + offset);
|
|
|
|
|
2020-03-31 00:51:29 +08:00
|
|
|
skb_release_head_state(nskb);
|
2021-09-03 06:56:23 +08:00
|
|
|
__copy_skb_header(nskb, skb);
|
2020-01-25 18:26:44 +08:00
|
|
|
|
|
|
|
skb_headers_offset_update(nskb, skb_headroom(nskb) - skb_headroom(skb));
|
|
|
|
skb_copy_from_linear_data_offset(skb, -tnl_hlen,
|
|
|
|
nskb->data - tnl_hlen,
|
|
|
|
offset + tnl_hlen);
|
|
|
|
|
|
|
|
if (skb_needs_linearize(nskb, features) &&
|
|
|
|
__skb_linearize(nskb))
|
|
|
|
goto err_linearize;
|
|
|
|
|
|
|
|
} while (list_skb);
|
|
|
|
|
|
|
|
skb->truesize = skb->truesize - delta_truesize;
|
|
|
|
skb->data_len = skb->data_len - delta_len;
|
|
|
|
skb->len = skb->len - delta_len;
|
|
|
|
|
|
|
|
skb_gso_reset(skb);
|
|
|
|
|
|
|
|
skb->prev = tail;
|
|
|
|
|
|
|
|
if (skb_needs_linearize(skb, features) &&
|
|
|
|
__skb_linearize(skb))
|
|
|
|
goto err_linearize;
|
|
|
|
|
|
|
|
skb_get(skb);
|
|
|
|
|
|
|
|
return skb;
|
|
|
|
|
|
|
|
err_linearize:
|
|
|
|
kfree_skb_list(skb->next);
|
|
|
|
skb->next = NULL;
|
|
|
|
return ERR_PTR(-ENOMEM);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(skb_segment_list);
|
|
|
|
|
2006-06-22 18:02:40 +08:00
|
|
|
/**
|
|
|
|
* skb_segment - Perform protocol segmentation on skb.
|
2014-03-11 00:29:19 +08:00
|
|
|
* @head_skb: buffer to segment
|
2006-06-28 04:22:38 +08:00
|
|
|
* @features: features for the output path (see dev->features)
|
2006-06-22 18:02:40 +08:00
|
|
|
*
|
|
|
|
* This function performs segmentation on the given skb. It returns
|
2008-04-14 12:52:48 +08:00
|
|
|
* a pointer to the first in a list of new skbs for the segments.
|
|
|
|
* In case of error it returns ERR_PTR(err).
|
2006-06-22 18:02:40 +08:00
|
|
|
*/
|
2014-03-11 00:29:19 +08:00
|
|
|
struct sk_buff *skb_segment(struct sk_buff *head_skb,
|
|
|
|
netdev_features_t features)
|
2006-06-22 18:02:40 +08:00
|
|
|
{
|
|
|
|
struct sk_buff *segs = NULL;
|
|
|
|
struct sk_buff *tail = NULL;
|
2014-03-11 01:27:59 +08:00
|
|
|
struct sk_buff *list_skb = skb_shinfo(head_skb)->frag_list;
|
2014-03-11 00:29:19 +08:00
|
|
|
skb_frag_t *frag = skb_shinfo(head_skb)->frags;
|
|
|
|
unsigned int mss = skb_shinfo(head_skb)->gso_size;
|
|
|
|
unsigned int doffset = head_skb->data - skb_mac_header(head_skb);
|
2014-03-11 01:28:08 +08:00
|
|
|
struct sk_buff *frag_skb = head_skb;
|
2006-06-22 18:02:40 +08:00
|
|
|
unsigned int offset = doffset;
|
2014-03-11 00:29:19 +08:00
|
|
|
unsigned int tnl_hlen = skb_tnl_header_len(head_skb);
|
2016-04-11 09:45:03 +08:00
|
|
|
unsigned int partial_segs = 0;
|
2006-06-22 18:02:40 +08:00
|
|
|
unsigned int headroom;
|
2016-04-11 09:45:03 +08:00
|
|
|
unsigned int len = head_skb->len;
|
2013-03-07 17:28:01 +08:00
|
|
|
__be16 proto;
|
2016-05-03 00:38:18 +08:00
|
|
|
bool csum, sg;
|
2014-03-11 00:29:19 +08:00
|
|
|
int nfrags = skb_shinfo(head_skb)->nr_frags;
|
2006-06-22 18:02:40 +08:00
|
|
|
int err = -ENOMEM;
|
|
|
|
int i = 0;
|
|
|
|
int pos;
|
|
|
|
|
net: gso: Fix skb_segment splat when splitting gso_size mangled skb having linear-headed frag_list
Historically, support for frag_list packets entering skb_segment() was
limited to frag_list members terminating on exact same gso_size
boundaries. This is verified with a BUG_ON since commit 89319d3801d1
("net: Add frag_list support to skb_segment"), quote:
As such we require all frag_list members terminate on exact MSS
boundaries. This is checked using BUG_ON.
As there should only be one producer in the kernel of such packets,
namely GRO, this requirement should not be difficult to maintain.
However, since commit 6578171a7ff0 ("bpf: add bpf_skb_change_proto helper"),
the "exact MSS boundaries" assumption no longer holds:
An eBPF program using bpf_skb_change_proto() DOES modify 'gso_size', but
leaves the frag_list members as originally merged by GRO with the
original 'gso_size'. Example of such programs are bpf-based NAT46 or
NAT64.
This lead to a kernel BUG_ON for flows involving:
- GRO generating a frag_list skb
- bpf program performing bpf_skb_change_proto() or bpf_skb_adjust_room()
- skb_segment() of the skb
See example BUG_ON reports in [0].
In commit 13acc94eff12 ("net: permit skb_segment on head_frag frag_list skb"),
skb_segment() was modified to support the "gso_size mangling" case of
a frag_list GRO'ed skb, but *only* for frag_list members having
head_frag==true (having a page-fragment head).
Alas, GRO packets having frag_list members with a linear kmalloced head
(head_frag==false) still hit the BUG_ON.
This commit adds support to skb_segment() for a 'head_skb' packet having
a frag_list whose members are *non* head_frag, with gso_size mangled, by
disabling SG and thus falling-back to copying the data from the given
'head_skb' into the generated segmented skbs - as suggested by Willem de
Bruijn [1].
Since this approach involves the penalty of skb_copy_and_csum_bits()
when building the segments, care was taken in order to enable this
solution only when required:
- untrusted gso_size, by testing SKB_GSO_DODGY is set
(SKB_GSO_DODGY is set by any gso_size mangling functions in
net/core/filter.c)
- the frag_list is non empty, its item is a non head_frag, *and* the
headlen of the given 'head_skb' does not match the gso_size.
[0]
https://lore.kernel.org/netdev/20190826170724.25ff616f@pixies/
https://lore.kernel.org/netdev/9265b93f-253d-6b8c-f2b8-4b54eff1835c@fb.com/
[1]
https://lore.kernel.org/netdev/CA+FuTSfVsgNDi7c=GUU8nMg2hWxF2SjCNLXetHeVPdnxAW5K-w@mail.gmail.com/
Fixes: 6578171a7ff0 ("bpf: add bpf_skb_change_proto helper")
Suggested-by: Willem de Bruijn <willemdebruijn.kernel@gmail.com>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: Eric Dumazet <eric.dumazet@gmail.com>
Cc: Alexander Duyck <alexander.duyck@gmail.com>
Signed-off-by: Shmulik Ladkani <shmulik.ladkani@gmail.com>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Reviewed-by: Alexander Duyck <alexander.h.duyck@linux.intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2019-09-06 17:23:50 +08:00
|
|
|
if (list_skb && !list_skb->head_frag && skb_headlen(list_skb) &&
|
|
|
|
(skb_shinfo(head_skb)->gso_type & SKB_GSO_DODGY)) {
|
|
|
|
/* gso_size is untrusted, and we have a frag_list with a linear
|
|
|
|
* non head_frag head.
|
|
|
|
*
|
|
|
|
* (we assume checking the first list_skb member suffices;
|
|
|
|
* i.e if either of the list_skb members have non head_frag
|
|
|
|
* head, then the first one has too).
|
|
|
|
*
|
|
|
|
* If head_skb's headlen does not fit requested gso_size, it
|
|
|
|
* means that the frag_list members do NOT terminate on exact
|
|
|
|
* gso_size boundaries. Hence we cannot perform skb_frag_t page
|
|
|
|
* sharing. Therefore we must fallback to copying the frag_list
|
|
|
|
* skbs; we do so by disabling SG.
|
|
|
|
*/
|
|
|
|
if (mss != GSO_BY_FRAGS && mss != skb_headlen(head_skb))
|
|
|
|
features &= ~NETIF_F_SG;
|
|
|
|
}
|
|
|
|
|
2014-06-09 14:48:54 +08:00
|
|
|
__skb_push(head_skb, doffset);
|
2020-08-01 17:36:05 +08:00
|
|
|
proto = skb_network_protocol(head_skb, NULL);
|
2013-03-07 17:28:01 +08:00
|
|
|
if (unlikely(!proto))
|
|
|
|
return ERR_PTR(-EINVAL);
|
|
|
|
|
2016-05-03 00:38:18 +08:00
|
|
|
sg = !!(features & NETIF_F_SG);
|
2016-02-06 07:28:26 +08:00
|
|
|
csum = !!can_checksum_protocol(features, proto);
|
2014-06-05 08:20:02 +08:00
|
|
|
|
2016-09-19 18:58:47 +08:00
|
|
|
if (sg && csum && (mss != GSO_BY_FRAGS)) {
|
|
|
|
if (!(features & NETIF_F_GSO_PARTIAL)) {
|
|
|
|
struct sk_buff *iter;
|
2017-04-20 02:26:07 +08:00
|
|
|
unsigned int frag_len;
|
2016-09-19 18:58:47 +08:00
|
|
|
|
|
|
|
if (!list_skb ||
|
|
|
|
!net_gso_ok(features, skb_shinfo(head_skb)->gso_type))
|
|
|
|
goto normal;
|
|
|
|
|
2017-04-20 02:26:07 +08:00
|
|
|
/* If we get here then all the required
|
|
|
|
* GSO features except frag_list are supported.
|
|
|
|
* Try to split the SKB to multiple GSO SKBs
|
|
|
|
* with no frag_list.
|
|
|
|
* Currently we can do that only when the buffers don't
|
|
|
|
* have a linear part and all the buffers except
|
|
|
|
* the last are of the same length.
|
2016-09-19 18:58:47 +08:00
|
|
|
*/
|
2017-04-20 02:26:07 +08:00
|
|
|
frag_len = list_skb->len;
|
2016-09-19 18:58:47 +08:00
|
|
|
skb_walk_frags(head_skb, iter) {
|
2017-04-20 02:26:07 +08:00
|
|
|
if (frag_len != iter->len && iter->next)
|
|
|
|
goto normal;
|
2017-04-08 07:07:08 +08:00
|
|
|
if (skb_headlen(iter) && !iter->head_frag)
|
2016-09-19 18:58:47 +08:00
|
|
|
goto normal;
|
|
|
|
|
|
|
|
len -= iter->len;
|
|
|
|
}
|
2017-04-20 02:26:07 +08:00
|
|
|
|
|
|
|
if (len != frag_len)
|
|
|
|
goto normal;
|
2016-09-19 18:58:47 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/* GSO partial only requires that we trim off any excess that
|
|
|
|
* doesn't fit into an MSS sized block, so take care of that
|
|
|
|
* now.
|
|
|
|
*/
|
2016-04-11 09:45:03 +08:00
|
|
|
partial_segs = len / mss;
|
2016-05-03 00:38:12 +08:00
|
|
|
if (partial_segs > 1)
|
|
|
|
mss *= partial_segs;
|
|
|
|
else
|
|
|
|
partial_segs = 0;
|
2016-04-11 09:45:03 +08:00
|
|
|
}
|
|
|
|
|
2016-09-19 18:58:47 +08:00
|
|
|
normal:
|
2014-03-11 00:29:19 +08:00
|
|
|
headroom = skb_headroom(head_skb);
|
|
|
|
pos = skb_headlen(head_skb);
|
2006-06-22 18:02:40 +08:00
|
|
|
|
|
|
|
do {
|
|
|
|
struct sk_buff *nskb;
|
2014-03-11 00:29:04 +08:00
|
|
|
skb_frag_t *nskb_frag;
|
2006-10-30 07:59:41 +08:00
|
|
|
int hsize;
|
2006-06-22 18:02:40 +08:00
|
|
|
int size;
|
|
|
|
|
2016-06-03 02:05:40 +08:00
|
|
|
if (unlikely(mss == GSO_BY_FRAGS)) {
|
|
|
|
len = list_skb->len;
|
|
|
|
} else {
|
|
|
|
len = head_skb->len - offset;
|
|
|
|
if (len > mss)
|
|
|
|
len = mss;
|
|
|
|
}
|
2006-06-22 18:02:40 +08:00
|
|
|
|
2014-03-11 00:29:19 +08:00
|
|
|
hsize = skb_headlen(head_skb) - offset;
|
2006-06-22 18:02:40 +08:00
|
|
|
|
2021-01-15 17:36:38 +08:00
|
|
|
if (hsize <= 0 && i >= nfrags && skb_headlen(list_skb) &&
|
2014-03-11 01:27:59 +08:00
|
|
|
(skb_headlen(list_skb) == len || sg)) {
|
|
|
|
BUG_ON(skb_headlen(list_skb) > len);
|
2013-11-22 03:10:04 +08:00
|
|
|
|
|
|
|
i = 0;
|
2014-03-11 01:27:59 +08:00
|
|
|
nfrags = skb_shinfo(list_skb)->nr_frags;
|
|
|
|
frag = skb_shinfo(list_skb)->frags;
|
2014-03-11 01:28:08 +08:00
|
|
|
frag_skb = list_skb;
|
2014-03-11 01:27:59 +08:00
|
|
|
pos += skb_headlen(list_skb);
|
2013-11-22 03:10:04 +08:00
|
|
|
|
|
|
|
while (pos < offset + len) {
|
|
|
|
BUG_ON(i >= nfrags);
|
|
|
|
|
2014-03-11 00:29:14 +08:00
|
|
|
size = skb_frag_size(frag);
|
2013-11-22 03:10:04 +08:00
|
|
|
if (pos + size > offset + len)
|
|
|
|
break;
|
|
|
|
|
|
|
|
i++;
|
|
|
|
pos += size;
|
2014-03-11 00:29:14 +08:00
|
|
|
frag++;
|
2013-11-22 03:10:04 +08:00
|
|
|
}
|
2008-12-16 15:26:06 +08:00
|
|
|
|
2014-03-11 01:27:59 +08:00
|
|
|
nskb = skb_clone(list_skb, GFP_ATOMIC);
|
|
|
|
list_skb = list_skb->next;
|
2008-12-16 15:26:06 +08:00
|
|
|
|
|
|
|
if (unlikely(!nskb))
|
|
|
|
goto err;
|
|
|
|
|
2013-11-22 03:10:04 +08:00
|
|
|
if (unlikely(pskb_trim(nskb, len))) {
|
|
|
|
kfree_skb(nskb);
|
|
|
|
goto err;
|
|
|
|
}
|
|
|
|
|
2012-05-04 22:26:56 +08:00
|
|
|
hsize = skb_end_offset(nskb);
|
2008-12-16 15:26:06 +08:00
|
|
|
if (skb_cow_head(nskb, doffset + headroom)) {
|
|
|
|
kfree_skb(nskb);
|
|
|
|
goto err;
|
|
|
|
}
|
|
|
|
|
2012-05-04 22:26:56 +08:00
|
|
|
nskb->truesize += skb_end_offset(nskb) - hsize;
|
2008-12-16 15:26:06 +08:00
|
|
|
skb_release_head_state(nskb);
|
|
|
|
__skb_push(nskb, doffset);
|
|
|
|
} else {
|
2021-01-20 00:56:56 +08:00
|
|
|
if (hsize < 0)
|
|
|
|
hsize = 0;
|
2021-01-15 17:36:38 +08:00
|
|
|
if (hsize > len || !sg)
|
|
|
|
hsize = len;
|
|
|
|
|
2012-08-01 07:44:19 +08:00
|
|
|
nskb = __alloc_skb(hsize + doffset + headroom,
|
2014-03-11 00:29:19 +08:00
|
|
|
GFP_ATOMIC, skb_alloc_rx_flag(head_skb),
|
2012-08-01 07:44:19 +08:00
|
|
|
NUMA_NO_NODE);
|
2008-12-16 15:26:06 +08:00
|
|
|
|
|
|
|
if (unlikely(!nskb))
|
|
|
|
goto err;
|
|
|
|
|
|
|
|
skb_reserve(nskb, headroom);
|
|
|
|
__skb_put(nskb, doffset);
|
|
|
|
}
|
2006-06-22 18:02:40 +08:00
|
|
|
|
|
|
|
if (segs)
|
|
|
|
tail->next = nskb;
|
|
|
|
else
|
|
|
|
segs = nskb;
|
|
|
|
tail = nskb;
|
|
|
|
|
2014-03-11 00:29:19 +08:00
|
|
|
__copy_skb_header(nskb, head_skb);
|
2006-06-22 18:02:40 +08:00
|
|
|
|
2013-10-20 02:42:54 +08:00
|
|
|
skb_headers_offset_update(nskb, skb_headroom(nskb) - headroom);
|
2014-07-31 22:33:06 +08:00
|
|
|
skb_reset_mac_len(nskb);
|
2013-02-14 22:02:41 +08:00
|
|
|
|
2014-03-11 00:29:19 +08:00
|
|
|
skb_copy_from_linear_data_offset(head_skb, -tnl_hlen,
|
2013-02-14 22:02:41 +08:00
|
|
|
nskb->data - tnl_hlen,
|
|
|
|
doffset + tnl_hlen);
|
2008-12-16 15:26:06 +08:00
|
|
|
|
2013-11-22 03:10:04 +08:00
|
|
|
if (nskb->len == len + doffset)
|
2013-05-19 23:46:49 +08:00
|
|
|
goto perform_csum_check;
|
2008-12-16 15:26:06 +08:00
|
|
|
|
2016-02-06 07:27:43 +08:00
|
|
|
if (!sg) {
|
2020-03-17 16:38:38 +08:00
|
|
|
if (!csum) {
|
|
|
|
if (!nskb->remcsum_offload)
|
|
|
|
nskb->ip_summed = CHECKSUM_NONE;
|
|
|
|
SKB_GSO_CB(nskb)->csum =
|
|
|
|
skb_copy_and_csum_bits(head_skb, offset,
|
|
|
|
skb_put(nskb,
|
|
|
|
len),
|
2020-07-11 08:07:10 +08:00
|
|
|
len);
|
2020-03-17 16:38:38 +08:00
|
|
|
SKB_GSO_CB(nskb)->csum_start =
|
|
|
|
skb_headroom(nskb) + doffset;
|
|
|
|
} else {
|
|
|
|
skb_copy_bits(head_skb, offset,
|
|
|
|
skb_put(nskb, len),
|
|
|
|
len);
|
|
|
|
}
|
2006-06-22 18:02:40 +08:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
2014-03-11 00:29:04 +08:00
|
|
|
nskb_frag = skb_shinfo(nskb)->frags;
|
2006-06-22 18:02:40 +08:00
|
|
|
|
2014-03-11 00:29:19 +08:00
|
|
|
skb_copy_from_linear_data_offset(head_skb, offset,
|
2007-03-28 05:55:52 +08:00
|
|
|
skb_put(nskb, hsize), hsize);
|
2006-06-22 18:02:40 +08:00
|
|
|
|
2021-01-07 06:18:38 +08:00
|
|
|
skb_shinfo(nskb)->flags |= skb_shinfo(head_skb)->flags &
|
|
|
|
SKBFL_SHARED_FRAG;
|
2013-01-26 04:34:37 +08:00
|
|
|
|
2017-12-23 08:00:17 +08:00
|
|
|
if (skb_orphan_frags(frag_skb, GFP_ATOMIC) ||
|
|
|
|
skb_zerocopy_clone(nskb, frag_skb, GFP_ATOMIC))
|
|
|
|
goto err;
|
|
|
|
|
2013-11-22 03:10:04 +08:00
|
|
|
while (pos < offset + len) {
|
|
|
|
if (i >= nfrags) {
|
|
|
|
i = 0;
|
2014-03-11 01:27:59 +08:00
|
|
|
nfrags = skb_shinfo(list_skb)->nr_frags;
|
|
|
|
frag = skb_shinfo(list_skb)->frags;
|
2014-03-11 01:28:08 +08:00
|
|
|
frag_skb = list_skb;
|
2018-03-22 07:31:03 +08:00
|
|
|
if (!skb_headlen(list_skb)) {
|
|
|
|
BUG_ON(!nfrags);
|
|
|
|
} else {
|
|
|
|
BUG_ON(!list_skb->head_frag);
|
2013-11-22 03:10:04 +08:00
|
|
|
|
2018-03-22 07:31:03 +08:00
|
|
|
/* to make room for head_frag. */
|
|
|
|
i--;
|
|
|
|
frag--;
|
|
|
|
}
|
2017-12-23 08:00:17 +08:00
|
|
|
if (skb_orphan_frags(frag_skb, GFP_ATOMIC) ||
|
|
|
|
skb_zerocopy_clone(nskb, frag_skb,
|
|
|
|
GFP_ATOMIC))
|
|
|
|
goto err;
|
|
|
|
|
2014-03-11 01:27:59 +08:00
|
|
|
list_skb = list_skb->next;
|
2013-11-22 03:10:04 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
if (unlikely(skb_shinfo(nskb)->nr_frags >=
|
|
|
|
MAX_SKB_FRAGS)) {
|
|
|
|
net_warn_ratelimited(
|
|
|
|
"skb_segment: too many frags: %u %u\n",
|
|
|
|
pos, mss);
|
2018-07-20 07:04:38 +08:00
|
|
|
err = -EINVAL;
|
2013-11-22 03:10:04 +08:00
|
|
|
goto err;
|
|
|
|
}
|
|
|
|
|
2018-03-22 07:31:03 +08:00
|
|
|
*nskb_frag = (i < 0) ? skb_head_frag_to_page_desc(frag_skb) : *frag;
|
2014-03-11 00:29:04 +08:00
|
|
|
__skb_frag_ref(nskb_frag);
|
|
|
|
size = skb_frag_size(nskb_frag);
|
2006-06-22 18:02:40 +08:00
|
|
|
|
|
|
|
if (pos < offset) {
|
2019-07-30 22:40:33 +08:00
|
|
|
skb_frag_off_add(nskb_frag, offset - pos);
|
2014-03-11 00:29:04 +08:00
|
|
|
skb_frag_size_sub(nskb_frag, offset - pos);
|
2006-06-22 18:02:40 +08:00
|
|
|
}
|
|
|
|
|
2008-12-16 15:26:06 +08:00
|
|
|
skb_shinfo(nskb)->nr_frags++;
|
2006-06-22 18:02:40 +08:00
|
|
|
|
|
|
|
if (pos + size <= offset + len) {
|
|
|
|
i++;
|
2014-03-11 00:29:14 +08:00
|
|
|
frag++;
|
2006-06-22 18:02:40 +08:00
|
|
|
pos += size;
|
|
|
|
} else {
|
2014-03-11 00:29:04 +08:00
|
|
|
skb_frag_size_sub(nskb_frag, pos + size - (offset + len));
|
2008-12-16 15:26:06 +08:00
|
|
|
goto skip_fraglist;
|
2006-06-22 18:02:40 +08:00
|
|
|
}
|
|
|
|
|
2014-03-11 00:29:04 +08:00
|
|
|
nskb_frag++;
|
2006-06-22 18:02:40 +08:00
|
|
|
}
|
|
|
|
|
2008-12-16 15:26:06 +08:00
|
|
|
skip_fraglist:
|
2006-06-22 18:02:40 +08:00
|
|
|
nskb->data_len = len - hsize;
|
|
|
|
nskb->len += nskb->data_len;
|
|
|
|
nskb->truesize += nskb->data_len;
|
2013-03-07 17:28:01 +08:00
|
|
|
|
2013-05-19 23:46:49 +08:00
|
|
|
perform_csum_check:
|
2016-02-06 07:27:43 +08:00
|
|
|
if (!csum) {
|
2018-07-20 07:04:38 +08:00
|
|
|
if (skb_has_shared_frag(nskb) &&
|
|
|
|
__skb_linearize(nskb))
|
|
|
|
goto err;
|
|
|
|
|
2016-02-06 07:27:43 +08:00
|
|
|
if (!nskb->remcsum_offload)
|
|
|
|
nskb->ip_summed = CHECKSUM_NONE;
|
2016-02-06 07:27:37 +08:00
|
|
|
SKB_GSO_CB(nskb)->csum =
|
|
|
|
skb_checksum(nskb, doffset,
|
|
|
|
nskb->len - doffset, 0);
|
2014-06-05 08:20:02 +08:00
|
|
|
SKB_GSO_CB(nskb)->csum_start =
|
2016-02-06 07:27:37 +08:00
|
|
|
skb_headroom(nskb) + doffset;
|
2013-03-07 17:28:01 +08:00
|
|
|
}
|
2014-03-11 00:29:19 +08:00
|
|
|
} while ((offset += len) < head_skb->len);
|
2006-06-22 18:02:40 +08:00
|
|
|
|
2014-10-04 11:59:19 +08:00
|
|
|
/* Some callers want to get the end of the list.
|
|
|
|
* Put it in segs->prev to avoid walking the list.
|
|
|
|
* (see validate_xmit_skb_list() for example)
|
|
|
|
*/
|
|
|
|
segs->prev = tail;
|
2014-10-28 01:30:51 +08:00
|
|
|
|
2016-04-11 09:45:03 +08:00
|
|
|
if (partial_segs) {
|
2016-09-19 18:58:47 +08:00
|
|
|
struct sk_buff *iter;
|
2016-04-11 09:45:03 +08:00
|
|
|
int type = skb_shinfo(head_skb)->gso_type;
|
2016-09-19 18:58:47 +08:00
|
|
|
unsigned short gso_size = skb_shinfo(head_skb)->gso_size;
|
2016-04-11 09:45:03 +08:00
|
|
|
|
|
|
|
/* Update type to add partial and then remove dodgy if set */
|
2016-09-19 18:58:47 +08:00
|
|
|
type |= (features & NETIF_F_GSO_PARTIAL) / NETIF_F_GSO_PARTIAL * SKB_GSO_PARTIAL;
|
2016-04-11 09:45:03 +08:00
|
|
|
type &= ~SKB_GSO_DODGY;
|
|
|
|
|
|
|
|
/* Update GSO info and prepare to start updating headers on
|
|
|
|
* our way back down the stack of protocols.
|
|
|
|
*/
|
2016-09-19 18:58:47 +08:00
|
|
|
for (iter = segs; iter; iter = iter->next) {
|
|
|
|
skb_shinfo(iter)->gso_size = gso_size;
|
|
|
|
skb_shinfo(iter)->gso_segs = partial_segs;
|
|
|
|
skb_shinfo(iter)->gso_type = type;
|
|
|
|
SKB_GSO_CB(iter)->data_offset = skb_headroom(iter) + doffset;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (tail->len - doffset <= gso_size)
|
|
|
|
skb_shinfo(tail)->gso_size = 0;
|
|
|
|
else if (tail != segs)
|
|
|
|
skb_shinfo(tail)->gso_segs = DIV_ROUND_UP(tail->len - doffset, gso_size);
|
2016-04-11 09:45:03 +08:00
|
|
|
}
|
|
|
|
|
2014-10-28 01:30:51 +08:00
|
|
|
/* Following permits correct backpressure, for protocols
|
|
|
|
* using skb_set_owner_w().
|
|
|
|
* Idea is to tranfert ownership from head_skb to last segment.
|
|
|
|
*/
|
|
|
|
if (head_skb->destructor == sock_wfree) {
|
|
|
|
swap(tail->truesize, head_skb->truesize);
|
|
|
|
swap(tail->destructor, head_skb->destructor);
|
|
|
|
swap(tail->sk, head_skb->sk);
|
|
|
|
}
|
2006-06-22 18:02:40 +08:00
|
|
|
return segs;
|
|
|
|
|
|
|
|
err:
|
2013-12-21 06:29:08 +08:00
|
|
|
kfree_skb_list(segs);
|
2006-06-22 18:02:40 +08:00
|
|
|
return ERR_PTR(err);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(skb_segment);
|
|
|
|
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
#ifdef CONFIG_SKB_EXTENSIONS
|
|
|
|
#define SKB_EXT_ALIGN_VALUE 8
|
|
|
|
#define SKB_EXT_CHUNKSIZEOF(x) (ALIGN((sizeof(x)), SKB_EXT_ALIGN_VALUE) / SKB_EXT_ALIGN_VALUE)
|
|
|
|
|
|
|
|
static const u8 skb_ext_type_len[] = {
|
|
|
|
#if IS_ENABLED(CONFIG_BRIDGE_NETFILTER)
|
|
|
|
[SKB_EXT_BRIDGE_NF] = SKB_EXT_CHUNKSIZEOF(struct nf_bridge_info),
|
|
|
|
#endif
|
2018-12-19 00:15:27 +08:00
|
|
|
#ifdef CONFIG_XFRM
|
|
|
|
[SKB_EXT_SEC_PATH] = SKB_EXT_CHUNKSIZEOF(struct sec_path),
|
|
|
|
#endif
|
net: openvswitch: Set OvS recirc_id from tc chain index
Offloaded OvS datapath rules are translated one to one to tc rules,
for example the following simplified OvS rule:
recirc_id(0),in_port(dev1),eth_type(0x0800),ct_state(-trk) actions:ct(),recirc(2)
Will be translated to the following tc rule:
$ tc filter add dev dev1 ingress \
prio 1 chain 0 proto ip \
flower tcp ct_state -trk \
action ct pipe \
action goto chain 2
Received packets will first travel though tc, and if they aren't stolen
by it, like in the above rule, they will continue to OvS datapath.
Since we already did some actions (action ct in this case) which might
modify the packets, and updated action stats, we would like to continue
the proccessing with the correct recirc_id in OvS (here recirc_id(2))
where we left off.
To support this, introduce a new skb extension for tc, which
will be used for translating tc chain to ovs recirc_id to
handle these miss cases. Last tc chain index will be set
by tc goto chain action and read by OvS datapath.
Signed-off-by: Paul Blakey <paulb@mellanox.com>
Signed-off-by: Vlad Buslov <vladbu@mellanox.com>
Acked-by: Jiri Pirko <jiri@mellanox.com>
Acked-by: Pravin B Shelar <pshelar@ovn.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2019-09-04 21:56:37 +08:00
|
|
|
#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT)
|
|
|
|
[TC_SKB_EXT] = SKB_EXT_CHUNKSIZEOF(struct tc_skb_ext),
|
|
|
|
#endif
|
2020-01-09 23:59:19 +08:00
|
|
|
#if IS_ENABLED(CONFIG_MPTCP)
|
|
|
|
[SKB_EXT_MPTCP] = SKB_EXT_CHUNKSIZEOF(struct mptcp_ext),
|
|
|
|
#endif
|
2021-10-29 11:01:44 +08:00
|
|
|
#if IS_ENABLED(CONFIG_MCTP_FLOWS)
|
|
|
|
[SKB_EXT_MCTP] = SKB_EXT_CHUNKSIZEOF(struct mctp_flow),
|
|
|
|
#endif
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
};
|
|
|
|
|
|
|
|
static __always_inline unsigned int skb_ext_total_length(void)
|
|
|
|
{
|
|
|
|
return SKB_EXT_CHUNKSIZEOF(struct skb_ext) +
|
|
|
|
#if IS_ENABLED(CONFIG_BRIDGE_NETFILTER)
|
|
|
|
skb_ext_type_len[SKB_EXT_BRIDGE_NF] +
|
2018-12-19 00:15:27 +08:00
|
|
|
#endif
|
|
|
|
#ifdef CONFIG_XFRM
|
|
|
|
skb_ext_type_len[SKB_EXT_SEC_PATH] +
|
net: openvswitch: Set OvS recirc_id from tc chain index
Offloaded OvS datapath rules are translated one to one to tc rules,
for example the following simplified OvS rule:
recirc_id(0),in_port(dev1),eth_type(0x0800),ct_state(-trk) actions:ct(),recirc(2)
Will be translated to the following tc rule:
$ tc filter add dev dev1 ingress \
prio 1 chain 0 proto ip \
flower tcp ct_state -trk \
action ct pipe \
action goto chain 2
Received packets will first travel though tc, and if they aren't stolen
by it, like in the above rule, they will continue to OvS datapath.
Since we already did some actions (action ct in this case) which might
modify the packets, and updated action stats, we would like to continue
the proccessing with the correct recirc_id in OvS (here recirc_id(2))
where we left off.
To support this, introduce a new skb extension for tc, which
will be used for translating tc chain to ovs recirc_id to
handle these miss cases. Last tc chain index will be set
by tc goto chain action and read by OvS datapath.
Signed-off-by: Paul Blakey <paulb@mellanox.com>
Signed-off-by: Vlad Buslov <vladbu@mellanox.com>
Acked-by: Jiri Pirko <jiri@mellanox.com>
Acked-by: Pravin B Shelar <pshelar@ovn.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2019-09-04 21:56:37 +08:00
|
|
|
#endif
|
|
|
|
#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT)
|
|
|
|
skb_ext_type_len[TC_SKB_EXT] +
|
2020-01-09 23:59:19 +08:00
|
|
|
#endif
|
|
|
|
#if IS_ENABLED(CONFIG_MPTCP)
|
|
|
|
skb_ext_type_len[SKB_EXT_MPTCP] +
|
2021-10-29 11:01:44 +08:00
|
|
|
#endif
|
|
|
|
#if IS_ENABLED(CONFIG_MCTP_FLOWS)
|
|
|
|
skb_ext_type_len[SKB_EXT_MCTP] +
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
#endif
|
|
|
|
0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void skb_extensions_init(void)
|
|
|
|
{
|
|
|
|
BUILD_BUG_ON(SKB_EXT_NUM >= 8);
|
|
|
|
BUILD_BUG_ON(skb_ext_total_length() > 255);
|
|
|
|
|
|
|
|
skbuff_ext_cache = kmem_cache_create("skbuff_ext_cache",
|
|
|
|
SKB_EXT_ALIGN_VALUE * skb_ext_total_length(),
|
|
|
|
0,
|
|
|
|
SLAB_HWCACHE_ALIGN|SLAB_PANIC,
|
|
|
|
NULL);
|
|
|
|
}
|
|
|
|
#else
|
|
|
|
static void skb_extensions_init(void) {}
|
|
|
|
#endif
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
void __init skb_init(void)
|
|
|
|
{
|
2018-02-08 09:44:38 +08:00
|
|
|
skbuff_head_cache = kmem_cache_create_usercopy("skbuff_head_cache",
|
2005-04-17 06:20:36 +08:00
|
|
|
sizeof(struct sk_buff),
|
|
|
|
0,
|
2006-08-27 10:25:52 +08:00
|
|
|
SLAB_HWCACHE_ALIGN|SLAB_PANIC,
|
2018-02-08 09:44:38 +08:00
|
|
|
offsetof(struct sk_buff, cb),
|
|
|
|
sizeof_field(struct sk_buff, cb),
|
2007-07-20 09:11:58 +08:00
|
|
|
NULL);
|
2005-08-18 05:57:30 +08:00
|
|
|
skbuff_fclone_cache = kmem_cache_create("skbuff_fclone_cache",
|
2014-09-30 04:29:15 +08:00
|
|
|
sizeof(struct sk_buff_fclones),
|
2005-08-18 05:57:30 +08:00
|
|
|
0,
|
2006-08-27 10:25:52 +08:00
|
|
|
SLAB_HWCACHE_ALIGN|SLAB_PANIC,
|
2007-07-20 09:11:58 +08:00
|
|
|
NULL);
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
skb_extensions_init();
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
2007-10-31 12:29:29 +08:00
|
|
|
static int
|
skbuff: return -EMSGSIZE in skb_to_sgvec to prevent overflow
This is a defense-in-depth measure in response to bugs like
4d6fa57b4dab ("macsec: avoid heap overflow in skb_to_sgvec"). There's
not only a potential overflow of sglist items, but also a stack overflow
potential, so we fix this by limiting the amount of recursion this function
is allowed to do. Not actually providing a bounded base case is a future
disaster that we can easily avoid here.
As a small matter of house keeping, we take this opportunity to move the
documentation comment over the actual function the documentation is for.
While this could be implemented by using an explicit stack of skbuffs,
when implementing this, the function complexity increased considerably,
and I don't think such complexity and bloat is actually worth it. So,
instead I built this and tested it on x86, x86_64, ARM, ARM64, and MIPS,
and measured the stack usage there. I also reverted the recent MIPS
changes that give it a separate IRQ stack, so that I could experience
some worst-case situations. I found that limiting it to 24 layers deep
yielded a good stack usage with room for safety, as well as being much
deeper than any driver actually ever creates.
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: David Howells <dhowells@redhat.com>
Cc: Sabrina Dubroca <sd@queasysnail.net>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Jason Wang <jasowang@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-06-04 10:16:22 +08:00
|
|
|
__skb_to_sgvec(struct sk_buff *skb, struct scatterlist *sg, int offset, int len,
|
|
|
|
unsigned int recursion_level)
|
2007-04-03 11:19:53 +08:00
|
|
|
{
|
2007-04-28 06:21:23 +08:00
|
|
|
int start = skb_headlen(skb);
|
|
|
|
int i, copy = start - offset;
|
2009-06-09 15:18:59 +08:00
|
|
|
struct sk_buff *frag_iter;
|
2007-04-03 11:19:53 +08:00
|
|
|
int elt = 0;
|
|
|
|
|
skbuff: return -EMSGSIZE in skb_to_sgvec to prevent overflow
This is a defense-in-depth measure in response to bugs like
4d6fa57b4dab ("macsec: avoid heap overflow in skb_to_sgvec"). There's
not only a potential overflow of sglist items, but also a stack overflow
potential, so we fix this by limiting the amount of recursion this function
is allowed to do. Not actually providing a bounded base case is a future
disaster that we can easily avoid here.
As a small matter of house keeping, we take this opportunity to move the
documentation comment over the actual function the documentation is for.
While this could be implemented by using an explicit stack of skbuffs,
when implementing this, the function complexity increased considerably,
and I don't think such complexity and bloat is actually worth it. So,
instead I built this and tested it on x86, x86_64, ARM, ARM64, and MIPS,
and measured the stack usage there. I also reverted the recent MIPS
changes that give it a separate IRQ stack, so that I could experience
some worst-case situations. I found that limiting it to 24 layers deep
yielded a good stack usage with room for safety, as well as being much
deeper than any driver actually ever creates.
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: David Howells <dhowells@redhat.com>
Cc: Sabrina Dubroca <sd@queasysnail.net>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Jason Wang <jasowang@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-06-04 10:16:22 +08:00
|
|
|
if (unlikely(recursion_level >= 24))
|
|
|
|
return -EMSGSIZE;
|
|
|
|
|
2007-04-03 11:19:53 +08:00
|
|
|
if (copy > 0) {
|
|
|
|
if (copy > len)
|
|
|
|
copy = len;
|
2007-10-24 17:20:47 +08:00
|
|
|
sg_set_buf(sg, skb->data + offset, copy);
|
2007-04-03 11:19:53 +08:00
|
|
|
elt++;
|
|
|
|
if ((len -= copy) == 0)
|
|
|
|
return elt;
|
|
|
|
offset += copy;
|
|
|
|
}
|
|
|
|
|
|
|
|
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
|
2007-04-28 06:21:23 +08:00
|
|
|
int end;
|
2007-04-03 11:19:53 +08:00
|
|
|
|
2008-07-26 12:43:18 +08:00
|
|
|
WARN_ON(start > offset + len);
|
2007-04-28 06:21:23 +08:00
|
|
|
|
2011-10-19 05:00:24 +08:00
|
|
|
end = start + skb_frag_size(&skb_shinfo(skb)->frags[i]);
|
2007-04-03 11:19:53 +08:00
|
|
|
if ((copy = end - offset) > 0) {
|
|
|
|
skb_frag_t *frag = &skb_shinfo(skb)->frags[i];
|
skbuff: return -EMSGSIZE in skb_to_sgvec to prevent overflow
This is a defense-in-depth measure in response to bugs like
4d6fa57b4dab ("macsec: avoid heap overflow in skb_to_sgvec"). There's
not only a potential overflow of sglist items, but also a stack overflow
potential, so we fix this by limiting the amount of recursion this function
is allowed to do. Not actually providing a bounded base case is a future
disaster that we can easily avoid here.
As a small matter of house keeping, we take this opportunity to move the
documentation comment over the actual function the documentation is for.
While this could be implemented by using an explicit stack of skbuffs,
when implementing this, the function complexity increased considerably,
and I don't think such complexity and bloat is actually worth it. So,
instead I built this and tested it on x86, x86_64, ARM, ARM64, and MIPS,
and measured the stack usage there. I also reverted the recent MIPS
changes that give it a separate IRQ stack, so that I could experience
some worst-case situations. I found that limiting it to 24 layers deep
yielded a good stack usage with room for safety, as well as being much
deeper than any driver actually ever creates.
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: David Howells <dhowells@redhat.com>
Cc: Sabrina Dubroca <sd@queasysnail.net>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Jason Wang <jasowang@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-06-04 10:16:22 +08:00
|
|
|
if (unlikely(elt && sg_is_last(&sg[elt - 1])))
|
|
|
|
return -EMSGSIZE;
|
2007-04-03 11:19:53 +08:00
|
|
|
|
|
|
|
if (copy > len)
|
|
|
|
copy = len;
|
2011-08-23 07:44:58 +08:00
|
|
|
sg_set_page(&sg[elt], skb_frag_page(frag), copy,
|
2019-07-30 22:40:33 +08:00
|
|
|
skb_frag_off(frag) + offset - start);
|
2007-04-03 11:19:53 +08:00
|
|
|
elt++;
|
|
|
|
if (!(len -= copy))
|
|
|
|
return elt;
|
|
|
|
offset += copy;
|
|
|
|
}
|
2007-04-28 06:21:23 +08:00
|
|
|
start = end;
|
2007-04-03 11:19:53 +08:00
|
|
|
}
|
|
|
|
|
2009-06-09 15:18:59 +08:00
|
|
|
skb_walk_frags(skb, frag_iter) {
|
skbuff: return -EMSGSIZE in skb_to_sgvec to prevent overflow
This is a defense-in-depth measure in response to bugs like
4d6fa57b4dab ("macsec: avoid heap overflow in skb_to_sgvec"). There's
not only a potential overflow of sglist items, but also a stack overflow
potential, so we fix this by limiting the amount of recursion this function
is allowed to do. Not actually providing a bounded base case is a future
disaster that we can easily avoid here.
As a small matter of house keeping, we take this opportunity to move the
documentation comment over the actual function the documentation is for.
While this could be implemented by using an explicit stack of skbuffs,
when implementing this, the function complexity increased considerably,
and I don't think such complexity and bloat is actually worth it. So,
instead I built this and tested it on x86, x86_64, ARM, ARM64, and MIPS,
and measured the stack usage there. I also reverted the recent MIPS
changes that give it a separate IRQ stack, so that I could experience
some worst-case situations. I found that limiting it to 24 layers deep
yielded a good stack usage with room for safety, as well as being much
deeper than any driver actually ever creates.
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: David Howells <dhowells@redhat.com>
Cc: Sabrina Dubroca <sd@queasysnail.net>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Jason Wang <jasowang@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-06-04 10:16:22 +08:00
|
|
|
int end, ret;
|
2007-04-28 06:21:23 +08:00
|
|
|
|
2009-06-09 15:18:59 +08:00
|
|
|
WARN_ON(start > offset + len);
|
2007-04-03 11:19:53 +08:00
|
|
|
|
2009-06-09 15:18:59 +08:00
|
|
|
end = start + frag_iter->len;
|
|
|
|
if ((copy = end - offset) > 0) {
|
skbuff: return -EMSGSIZE in skb_to_sgvec to prevent overflow
This is a defense-in-depth measure in response to bugs like
4d6fa57b4dab ("macsec: avoid heap overflow in skb_to_sgvec"). There's
not only a potential overflow of sglist items, but also a stack overflow
potential, so we fix this by limiting the amount of recursion this function
is allowed to do. Not actually providing a bounded base case is a future
disaster that we can easily avoid here.
As a small matter of house keeping, we take this opportunity to move the
documentation comment over the actual function the documentation is for.
While this could be implemented by using an explicit stack of skbuffs,
when implementing this, the function complexity increased considerably,
and I don't think such complexity and bloat is actually worth it. So,
instead I built this and tested it on x86, x86_64, ARM, ARM64, and MIPS,
and measured the stack usage there. I also reverted the recent MIPS
changes that give it a separate IRQ stack, so that I could experience
some worst-case situations. I found that limiting it to 24 layers deep
yielded a good stack usage with room for safety, as well as being much
deeper than any driver actually ever creates.
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: David Howells <dhowells@redhat.com>
Cc: Sabrina Dubroca <sd@queasysnail.net>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Jason Wang <jasowang@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-06-04 10:16:22 +08:00
|
|
|
if (unlikely(elt && sg_is_last(&sg[elt - 1])))
|
|
|
|
return -EMSGSIZE;
|
|
|
|
|
2009-06-09 15:18:59 +08:00
|
|
|
if (copy > len)
|
|
|
|
copy = len;
|
skbuff: return -EMSGSIZE in skb_to_sgvec to prevent overflow
This is a defense-in-depth measure in response to bugs like
4d6fa57b4dab ("macsec: avoid heap overflow in skb_to_sgvec"). There's
not only a potential overflow of sglist items, but also a stack overflow
potential, so we fix this by limiting the amount of recursion this function
is allowed to do. Not actually providing a bounded base case is a future
disaster that we can easily avoid here.
As a small matter of house keeping, we take this opportunity to move the
documentation comment over the actual function the documentation is for.
While this could be implemented by using an explicit stack of skbuffs,
when implementing this, the function complexity increased considerably,
and I don't think such complexity and bloat is actually worth it. So,
instead I built this and tested it on x86, x86_64, ARM, ARM64, and MIPS,
and measured the stack usage there. I also reverted the recent MIPS
changes that give it a separate IRQ stack, so that I could experience
some worst-case situations. I found that limiting it to 24 layers deep
yielded a good stack usage with room for safety, as well as being much
deeper than any driver actually ever creates.
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: David Howells <dhowells@redhat.com>
Cc: Sabrina Dubroca <sd@queasysnail.net>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Jason Wang <jasowang@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-06-04 10:16:22 +08:00
|
|
|
ret = __skb_to_sgvec(frag_iter, sg+elt, offset - start,
|
|
|
|
copy, recursion_level + 1);
|
|
|
|
if (unlikely(ret < 0))
|
|
|
|
return ret;
|
|
|
|
elt += ret;
|
2009-06-09 15:18:59 +08:00
|
|
|
if ((len -= copy) == 0)
|
|
|
|
return elt;
|
|
|
|
offset += copy;
|
2007-04-03 11:19:53 +08:00
|
|
|
}
|
2009-06-09 15:18:59 +08:00
|
|
|
start = end;
|
2007-04-03 11:19:53 +08:00
|
|
|
}
|
|
|
|
BUG_ON(len);
|
|
|
|
return elt;
|
|
|
|
}
|
|
|
|
|
skbuff: return -EMSGSIZE in skb_to_sgvec to prevent overflow
This is a defense-in-depth measure in response to bugs like
4d6fa57b4dab ("macsec: avoid heap overflow in skb_to_sgvec"). There's
not only a potential overflow of sglist items, but also a stack overflow
potential, so we fix this by limiting the amount of recursion this function
is allowed to do. Not actually providing a bounded base case is a future
disaster that we can easily avoid here.
As a small matter of house keeping, we take this opportunity to move the
documentation comment over the actual function the documentation is for.
While this could be implemented by using an explicit stack of skbuffs,
when implementing this, the function complexity increased considerably,
and I don't think such complexity and bloat is actually worth it. So,
instead I built this and tested it on x86, x86_64, ARM, ARM64, and MIPS,
and measured the stack usage there. I also reverted the recent MIPS
changes that give it a separate IRQ stack, so that I could experience
some worst-case situations. I found that limiting it to 24 layers deep
yielded a good stack usage with room for safety, as well as being much
deeper than any driver actually ever creates.
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: David Howells <dhowells@redhat.com>
Cc: Sabrina Dubroca <sd@queasysnail.net>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Jason Wang <jasowang@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-06-04 10:16:22 +08:00
|
|
|
/**
|
|
|
|
* skb_to_sgvec - Fill a scatter-gather list from a socket buffer
|
|
|
|
* @skb: Socket buffer containing the buffers to be mapped
|
|
|
|
* @sg: The scatter-gather list to map into
|
|
|
|
* @offset: The offset into the buffer's contents to start mapping
|
|
|
|
* @len: Length of buffer space to be mapped
|
|
|
|
*
|
|
|
|
* Fill the specified scatter-gather list with mappings/pointers into a
|
|
|
|
* region of the buffer space attached to a socket buffer. Returns either
|
|
|
|
* the number of scatterlist items used, or -EMSGSIZE if the contents
|
|
|
|
* could not fit.
|
|
|
|
*/
|
|
|
|
int skb_to_sgvec(struct sk_buff *skb, struct scatterlist *sg, int offset, int len)
|
|
|
|
{
|
|
|
|
int nsg = __skb_to_sgvec(skb, sg, offset, len, 0);
|
|
|
|
|
|
|
|
if (nsg <= 0)
|
|
|
|
return nsg;
|
|
|
|
|
|
|
|
sg_mark_end(&sg[nsg - 1]);
|
|
|
|
|
|
|
|
return nsg;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(skb_to_sgvec);
|
|
|
|
|
2014-01-18 09:54:23 +08:00
|
|
|
/* As compared with skb_to_sgvec, skb_to_sgvec_nomark only map skb to given
|
|
|
|
* sglist without mark the sg which contain last skb data as the end.
|
|
|
|
* So the caller can mannipulate sg list as will when padding new data after
|
|
|
|
* the first call without calling sg_unmark_end to expend sg list.
|
|
|
|
*
|
|
|
|
* Scenario to use skb_to_sgvec_nomark:
|
|
|
|
* 1. sg_init_table
|
|
|
|
* 2. skb_to_sgvec_nomark(payload1)
|
|
|
|
* 3. skb_to_sgvec_nomark(payload2)
|
|
|
|
*
|
|
|
|
* This is equivalent to:
|
|
|
|
* 1. sg_init_table
|
|
|
|
* 2. skb_to_sgvec(payload1)
|
|
|
|
* 3. sg_unmark_end
|
|
|
|
* 4. skb_to_sgvec(payload2)
|
|
|
|
*
|
|
|
|
* When mapping mutilple payload conditionally, skb_to_sgvec_nomark
|
|
|
|
* is more preferable.
|
|
|
|
*/
|
|
|
|
int skb_to_sgvec_nomark(struct sk_buff *skb, struct scatterlist *sg,
|
|
|
|
int offset, int len)
|
|
|
|
{
|
skbuff: return -EMSGSIZE in skb_to_sgvec to prevent overflow
This is a defense-in-depth measure in response to bugs like
4d6fa57b4dab ("macsec: avoid heap overflow in skb_to_sgvec"). There's
not only a potential overflow of sglist items, but also a stack overflow
potential, so we fix this by limiting the amount of recursion this function
is allowed to do. Not actually providing a bounded base case is a future
disaster that we can easily avoid here.
As a small matter of house keeping, we take this opportunity to move the
documentation comment over the actual function the documentation is for.
While this could be implemented by using an explicit stack of skbuffs,
when implementing this, the function complexity increased considerably,
and I don't think such complexity and bloat is actually worth it. So,
instead I built this and tested it on x86, x86_64, ARM, ARM64, and MIPS,
and measured the stack usage there. I also reverted the recent MIPS
changes that give it a separate IRQ stack, so that I could experience
some worst-case situations. I found that limiting it to 24 layers deep
yielded a good stack usage with room for safety, as well as being much
deeper than any driver actually ever creates.
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
Cc: Steffen Klassert <steffen.klassert@secunet.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: David Howells <dhowells@redhat.com>
Cc: Sabrina Dubroca <sd@queasysnail.net>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Jason Wang <jasowang@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-06-04 10:16:22 +08:00
|
|
|
return __skb_to_sgvec(skb, sg, offset, len, 0);
|
2014-01-18 09:54:23 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(skb_to_sgvec_nomark);
|
|
|
|
|
2007-10-31 12:29:29 +08:00
|
|
|
|
|
|
|
|
2007-04-03 11:19:53 +08:00
|
|
|
/**
|
|
|
|
* skb_cow_data - Check that a socket buffer's data buffers are writable
|
|
|
|
* @skb: The socket buffer to check.
|
|
|
|
* @tailbits: Amount of trailing space to be added
|
|
|
|
* @trailer: Returned pointer to the skb where the @tailbits space begins
|
|
|
|
*
|
|
|
|
* Make sure that the data buffers attached to a socket buffer are
|
|
|
|
* writable. If they are not, private copies are made of the data buffers
|
|
|
|
* and the socket buffer is set to use these instead.
|
|
|
|
*
|
|
|
|
* If @tailbits is given, make sure that there is space to write @tailbits
|
|
|
|
* bytes of data beyond current end of socket buffer. @trailer will be
|
|
|
|
* set to point to the skb in which this space begins.
|
|
|
|
*
|
|
|
|
* The number of scatterlist elements required to completely map the
|
|
|
|
* COW'd and extended socket buffer will be returned.
|
|
|
|
*/
|
|
|
|
int skb_cow_data(struct sk_buff *skb, int tailbits, struct sk_buff **trailer)
|
|
|
|
{
|
|
|
|
int copyflag;
|
|
|
|
int elt;
|
|
|
|
struct sk_buff *skb1, **skb_p;
|
|
|
|
|
|
|
|
/* If skb is cloned or its head is paged, reallocate
|
|
|
|
* head pulling out all the pages (pages are considered not writable
|
|
|
|
* at the moment even if they are anonymous).
|
|
|
|
*/
|
|
|
|
if ((skb_cloned(skb) || skb_shinfo(skb)->nr_frags) &&
|
2020-08-01 17:30:23 +08:00
|
|
|
!__pskb_pull_tail(skb, __skb_pagelen(skb)))
|
2007-04-03 11:19:53 +08:00
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
/* Easy case. Most of packets will go this way. */
|
2010-08-23 15:13:46 +08:00
|
|
|
if (!skb_has_frag_list(skb)) {
|
2007-04-03 11:19:53 +08:00
|
|
|
/* A little of trouble, not enough of space for trailer.
|
|
|
|
* This should not happen, when stack is tuned to generate
|
|
|
|
* good frames. OK, on miss we reallocate and reserve even more
|
|
|
|
* space, 128 bytes is fair. */
|
|
|
|
|
|
|
|
if (skb_tailroom(skb) < tailbits &&
|
|
|
|
pskb_expand_head(skb, 0, tailbits-skb_tailroom(skb)+128, GFP_ATOMIC))
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
/* Voila! */
|
|
|
|
*trailer = skb;
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Misery. We are in troubles, going to mincer fragments... */
|
|
|
|
|
|
|
|
elt = 1;
|
|
|
|
skb_p = &skb_shinfo(skb)->frag_list;
|
|
|
|
copyflag = 0;
|
|
|
|
|
|
|
|
while ((skb1 = *skb_p) != NULL) {
|
|
|
|
int ntail = 0;
|
|
|
|
|
|
|
|
/* The fragment is partially pulled by someone,
|
|
|
|
* this can happen on input. Copy it and everything
|
|
|
|
* after it. */
|
|
|
|
|
|
|
|
if (skb_shared(skb1))
|
|
|
|
copyflag = 1;
|
|
|
|
|
|
|
|
/* If the skb is the last, worry about trailer. */
|
|
|
|
|
|
|
|
if (skb1->next == NULL && tailbits) {
|
|
|
|
if (skb_shinfo(skb1)->nr_frags ||
|
2010-08-23 15:13:46 +08:00
|
|
|
skb_has_frag_list(skb1) ||
|
2007-04-03 11:19:53 +08:00
|
|
|
skb_tailroom(skb1) < tailbits)
|
|
|
|
ntail = tailbits + 128;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (copyflag ||
|
|
|
|
skb_cloned(skb1) ||
|
|
|
|
ntail ||
|
|
|
|
skb_shinfo(skb1)->nr_frags ||
|
2010-08-23 15:13:46 +08:00
|
|
|
skb_has_frag_list(skb1)) {
|
2007-04-03 11:19:53 +08:00
|
|
|
struct sk_buff *skb2;
|
|
|
|
|
|
|
|
/* Fuck, we are miserable poor guys... */
|
|
|
|
if (ntail == 0)
|
|
|
|
skb2 = skb_copy(skb1, GFP_ATOMIC);
|
|
|
|
else
|
|
|
|
skb2 = skb_copy_expand(skb1,
|
|
|
|
skb_headroom(skb1),
|
|
|
|
ntail,
|
|
|
|
GFP_ATOMIC);
|
|
|
|
if (unlikely(skb2 == NULL))
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
if (skb1->sk)
|
|
|
|
skb_set_owner_w(skb2, skb1->sk);
|
|
|
|
|
|
|
|
/* Looking around. Are we still alive?
|
|
|
|
* OK, link new skb, drop old one */
|
|
|
|
|
|
|
|
skb2->next = skb1->next;
|
|
|
|
*skb_p = skb2;
|
|
|
|
kfree_skb(skb1);
|
|
|
|
skb1 = skb2;
|
|
|
|
}
|
|
|
|
elt++;
|
|
|
|
*trailer = skb1;
|
|
|
|
skb_p = &skb1->next;
|
|
|
|
}
|
|
|
|
|
|
|
|
return elt;
|
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL_GPL(skb_cow_data);
|
2007-04-03 11:19:53 +08:00
|
|
|
|
2010-06-01 14:44:05 +08:00
|
|
|
static void sock_rmem_free(struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
struct sock *sk = skb->sk;
|
|
|
|
|
|
|
|
atomic_sub(skb->truesize, &sk->sk_rmem_alloc);
|
|
|
|
}
|
|
|
|
|
2017-03-19 05:02:59 +08:00
|
|
|
static void skb_set_err_queue(struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
/* pkt_type of skbs received on local sockets is never PACKET_OUTGOING.
|
|
|
|
* So, it is safe to (mis)use it to mark skbs on the error queue.
|
|
|
|
*/
|
|
|
|
skb->pkt_type = PACKET_OUTGOING;
|
|
|
|
BUILD_BUG_ON(PACKET_OUTGOING == 0);
|
|
|
|
}
|
|
|
|
|
2010-06-01 14:44:05 +08:00
|
|
|
/*
|
|
|
|
* Note: We dont mem charge error packets (no sk_forward_alloc changes)
|
|
|
|
*/
|
|
|
|
int sock_queue_err_skb(struct sock *sk, struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
if (atomic_read(&sk->sk_rmem_alloc) + skb->truesize >=
|
2019-10-11 11:17:44 +08:00
|
|
|
(unsigned int)READ_ONCE(sk->sk_rcvbuf))
|
2010-06-01 14:44:05 +08:00
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
skb_orphan(skb);
|
|
|
|
skb->sk = sk;
|
|
|
|
skb->destructor = sock_rmem_free;
|
|
|
|
atomic_add(skb->truesize, &sk->sk_rmem_alloc);
|
2017-03-19 05:02:59 +08:00
|
|
|
skb_set_err_queue(skb);
|
2010-06-01 14:44:05 +08:00
|
|
|
|
2011-05-18 14:21:31 +08:00
|
|
|
/* before exiting rcu section, make sure dst is refcounted */
|
|
|
|
skb_dst_force(skb);
|
|
|
|
|
2010-06-01 14:44:05 +08:00
|
|
|
skb_queue_tail(&sk->sk_error_queue, skb);
|
|
|
|
if (!sock_flag(sk, SOCK_DEAD))
|
2021-06-28 06:48:21 +08:00
|
|
|
sk_error_report(sk);
|
2010-06-01 14:44:05 +08:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(sock_queue_err_skb);
|
|
|
|
|
2016-12-01 03:01:08 +08:00
|
|
|
static bool is_icmp_err_skb(const struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
return skb && (SKB_EXT_ERR(skb)->ee.ee_origin == SO_EE_ORIGIN_ICMP ||
|
|
|
|
SKB_EXT_ERR(skb)->ee.ee_origin == SO_EE_ORIGIN_ICMP6);
|
|
|
|
}
|
|
|
|
|
2014-09-01 09:30:27 +08:00
|
|
|
struct sk_buff *sock_dequeue_err_skb(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct sk_buff_head *q = &sk->sk_error_queue;
|
2016-12-01 03:01:08 +08:00
|
|
|
struct sk_buff *skb, *skb_next = NULL;
|
|
|
|
bool icmp_next = false;
|
2015-02-18 21:47:55 +08:00
|
|
|
unsigned long flags;
|
2014-09-01 09:30:27 +08:00
|
|
|
|
2015-02-18 21:47:55 +08:00
|
|
|
spin_lock_irqsave(&q->lock, flags);
|
2014-09-01 09:30:27 +08:00
|
|
|
skb = __skb_dequeue(q);
|
2017-06-03 00:38:22 +08:00
|
|
|
if (skb && (skb_next = skb_peek(q))) {
|
2016-12-01 03:01:08 +08:00
|
|
|
icmp_next = is_icmp_err_skb(skb_next);
|
2017-06-03 00:38:22 +08:00
|
|
|
if (icmp_next)
|
2020-11-26 23:12:20 +08:00
|
|
|
sk->sk_err = SKB_EXT_ERR(skb_next)->ee.ee_errno;
|
2017-06-03 00:38:22 +08:00
|
|
|
}
|
2015-02-18 21:47:55 +08:00
|
|
|
spin_unlock_irqrestore(&q->lock, flags);
|
2014-09-01 09:30:27 +08:00
|
|
|
|
2016-12-01 03:01:08 +08:00
|
|
|
if (is_icmp_err_skb(skb) && !icmp_next)
|
|
|
|
sk->sk_err = 0;
|
|
|
|
|
|
|
|
if (skb_next)
|
2021-06-28 06:48:21 +08:00
|
|
|
sk_error_report(sk);
|
2014-09-01 09:30:27 +08:00
|
|
|
|
|
|
|
return skb;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(sock_dequeue_err_skb);
|
|
|
|
|
2014-09-11 06:05:26 +08:00
|
|
|
/**
|
|
|
|
* skb_clone_sk - create clone of skb, and take reference to socket
|
|
|
|
* @skb: the skb to clone
|
|
|
|
*
|
|
|
|
* This function creates a clone of a buffer that holds a reference on
|
|
|
|
* sk_refcnt. Buffers created via this function are meant to be
|
|
|
|
* returned using sock_queue_err_skb, or free via kfree_skb.
|
|
|
|
*
|
|
|
|
* When passing buffers allocated with this function to sock_queue_err_skb
|
|
|
|
* it is necessary to wrap the call with sock_hold/sock_put in order to
|
|
|
|
* prevent the socket from being released prior to being enqueued on
|
|
|
|
* the sk_error_queue.
|
|
|
|
*/
|
2014-09-05 01:31:35 +08:00
|
|
|
struct sk_buff *skb_clone_sk(struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
struct sock *sk = skb->sk;
|
|
|
|
struct sk_buff *clone;
|
|
|
|
|
2017-06-30 18:08:01 +08:00
|
|
|
if (!sk || !refcount_inc_not_zero(&sk->sk_refcnt))
|
2014-09-05 01:31:35 +08:00
|
|
|
return NULL;
|
|
|
|
|
|
|
|
clone = skb_clone(skb, GFP_ATOMIC);
|
|
|
|
if (!clone) {
|
|
|
|
sock_put(sk);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
clone->sk = sk;
|
|
|
|
clone->destructor = sock_efree;
|
|
|
|
|
|
|
|
return clone;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_clone_sk);
|
|
|
|
|
2014-09-05 01:31:10 +08:00
|
|
|
static void __skb_complete_tx_timestamp(struct sk_buff *skb,
|
|
|
|
struct sock *sk,
|
2017-03-19 05:03:00 +08:00
|
|
|
int tstype,
|
|
|
|
bool opt_stats)
|
2009-02-12 13:03:37 +08:00
|
|
|
{
|
|
|
|
struct sock_exterr_skb *serr;
|
|
|
|
int err;
|
|
|
|
|
2017-03-19 05:03:00 +08:00
|
|
|
BUILD_BUG_ON(sizeof(struct sock_exterr_skb) > sizeof(skb->cb));
|
|
|
|
|
2009-02-12 13:03:37 +08:00
|
|
|
serr = SKB_EXT_ERR(skb);
|
|
|
|
memset(serr, 0, sizeof(*serr));
|
|
|
|
serr->ee.ee_errno = ENOMSG;
|
|
|
|
serr->ee.ee_origin = SO_EE_ORIGIN_TIMESTAMPING;
|
2014-08-05 10:11:48 +08:00
|
|
|
serr->ee.ee_info = tstype;
|
2017-03-19 05:03:00 +08:00
|
|
|
serr->opt_stats = opt_stats;
|
2017-04-13 07:24:35 +08:00
|
|
|
serr->header.h4.iif = skb->dev ? skb->dev->ifindex : 0;
|
net-timestamp: TCP timestamping
TCP timestamping extends SO_TIMESTAMPING to bytestreams.
Bytestreams do not have a 1:1 relationship between send() buffers and
network packets. The feature interprets a send call on a bytestream as
a request for a timestamp for the last byte in that send() buffer.
The choice corresponds to a request for a timestamp when all bytes in
the buffer have been sent. That assumption depends on in-order kernel
transmission. This is the common case. That said, it is possible to
construct a traffic shaping tree that would result in reordering.
The guarantee is strong, then, but not ironclad.
This implementation supports send and sendpages (splice). GSO replaces
one large packet with multiple smaller packets. This patch also copies
the option into the correct smaller packet.
This patch does not yet support timestamping on data in an initial TCP
Fast Open SYN, because that takes a very different data path.
If ID generation in ee_data is enabled, bytestream timestamps return a
byte offset, instead of the packet counter for datagrams.
The implementation supports a single timestamp per packet. It silenty
replaces requests for previous timestamps. To avoid missing tstamps,
flush the tcp queue by disabling Nagle, cork and autocork. Missing
tstamps can be detected by offset when the ee_data ID is enabled.
Implementation details:
- On GSO, the timestamping code can be included in the main loop. I
moved it into its own loop to reduce the impact on the common case
to a single branch.
- To avoid leaking the absolute seqno to userspace, the offset
returned in ee_data must always be relative. It is an offset between
an skb and sk field. The first is always set (also for GSO & ACK).
The second must also never be uninitialized. Only allow the ID
option on sockets in the ESTABLISHED state, for which the seqno
is available. Never reset it to zero (instead, move it to the
current seqno when reenabling the option).
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-05 10:11:49 +08:00
|
|
|
if (sk->sk_tsflags & SOF_TIMESTAMPING_OPT_ID) {
|
2014-08-05 10:11:47 +08:00
|
|
|
serr->ee.ee_data = skb_shinfo(skb)->tskey;
|
2021-11-16 03:02:33 +08:00
|
|
|
if (sk_is_tcp(sk))
|
2022-02-18 01:05:02 +08:00
|
|
|
serr->ee.ee_data -= atomic_read(&sk->sk_tskey);
|
net-timestamp: TCP timestamping
TCP timestamping extends SO_TIMESTAMPING to bytestreams.
Bytestreams do not have a 1:1 relationship between send() buffers and
network packets. The feature interprets a send call on a bytestream as
a request for a timestamp for the last byte in that send() buffer.
The choice corresponds to a request for a timestamp when all bytes in
the buffer have been sent. That assumption depends on in-order kernel
transmission. This is the common case. That said, it is possible to
construct a traffic shaping tree that would result in reordering.
The guarantee is strong, then, but not ironclad.
This implementation supports send and sendpages (splice). GSO replaces
one large packet with multiple smaller packets. This patch also copies
the option into the correct smaller packet.
This patch does not yet support timestamping on data in an initial TCP
Fast Open SYN, because that takes a very different data path.
If ID generation in ee_data is enabled, bytestream timestamps return a
byte offset, instead of the packet counter for datagrams.
The implementation supports a single timestamp per packet. It silenty
replaces requests for previous timestamps. To avoid missing tstamps,
flush the tcp queue by disabling Nagle, cork and autocork. Missing
tstamps can be detected by offset when the ee_data ID is enabled.
Implementation details:
- On GSO, the timestamping code can be included in the main loop. I
moved it into its own loop to reduce the impact on the common case
to a single branch.
- To avoid leaking the absolute seqno to userspace, the offset
returned in ee_data must always be relative. It is an offset between
an skb and sk field. The first is always set (also for GSO & ACK).
The second must also never be uninitialized. Only allow the ID
option on sockets in the ESTABLISHED state, for which the seqno
is available. Never reset it to zero (instead, move it to the
current seqno when reenabling the option).
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-05 10:11:49 +08:00
|
|
|
}
|
2010-05-29 15:20:48 +08:00
|
|
|
|
2009-02-12 13:03:37 +08:00
|
|
|
err = sock_queue_err_skb(sk, skb);
|
2010-05-29 15:20:48 +08:00
|
|
|
|
2009-02-12 13:03:37 +08:00
|
|
|
if (err)
|
|
|
|
kfree_skb(skb);
|
|
|
|
}
|
2014-09-05 01:31:10 +08:00
|
|
|
|
2015-01-31 02:29:32 +08:00
|
|
|
static bool skb_may_tx_timestamp(struct sock *sk, bool tsonly)
|
|
|
|
{
|
|
|
|
bool ret;
|
|
|
|
|
|
|
|
if (likely(sysctl_tstamp_allow_data || tsonly))
|
|
|
|
return true;
|
|
|
|
|
|
|
|
read_lock_bh(&sk->sk_callback_lock);
|
|
|
|
ret = sk->sk_socket && sk->sk_socket->file &&
|
|
|
|
file_ns_capable(sk->sk_socket->file, &init_user_ns, CAP_NET_RAW);
|
|
|
|
read_unlock_bh(&sk->sk_callback_lock);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2014-09-05 01:31:10 +08:00
|
|
|
void skb_complete_tx_timestamp(struct sk_buff *skb,
|
|
|
|
struct skb_shared_hwtstamps *hwtstamps)
|
|
|
|
{
|
|
|
|
struct sock *sk = skb->sk;
|
|
|
|
|
2015-01-31 02:29:32 +08:00
|
|
|
if (!skb_may_tx_timestamp(sk, false))
|
2017-12-14 03:41:06 +08:00
|
|
|
goto err;
|
2015-01-31 02:29:32 +08:00
|
|
|
|
2017-03-04 13:01:03 +08:00
|
|
|
/* Take a reference to prevent skb_orphan() from freeing the socket,
|
|
|
|
* but only if the socket refcount is not zero.
|
|
|
|
*/
|
2017-06-30 18:08:01 +08:00
|
|
|
if (likely(refcount_inc_not_zero(&sk->sk_refcnt))) {
|
2017-03-04 13:01:03 +08:00
|
|
|
*skb_hwtstamps(skb) = *hwtstamps;
|
2017-03-19 05:03:00 +08:00
|
|
|
__skb_complete_tx_timestamp(skb, sk, SCM_TSTAMP_SND, false);
|
2017-03-04 13:01:03 +08:00
|
|
|
sock_put(sk);
|
2017-12-14 03:41:06 +08:00
|
|
|
return;
|
2017-03-04 13:01:03 +08:00
|
|
|
}
|
2017-12-14 03:41:06 +08:00
|
|
|
|
|
|
|
err:
|
|
|
|
kfree_skb(skb);
|
2014-09-05 01:31:10 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(skb_complete_tx_timestamp);
|
|
|
|
|
|
|
|
void __skb_tstamp_tx(struct sk_buff *orig_skb,
|
2021-01-21 04:41:55 +08:00
|
|
|
const struct sk_buff *ack_skb,
|
2014-09-05 01:31:10 +08:00
|
|
|
struct skb_shared_hwtstamps *hwtstamps,
|
|
|
|
struct sock *sk, int tstype)
|
|
|
|
{
|
|
|
|
struct sk_buff *skb;
|
2017-03-19 05:03:00 +08:00
|
|
|
bool tsonly, opt_stats = false;
|
2014-09-05 01:31:10 +08:00
|
|
|
|
2015-03-12 03:43:55 +08:00
|
|
|
if (!sk)
|
|
|
|
return;
|
|
|
|
|
2017-05-19 23:52:40 +08:00
|
|
|
if (!hwtstamps && !(sk->sk_tsflags & SOF_TIMESTAMPING_OPT_TX_SWHW) &&
|
|
|
|
skb_shinfo(orig_skb)->tx_flags & SKBTX_IN_PROGRESS)
|
|
|
|
return;
|
|
|
|
|
2015-03-12 03:43:55 +08:00
|
|
|
tsonly = sk->sk_tsflags & SOF_TIMESTAMPING_OPT_TSONLY;
|
|
|
|
if (!skb_may_tx_timestamp(sk, tsonly))
|
2014-09-05 01:31:10 +08:00
|
|
|
return;
|
|
|
|
|
2016-11-28 15:07:18 +08:00
|
|
|
if (tsonly) {
|
|
|
|
#ifdef CONFIG_INET
|
|
|
|
if ((sk->sk_tsflags & SOF_TIMESTAMPING_OPT_STATS) &&
|
2021-11-16 03:02:33 +08:00
|
|
|
sk_is_tcp(sk)) {
|
2021-01-21 04:41:55 +08:00
|
|
|
skb = tcp_get_timestamping_opt_stats(sk, orig_skb,
|
|
|
|
ack_skb);
|
2017-03-19 05:03:00 +08:00
|
|
|
opt_stats = true;
|
|
|
|
} else
|
2016-11-28 15:07:18 +08:00
|
|
|
#endif
|
|
|
|
skb = alloc_skb(0, GFP_ATOMIC);
|
|
|
|
} else {
|
2015-01-31 02:29:31 +08:00
|
|
|
skb = skb_clone(orig_skb, GFP_ATOMIC);
|
2016-11-28 15:07:18 +08:00
|
|
|
}
|
2014-09-05 01:31:10 +08:00
|
|
|
if (!skb)
|
|
|
|
return;
|
|
|
|
|
2015-01-31 02:29:31 +08:00
|
|
|
if (tsonly) {
|
2017-06-08 23:35:03 +08:00
|
|
|
skb_shinfo(skb)->tx_flags |= skb_shinfo(orig_skb)->tx_flags &
|
|
|
|
SKBTX_ANY_TSTAMP;
|
2015-01-31 02:29:31 +08:00
|
|
|
skb_shinfo(skb)->tskey = skb_shinfo(orig_skb)->tskey;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (hwtstamps)
|
|
|
|
*skb_hwtstamps(skb) = *hwtstamps;
|
|
|
|
else
|
2022-03-03 03:55:44 +08:00
|
|
|
__net_timestamp(skb);
|
2015-01-31 02:29:31 +08:00
|
|
|
|
2017-03-19 05:03:00 +08:00
|
|
|
__skb_complete_tx_timestamp(skb, sk, tstype, opt_stats);
|
2014-09-05 01:31:10 +08:00
|
|
|
}
|
2014-08-05 10:11:48 +08:00
|
|
|
EXPORT_SYMBOL_GPL(__skb_tstamp_tx);
|
|
|
|
|
|
|
|
void skb_tstamp_tx(struct sk_buff *orig_skb,
|
|
|
|
struct skb_shared_hwtstamps *hwtstamps)
|
|
|
|
{
|
2021-01-21 04:41:55 +08:00
|
|
|
return __skb_tstamp_tx(orig_skb, NULL, hwtstamps, orig_skb->sk,
|
2014-08-05 10:11:48 +08:00
|
|
|
SCM_TSTAMP_SND);
|
|
|
|
}
|
2009-02-12 13:03:37 +08:00
|
|
|
EXPORT_SYMBOL_GPL(skb_tstamp_tx);
|
|
|
|
|
2011-11-09 17:15:42 +08:00
|
|
|
void skb_complete_wifi_ack(struct sk_buff *skb, bool acked)
|
|
|
|
{
|
|
|
|
struct sock *sk = skb->sk;
|
|
|
|
struct sock_exterr_skb *serr;
|
2017-03-04 13:01:02 +08:00
|
|
|
int err = 1;
|
2011-11-09 17:15:42 +08:00
|
|
|
|
|
|
|
skb->wifi_acked_valid = 1;
|
|
|
|
skb->wifi_acked = acked;
|
|
|
|
|
|
|
|
serr = SKB_EXT_ERR(skb);
|
|
|
|
memset(serr, 0, sizeof(*serr));
|
|
|
|
serr->ee.ee_errno = ENOMSG;
|
|
|
|
serr->ee.ee_origin = SO_EE_ORIGIN_TXSTATUS;
|
|
|
|
|
2017-03-04 13:01:02 +08:00
|
|
|
/* Take a reference to prevent skb_orphan() from freeing the socket,
|
|
|
|
* but only if the socket refcount is not zero.
|
|
|
|
*/
|
2017-06-30 18:08:01 +08:00
|
|
|
if (likely(refcount_inc_not_zero(&sk->sk_refcnt))) {
|
2017-03-04 13:01:02 +08:00
|
|
|
err = sock_queue_err_skb(sk, skb);
|
|
|
|
sock_put(sk);
|
|
|
|
}
|
2011-11-09 17:15:42 +08:00
|
|
|
if (err)
|
|
|
|
kfree_skb(skb);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(skb_complete_wifi_ack);
|
|
|
|
|
2008-02-05 12:49:54 +08:00
|
|
|
/**
|
|
|
|
* skb_partial_csum_set - set up and verify partial csum values for packet
|
|
|
|
* @skb: the skb to set
|
|
|
|
* @start: the number of bytes after skb->data to start checksumming.
|
|
|
|
* @off: the offset from start to place the checksum.
|
|
|
|
*
|
|
|
|
* For untrusted partially-checksummed packets, we need to make sure the values
|
|
|
|
* for skb->csum_start and skb->csum_offset are valid so we don't oops.
|
|
|
|
*
|
|
|
|
* This function checks and sets those values and skb->ip_summed: if this
|
|
|
|
* returns false you should drop the packet.
|
|
|
|
*/
|
|
|
|
bool skb_partial_csum_set(struct sk_buff *skb, u16 start, u16 off)
|
|
|
|
{
|
2018-10-10 21:59:35 +08:00
|
|
|
u32 csum_end = (u32)start + (u32)off + sizeof(__sum16);
|
|
|
|
u32 csum_start = skb_headroom(skb) + (u32)start;
|
|
|
|
|
|
|
|
if (unlikely(csum_start > U16_MAX || csum_end > skb_headlen(skb))) {
|
|
|
|
net_warn_ratelimited("bad partial csum: csum=%u/%u headroom=%u headlen=%u\n",
|
|
|
|
start, off, skb_headroom(skb), skb_headlen(skb));
|
2008-02-05 12:49:54 +08:00
|
|
|
return false;
|
|
|
|
}
|
|
|
|
skb->ip_summed = CHECKSUM_PARTIAL;
|
2018-10-10 21:59:35 +08:00
|
|
|
skb->csum_start = csum_start;
|
2008-02-05 12:49:54 +08:00
|
|
|
skb->csum_offset = off;
|
2013-03-27 07:11:20 +08:00
|
|
|
skb_set_transport_header(skb, start);
|
2008-02-05 12:49:54 +08:00
|
|
|
return true;
|
|
|
|
}
|
2009-02-10 18:09:24 +08:00
|
|
|
EXPORT_SYMBOL_GPL(skb_partial_csum_set);
|
2008-02-05 12:49:54 +08:00
|
|
|
|
2014-01-09 18:02:46 +08:00
|
|
|
static int skb_maybe_pull_tail(struct sk_buff *skb, unsigned int len,
|
|
|
|
unsigned int max)
|
|
|
|
{
|
|
|
|
if (skb_headlen(skb) >= len)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
/* If we need to pullup then pullup to the max, so we
|
|
|
|
* won't need to do it again.
|
|
|
|
*/
|
|
|
|
if (max > skb->len)
|
|
|
|
max = skb->len;
|
|
|
|
|
|
|
|
if (__pskb_pull_tail(skb, max - skb_headlen(skb)) == NULL)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
if (skb_headlen(skb) < len)
|
|
|
|
return -EPROTO;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2014-03-11 21:56:05 +08:00
|
|
|
#define MAX_TCP_HDR_LEN (15 * 4)
|
|
|
|
|
|
|
|
static __sum16 *skb_checksum_setup_ip(struct sk_buff *skb,
|
|
|
|
typeof(IPPROTO_IP) proto,
|
|
|
|
unsigned int off)
|
|
|
|
{
|
2020-02-20 14:23:04 +08:00
|
|
|
int err;
|
2014-03-11 21:56:05 +08:00
|
|
|
|
2020-02-20 14:23:04 +08:00
|
|
|
switch (proto) {
|
2014-03-11 21:56:05 +08:00
|
|
|
case IPPROTO_TCP:
|
|
|
|
err = skb_maybe_pull_tail(skb, off + sizeof(struct tcphdr),
|
|
|
|
off + MAX_TCP_HDR_LEN);
|
|
|
|
if (!err && !skb_partial_csum_set(skb, off,
|
|
|
|
offsetof(struct tcphdr,
|
|
|
|
check)))
|
|
|
|
err = -EPROTO;
|
|
|
|
return err ? ERR_PTR(err) : &tcp_hdr(skb)->check;
|
|
|
|
|
|
|
|
case IPPROTO_UDP:
|
|
|
|
err = skb_maybe_pull_tail(skb, off + sizeof(struct udphdr),
|
|
|
|
off + sizeof(struct udphdr));
|
|
|
|
if (!err && !skb_partial_csum_set(skb, off,
|
|
|
|
offsetof(struct udphdr,
|
|
|
|
check)))
|
|
|
|
err = -EPROTO;
|
|
|
|
return err ? ERR_PTR(err) : &udp_hdr(skb)->check;
|
|
|
|
}
|
|
|
|
|
|
|
|
return ERR_PTR(-EPROTO);
|
|
|
|
}
|
|
|
|
|
2014-01-09 18:02:46 +08:00
|
|
|
/* This value should be large enough to cover a tagged ethernet header plus
|
|
|
|
* maximally sized IP and TCP or UDP headers.
|
|
|
|
*/
|
|
|
|
#define MAX_IP_HDR_LEN 128
|
|
|
|
|
2014-03-11 21:56:05 +08:00
|
|
|
static int skb_checksum_setup_ipv4(struct sk_buff *skb, bool recalculate)
|
2014-01-09 18:02:46 +08:00
|
|
|
{
|
|
|
|
unsigned int off;
|
|
|
|
bool fragment;
|
2014-03-11 21:56:05 +08:00
|
|
|
__sum16 *csum;
|
2014-01-09 18:02:46 +08:00
|
|
|
int err;
|
|
|
|
|
|
|
|
fragment = false;
|
|
|
|
|
|
|
|
err = skb_maybe_pull_tail(skb,
|
|
|
|
sizeof(struct iphdr),
|
|
|
|
MAX_IP_HDR_LEN);
|
|
|
|
if (err < 0)
|
|
|
|
goto out;
|
|
|
|
|
2020-08-06 19:57:18 +08:00
|
|
|
if (ip_is_fragment(ip_hdr(skb)))
|
2014-01-09 18:02:46 +08:00
|
|
|
fragment = true;
|
|
|
|
|
|
|
|
off = ip_hdrlen(skb);
|
|
|
|
|
|
|
|
err = -EPROTO;
|
|
|
|
|
|
|
|
if (fragment)
|
|
|
|
goto out;
|
|
|
|
|
2014-03-11 21:56:05 +08:00
|
|
|
csum = skb_checksum_setup_ip(skb, ip_hdr(skb)->protocol, off);
|
|
|
|
if (IS_ERR(csum))
|
|
|
|
return PTR_ERR(csum);
|
2014-01-09 18:02:46 +08:00
|
|
|
|
2014-03-11 21:56:05 +08:00
|
|
|
if (recalculate)
|
|
|
|
*csum = ~csum_tcpudp_magic(ip_hdr(skb)->saddr,
|
|
|
|
ip_hdr(skb)->daddr,
|
|
|
|
skb->len - off,
|
|
|
|
ip_hdr(skb)->protocol, 0);
|
2014-01-09 18:02:46 +08:00
|
|
|
err = 0;
|
|
|
|
|
|
|
|
out:
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* This value should be large enough to cover a tagged ethernet header plus
|
|
|
|
* an IPv6 header, all options, and a maximal TCP or UDP header.
|
|
|
|
*/
|
|
|
|
#define MAX_IPV6_HDR_LEN 256
|
|
|
|
|
|
|
|
#define OPT_HDR(type, skb, off) \
|
|
|
|
(type *)(skb_network_header(skb) + (off))
|
|
|
|
|
|
|
|
static int skb_checksum_setup_ipv6(struct sk_buff *skb, bool recalculate)
|
|
|
|
{
|
|
|
|
int err;
|
|
|
|
u8 nexthdr;
|
|
|
|
unsigned int off;
|
|
|
|
unsigned int len;
|
|
|
|
bool fragment;
|
|
|
|
bool done;
|
2014-03-11 21:56:05 +08:00
|
|
|
__sum16 *csum;
|
2014-01-09 18:02:46 +08:00
|
|
|
|
|
|
|
fragment = false;
|
|
|
|
done = false;
|
|
|
|
|
|
|
|
off = sizeof(struct ipv6hdr);
|
|
|
|
|
|
|
|
err = skb_maybe_pull_tail(skb, off, MAX_IPV6_HDR_LEN);
|
|
|
|
if (err < 0)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
nexthdr = ipv6_hdr(skb)->nexthdr;
|
|
|
|
|
|
|
|
len = sizeof(struct ipv6hdr) + ntohs(ipv6_hdr(skb)->payload_len);
|
|
|
|
while (off <= len && !done) {
|
|
|
|
switch (nexthdr) {
|
|
|
|
case IPPROTO_DSTOPTS:
|
|
|
|
case IPPROTO_HOPOPTS:
|
|
|
|
case IPPROTO_ROUTING: {
|
|
|
|
struct ipv6_opt_hdr *hp;
|
|
|
|
|
|
|
|
err = skb_maybe_pull_tail(skb,
|
|
|
|
off +
|
|
|
|
sizeof(struct ipv6_opt_hdr),
|
|
|
|
MAX_IPV6_HDR_LEN);
|
|
|
|
if (err < 0)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
hp = OPT_HDR(struct ipv6_opt_hdr, skb, off);
|
|
|
|
nexthdr = hp->nexthdr;
|
|
|
|
off += ipv6_optlen(hp);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
case IPPROTO_AH: {
|
|
|
|
struct ip_auth_hdr *hp;
|
|
|
|
|
|
|
|
err = skb_maybe_pull_tail(skb,
|
|
|
|
off +
|
|
|
|
sizeof(struct ip_auth_hdr),
|
|
|
|
MAX_IPV6_HDR_LEN);
|
|
|
|
if (err < 0)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
hp = OPT_HDR(struct ip_auth_hdr, skb, off);
|
|
|
|
nexthdr = hp->nexthdr;
|
|
|
|
off += ipv6_authlen(hp);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
case IPPROTO_FRAGMENT: {
|
|
|
|
struct frag_hdr *hp;
|
|
|
|
|
|
|
|
err = skb_maybe_pull_tail(skb,
|
|
|
|
off +
|
|
|
|
sizeof(struct frag_hdr),
|
|
|
|
MAX_IPV6_HDR_LEN);
|
|
|
|
if (err < 0)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
hp = OPT_HDR(struct frag_hdr, skb, off);
|
|
|
|
|
|
|
|
if (hp->frag_off & htons(IP6_OFFSET | IP6_MF))
|
|
|
|
fragment = true;
|
|
|
|
|
|
|
|
nexthdr = hp->nexthdr;
|
|
|
|
off += sizeof(struct frag_hdr);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
default:
|
|
|
|
done = true;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
err = -EPROTO;
|
|
|
|
|
|
|
|
if (!done || fragment)
|
|
|
|
goto out;
|
|
|
|
|
2014-03-11 21:56:05 +08:00
|
|
|
csum = skb_checksum_setup_ip(skb, nexthdr, off);
|
|
|
|
if (IS_ERR(csum))
|
|
|
|
return PTR_ERR(csum);
|
2014-01-09 18:02:46 +08:00
|
|
|
|
2014-03-11 21:56:05 +08:00
|
|
|
if (recalculate)
|
|
|
|
*csum = ~csum_ipv6_magic(&ipv6_hdr(skb)->saddr,
|
|
|
|
&ipv6_hdr(skb)->daddr,
|
|
|
|
skb->len - off, nexthdr, 0);
|
2014-01-09 18:02:46 +08:00
|
|
|
err = 0;
|
|
|
|
|
|
|
|
out:
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* skb_checksum_setup - set up partial checksum offset
|
|
|
|
* @skb: the skb to set up
|
|
|
|
* @recalculate: if true the pseudo-header checksum will be recalculated
|
|
|
|
*/
|
|
|
|
int skb_checksum_setup(struct sk_buff *skb, bool recalculate)
|
|
|
|
{
|
|
|
|
int err;
|
|
|
|
|
|
|
|
switch (skb->protocol) {
|
|
|
|
case htons(ETH_P_IP):
|
2014-03-11 21:56:05 +08:00
|
|
|
err = skb_checksum_setup_ipv4(skb, recalculate);
|
2014-01-09 18:02:46 +08:00
|
|
|
break;
|
|
|
|
|
|
|
|
case htons(ETH_P_IPV6):
|
|
|
|
err = skb_checksum_setup_ipv6(skb, recalculate);
|
|
|
|
break;
|
|
|
|
|
|
|
|
default:
|
|
|
|
err = -EPROTO;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_checksum_setup);
|
|
|
|
|
2015-05-02 20:01:07 +08:00
|
|
|
/**
|
|
|
|
* skb_checksum_maybe_trim - maybe trims the given skb
|
|
|
|
* @skb: the skb to check
|
|
|
|
* @transport_len: the data length beyond the network header
|
|
|
|
*
|
|
|
|
* Checks whether the given skb has data beyond the given transport length.
|
|
|
|
* If so, returns a cloned skb trimmed to this transport length.
|
|
|
|
* Otherwise returns the provided skb. Returns NULL in error cases
|
|
|
|
* (e.g. transport_len exceeds skb length or out-of-memory).
|
|
|
|
*
|
2015-08-13 11:54:07 +08:00
|
|
|
* Caller needs to set the skb transport header and free any returned skb if it
|
|
|
|
* differs from the provided skb.
|
2015-05-02 20:01:07 +08:00
|
|
|
*/
|
|
|
|
static struct sk_buff *skb_checksum_maybe_trim(struct sk_buff *skb,
|
|
|
|
unsigned int transport_len)
|
|
|
|
{
|
|
|
|
struct sk_buff *skb_chk;
|
|
|
|
unsigned int len = skb_transport_offset(skb) + transport_len;
|
|
|
|
int ret;
|
|
|
|
|
2015-08-13 11:54:07 +08:00
|
|
|
if (skb->len < len)
|
2015-05-02 20:01:07 +08:00
|
|
|
return NULL;
|
2015-08-13 11:54:07 +08:00
|
|
|
else if (skb->len == len)
|
2015-05-02 20:01:07 +08:00
|
|
|
return skb;
|
|
|
|
|
|
|
|
skb_chk = skb_clone(skb, GFP_ATOMIC);
|
|
|
|
if (!skb_chk)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
ret = pskb_trim_rcsum(skb_chk, len);
|
|
|
|
if (ret) {
|
|
|
|
kfree_skb(skb_chk);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
return skb_chk;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* skb_checksum_trimmed - validate checksum of an skb
|
|
|
|
* @skb: the skb to check
|
|
|
|
* @transport_len: the data length beyond the network header
|
|
|
|
* @skb_chkf: checksum function to use
|
|
|
|
*
|
|
|
|
* Applies the given checksum function skb_chkf to the provided skb.
|
|
|
|
* Returns a checked and maybe trimmed skb. Returns NULL on error.
|
|
|
|
*
|
|
|
|
* If the skb has data beyond the given transport length, then a
|
|
|
|
* trimmed & cloned skb is checked and returned.
|
|
|
|
*
|
2015-08-13 11:54:07 +08:00
|
|
|
* Caller needs to set the skb transport header and free any returned skb if it
|
|
|
|
* differs from the provided skb.
|
2015-05-02 20:01:07 +08:00
|
|
|
*/
|
|
|
|
struct sk_buff *skb_checksum_trimmed(struct sk_buff *skb,
|
|
|
|
unsigned int transport_len,
|
|
|
|
__sum16(*skb_chkf)(struct sk_buff *skb))
|
|
|
|
{
|
|
|
|
struct sk_buff *skb_chk;
|
|
|
|
unsigned int offset = skb_transport_offset(skb);
|
2015-05-05 06:19:35 +08:00
|
|
|
__sum16 ret;
|
2015-05-02 20:01:07 +08:00
|
|
|
|
|
|
|
skb_chk = skb_checksum_maybe_trim(skb, transport_len);
|
|
|
|
if (!skb_chk)
|
2015-08-13 11:54:07 +08:00
|
|
|
goto err;
|
2015-05-02 20:01:07 +08:00
|
|
|
|
2015-08-13 11:54:07 +08:00
|
|
|
if (!pskb_may_pull(skb_chk, offset))
|
|
|
|
goto err;
|
2015-05-02 20:01:07 +08:00
|
|
|
|
2016-02-24 11:21:42 +08:00
|
|
|
skb_pull_rcsum(skb_chk, offset);
|
2015-05-02 20:01:07 +08:00
|
|
|
ret = skb_chkf(skb_chk);
|
2016-02-24 11:21:42 +08:00
|
|
|
skb_push_rcsum(skb_chk, offset);
|
2015-05-02 20:01:07 +08:00
|
|
|
|
2015-08-13 11:54:07 +08:00
|
|
|
if (ret)
|
|
|
|
goto err;
|
2015-05-02 20:01:07 +08:00
|
|
|
|
|
|
|
return skb_chk;
|
2015-08-13 11:54:07 +08:00
|
|
|
|
|
|
|
err:
|
|
|
|
if (skb_chk && skb_chk != skb)
|
|
|
|
kfree_skb(skb_chk);
|
|
|
|
|
|
|
|
return NULL;
|
|
|
|
|
2015-05-02 20:01:07 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_checksum_trimmed);
|
|
|
|
|
2008-06-20 07:22:28 +08:00
|
|
|
void __skb_warn_lro_forwarding(const struct sk_buff *skb)
|
|
|
|
{
|
2012-05-14 05:56:26 +08:00
|
|
|
net_warn_ratelimited("%s: received packets cannot be forwarded while LRO is enabled\n",
|
|
|
|
skb->dev->name);
|
2008-06-20 07:22:28 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(__skb_warn_lro_forwarding);
|
2012-05-19 11:02:02 +08:00
|
|
|
|
|
|
|
void kfree_skb_partial(struct sk_buff *skb, bool head_stolen)
|
|
|
|
{
|
2012-10-22 17:03:40 +08:00
|
|
|
if (head_stolen) {
|
|
|
|
skb_release_head_state(skb);
|
2012-05-19 11:02:02 +08:00
|
|
|
kmem_cache_free(skbuff_head_cache, skb);
|
2012-10-22 17:03:40 +08:00
|
|
|
} else {
|
2012-05-19 11:02:02 +08:00
|
|
|
__kfree_skb(skb);
|
2012-10-22 17:03:40 +08:00
|
|
|
}
|
2012-05-19 11:02:02 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(kfree_skb_partial);
|
|
|
|
|
|
|
|
/**
|
|
|
|
* skb_try_coalesce - try to merge skb to prior one
|
|
|
|
* @to: prior buffer
|
|
|
|
* @from: buffer to add
|
|
|
|
* @fragstolen: pointer to boolean
|
2012-06-08 22:01:44 +08:00
|
|
|
* @delta_truesize: how much more was allocated than was requested
|
2012-05-19 11:02:02 +08:00
|
|
|
*/
|
|
|
|
bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from,
|
|
|
|
bool *fragstolen, int *delta_truesize)
|
|
|
|
{
|
2017-10-05 01:48:35 +08:00
|
|
|
struct skb_shared_info *to_shinfo, *from_shinfo;
|
2012-05-19 11:02:02 +08:00
|
|
|
int i, delta, len = from->len;
|
|
|
|
|
|
|
|
*fragstolen = false;
|
|
|
|
|
|
|
|
if (skb_cloned(to))
|
|
|
|
return false;
|
|
|
|
|
2021-06-08 03:02:38 +08:00
|
|
|
/* The page pool signature of struct page will eventually figure out
|
|
|
|
* which pages can be recycled or not but for now let's prohibit slab
|
|
|
|
* allocated and page_pool allocated SKBs from being coalesced.
|
|
|
|
*/
|
|
|
|
if (to->pp_recycle != from->pp_recycle)
|
|
|
|
return false;
|
|
|
|
|
2012-05-19 11:02:02 +08:00
|
|
|
if (len <= skb_tailroom(to)) {
|
2014-09-15 19:19:52 +08:00
|
|
|
if (len)
|
|
|
|
BUG_ON(skb_copy_bits(from, 0, skb_put(to, len), len));
|
2012-05-19 11:02:02 +08:00
|
|
|
*delta_truesize = 0;
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
2017-10-05 01:48:35 +08:00
|
|
|
to_shinfo = skb_shinfo(to);
|
|
|
|
from_shinfo = skb_shinfo(from);
|
|
|
|
if (to_shinfo->frag_list || from_shinfo->frag_list)
|
2012-05-19 11:02:02 +08:00
|
|
|
return false;
|
sock: enable MSG_ZEROCOPY
Prepare the datapath for refcounted ubuf_info. Clone ubuf_info with
skb_zerocopy_clone() wherever needed due to skb split, merge, resize
or clone.
Split skb_orphan_frags into two variants. The split, merge, .. paths
support reference counted zerocopy buffers, so do not do a deep copy.
Add skb_orphan_frags_rx for paths that may loop packets to receive
sockets. That is not allowed, as it may cause unbounded latency.
Deep copy all zerocopy copy buffers, ref-counted or not, in this path.
The exact locations to modify were chosen by exhaustively searching
through all code that might modify skb_frag references and/or the
the SKBTX_DEV_ZEROCOPY tx_flags bit.
The changes err on the safe side, in two ways.
(1) legacy ubuf_info paths virtio and tap are not modified. They keep
a 1:1 ubuf_info to sk_buff relationship. Calls to skb_orphan_frags
still call skb_copy_ubufs and thus copy frags in this case.
(2) not all copies deep in the stack are addressed yet. skb_shift,
skb_split and skb_try_coalesce can be refined to avoid copying.
These are not in the hot path and this patch is hairy enough as
is, so that is left for future refinement.
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-04 04:29:41 +08:00
|
|
|
if (skb_zcopy(to) || skb_zcopy(from))
|
|
|
|
return false;
|
2012-05-19 11:02:02 +08:00
|
|
|
|
|
|
|
if (skb_headlen(from) != 0) {
|
|
|
|
struct page *page;
|
|
|
|
unsigned int offset;
|
|
|
|
|
2017-10-05 01:48:35 +08:00
|
|
|
if (to_shinfo->nr_frags +
|
|
|
|
from_shinfo->nr_frags >= MAX_SKB_FRAGS)
|
2012-05-19 11:02:02 +08:00
|
|
|
return false;
|
|
|
|
|
|
|
|
if (skb_head_is_locked(from))
|
|
|
|
return false;
|
|
|
|
|
|
|
|
delta = from->truesize - SKB_DATA_ALIGN(sizeof(struct sk_buff));
|
|
|
|
|
|
|
|
page = virt_to_head_page(from->head);
|
|
|
|
offset = from->data - (unsigned char *)page_address(page);
|
|
|
|
|
2017-10-05 01:48:35 +08:00
|
|
|
skb_fill_page_desc(to, to_shinfo->nr_frags,
|
2012-05-19 11:02:02 +08:00
|
|
|
page, offset, skb_headlen(from));
|
|
|
|
*fragstolen = true;
|
|
|
|
} else {
|
2017-10-05 01:48:35 +08:00
|
|
|
if (to_shinfo->nr_frags +
|
|
|
|
from_shinfo->nr_frags > MAX_SKB_FRAGS)
|
2012-05-19 11:02:02 +08:00
|
|
|
return false;
|
|
|
|
|
2012-09-29 04:15:30 +08:00
|
|
|
delta = from->truesize - SKB_TRUESIZE(skb_end_offset(from));
|
2012-05-19 11:02:02 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
WARN_ON_ONCE(delta < len);
|
|
|
|
|
2017-10-05 01:48:35 +08:00
|
|
|
memcpy(to_shinfo->frags + to_shinfo->nr_frags,
|
|
|
|
from_shinfo->frags,
|
|
|
|
from_shinfo->nr_frags * sizeof(skb_frag_t));
|
|
|
|
to_shinfo->nr_frags += from_shinfo->nr_frags;
|
2012-05-19 11:02:02 +08:00
|
|
|
|
|
|
|
if (!skb_cloned(from))
|
2017-10-05 01:48:35 +08:00
|
|
|
from_shinfo->nr_frags = 0;
|
2012-05-19 11:02:02 +08:00
|
|
|
|
2012-09-19 00:53:21 +08:00
|
|
|
/* if the skb is not cloned this does nothing
|
|
|
|
* since we set nr_frags to 0.
|
|
|
|
*/
|
2017-10-05 01:48:35 +08:00
|
|
|
for (i = 0; i < from_shinfo->nr_frags; i++)
|
|
|
|
__skb_frag_ref(&from_shinfo->frags[i]);
|
2012-05-19 11:02:02 +08:00
|
|
|
|
|
|
|
to->truesize += delta;
|
|
|
|
to->len += len;
|
|
|
|
to->data_len += len;
|
|
|
|
|
|
|
|
*delta_truesize = delta;
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_try_coalesce);
|
2013-06-26 22:11:27 +08:00
|
|
|
|
|
|
|
/**
|
2013-09-02 21:34:56 +08:00
|
|
|
* skb_scrub_packet - scrub an skb
|
2013-06-26 22:11:27 +08:00
|
|
|
*
|
|
|
|
* @skb: buffer to clean
|
2013-09-02 21:34:56 +08:00
|
|
|
* @xnet: packet is crossing netns
|
|
|
|
*
|
|
|
|
* skb_scrub_packet can be used after encapsulating or decapsulting a packet
|
|
|
|
* into/from a tunnel. Some information have to be cleared during these
|
|
|
|
* operations.
|
|
|
|
* skb_scrub_packet can also be used to clean a skb before injecting it in
|
|
|
|
* another namespace (@xnet == true). We have to clear all information in the
|
|
|
|
* skb that could impact namespace isolation.
|
2013-06-26 22:11:27 +08:00
|
|
|
*/
|
2013-09-02 21:34:56 +08:00
|
|
|
void skb_scrub_packet(struct sk_buff *skb, bool xnet)
|
2013-06-26 22:11:27 +08:00
|
|
|
{
|
|
|
|
skb->pkt_type = PACKET_HOST;
|
|
|
|
skb->skb_iif = 0;
|
2014-05-05 07:39:18 +08:00
|
|
|
skb->ignore_df = 0;
|
2013-06-26 22:11:27 +08:00
|
|
|
skb_dst_drop(skb);
|
2019-09-27 02:37:05 +08:00
|
|
|
skb_ext_reset(skb);
|
2019-09-30 02:54:03 +08:00
|
|
|
nf_reset_ct(skb);
|
2013-06-26 22:11:27 +08:00
|
|
|
nf_reset_trace(skb);
|
skbuff: Do not scrub skb mark within the same name space
On Wed, Apr 15, 2015 at 05:41:26PM +0200, Nicolas Dichtel wrote:
> Le 15/04/2015 15:57, Herbert Xu a écrit :
> >On Wed, Apr 15, 2015 at 06:22:29PM +0800, Herbert Xu wrote:
> [snip]
> >Subject: skbuff: Do not scrub skb mark within the same name space
> >
> >The commit ea23192e8e577dfc51e0f4fc5ca113af334edff9 ("tunnels:
> Maybe add a Fixes tag?
> Fixes: ea23192e8e57 ("tunnels: harmonize cleanup done on skb on rx path")
>
> >harmonize cleanup done on skb on rx path") broke anyone trying to
> >use netfilter marking across IPv4 tunnels. While most of the
> >fields that are cleared by skb_scrub_packet don't matter, the
> >netfilter mark must be preserved.
> >
> >This patch rearranges skb_scurb_packet to preserve the mark field.
> nit: s/scurb/scrub
>
> Else it's fine for me.
Sure.
PS I used the wrong email for James the first time around. So
let me repeat the question here. Should secmark be preserved
or cleared across tunnels within the same name space? In fact,
do our security models even support name spaces?
---8<---
The commit ea23192e8e577dfc51e0f4fc5ca113af334edff9 ("tunnels:
harmonize cleanup done on skb on rx path") broke anyone trying to
use netfilter marking across IPv4 tunnels. While most of the
fields that are cleared by skb_scrub_packet don't matter, the
netfilter mark must be preserved.
This patch rearranges skb_scrub_packet to preserve the mark field.
Fixes: ea23192e8e57 ("tunnels: harmonize cleanup done on skb on rx path")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Acked-by: Thomas Graf <tgraf@suug.ch>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-04-16 09:03:27 +08:00
|
|
|
|
2018-11-20 00:11:07 +08:00
|
|
|
#ifdef CONFIG_NET_SWITCHDEV
|
|
|
|
skb->offload_fwd_mark = 0;
|
2018-12-04 16:15:10 +08:00
|
|
|
skb->offload_l3_fwd_mark = 0;
|
2018-11-20 00:11:07 +08:00
|
|
|
#endif
|
|
|
|
|
skbuff: Do not scrub skb mark within the same name space
On Wed, Apr 15, 2015 at 05:41:26PM +0200, Nicolas Dichtel wrote:
> Le 15/04/2015 15:57, Herbert Xu a écrit :
> >On Wed, Apr 15, 2015 at 06:22:29PM +0800, Herbert Xu wrote:
> [snip]
> >Subject: skbuff: Do not scrub skb mark within the same name space
> >
> >The commit ea23192e8e577dfc51e0f4fc5ca113af334edff9 ("tunnels:
> Maybe add a Fixes tag?
> Fixes: ea23192e8e57 ("tunnels: harmonize cleanup done on skb on rx path")
>
> >harmonize cleanup done on skb on rx path") broke anyone trying to
> >use netfilter marking across IPv4 tunnels. While most of the
> >fields that are cleared by skb_scrub_packet don't matter, the
> >netfilter mark must be preserved.
> >
> >This patch rearranges skb_scurb_packet to preserve the mark field.
> nit: s/scurb/scrub
>
> Else it's fine for me.
Sure.
PS I used the wrong email for James the first time around. So
let me repeat the question here. Should secmark be preserved
or cleared across tunnels within the same name space? In fact,
do our security models even support name spaces?
---8<---
The commit ea23192e8e577dfc51e0f4fc5ca113af334edff9 ("tunnels:
harmonize cleanup done on skb on rx path") broke anyone trying to
use netfilter marking across IPv4 tunnels. While most of the
fields that are cleared by skb_scrub_packet don't matter, the
netfilter mark must be preserved.
This patch rearranges skb_scrub_packet to preserve the mark field.
Fixes: ea23192e8e57 ("tunnels: harmonize cleanup done on skb on rx path")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Acked-by: Thomas Graf <tgraf@suug.ch>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-04-16 09:03:27 +08:00
|
|
|
if (!xnet)
|
|
|
|
return;
|
|
|
|
|
2017-10-26 16:57:05 +08:00
|
|
|
ipvs_reset(skb);
|
skbuff: Do not scrub skb mark within the same name space
On Wed, Apr 15, 2015 at 05:41:26PM +0200, Nicolas Dichtel wrote:
> Le 15/04/2015 15:57, Herbert Xu a écrit :
> >On Wed, Apr 15, 2015 at 06:22:29PM +0800, Herbert Xu wrote:
> [snip]
> >Subject: skbuff: Do not scrub skb mark within the same name space
> >
> >The commit ea23192e8e577dfc51e0f4fc5ca113af334edff9 ("tunnels:
> Maybe add a Fixes tag?
> Fixes: ea23192e8e57 ("tunnels: harmonize cleanup done on skb on rx path")
>
> >harmonize cleanup done on skb on rx path") broke anyone trying to
> >use netfilter marking across IPv4 tunnels. While most of the
> >fields that are cleared by skb_scrub_packet don't matter, the
> >netfilter mark must be preserved.
> >
> >This patch rearranges skb_scurb_packet to preserve the mark field.
> nit: s/scurb/scrub
>
> Else it's fine for me.
Sure.
PS I used the wrong email for James the first time around. So
let me repeat the question here. Should secmark be preserved
or cleared across tunnels within the same name space? In fact,
do our security models even support name spaces?
---8<---
The commit ea23192e8e577dfc51e0f4fc5ca113af334edff9 ("tunnels:
harmonize cleanup done on skb on rx path") broke anyone trying to
use netfilter marking across IPv4 tunnels. While most of the
fields that are cleared by skb_scrub_packet don't matter, the
netfilter mark must be preserved.
This patch rearranges skb_scrub_packet to preserve the mark field.
Fixes: ea23192e8e57 ("tunnels: harmonize cleanup done on skb on rx path")
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Acked-by: Thomas Graf <tgraf@suug.ch>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-04-16 09:03:27 +08:00
|
|
|
skb->mark = 0;
|
2022-03-03 03:55:31 +08:00
|
|
|
skb_clear_tstamp(skb);
|
2013-06-26 22:11:27 +08:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(skb_scrub_packet);
|
2014-01-26 17:58:16 +08:00
|
|
|
|
|
|
|
/**
|
|
|
|
* skb_gso_transport_seglen - Return length of individual segments of a gso packet
|
|
|
|
*
|
|
|
|
* @skb: GSO skb
|
|
|
|
*
|
|
|
|
* skb_gso_transport_seglen is used to determine the real size of the
|
|
|
|
* individual segments, including Layer4 headers (TCP/UDP).
|
|
|
|
*
|
|
|
|
* The MAC/L2 or network (IP, IPv6) headers are not accounted for.
|
|
|
|
*/
|
2018-03-01 14:13:40 +08:00
|
|
|
static unsigned int skb_gso_transport_seglen(const struct sk_buff *skb)
|
2014-01-26 17:58:16 +08:00
|
|
|
{
|
|
|
|
const struct skb_shared_info *shinfo = skb_shinfo(skb);
|
2014-10-20 19:49:18 +08:00
|
|
|
unsigned int thlen = 0;
|
2014-01-26 17:58:16 +08:00
|
|
|
|
2014-10-20 19:49:18 +08:00
|
|
|
if (skb->encapsulation) {
|
|
|
|
thlen = skb_inner_transport_header(skb) -
|
|
|
|
skb_transport_header(skb);
|
2014-04-09 16:28:50 +08:00
|
|
|
|
2014-10-20 19:49:18 +08:00
|
|
|
if (likely(shinfo->gso_type & (SKB_GSO_TCPV4 | SKB_GSO_TCPV6)))
|
|
|
|
thlen += inner_tcp_hdrlen(skb);
|
|
|
|
} else if (likely(shinfo->gso_type & (SKB_GSO_TCPV4 | SKB_GSO_TCPV6))) {
|
|
|
|
thlen = tcp_hdrlen(skb);
|
2018-03-09 11:06:09 +08:00
|
|
|
} else if (unlikely(skb_is_gso_sctp(skb))) {
|
2016-06-03 02:05:43 +08:00
|
|
|
thlen = sizeof(struct sctphdr);
|
2018-04-27 01:42:16 +08:00
|
|
|
} else if (shinfo->gso_type & SKB_GSO_UDP_L4) {
|
|
|
|
thlen = sizeof(struct udphdr);
|
2014-10-20 19:49:18 +08:00
|
|
|
}
|
2014-04-09 16:28:50 +08:00
|
|
|
/* UFO sets gso_size to the size of the fragmentation
|
|
|
|
* payload, i.e. the size of the L4 (UDP) header is already
|
|
|
|
* accounted for.
|
|
|
|
*/
|
2014-10-20 19:49:18 +08:00
|
|
|
return thlen + shinfo->gso_size;
|
2014-01-26 17:58:16 +08:00
|
|
|
}
|
2018-03-01 14:13:40 +08:00
|
|
|
|
|
|
|
/**
|
|
|
|
* skb_gso_network_seglen - Return length of individual segments of a gso packet
|
|
|
|
*
|
|
|
|
* @skb: GSO skb
|
|
|
|
*
|
|
|
|
* skb_gso_network_seglen is used to determine the real size of the
|
|
|
|
* individual segments, including Layer3 (IP, IPv6) and L4 headers (TCP/UDP).
|
|
|
|
*
|
|
|
|
* The MAC/L2 header is not accounted for.
|
|
|
|
*/
|
|
|
|
static unsigned int skb_gso_network_seglen(const struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
unsigned int hdr_len = skb_transport_header(skb) -
|
|
|
|
skb_network_header(skb);
|
|
|
|
|
|
|
|
return hdr_len + skb_gso_transport_seglen(skb);
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* skb_gso_mac_seglen - Return length of individual segments of a gso packet
|
|
|
|
*
|
|
|
|
* @skb: GSO skb
|
|
|
|
*
|
|
|
|
* skb_gso_mac_seglen is used to determine the real size of the
|
|
|
|
* individual segments, including MAC/L2, Layer3 (IP, IPv6) and L4
|
|
|
|
* headers (TCP/UDP).
|
|
|
|
*/
|
|
|
|
static unsigned int skb_gso_mac_seglen(const struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
unsigned int hdr_len = skb_transport_header(skb) - skb_mac_header(skb);
|
|
|
|
|
|
|
|
return hdr_len + skb_gso_transport_seglen(skb);
|
|
|
|
}
|
net: Always untag vlan-tagged traffic on input.
Currently the functionality to untag traffic on input resides
as part of the vlan module and is build only when VLAN support
is enabled in the kernel. When VLAN is disabled, the function
vlan_untag() turns into a stub and doesn't really untag the
packets. This seems to create an interesting interaction
between VMs supporting checksum offloading and some network drivers.
There are some drivers that do not allow the user to change
tx-vlan-offload feature of the driver. These drivers also seem
to assume that any VLAN-tagged traffic they transmit will
have the vlan information in the vlan_tci and not in the vlan
header already in the skb. When transmitting skbs that already
have tagged data with partial checksum set, the checksum doesn't
appear to be updated correctly by the card thus resulting in a
failure to establish TCP connections.
The following is a packet trace taken on the receiver where a
sender is a VM with a VLAN configued. The host VM is running on
doest not have VLAN support and the outging interface on the
host is tg3:
10:12:43.503055 52:54:00:ae:42:3f > 28:d2:44:7d:c2:de, ethertype 802.1Q
(0x8100), length 78: vlan 100, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 27243,
offset 0, flags [DF], proto TCP (6), length 60)
10.0.100.1.58545 > 10.0.100.10.ircu-2: Flags [S], cksum 0xdc39 (incorrect
-> 0x48d9), seq 1069378582, win 29200, options [mss 1460,sackOK,TS val
4294837885 ecr 0,nop,wscale 7], length 0
10:12:44.505556 52:54:00:ae:42:3f > 28:d2:44:7d:c2:de, ethertype 802.1Q
(0x8100), length 78: vlan 100, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 27244,
offset 0, flags [DF], proto TCP (6), length 60)
10.0.100.1.58545 > 10.0.100.10.ircu-2: Flags [S], cksum 0xdc39 (incorrect
-> 0x44ee), seq 1069378582, win 29200, options [mss 1460,sackOK,TS val
4294838888 ecr 0,nop,wscale 7], length 0
This connection finally times out.
I've only access to the TG3 hardware in this configuration thus have
only tested this with TG3 driver. There are a lot of other drivers
that do not permit user changes to vlan acceleration features, and
I don't know if they all suffere from a similar issue.
The patch attempt to fix this another way. It moves the vlan header
stipping code out of the vlan module and always builds it into the
kernel network core. This way, even if vlan is not supported on
a virtualizatoin host, the virtual machines running on top of such
host will still work with VLANs enabled.
CC: Patrick McHardy <kaber@trash.net>
CC: Nithin Nayak Sujir <nsujir@broadcom.com>
CC: Michael Chan <mchan@broadcom.com>
CC: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Vladislav Yasevich <vyasevic@redhat.com>
Acked-by: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-09 02:42:13 +08:00
|
|
|
|
2016-06-03 02:05:41 +08:00
|
|
|
/**
|
2018-01-31 11:15:33 +08:00
|
|
|
* skb_gso_size_check - check the skb size, considering GSO_BY_FRAGS
|
2016-06-03 02:05:41 +08:00
|
|
|
*
|
2018-01-31 11:15:33 +08:00
|
|
|
* There are a couple of instances where we have a GSO skb, and we
|
|
|
|
* want to determine what size it would be after it is segmented.
|
2016-06-03 02:05:41 +08:00
|
|
|
*
|
2018-01-31 11:15:33 +08:00
|
|
|
* We might want to check:
|
|
|
|
* - L3+L4+payload size (e.g. IP forwarding)
|
|
|
|
* - L2+L3+L4+payload size (e.g. sanity check before passing to driver)
|
|
|
|
*
|
|
|
|
* This is a helper to do that correctly considering GSO_BY_FRAGS.
|
|
|
|
*
|
2018-10-31 20:16:58 +08:00
|
|
|
* @skb: GSO skb
|
|
|
|
*
|
2018-01-31 11:15:33 +08:00
|
|
|
* @seg_len: The segmented length (from skb_gso_*_seglen). In the
|
|
|
|
* GSO_BY_FRAGS case this will be [header sizes + GSO_BY_FRAGS].
|
|
|
|
*
|
|
|
|
* @max_len: The maximum permissible length.
|
|
|
|
*
|
|
|
|
* Returns true if the segmented length <= max length.
|
2016-06-03 02:05:41 +08:00
|
|
|
*/
|
2018-01-31 11:15:33 +08:00
|
|
|
static inline bool skb_gso_size_check(const struct sk_buff *skb,
|
|
|
|
unsigned int seg_len,
|
|
|
|
unsigned int max_len) {
|
2016-06-03 02:05:41 +08:00
|
|
|
const struct skb_shared_info *shinfo = skb_shinfo(skb);
|
|
|
|
const struct sk_buff *iter;
|
|
|
|
|
|
|
|
if (shinfo->gso_size != GSO_BY_FRAGS)
|
2018-01-31 11:15:33 +08:00
|
|
|
return seg_len <= max_len;
|
2016-06-03 02:05:41 +08:00
|
|
|
|
|
|
|
/* Undo this so we can re-use header sizes */
|
2018-01-31 11:15:33 +08:00
|
|
|
seg_len -= GSO_BY_FRAGS;
|
2016-06-03 02:05:41 +08:00
|
|
|
|
|
|
|
skb_walk_frags(skb, iter) {
|
2018-01-31 11:15:33 +08:00
|
|
|
if (seg_len + skb_headlen(iter) > max_len)
|
2016-06-03 02:05:41 +08:00
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
|
|
|
return true;
|
|
|
|
}
|
2018-01-31 11:15:33 +08:00
|
|
|
|
|
|
|
/**
|
2018-03-01 14:13:37 +08:00
|
|
|
* skb_gso_validate_network_len - Will a split GSO skb fit into a given MTU?
|
2018-01-31 11:15:33 +08:00
|
|
|
*
|
|
|
|
* @skb: GSO skb
|
|
|
|
* @mtu: MTU to validate against
|
|
|
|
*
|
2018-03-01 14:13:37 +08:00
|
|
|
* skb_gso_validate_network_len validates if a given skb will fit a
|
|
|
|
* wanted MTU once split. It considers L3 headers, L4 headers, and the
|
|
|
|
* payload.
|
2018-01-31 11:15:33 +08:00
|
|
|
*/
|
2018-03-01 14:13:37 +08:00
|
|
|
bool skb_gso_validate_network_len(const struct sk_buff *skb, unsigned int mtu)
|
2018-01-31 11:15:33 +08:00
|
|
|
{
|
|
|
|
return skb_gso_size_check(skb, skb_gso_network_seglen(skb), mtu);
|
|
|
|
}
|
2018-03-01 14:13:37 +08:00
|
|
|
EXPORT_SYMBOL_GPL(skb_gso_validate_network_len);
|
2016-06-03 02:05:41 +08:00
|
|
|
|
2018-01-31 11:15:33 +08:00
|
|
|
/**
|
|
|
|
* skb_gso_validate_mac_len - Will a split GSO skb fit in a given length?
|
|
|
|
*
|
|
|
|
* @skb: GSO skb
|
|
|
|
* @len: length to validate against
|
|
|
|
*
|
|
|
|
* skb_gso_validate_mac_len validates if a given skb will fit a wanted
|
|
|
|
* length once split, including L2, L3 and L4 headers and the payload.
|
|
|
|
*/
|
|
|
|
bool skb_gso_validate_mac_len(const struct sk_buff *skb, unsigned int len)
|
|
|
|
{
|
|
|
|
return skb_gso_size_check(skb, skb_gso_mac_seglen(skb), len);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(skb_gso_validate_mac_len);
|
|
|
|
|
net: Always untag vlan-tagged traffic on input.
Currently the functionality to untag traffic on input resides
as part of the vlan module and is build only when VLAN support
is enabled in the kernel. When VLAN is disabled, the function
vlan_untag() turns into a stub and doesn't really untag the
packets. This seems to create an interesting interaction
between VMs supporting checksum offloading and some network drivers.
There are some drivers that do not allow the user to change
tx-vlan-offload feature of the driver. These drivers also seem
to assume that any VLAN-tagged traffic they transmit will
have the vlan information in the vlan_tci and not in the vlan
header already in the skb. When transmitting skbs that already
have tagged data with partial checksum set, the checksum doesn't
appear to be updated correctly by the card thus resulting in a
failure to establish TCP connections.
The following is a packet trace taken on the receiver where a
sender is a VM with a VLAN configued. The host VM is running on
doest not have VLAN support and the outging interface on the
host is tg3:
10:12:43.503055 52:54:00:ae:42:3f > 28:d2:44:7d:c2:de, ethertype 802.1Q
(0x8100), length 78: vlan 100, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 27243,
offset 0, flags [DF], proto TCP (6), length 60)
10.0.100.1.58545 > 10.0.100.10.ircu-2: Flags [S], cksum 0xdc39 (incorrect
-> 0x48d9), seq 1069378582, win 29200, options [mss 1460,sackOK,TS val
4294837885 ecr 0,nop,wscale 7], length 0
10:12:44.505556 52:54:00:ae:42:3f > 28:d2:44:7d:c2:de, ethertype 802.1Q
(0x8100), length 78: vlan 100, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 27244,
offset 0, flags [DF], proto TCP (6), length 60)
10.0.100.1.58545 > 10.0.100.10.ircu-2: Flags [S], cksum 0xdc39 (incorrect
-> 0x44ee), seq 1069378582, win 29200, options [mss 1460,sackOK,TS val
4294838888 ecr 0,nop,wscale 7], length 0
This connection finally times out.
I've only access to the TG3 hardware in this configuration thus have
only tested this with TG3 driver. There are a lot of other drivers
that do not permit user changes to vlan acceleration features, and
I don't know if they all suffere from a similar issue.
The patch attempt to fix this another way. It moves the vlan header
stipping code out of the vlan module and always builds it into the
kernel network core. This way, even if vlan is not supported on
a virtualizatoin host, the virtual machines running on top of such
host will still work with VLANs enabled.
CC: Patrick McHardy <kaber@trash.net>
CC: Nithin Nayak Sujir <nsujir@broadcom.com>
CC: Michael Chan <mchan@broadcom.com>
CC: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Vladislav Yasevich <vyasevic@redhat.com>
Acked-by: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-09 02:42:13 +08:00
|
|
|
static struct sk_buff *skb_reorder_vlan_header(struct sk_buff *skb)
|
|
|
|
{
|
2019-04-16 09:22:28 +08:00
|
|
|
int mac_len, meta_len;
|
|
|
|
void *meta;
|
2018-03-13 13:51:27 +08:00
|
|
|
|
net: Always untag vlan-tagged traffic on input.
Currently the functionality to untag traffic on input resides
as part of the vlan module and is build only when VLAN support
is enabled in the kernel. When VLAN is disabled, the function
vlan_untag() turns into a stub and doesn't really untag the
packets. This seems to create an interesting interaction
between VMs supporting checksum offloading and some network drivers.
There are some drivers that do not allow the user to change
tx-vlan-offload feature of the driver. These drivers also seem
to assume that any VLAN-tagged traffic they transmit will
have the vlan information in the vlan_tci and not in the vlan
header already in the skb. When transmitting skbs that already
have tagged data with partial checksum set, the checksum doesn't
appear to be updated correctly by the card thus resulting in a
failure to establish TCP connections.
The following is a packet trace taken on the receiver where a
sender is a VM with a VLAN configued. The host VM is running on
doest not have VLAN support and the outging interface on the
host is tg3:
10:12:43.503055 52:54:00:ae:42:3f > 28:d2:44:7d:c2:de, ethertype 802.1Q
(0x8100), length 78: vlan 100, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 27243,
offset 0, flags [DF], proto TCP (6), length 60)
10.0.100.1.58545 > 10.0.100.10.ircu-2: Flags [S], cksum 0xdc39 (incorrect
-> 0x48d9), seq 1069378582, win 29200, options [mss 1460,sackOK,TS val
4294837885 ecr 0,nop,wscale 7], length 0
10:12:44.505556 52:54:00:ae:42:3f > 28:d2:44:7d:c2:de, ethertype 802.1Q
(0x8100), length 78: vlan 100, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 27244,
offset 0, flags [DF], proto TCP (6), length 60)
10.0.100.1.58545 > 10.0.100.10.ircu-2: Flags [S], cksum 0xdc39 (incorrect
-> 0x44ee), seq 1069378582, win 29200, options [mss 1460,sackOK,TS val
4294838888 ecr 0,nop,wscale 7], length 0
This connection finally times out.
I've only access to the TG3 hardware in this configuration thus have
only tested this with TG3 driver. There are a lot of other drivers
that do not permit user changes to vlan acceleration features, and
I don't know if they all suffere from a similar issue.
The patch attempt to fix this another way. It moves the vlan header
stipping code out of the vlan module and always builds it into the
kernel network core. This way, even if vlan is not supported on
a virtualizatoin host, the virtual machines running on top of such
host will still work with VLANs enabled.
CC: Patrick McHardy <kaber@trash.net>
CC: Nithin Nayak Sujir <nsujir@broadcom.com>
CC: Michael Chan <mchan@broadcom.com>
CC: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Vladislav Yasevich <vyasevic@redhat.com>
Acked-by: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-09 02:42:13 +08:00
|
|
|
if (skb_cow(skb, skb_headroom(skb)) < 0) {
|
|
|
|
kfree_skb(skb);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
2018-03-13 13:51:27 +08:00
|
|
|
mac_len = skb->data - skb_mac_header(skb);
|
2018-03-29 18:05:29 +08:00
|
|
|
if (likely(mac_len > VLAN_HLEN + ETH_TLEN)) {
|
|
|
|
memmove(skb_mac_header(skb) + VLAN_HLEN, skb_mac_header(skb),
|
|
|
|
mac_len - VLAN_HLEN - ETH_TLEN);
|
|
|
|
}
|
2019-04-16 09:22:28 +08:00
|
|
|
|
|
|
|
meta_len = skb_metadata_len(skb);
|
|
|
|
if (meta_len) {
|
|
|
|
meta = skb_metadata_end(skb) - meta_len;
|
|
|
|
memmove(meta + VLAN_HLEN, meta, meta_len);
|
|
|
|
}
|
|
|
|
|
net: Always untag vlan-tagged traffic on input.
Currently the functionality to untag traffic on input resides
as part of the vlan module and is build only when VLAN support
is enabled in the kernel. When VLAN is disabled, the function
vlan_untag() turns into a stub and doesn't really untag the
packets. This seems to create an interesting interaction
between VMs supporting checksum offloading and some network drivers.
There are some drivers that do not allow the user to change
tx-vlan-offload feature of the driver. These drivers also seem
to assume that any VLAN-tagged traffic they transmit will
have the vlan information in the vlan_tci and not in the vlan
header already in the skb. When transmitting skbs that already
have tagged data with partial checksum set, the checksum doesn't
appear to be updated correctly by the card thus resulting in a
failure to establish TCP connections.
The following is a packet trace taken on the receiver where a
sender is a VM with a VLAN configued. The host VM is running on
doest not have VLAN support and the outging interface on the
host is tg3:
10:12:43.503055 52:54:00:ae:42:3f > 28:d2:44:7d:c2:de, ethertype 802.1Q
(0x8100), length 78: vlan 100, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 27243,
offset 0, flags [DF], proto TCP (6), length 60)
10.0.100.1.58545 > 10.0.100.10.ircu-2: Flags [S], cksum 0xdc39 (incorrect
-> 0x48d9), seq 1069378582, win 29200, options [mss 1460,sackOK,TS val
4294837885 ecr 0,nop,wscale 7], length 0
10:12:44.505556 52:54:00:ae:42:3f > 28:d2:44:7d:c2:de, ethertype 802.1Q
(0x8100), length 78: vlan 100, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 27244,
offset 0, flags [DF], proto TCP (6), length 60)
10.0.100.1.58545 > 10.0.100.10.ircu-2: Flags [S], cksum 0xdc39 (incorrect
-> 0x44ee), seq 1069378582, win 29200, options [mss 1460,sackOK,TS val
4294838888 ecr 0,nop,wscale 7], length 0
This connection finally times out.
I've only access to the TG3 hardware in this configuration thus have
only tested this with TG3 driver. There are a lot of other drivers
that do not permit user changes to vlan acceleration features, and
I don't know if they all suffere from a similar issue.
The patch attempt to fix this another way. It moves the vlan header
stipping code out of the vlan module and always builds it into the
kernel network core. This way, even if vlan is not supported on
a virtualizatoin host, the virtual machines running on top of such
host will still work with VLANs enabled.
CC: Patrick McHardy <kaber@trash.net>
CC: Nithin Nayak Sujir <nsujir@broadcom.com>
CC: Michael Chan <mchan@broadcom.com>
CC: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Vladislav Yasevich <vyasevic@redhat.com>
Acked-by: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-09 02:42:13 +08:00
|
|
|
skb->mac_header += VLAN_HLEN;
|
|
|
|
return skb;
|
|
|
|
}
|
|
|
|
|
|
|
|
struct sk_buff *skb_vlan_untag(struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
struct vlan_hdr *vhdr;
|
|
|
|
u16 vlan_tci;
|
|
|
|
|
2015-01-14 00:13:44 +08:00
|
|
|
if (unlikely(skb_vlan_tag_present(skb))) {
|
net: Always untag vlan-tagged traffic on input.
Currently the functionality to untag traffic on input resides
as part of the vlan module and is build only when VLAN support
is enabled in the kernel. When VLAN is disabled, the function
vlan_untag() turns into a stub and doesn't really untag the
packets. This seems to create an interesting interaction
between VMs supporting checksum offloading and some network drivers.
There are some drivers that do not allow the user to change
tx-vlan-offload feature of the driver. These drivers also seem
to assume that any VLAN-tagged traffic they transmit will
have the vlan information in the vlan_tci and not in the vlan
header already in the skb. When transmitting skbs that already
have tagged data with partial checksum set, the checksum doesn't
appear to be updated correctly by the card thus resulting in a
failure to establish TCP connections.
The following is a packet trace taken on the receiver where a
sender is a VM with a VLAN configued. The host VM is running on
doest not have VLAN support and the outging interface on the
host is tg3:
10:12:43.503055 52:54:00:ae:42:3f > 28:d2:44:7d:c2:de, ethertype 802.1Q
(0x8100), length 78: vlan 100, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 27243,
offset 0, flags [DF], proto TCP (6), length 60)
10.0.100.1.58545 > 10.0.100.10.ircu-2: Flags [S], cksum 0xdc39 (incorrect
-> 0x48d9), seq 1069378582, win 29200, options [mss 1460,sackOK,TS val
4294837885 ecr 0,nop,wscale 7], length 0
10:12:44.505556 52:54:00:ae:42:3f > 28:d2:44:7d:c2:de, ethertype 802.1Q
(0x8100), length 78: vlan 100, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 27244,
offset 0, flags [DF], proto TCP (6), length 60)
10.0.100.1.58545 > 10.0.100.10.ircu-2: Flags [S], cksum 0xdc39 (incorrect
-> 0x44ee), seq 1069378582, win 29200, options [mss 1460,sackOK,TS val
4294838888 ecr 0,nop,wscale 7], length 0
This connection finally times out.
I've only access to the TG3 hardware in this configuration thus have
only tested this with TG3 driver. There are a lot of other drivers
that do not permit user changes to vlan acceleration features, and
I don't know if they all suffere from a similar issue.
The patch attempt to fix this another way. It moves the vlan header
stipping code out of the vlan module and always builds it into the
kernel network core. This way, even if vlan is not supported on
a virtualizatoin host, the virtual machines running on top of such
host will still work with VLANs enabled.
CC: Patrick McHardy <kaber@trash.net>
CC: Nithin Nayak Sujir <nsujir@broadcom.com>
CC: Michael Chan <mchan@broadcom.com>
CC: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Vladislav Yasevich <vyasevic@redhat.com>
Acked-by: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-09 02:42:13 +08:00
|
|
|
/* vlan_tci is already set-up so leave this for another time */
|
|
|
|
return skb;
|
|
|
|
}
|
|
|
|
|
|
|
|
skb = skb_share_check(skb, GFP_ATOMIC);
|
|
|
|
if (unlikely(!skb))
|
|
|
|
goto err_free;
|
2020-08-15 16:44:31 +08:00
|
|
|
/* We may access the two bytes after vlan_hdr in vlan_set_encap_proto(). */
|
|
|
|
if (unlikely(!pskb_may_pull(skb, VLAN_HLEN + sizeof(unsigned short))))
|
net: Always untag vlan-tagged traffic on input.
Currently the functionality to untag traffic on input resides
as part of the vlan module and is build only when VLAN support
is enabled in the kernel. When VLAN is disabled, the function
vlan_untag() turns into a stub and doesn't really untag the
packets. This seems to create an interesting interaction
between VMs supporting checksum offloading and some network drivers.
There are some drivers that do not allow the user to change
tx-vlan-offload feature of the driver. These drivers also seem
to assume that any VLAN-tagged traffic they transmit will
have the vlan information in the vlan_tci and not in the vlan
header already in the skb. When transmitting skbs that already
have tagged data with partial checksum set, the checksum doesn't
appear to be updated correctly by the card thus resulting in a
failure to establish TCP connections.
The following is a packet trace taken on the receiver where a
sender is a VM with a VLAN configued. The host VM is running on
doest not have VLAN support and the outging interface on the
host is tg3:
10:12:43.503055 52:54:00:ae:42:3f > 28:d2:44:7d:c2:de, ethertype 802.1Q
(0x8100), length 78: vlan 100, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 27243,
offset 0, flags [DF], proto TCP (6), length 60)
10.0.100.1.58545 > 10.0.100.10.ircu-2: Flags [S], cksum 0xdc39 (incorrect
-> 0x48d9), seq 1069378582, win 29200, options [mss 1460,sackOK,TS val
4294837885 ecr 0,nop,wscale 7], length 0
10:12:44.505556 52:54:00:ae:42:3f > 28:d2:44:7d:c2:de, ethertype 802.1Q
(0x8100), length 78: vlan 100, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 27244,
offset 0, flags [DF], proto TCP (6), length 60)
10.0.100.1.58545 > 10.0.100.10.ircu-2: Flags [S], cksum 0xdc39 (incorrect
-> 0x44ee), seq 1069378582, win 29200, options [mss 1460,sackOK,TS val
4294838888 ecr 0,nop,wscale 7], length 0
This connection finally times out.
I've only access to the TG3 hardware in this configuration thus have
only tested this with TG3 driver. There are a lot of other drivers
that do not permit user changes to vlan acceleration features, and
I don't know if they all suffere from a similar issue.
The patch attempt to fix this another way. It moves the vlan header
stipping code out of the vlan module and always builds it into the
kernel network core. This way, even if vlan is not supported on
a virtualizatoin host, the virtual machines running on top of such
host will still work with VLANs enabled.
CC: Patrick McHardy <kaber@trash.net>
CC: Nithin Nayak Sujir <nsujir@broadcom.com>
CC: Michael Chan <mchan@broadcom.com>
CC: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Vladislav Yasevich <vyasevic@redhat.com>
Acked-by: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-09 02:42:13 +08:00
|
|
|
goto err_free;
|
|
|
|
|
|
|
|
vhdr = (struct vlan_hdr *)skb->data;
|
|
|
|
vlan_tci = ntohs(vhdr->h_vlan_TCI);
|
|
|
|
__vlan_hwaccel_put_tag(skb, skb->protocol, vlan_tci);
|
|
|
|
|
|
|
|
skb_pull_rcsum(skb, VLAN_HLEN);
|
|
|
|
vlan_set_encap_proto(skb, vhdr);
|
|
|
|
|
|
|
|
skb = skb_reorder_vlan_header(skb);
|
|
|
|
if (unlikely(!skb))
|
|
|
|
goto err_free;
|
|
|
|
|
|
|
|
skb_reset_network_header(skb);
|
2020-11-10 07:47:23 +08:00
|
|
|
if (!skb_transport_header_was_set(skb))
|
|
|
|
skb_reset_transport_header(skb);
|
net: Always untag vlan-tagged traffic on input.
Currently the functionality to untag traffic on input resides
as part of the vlan module and is build only when VLAN support
is enabled in the kernel. When VLAN is disabled, the function
vlan_untag() turns into a stub and doesn't really untag the
packets. This seems to create an interesting interaction
between VMs supporting checksum offloading and some network drivers.
There are some drivers that do not allow the user to change
tx-vlan-offload feature of the driver. These drivers also seem
to assume that any VLAN-tagged traffic they transmit will
have the vlan information in the vlan_tci and not in the vlan
header already in the skb. When transmitting skbs that already
have tagged data with partial checksum set, the checksum doesn't
appear to be updated correctly by the card thus resulting in a
failure to establish TCP connections.
The following is a packet trace taken on the receiver where a
sender is a VM with a VLAN configued. The host VM is running on
doest not have VLAN support and the outging interface on the
host is tg3:
10:12:43.503055 52:54:00:ae:42:3f > 28:d2:44:7d:c2:de, ethertype 802.1Q
(0x8100), length 78: vlan 100, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 27243,
offset 0, flags [DF], proto TCP (6), length 60)
10.0.100.1.58545 > 10.0.100.10.ircu-2: Flags [S], cksum 0xdc39 (incorrect
-> 0x48d9), seq 1069378582, win 29200, options [mss 1460,sackOK,TS val
4294837885 ecr 0,nop,wscale 7], length 0
10:12:44.505556 52:54:00:ae:42:3f > 28:d2:44:7d:c2:de, ethertype 802.1Q
(0x8100), length 78: vlan 100, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 27244,
offset 0, flags [DF], proto TCP (6), length 60)
10.0.100.1.58545 > 10.0.100.10.ircu-2: Flags [S], cksum 0xdc39 (incorrect
-> 0x44ee), seq 1069378582, win 29200, options [mss 1460,sackOK,TS val
4294838888 ecr 0,nop,wscale 7], length 0
This connection finally times out.
I've only access to the TG3 hardware in this configuration thus have
only tested this with TG3 driver. There are a lot of other drivers
that do not permit user changes to vlan acceleration features, and
I don't know if they all suffere from a similar issue.
The patch attempt to fix this another way. It moves the vlan header
stipping code out of the vlan module and always builds it into the
kernel network core. This way, even if vlan is not supported on
a virtualizatoin host, the virtual machines running on top of such
host will still work with VLANs enabled.
CC: Patrick McHardy <kaber@trash.net>
CC: Nithin Nayak Sujir <nsujir@broadcom.com>
CC: Michael Chan <mchan@broadcom.com>
CC: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Vladislav Yasevich <vyasevic@redhat.com>
Acked-by: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-09 02:42:13 +08:00
|
|
|
skb_reset_mac_len(skb);
|
|
|
|
|
|
|
|
return skb;
|
|
|
|
|
|
|
|
err_free:
|
|
|
|
kfree_skb(skb);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_vlan_untag);
|
2014-09-17 19:49:49 +08:00
|
|
|
|
2014-11-19 21:05:01 +08:00
|
|
|
int skb_ensure_writable(struct sk_buff *skb, int write_len)
|
|
|
|
{
|
|
|
|
if (!pskb_may_pull(skb, write_len))
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
if (!skb_cloned(skb) || skb_clone_writable(skb, write_len))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
return pskb_expand_head(skb, 0, 0, GFP_ATOMIC);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_ensure_writable);
|
|
|
|
|
2016-09-20 00:11:09 +08:00
|
|
|
/* remove VLAN header from packet and update csum accordingly.
|
|
|
|
* expects a non skb_vlan_tag_present skb with a vlan tag payload
|
|
|
|
*/
|
|
|
|
int __skb_vlan_pop(struct sk_buff *skb, u16 *vlan_tci)
|
2014-11-19 21:05:02 +08:00
|
|
|
{
|
|
|
|
struct vlan_hdr *vhdr;
|
2016-09-29 17:10:41 +08:00
|
|
|
int offset = skb->data - skb_mac_header(skb);
|
2014-11-19 21:05:02 +08:00
|
|
|
int err;
|
|
|
|
|
2016-09-29 17:10:41 +08:00
|
|
|
if (WARN_ONCE(offset,
|
|
|
|
"__skb_vlan_pop got skb with skb->data not at mac header (offset %d)\n",
|
|
|
|
offset)) {
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
2014-11-19 21:05:02 +08:00
|
|
|
err = skb_ensure_writable(skb, VLAN_ETH_HLEN);
|
|
|
|
if (unlikely(err))
|
2016-09-29 17:10:41 +08:00
|
|
|
return err;
|
2014-11-19 21:05:02 +08:00
|
|
|
|
|
|
|
skb_postpull_rcsum(skb, skb->data + (2 * ETH_ALEN), VLAN_HLEN);
|
|
|
|
|
|
|
|
vhdr = (struct vlan_hdr *)(skb->data + ETH_HLEN);
|
|
|
|
*vlan_tci = ntohs(vhdr->h_vlan_TCI);
|
|
|
|
|
|
|
|
memmove(skb->data + VLAN_HLEN, skb->data, 2 * ETH_ALEN);
|
|
|
|
__skb_pull(skb, VLAN_HLEN);
|
|
|
|
|
|
|
|
vlan_set_encap_proto(skb, vhdr);
|
|
|
|
skb->mac_header += VLAN_HLEN;
|
|
|
|
|
|
|
|
if (skb_network_offset(skb) < ETH_HLEN)
|
|
|
|
skb_set_network_header(skb, ETH_HLEN);
|
|
|
|
|
|
|
|
skb_reset_mac_len(skb);
|
|
|
|
|
|
|
|
return err;
|
|
|
|
}
|
2016-09-20 00:11:09 +08:00
|
|
|
EXPORT_SYMBOL(__skb_vlan_pop);
|
2014-11-19 21:05:02 +08:00
|
|
|
|
2016-09-29 17:10:41 +08:00
|
|
|
/* Pop a vlan tag either from hwaccel or from payload.
|
|
|
|
* Expects skb->data at mac header.
|
|
|
|
*/
|
2014-11-19 21:05:02 +08:00
|
|
|
int skb_vlan_pop(struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
u16 vlan_tci;
|
|
|
|
__be16 vlan_proto;
|
|
|
|
int err;
|
|
|
|
|
2015-01-14 00:13:44 +08:00
|
|
|
if (likely(skb_vlan_tag_present(skb))) {
|
2018-11-09 07:18:02 +08:00
|
|
|
__vlan_hwaccel_clear_tag(skb);
|
2014-11-19 21:05:02 +08:00
|
|
|
} else {
|
2016-09-20 17:48:37 +08:00
|
|
|
if (unlikely(!eth_type_vlan(skb->protocol)))
|
2014-11-19 21:05:02 +08:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
err = __skb_vlan_pop(skb, &vlan_tci);
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
/* move next vlan tag to hw accel tag */
|
2016-09-20 17:48:37 +08:00
|
|
|
if (likely(!eth_type_vlan(skb->protocol)))
|
2014-11-19 21:05:02 +08:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
vlan_proto = skb->protocol;
|
|
|
|
err = __skb_vlan_pop(skb, &vlan_tci);
|
|
|
|
if (unlikely(err))
|
|
|
|
return err;
|
|
|
|
|
|
|
|
__vlan_hwaccel_put_tag(skb, vlan_proto, vlan_tci);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_vlan_pop);
|
|
|
|
|
2016-09-29 17:10:41 +08:00
|
|
|
/* Push a vlan tag either into hwaccel or into payload (if hwaccel tag present).
|
|
|
|
* Expects skb->data at mac header.
|
|
|
|
*/
|
2014-11-19 21:05:02 +08:00
|
|
|
int skb_vlan_push(struct sk_buff *skb, __be16 vlan_proto, u16 vlan_tci)
|
|
|
|
{
|
2015-01-14 00:13:44 +08:00
|
|
|
if (skb_vlan_tag_present(skb)) {
|
2016-09-29 17:10:41 +08:00
|
|
|
int offset = skb->data - skb_mac_header(skb);
|
2014-11-19 21:05:02 +08:00
|
|
|
int err;
|
|
|
|
|
2016-09-29 17:10:41 +08:00
|
|
|
if (WARN_ONCE(offset,
|
|
|
|
"skb_vlan_push got skb with skb->data not at mac header (offset %d)\n",
|
|
|
|
offset)) {
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
2014-11-19 21:05:02 +08:00
|
|
|
err = __vlan_insert_tag(skb, skb->vlan_proto,
|
2015-01-14 00:13:44 +08:00
|
|
|
skb_vlan_tag_get(skb));
|
2016-09-29 17:10:41 +08:00
|
|
|
if (err)
|
2014-11-19 21:05:02 +08:00
|
|
|
return err;
|
2016-04-16 08:27:58 +08:00
|
|
|
|
2014-11-19 21:05:02 +08:00
|
|
|
skb->protocol = skb->vlan_proto;
|
|
|
|
skb->mac_len += VLAN_HLEN;
|
|
|
|
|
2016-02-20 07:29:30 +08:00
|
|
|
skb_postpush_rcsum(skb, skb->data + (2 * ETH_ALEN), VLAN_HLEN);
|
2014-11-19 21:05:02 +08:00
|
|
|
}
|
|
|
|
__vlan_hwaccel_put_tag(skb, vlan_proto, vlan_tci);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_vlan_push);
|
|
|
|
|
2020-10-03 06:44:28 +08:00
|
|
|
/**
|
|
|
|
* skb_eth_pop() - Drop the Ethernet header at the head of a packet
|
|
|
|
*
|
|
|
|
* @skb: Socket buffer to modify
|
|
|
|
*
|
|
|
|
* Drop the Ethernet header of @skb.
|
|
|
|
*
|
|
|
|
* Expects that skb->data points to the mac header and that no VLAN tags are
|
|
|
|
* present.
|
|
|
|
*
|
|
|
|
* Returns 0 on success, -errno otherwise.
|
|
|
|
*/
|
|
|
|
int skb_eth_pop(struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
if (!pskb_may_pull(skb, ETH_HLEN) || skb_vlan_tagged(skb) ||
|
|
|
|
skb_network_offset(skb) < ETH_HLEN)
|
|
|
|
return -EPROTO;
|
|
|
|
|
|
|
|
skb_pull_rcsum(skb, ETH_HLEN);
|
|
|
|
skb_reset_mac_header(skb);
|
|
|
|
skb_reset_mac_len(skb);
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_eth_pop);
|
|
|
|
|
|
|
|
/**
|
|
|
|
* skb_eth_push() - Add a new Ethernet header at the head of a packet
|
|
|
|
*
|
|
|
|
* @skb: Socket buffer to modify
|
|
|
|
* @dst: Destination MAC address of the new header
|
|
|
|
* @src: Source MAC address of the new header
|
|
|
|
*
|
|
|
|
* Prepend @skb with a new Ethernet header.
|
|
|
|
*
|
|
|
|
* Expects that skb->data points to the mac header, which must be empty.
|
|
|
|
*
|
|
|
|
* Returns 0 on success, -errno otherwise.
|
|
|
|
*/
|
|
|
|
int skb_eth_push(struct sk_buff *skb, const unsigned char *dst,
|
|
|
|
const unsigned char *src)
|
|
|
|
{
|
|
|
|
struct ethhdr *eth;
|
|
|
|
int err;
|
|
|
|
|
|
|
|
if (skb_network_offset(skb) || skb_vlan_tag_present(skb))
|
|
|
|
return -EPROTO;
|
|
|
|
|
|
|
|
err = skb_cow_head(skb, sizeof(*eth));
|
|
|
|
if (err < 0)
|
|
|
|
return err;
|
|
|
|
|
|
|
|
skb_push(skb, sizeof(*eth));
|
|
|
|
skb_reset_mac_header(skb);
|
|
|
|
skb_reset_mac_len(skb);
|
|
|
|
|
|
|
|
eth = eth_hdr(skb);
|
|
|
|
ether_addr_copy(eth->h_dest, dst);
|
|
|
|
ether_addr_copy(eth->h_source, src);
|
|
|
|
eth->h_proto = skb->protocol;
|
|
|
|
|
|
|
|
skb_postpush_rcsum(skb, eth, sizeof(*eth));
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_eth_push);
|
|
|
|
|
2019-07-07 22:01:54 +08:00
|
|
|
/* Update the ethertype of hdr and the skb csum value if required. */
|
|
|
|
static void skb_mod_eth_type(struct sk_buff *skb, struct ethhdr *hdr,
|
|
|
|
__be16 ethertype)
|
|
|
|
{
|
|
|
|
if (skb->ip_summed == CHECKSUM_COMPLETE) {
|
|
|
|
__be16 diff[] = { ~hdr->h_proto, ethertype };
|
|
|
|
|
|
|
|
skb->csum = csum_partial((char *)diff, sizeof(diff), skb->csum);
|
|
|
|
}
|
|
|
|
|
|
|
|
hdr->h_proto = ethertype;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
2019-12-21 11:20:01 +08:00
|
|
|
* skb_mpls_push() - push a new MPLS header after mac_len bytes from start of
|
|
|
|
* the packet
|
2019-07-07 22:01:54 +08:00
|
|
|
*
|
|
|
|
* @skb: buffer
|
|
|
|
* @mpls_lse: MPLS label stack entry to push
|
|
|
|
* @mpls_proto: ethertype of the new MPLS header (expects 0x8847 or 0x8848)
|
2019-10-12 19:55:07 +08:00
|
|
|
* @mac_len: length of the MAC header
|
2019-12-21 11:20:01 +08:00
|
|
|
* @ethernet: flag to indicate if the resulting packet after skb_mpls_push is
|
|
|
|
* ethernet
|
2019-07-07 22:01:54 +08:00
|
|
|
*
|
|
|
|
* Expects skb->data at mac header.
|
|
|
|
*
|
|
|
|
* Returns 0 on success, -errno otherwise.
|
|
|
|
*/
|
2019-10-12 19:55:07 +08:00
|
|
|
int skb_mpls_push(struct sk_buff *skb, __be32 mpls_lse, __be16 mpls_proto,
|
net: Fixed updating of ethertype in skb_mpls_push()
The skb_mpls_push was not updating ethertype of an ethernet packet if
the packet was originally received from a non ARPHRD_ETHER device.
In the below OVS data path flow, since the device corresponding to
port 7 is an l3 device (ARPHRD_NONE) the skb_mpls_push function does
not update the ethertype of the packet even though the previous
push_eth action had added an ethernet header to the packet.
recirc_id(0),in_port(7),eth_type(0x0800),ipv4(tos=0/0xfc,ttl=64,frag=no),
actions:push_eth(src=00:00:00:00:00:00,dst=00:00:00:00:00:00),
push_mpls(label=13,tc=0,ttl=64,bos=1,eth_type=0x8847),4
Fixes: 8822e270d697 ("net: core: move push MPLS functionality from OvS to core helper")
Signed-off-by: Martin Varghese <martin.varghese@nokia.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2019-12-05 08:27:22 +08:00
|
|
|
int mac_len, bool ethernet)
|
2019-07-07 22:01:54 +08:00
|
|
|
{
|
|
|
|
struct mpls_shim_hdr *lse;
|
|
|
|
int err;
|
|
|
|
|
|
|
|
if (unlikely(!eth_p_mpls(mpls_proto)))
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
/* Networking stack does not allow simultaneous Tunnel and MPLS GSO. */
|
|
|
|
if (skb->encapsulation)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
err = skb_cow_head(skb, MPLS_HLEN);
|
|
|
|
if (unlikely(err))
|
|
|
|
return err;
|
|
|
|
|
|
|
|
if (!skb->inner_protocol) {
|
2019-12-21 11:20:01 +08:00
|
|
|
skb_set_inner_network_header(skb, skb_network_offset(skb));
|
2019-07-07 22:01:54 +08:00
|
|
|
skb_set_inner_protocol(skb, skb->protocol);
|
|
|
|
}
|
|
|
|
|
|
|
|
skb_push(skb, MPLS_HLEN);
|
|
|
|
memmove(skb_mac_header(skb) - MPLS_HLEN, skb_mac_header(skb),
|
2019-10-12 19:55:07 +08:00
|
|
|
mac_len);
|
2019-07-07 22:01:54 +08:00
|
|
|
skb_reset_mac_header(skb);
|
2019-10-12 19:55:07 +08:00
|
|
|
skb_set_network_header(skb, mac_len);
|
2019-12-21 11:20:01 +08:00
|
|
|
skb_reset_mac_len(skb);
|
2019-07-07 22:01:54 +08:00
|
|
|
|
|
|
|
lse = mpls_hdr(skb);
|
|
|
|
lse->label_stack_entry = mpls_lse;
|
|
|
|
skb_postpush_rcsum(skb, lse, MPLS_HLEN);
|
|
|
|
|
2020-10-03 03:53:08 +08:00
|
|
|
if (ethernet && mac_len >= ETH_HLEN)
|
2019-07-07 22:01:54 +08:00
|
|
|
skb_mod_eth_type(skb, eth_hdr(skb), mpls_proto);
|
|
|
|
skb->protocol = mpls_proto;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(skb_mpls_push);
|
|
|
|
|
2019-07-07 22:01:55 +08:00
|
|
|
/**
|
|
|
|
* skb_mpls_pop() - pop the outermost MPLS header
|
|
|
|
*
|
|
|
|
* @skb: buffer
|
|
|
|
* @next_proto: ethertype of header after popped MPLS header
|
2019-10-12 19:55:07 +08:00
|
|
|
* @mac_len: length of the MAC header
|
2019-12-21 11:20:23 +08:00
|
|
|
* @ethernet: flag to indicate if the packet is ethernet
|
2019-07-07 22:01:55 +08:00
|
|
|
*
|
|
|
|
* Expects skb->data at mac header.
|
|
|
|
*
|
|
|
|
* Returns 0 on success, -errno otherwise.
|
|
|
|
*/
|
Fixed updating of ethertype in function skb_mpls_pop
The skb_mpls_pop was not updating ethertype of an ethernet packet if the
packet was originally received from a non ARPHRD_ETHER device.
In the below OVS data path flow, since the device corresponding to port 7
is an l3 device (ARPHRD_NONE) the skb_mpls_pop function does not update
the ethertype of the packet even though the previous push_eth action had
added an ethernet header to the packet.
recirc_id(0),in_port(7),eth_type(0x8847),
mpls(label=12/0xfffff,tc=0/0,ttl=0/0x0,bos=1/1),
actions:push_eth(src=00:00:00:00:00:00,dst=00:00:00:00:00:00),
pop_mpls(eth_type=0x800),4
Fixes: ed246cee09b9 ("net: core: move pop MPLS functionality from OvS to core helper")
Signed-off-by: Martin Varghese <martin.varghese@nokia.com>
Acked-by: Pravin B Shelar <pshelar@ovn.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2019-12-02 13:19:51 +08:00
|
|
|
int skb_mpls_pop(struct sk_buff *skb, __be16 next_proto, int mac_len,
|
|
|
|
bool ethernet)
|
2019-07-07 22:01:55 +08:00
|
|
|
{
|
|
|
|
int err;
|
|
|
|
|
|
|
|
if (unlikely(!eth_p_mpls(skb->protocol)))
|
2019-10-12 19:55:06 +08:00
|
|
|
return 0;
|
2019-07-07 22:01:55 +08:00
|
|
|
|
2019-10-12 19:55:07 +08:00
|
|
|
err = skb_ensure_writable(skb, mac_len + MPLS_HLEN);
|
2019-07-07 22:01:55 +08:00
|
|
|
if (unlikely(err))
|
|
|
|
return err;
|
|
|
|
|
|
|
|
skb_postpull_rcsum(skb, mpls_hdr(skb), MPLS_HLEN);
|
|
|
|
memmove(skb_mac_header(skb) + MPLS_HLEN, skb_mac_header(skb),
|
2019-10-12 19:55:07 +08:00
|
|
|
mac_len);
|
2019-07-07 22:01:55 +08:00
|
|
|
|
|
|
|
__skb_pull(skb, MPLS_HLEN);
|
|
|
|
skb_reset_mac_header(skb);
|
2019-10-12 19:55:07 +08:00
|
|
|
skb_set_network_header(skb, mac_len);
|
2019-07-07 22:01:55 +08:00
|
|
|
|
2020-10-03 03:53:08 +08:00
|
|
|
if (ethernet && mac_len >= ETH_HLEN) {
|
2019-07-07 22:01:55 +08:00
|
|
|
struct ethhdr *hdr;
|
|
|
|
|
|
|
|
/* use mpls_hdr() to get ethertype to account for VLANs. */
|
|
|
|
hdr = (struct ethhdr *)((void *)mpls_hdr(skb) - ETH_HLEN);
|
|
|
|
skb_mod_eth_type(skb, hdr, next_proto);
|
|
|
|
}
|
|
|
|
skb->protocol = next_proto;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(skb_mpls_pop);
|
|
|
|
|
2019-07-07 22:01:56 +08:00
|
|
|
/**
|
|
|
|
* skb_mpls_update_lse() - modify outermost MPLS header and update csum
|
|
|
|
*
|
|
|
|
* @skb: buffer
|
|
|
|
* @mpls_lse: new MPLS label stack entry to update to
|
|
|
|
*
|
|
|
|
* Expects skb->data at mac header.
|
|
|
|
*
|
|
|
|
* Returns 0 on success, -errno otherwise.
|
|
|
|
*/
|
|
|
|
int skb_mpls_update_lse(struct sk_buff *skb, __be32 mpls_lse)
|
|
|
|
{
|
|
|
|
int err;
|
|
|
|
|
|
|
|
if (unlikely(!eth_p_mpls(skb->protocol)))
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
err = skb_ensure_writable(skb, skb->mac_len + MPLS_HLEN);
|
|
|
|
if (unlikely(err))
|
|
|
|
return err;
|
|
|
|
|
|
|
|
if (skb->ip_summed == CHECKSUM_COMPLETE) {
|
|
|
|
__be32 diff[] = { ~mpls_hdr(skb)->label_stack_entry, mpls_lse };
|
|
|
|
|
|
|
|
skb->csum = csum_partial((char *)diff, sizeof(diff), skb->csum);
|
|
|
|
}
|
|
|
|
|
|
|
|
mpls_hdr(skb)->label_stack_entry = mpls_lse;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(skb_mpls_update_lse);
|
|
|
|
|
2019-07-07 22:01:57 +08:00
|
|
|
/**
|
|
|
|
* skb_mpls_dec_ttl() - decrement the TTL of the outermost MPLS header
|
|
|
|
*
|
|
|
|
* @skb: buffer
|
|
|
|
*
|
|
|
|
* Expects skb->data at mac header.
|
|
|
|
*
|
|
|
|
* Returns 0 on success, -errno otherwise.
|
|
|
|
*/
|
|
|
|
int skb_mpls_dec_ttl(struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
u32 lse;
|
|
|
|
u8 ttl;
|
|
|
|
|
|
|
|
if (unlikely(!eth_p_mpls(skb->protocol)))
|
|
|
|
return -EINVAL;
|
|
|
|
|
2020-12-03 17:58:21 +08:00
|
|
|
if (!pskb_may_pull(skb, skb_network_offset(skb) + MPLS_HLEN))
|
|
|
|
return -ENOMEM;
|
|
|
|
|
2019-07-07 22:01:57 +08:00
|
|
|
lse = be32_to_cpu(mpls_hdr(skb)->label_stack_entry);
|
|
|
|
ttl = (lse & MPLS_LS_TTL_MASK) >> MPLS_LS_TTL_SHIFT;
|
|
|
|
if (!--ttl)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
lse &= ~MPLS_LS_TTL_MASK;
|
|
|
|
lse |= ttl << MPLS_LS_TTL_SHIFT;
|
|
|
|
|
|
|
|
return skb_mpls_update_lse(skb, cpu_to_be32(lse));
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(skb_mpls_dec_ttl);
|
|
|
|
|
2014-09-17 19:49:49 +08:00
|
|
|
/**
|
|
|
|
* alloc_skb_with_frags - allocate skb with page frags
|
|
|
|
*
|
2014-10-09 11:58:08 +08:00
|
|
|
* @header_len: size of linear part
|
|
|
|
* @data_len: needed length in frags
|
|
|
|
* @max_page_order: max page order desired.
|
|
|
|
* @errcode: pointer to error code if any
|
|
|
|
* @gfp_mask: allocation mask
|
2014-09-17 19:49:49 +08:00
|
|
|
*
|
|
|
|
* This can be used to allocate a paged skb, given a maximal order for frags.
|
|
|
|
*/
|
|
|
|
struct sk_buff *alloc_skb_with_frags(unsigned long header_len,
|
|
|
|
unsigned long data_len,
|
|
|
|
int max_page_order,
|
|
|
|
int *errcode,
|
|
|
|
gfp_t gfp_mask)
|
|
|
|
{
|
|
|
|
int npages = (data_len + (PAGE_SIZE - 1)) >> PAGE_SHIFT;
|
|
|
|
unsigned long chunk;
|
|
|
|
struct sk_buff *skb;
|
|
|
|
struct page *page;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
*errcode = -EMSGSIZE;
|
|
|
|
/* Note this test could be relaxed, if we succeed to allocate
|
|
|
|
* high order pages...
|
|
|
|
*/
|
|
|
|
if (npages > MAX_SKB_FRAGS)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
*errcode = -ENOBUFS;
|
net, skbuff: do not prefer skb allocation fails early
Commit dcda9b04713c ("mm, tree wide: replace __GFP_REPEAT by
__GFP_RETRY_MAYFAIL with more useful semantic") replaced __GFP_REPEAT in
alloc_skb_with_frags() with __GFP_RETRY_MAYFAIL when the allocation may
directly reclaim.
The previous behavior would require reclaim up to 1 << order pages for
skb aligned header_len of order > PAGE_ALLOC_COSTLY_ORDER before failing,
otherwise the allocations in alloc_skb() would loop in the page allocator
looking for memory. __GFP_RETRY_MAYFAIL makes both allocations failable
under memory pressure, including for the HEAD allocation.
This can cause, among many other things, write() to fail with ENOTCONN
during RPC when under memory pressure.
These allocations should succeed as they did previous to dcda9b04713c
even if it requires calling the oom killer and additional looping in the
page allocator to find memory. There is no way to specify the previous
behavior of __GFP_REPEAT, but it's unlikely to be necessary since the
previous behavior only guaranteed that 1 << order pages would be reclaimed
before failing for order > PAGE_ALLOC_COSTLY_ORDER. That reclaim is not
guaranteed to be contiguous memory, so repeating for such large orders is
usually not beneficial.
Removing the setting of __GFP_RETRY_MAYFAIL to restore the previous
behavior, specifically not allowing alloc_skb() to fail for small orders
and oom kill if necessary rather than allowing RPCs to fail.
Fixes: dcda9b04713c ("mm, tree wide: replace __GFP_REPEAT by __GFP_RETRY_MAYFAIL with more useful semantic")
Signed-off-by: David Rientjes <rientjes@google.com>
Reviewed-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2019-01-03 05:01:43 +08:00
|
|
|
skb = alloc_skb(header_len, gfp_mask);
|
2014-09-17 19:49:49 +08:00
|
|
|
if (!skb)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
skb->truesize += npages << PAGE_SHIFT;
|
|
|
|
|
|
|
|
for (i = 0; npages > 0; i++) {
|
|
|
|
int order = max_page_order;
|
|
|
|
|
|
|
|
while (order) {
|
|
|
|
if (npages >= 1 << order) {
|
2015-11-07 08:28:21 +08:00
|
|
|
page = alloc_pages((gfp_mask & ~__GFP_DIRECT_RECLAIM) |
|
2014-09-17 19:49:49 +08:00
|
|
|
__GFP_COMP |
|
2018-06-28 23:53:06 +08:00
|
|
|
__GFP_NOWARN,
|
2014-09-17 19:49:49 +08:00
|
|
|
order);
|
|
|
|
if (page)
|
|
|
|
goto fill_page;
|
|
|
|
/* Do not retry other high order allocations */
|
|
|
|
order = 1;
|
|
|
|
max_page_order = 0;
|
|
|
|
}
|
|
|
|
order--;
|
|
|
|
}
|
|
|
|
page = alloc_page(gfp_mask);
|
|
|
|
if (!page)
|
|
|
|
goto failure;
|
|
|
|
fill_page:
|
|
|
|
chunk = min_t(unsigned long, data_len,
|
|
|
|
PAGE_SIZE << order);
|
|
|
|
skb_fill_page_desc(skb, i, page, 0, chunk);
|
|
|
|
data_len -= chunk;
|
|
|
|
npages -= 1 << order;
|
|
|
|
}
|
|
|
|
return skb;
|
|
|
|
|
|
|
|
failure:
|
|
|
|
kfree_skb(skb);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(alloc_skb_with_frags);
|
skbuff: Add pskb_extract() helper function
A pattern of skb usage seen in modules such as RDS-TCP is to
extract `to_copy' bytes from the received TCP segment, starting
at some offset `off' into a new skb `clone'. This is done in
the ->data_ready callback, where the clone skb is queued up for rx on
the PF_RDS socket, while the parent TCP segment is returned unchanged
back to the TCP engine.
The existing code uses the sequence
clone = skb_clone(..);
pskb_pull(clone, off, ..);
pskb_trim(clone, to_copy, ..);
with the intention of discarding the first `off' bytes. However,
skb_clone() + pskb_pull() implies pksb_expand_head(), which ends
up doing a redundant memcpy of bytes that will then get discarded
in __pskb_pull_tail().
To avoid this inefficiency, this commit adds pskb_extract() that
creates the clone, and memcpy's only the relevant header/frag/frag_list
to the start of `clone'. pskb_trim() is then invoked to trim clone
down to the requested to_copy bytes.
Signed-off-by: Sowmini Varadhan <sowmini.varadhan@oracle.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-04-23 09:36:35 +08:00
|
|
|
|
|
|
|
/* carve out the first off bytes from skb when off < headlen */
|
|
|
|
static int pskb_carve_inside_header(struct sk_buff *skb, const u32 off,
|
|
|
|
const int headlen, gfp_t gfp_mask)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
int size = skb_end_offset(skb);
|
|
|
|
int new_hlen = headlen - off;
|
|
|
|
u8 *data;
|
|
|
|
|
|
|
|
size = SKB_DATA_ALIGN(size);
|
|
|
|
|
|
|
|
if (skb_pfmemalloc(skb))
|
|
|
|
gfp_mask |= __GFP_MEMALLOC;
|
|
|
|
data = kmalloc_reserve(size +
|
|
|
|
SKB_DATA_ALIGN(sizeof(struct skb_shared_info)),
|
|
|
|
gfp_mask, NUMA_NO_NODE, NULL);
|
|
|
|
if (!data)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
size = SKB_WITH_OVERHEAD(ksize(data));
|
|
|
|
|
|
|
|
/* Copy real data, and all frags */
|
|
|
|
skb_copy_from_linear_data_offset(skb, off, data, new_hlen);
|
|
|
|
skb->len -= off;
|
|
|
|
|
|
|
|
memcpy((struct skb_shared_info *)(data + size),
|
|
|
|
skb_shinfo(skb),
|
|
|
|
offsetof(struct skb_shared_info,
|
|
|
|
frags[skb_shinfo(skb)->nr_frags]));
|
|
|
|
if (skb_cloned(skb)) {
|
|
|
|
/* drop the old head gracefully */
|
|
|
|
if (skb_orphan_frags(skb, gfp_mask)) {
|
|
|
|
kfree(data);
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
for (i = 0; i < skb_shinfo(skb)->nr_frags; i++)
|
|
|
|
skb_frag_ref(skb, i);
|
|
|
|
if (skb_has_frag_list(skb))
|
|
|
|
skb_clone_fraglist(skb);
|
|
|
|
skb_release_data(skb);
|
|
|
|
} else {
|
|
|
|
/* we can reuse existing recount- all we did was
|
|
|
|
* relocate values
|
|
|
|
*/
|
|
|
|
skb_free_head(skb);
|
|
|
|
}
|
|
|
|
|
|
|
|
skb->head = data;
|
|
|
|
skb->data = data;
|
|
|
|
skb->head_frag = 0;
|
2022-02-22 11:21:12 +08:00
|
|
|
skb_set_end_offset(skb, size);
|
skbuff: Add pskb_extract() helper function
A pattern of skb usage seen in modules such as RDS-TCP is to
extract `to_copy' bytes from the received TCP segment, starting
at some offset `off' into a new skb `clone'. This is done in
the ->data_ready callback, where the clone skb is queued up for rx on
the PF_RDS socket, while the parent TCP segment is returned unchanged
back to the TCP engine.
The existing code uses the sequence
clone = skb_clone(..);
pskb_pull(clone, off, ..);
pskb_trim(clone, to_copy, ..);
with the intention of discarding the first `off' bytes. However,
skb_clone() + pskb_pull() implies pksb_expand_head(), which ends
up doing a redundant memcpy of bytes that will then get discarded
in __pskb_pull_tail().
To avoid this inefficiency, this commit adds pskb_extract() that
creates the clone, and memcpy's only the relevant header/frag/frag_list
to the start of `clone'. pskb_trim() is then invoked to trim clone
down to the requested to_copy bytes.
Signed-off-by: Sowmini Varadhan <sowmini.varadhan@oracle.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-04-23 09:36:35 +08:00
|
|
|
skb_set_tail_pointer(skb, skb_headlen(skb));
|
|
|
|
skb_headers_offset_update(skb, 0);
|
|
|
|
skb->cloned = 0;
|
|
|
|
skb->hdr_len = 0;
|
|
|
|
skb->nohdr = 0;
|
|
|
|
atomic_set(&skb_shinfo(skb)->dataref, 1);
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int pskb_carve(struct sk_buff *skb, const u32 off, gfp_t gfp);
|
|
|
|
|
|
|
|
/* carve out the first eat bytes from skb's frag_list. May recurse into
|
|
|
|
* pskb_carve()
|
|
|
|
*/
|
|
|
|
static int pskb_carve_frag_list(struct sk_buff *skb,
|
|
|
|
struct skb_shared_info *shinfo, int eat,
|
|
|
|
gfp_t gfp_mask)
|
|
|
|
{
|
|
|
|
struct sk_buff *list = shinfo->frag_list;
|
|
|
|
struct sk_buff *clone = NULL;
|
|
|
|
struct sk_buff *insp = NULL;
|
|
|
|
|
|
|
|
do {
|
|
|
|
if (!list) {
|
|
|
|
pr_err("Not enough bytes to eat. Want %d\n", eat);
|
|
|
|
return -EFAULT;
|
|
|
|
}
|
|
|
|
if (list->len <= eat) {
|
|
|
|
/* Eaten as whole. */
|
|
|
|
eat -= list->len;
|
|
|
|
list = list->next;
|
|
|
|
insp = list;
|
|
|
|
} else {
|
|
|
|
/* Eaten partially. */
|
|
|
|
if (skb_shared(list)) {
|
|
|
|
clone = skb_clone(list, gfp_mask);
|
|
|
|
if (!clone)
|
|
|
|
return -ENOMEM;
|
|
|
|
insp = list->next;
|
|
|
|
list = clone;
|
|
|
|
} else {
|
|
|
|
/* This may be pulled without problems. */
|
|
|
|
insp = list;
|
|
|
|
}
|
|
|
|
if (pskb_carve(list, eat, gfp_mask) < 0) {
|
|
|
|
kfree_skb(clone);
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
} while (eat);
|
|
|
|
|
|
|
|
/* Free pulled out fragments. */
|
|
|
|
while ((list = shinfo->frag_list) != insp) {
|
|
|
|
shinfo->frag_list = list->next;
|
2022-02-20 23:40:52 +08:00
|
|
|
consume_skb(list);
|
skbuff: Add pskb_extract() helper function
A pattern of skb usage seen in modules such as RDS-TCP is to
extract `to_copy' bytes from the received TCP segment, starting
at some offset `off' into a new skb `clone'. This is done in
the ->data_ready callback, where the clone skb is queued up for rx on
the PF_RDS socket, while the parent TCP segment is returned unchanged
back to the TCP engine.
The existing code uses the sequence
clone = skb_clone(..);
pskb_pull(clone, off, ..);
pskb_trim(clone, to_copy, ..);
with the intention of discarding the first `off' bytes. However,
skb_clone() + pskb_pull() implies pksb_expand_head(), which ends
up doing a redundant memcpy of bytes that will then get discarded
in __pskb_pull_tail().
To avoid this inefficiency, this commit adds pskb_extract() that
creates the clone, and memcpy's only the relevant header/frag/frag_list
to the start of `clone'. pskb_trim() is then invoked to trim clone
down to the requested to_copy bytes.
Signed-off-by: Sowmini Varadhan <sowmini.varadhan@oracle.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-04-23 09:36:35 +08:00
|
|
|
}
|
|
|
|
/* And insert new clone at head. */
|
|
|
|
if (clone) {
|
|
|
|
clone->next = list;
|
|
|
|
shinfo->frag_list = clone;
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* carve off first len bytes from skb. Split line (off) is in the
|
|
|
|
* non-linear part of skb
|
|
|
|
*/
|
|
|
|
static int pskb_carve_inside_nonlinear(struct sk_buff *skb, const u32 off,
|
|
|
|
int pos, gfp_t gfp_mask)
|
|
|
|
{
|
|
|
|
int i, k = 0;
|
|
|
|
int size = skb_end_offset(skb);
|
|
|
|
u8 *data;
|
|
|
|
const int nfrags = skb_shinfo(skb)->nr_frags;
|
|
|
|
struct skb_shared_info *shinfo;
|
|
|
|
|
|
|
|
size = SKB_DATA_ALIGN(size);
|
|
|
|
|
|
|
|
if (skb_pfmemalloc(skb))
|
|
|
|
gfp_mask |= __GFP_MEMALLOC;
|
|
|
|
data = kmalloc_reserve(size +
|
|
|
|
SKB_DATA_ALIGN(sizeof(struct skb_shared_info)),
|
|
|
|
gfp_mask, NUMA_NO_NODE, NULL);
|
|
|
|
if (!data)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
size = SKB_WITH_OVERHEAD(ksize(data));
|
|
|
|
|
|
|
|
memcpy((struct skb_shared_info *)(data + size),
|
2020-08-15 16:48:53 +08:00
|
|
|
skb_shinfo(skb), offsetof(struct skb_shared_info, frags[0]));
|
skbuff: Add pskb_extract() helper function
A pattern of skb usage seen in modules such as RDS-TCP is to
extract `to_copy' bytes from the received TCP segment, starting
at some offset `off' into a new skb `clone'. This is done in
the ->data_ready callback, where the clone skb is queued up for rx on
the PF_RDS socket, while the parent TCP segment is returned unchanged
back to the TCP engine.
The existing code uses the sequence
clone = skb_clone(..);
pskb_pull(clone, off, ..);
pskb_trim(clone, to_copy, ..);
with the intention of discarding the first `off' bytes. However,
skb_clone() + pskb_pull() implies pksb_expand_head(), which ends
up doing a redundant memcpy of bytes that will then get discarded
in __pskb_pull_tail().
To avoid this inefficiency, this commit adds pskb_extract() that
creates the clone, and memcpy's only the relevant header/frag/frag_list
to the start of `clone'. pskb_trim() is then invoked to trim clone
down to the requested to_copy bytes.
Signed-off-by: Sowmini Varadhan <sowmini.varadhan@oracle.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-04-23 09:36:35 +08:00
|
|
|
if (skb_orphan_frags(skb, gfp_mask)) {
|
|
|
|
kfree(data);
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
shinfo = (struct skb_shared_info *)(data + size);
|
|
|
|
for (i = 0; i < nfrags; i++) {
|
|
|
|
int fsize = skb_frag_size(&skb_shinfo(skb)->frags[i]);
|
|
|
|
|
|
|
|
if (pos + fsize > off) {
|
|
|
|
shinfo->frags[k] = skb_shinfo(skb)->frags[i];
|
|
|
|
|
|
|
|
if (pos < off) {
|
|
|
|
/* Split frag.
|
|
|
|
* We have two variants in this case:
|
|
|
|
* 1. Move all the frag to the second
|
|
|
|
* part, if it is possible. F.e.
|
|
|
|
* this approach is mandatory for TUX,
|
|
|
|
* where splitting is expensive.
|
|
|
|
* 2. Split is accurately. We make this.
|
|
|
|
*/
|
2019-07-30 22:40:33 +08:00
|
|
|
skb_frag_off_add(&shinfo->frags[0], off - pos);
|
skbuff: Add pskb_extract() helper function
A pattern of skb usage seen in modules such as RDS-TCP is to
extract `to_copy' bytes from the received TCP segment, starting
at some offset `off' into a new skb `clone'. This is done in
the ->data_ready callback, where the clone skb is queued up for rx on
the PF_RDS socket, while the parent TCP segment is returned unchanged
back to the TCP engine.
The existing code uses the sequence
clone = skb_clone(..);
pskb_pull(clone, off, ..);
pskb_trim(clone, to_copy, ..);
with the intention of discarding the first `off' bytes. However,
skb_clone() + pskb_pull() implies pksb_expand_head(), which ends
up doing a redundant memcpy of bytes that will then get discarded
in __pskb_pull_tail().
To avoid this inefficiency, this commit adds pskb_extract() that
creates the clone, and memcpy's only the relevant header/frag/frag_list
to the start of `clone'. pskb_trim() is then invoked to trim clone
down to the requested to_copy bytes.
Signed-off-by: Sowmini Varadhan <sowmini.varadhan@oracle.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-04-23 09:36:35 +08:00
|
|
|
skb_frag_size_sub(&shinfo->frags[0], off - pos);
|
|
|
|
}
|
|
|
|
skb_frag_ref(skb, i);
|
|
|
|
k++;
|
|
|
|
}
|
|
|
|
pos += fsize;
|
|
|
|
}
|
|
|
|
shinfo->nr_frags = k;
|
|
|
|
if (skb_has_frag_list(skb))
|
|
|
|
skb_clone_fraglist(skb);
|
|
|
|
|
2020-08-15 16:46:41 +08:00
|
|
|
/* split line is in frag list */
|
|
|
|
if (k == 0 && pskb_carve_frag_list(skb, shinfo, off - pos, gfp_mask)) {
|
|
|
|
/* skb_frag_unref() is not needed here as shinfo->nr_frags = 0. */
|
|
|
|
if (skb_has_frag_list(skb))
|
|
|
|
kfree_skb_list(skb_shinfo(skb)->frag_list);
|
|
|
|
kfree(data);
|
|
|
|
return -ENOMEM;
|
skbuff: Add pskb_extract() helper function
A pattern of skb usage seen in modules such as RDS-TCP is to
extract `to_copy' bytes from the received TCP segment, starting
at some offset `off' into a new skb `clone'. This is done in
the ->data_ready callback, where the clone skb is queued up for rx on
the PF_RDS socket, while the parent TCP segment is returned unchanged
back to the TCP engine.
The existing code uses the sequence
clone = skb_clone(..);
pskb_pull(clone, off, ..);
pskb_trim(clone, to_copy, ..);
with the intention of discarding the first `off' bytes. However,
skb_clone() + pskb_pull() implies pksb_expand_head(), which ends
up doing a redundant memcpy of bytes that will then get discarded
in __pskb_pull_tail().
To avoid this inefficiency, this commit adds pskb_extract() that
creates the clone, and memcpy's only the relevant header/frag/frag_list
to the start of `clone'. pskb_trim() is then invoked to trim clone
down to the requested to_copy bytes.
Signed-off-by: Sowmini Varadhan <sowmini.varadhan@oracle.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-04-23 09:36:35 +08:00
|
|
|
}
|
|
|
|
skb_release_data(skb);
|
|
|
|
|
|
|
|
skb->head = data;
|
|
|
|
skb->head_frag = 0;
|
|
|
|
skb->data = data;
|
2022-02-22 11:21:12 +08:00
|
|
|
skb_set_end_offset(skb, size);
|
skbuff: Add pskb_extract() helper function
A pattern of skb usage seen in modules such as RDS-TCP is to
extract `to_copy' bytes from the received TCP segment, starting
at some offset `off' into a new skb `clone'. This is done in
the ->data_ready callback, where the clone skb is queued up for rx on
the PF_RDS socket, while the parent TCP segment is returned unchanged
back to the TCP engine.
The existing code uses the sequence
clone = skb_clone(..);
pskb_pull(clone, off, ..);
pskb_trim(clone, to_copy, ..);
with the intention of discarding the first `off' bytes. However,
skb_clone() + pskb_pull() implies pksb_expand_head(), which ends
up doing a redundant memcpy of bytes that will then get discarded
in __pskb_pull_tail().
To avoid this inefficiency, this commit adds pskb_extract() that
creates the clone, and memcpy's only the relevant header/frag/frag_list
to the start of `clone'. pskb_trim() is then invoked to trim clone
down to the requested to_copy bytes.
Signed-off-by: Sowmini Varadhan <sowmini.varadhan@oracle.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-04-23 09:36:35 +08:00
|
|
|
skb_reset_tail_pointer(skb);
|
|
|
|
skb_headers_offset_update(skb, 0);
|
|
|
|
skb->cloned = 0;
|
|
|
|
skb->hdr_len = 0;
|
|
|
|
skb->nohdr = 0;
|
|
|
|
skb->len -= off;
|
|
|
|
skb->data_len = skb->len;
|
|
|
|
atomic_set(&skb_shinfo(skb)->dataref, 1);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* remove len bytes from the beginning of the skb */
|
|
|
|
static int pskb_carve(struct sk_buff *skb, const u32 len, gfp_t gfp)
|
|
|
|
{
|
|
|
|
int headlen = skb_headlen(skb);
|
|
|
|
|
|
|
|
if (len < headlen)
|
|
|
|
return pskb_carve_inside_header(skb, len, headlen, gfp);
|
|
|
|
else
|
|
|
|
return pskb_carve_inside_nonlinear(skb, len, headlen, gfp);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Extract to_copy bytes starting at off from skb, and return this in
|
|
|
|
* a new skb
|
|
|
|
*/
|
|
|
|
struct sk_buff *pskb_extract(struct sk_buff *skb, int off,
|
|
|
|
int to_copy, gfp_t gfp)
|
|
|
|
{
|
|
|
|
struct sk_buff *clone = skb_clone(skb, gfp);
|
|
|
|
|
|
|
|
if (!clone)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
if (pskb_carve(clone, off, gfp) < 0 ||
|
|
|
|
pskb_trim(clone, to_copy)) {
|
|
|
|
kfree_skb(clone);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
return clone;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(pskb_extract);
|
udp: under rx pressure, try to condense skbs
Under UDP flood, many softirq producers try to add packets to
UDP receive queue, and one user thread is burning one cpu trying
to dequeue packets as fast as possible.
Two parts of the per packet cost are :
- copying payload from kernel space to user space,
- freeing memory pieces associated with skb.
If socket is under pressure, softirq handler(s) can try to pull in
skb->head the payload of the packet if it fits.
Meaning the softirq handler(s) can free/reuse the page fragment
immediately, instead of letting udp_recvmsg() do this hundreds of usec
later, possibly from another node.
Additional gains :
- We reduce skb->truesize and thus can store more packets per SO_RCVBUF
- We avoid cache line misses at copyout() time and consume_skb() time,
and avoid one put_page() with potential alien freeing on NUMA hosts.
This comes at the cost of a copy, bounded to available tail room, which
is usually small. (We might have to fix GRO_MAX_HEAD which looks bigger
than necessary)
This patch gave me about 5 % increase in throughput in my tests.
skb_condense() helper could probably used in other contexts.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Paolo Abeni <pabeni@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-12-08 01:19:33 +08:00
|
|
|
|
|
|
|
/**
|
|
|
|
* skb_condense - try to get rid of fragments/frag_list if possible
|
|
|
|
* @skb: buffer
|
|
|
|
*
|
|
|
|
* Can be used to save memory before skb is added to a busy queue.
|
|
|
|
* If packet has bytes in frags and enough tail room in skb->head,
|
|
|
|
* pull all of them, so that we can free the frags right now and adjust
|
|
|
|
* truesize.
|
|
|
|
* Notes:
|
|
|
|
* We do not reallocate skb->head thus can not fail.
|
|
|
|
* Caller must re-evaluate skb->truesize if needed.
|
|
|
|
*/
|
|
|
|
void skb_condense(struct sk_buff *skb)
|
|
|
|
{
|
2016-12-10 00:02:05 +08:00
|
|
|
if (skb->data_len) {
|
|
|
|
if (skb->data_len > skb->end - skb->tail ||
|
|
|
|
skb_cloned(skb))
|
|
|
|
return;
|
udp: under rx pressure, try to condense skbs
Under UDP flood, many softirq producers try to add packets to
UDP receive queue, and one user thread is burning one cpu trying
to dequeue packets as fast as possible.
Two parts of the per packet cost are :
- copying payload from kernel space to user space,
- freeing memory pieces associated with skb.
If socket is under pressure, softirq handler(s) can try to pull in
skb->head the payload of the packet if it fits.
Meaning the softirq handler(s) can free/reuse the page fragment
immediately, instead of letting udp_recvmsg() do this hundreds of usec
later, possibly from another node.
Additional gains :
- We reduce skb->truesize and thus can store more packets per SO_RCVBUF
- We avoid cache line misses at copyout() time and consume_skb() time,
and avoid one put_page() with potential alien freeing on NUMA hosts.
This comes at the cost of a copy, bounded to available tail room, which
is usually small. (We might have to fix GRO_MAX_HEAD which looks bigger
than necessary)
This patch gave me about 5 % increase in throughput in my tests.
skb_condense() helper could probably used in other contexts.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Paolo Abeni <pabeni@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-12-08 01:19:33 +08:00
|
|
|
|
2016-12-10 00:02:05 +08:00
|
|
|
/* Nice, we can free page frag(s) right now */
|
|
|
|
__pskb_pull_tail(skb, skb->data_len);
|
|
|
|
}
|
|
|
|
/* At this point, skb->truesize might be over estimated,
|
|
|
|
* because skb had a fragment, and fragments do not tell
|
|
|
|
* their truesize.
|
|
|
|
* When we pulled its content into skb->head, fragment
|
|
|
|
* was freed, but __pskb_pull_tail() could not possibly
|
|
|
|
* adjust skb->truesize, not knowing the frag truesize.
|
udp: under rx pressure, try to condense skbs
Under UDP flood, many softirq producers try to add packets to
UDP receive queue, and one user thread is burning one cpu trying
to dequeue packets as fast as possible.
Two parts of the per packet cost are :
- copying payload from kernel space to user space,
- freeing memory pieces associated with skb.
If socket is under pressure, softirq handler(s) can try to pull in
skb->head the payload of the packet if it fits.
Meaning the softirq handler(s) can free/reuse the page fragment
immediately, instead of letting udp_recvmsg() do this hundreds of usec
later, possibly from another node.
Additional gains :
- We reduce skb->truesize and thus can store more packets per SO_RCVBUF
- We avoid cache line misses at copyout() time and consume_skb() time,
and avoid one put_page() with potential alien freeing on NUMA hosts.
This comes at the cost of a copy, bounded to available tail room, which
is usually small. (We might have to fix GRO_MAX_HEAD which looks bigger
than necessary)
This patch gave me about 5 % increase in throughput in my tests.
skb_condense() helper could probably used in other contexts.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Paolo Abeni <pabeni@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-12-08 01:19:33 +08:00
|
|
|
*/
|
|
|
|
skb->truesize = SKB_TRUESIZE(skb_end_offset(skb));
|
|
|
|
}
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
|
|
|
|
#ifdef CONFIG_SKB_EXTENSIONS
|
|
|
|
static void *skb_ext_get_ptr(struct skb_ext *ext, enum skb_ext_id id)
|
|
|
|
{
|
|
|
|
return (void *)ext + (ext->offset[id] * SKB_EXT_ALIGN_VALUE);
|
|
|
|
}
|
|
|
|
|
2020-01-09 23:59:24 +08:00
|
|
|
/**
|
|
|
|
* __skb_ext_alloc - allocate a new skb extensions storage
|
|
|
|
*
|
2020-05-16 16:46:23 +08:00
|
|
|
* @flags: See kmalloc().
|
|
|
|
*
|
2020-01-09 23:59:24 +08:00
|
|
|
* Returns the newly allocated pointer. The pointer can later attached to a
|
|
|
|
* skb via __skb_ext_set().
|
|
|
|
* Note: caller must handle the skb_ext as an opaque data.
|
|
|
|
*/
|
2020-05-16 16:46:23 +08:00
|
|
|
struct skb_ext *__skb_ext_alloc(gfp_t flags)
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
{
|
2020-05-16 16:46:23 +08:00
|
|
|
struct skb_ext *new = kmem_cache_alloc(skbuff_ext_cache, flags);
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
|
|
|
|
if (new) {
|
|
|
|
memset(new->offset, 0, sizeof(new->offset));
|
|
|
|
refcount_set(&new->refcnt, 1);
|
|
|
|
}
|
|
|
|
|
|
|
|
return new;
|
|
|
|
}
|
|
|
|
|
2018-12-19 00:15:27 +08:00
|
|
|
static struct skb_ext *skb_ext_maybe_cow(struct skb_ext *old,
|
|
|
|
unsigned int old_active)
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
{
|
|
|
|
struct skb_ext *new;
|
|
|
|
|
|
|
|
if (refcount_read(&old->refcnt) == 1)
|
|
|
|
return old;
|
|
|
|
|
|
|
|
new = kmem_cache_alloc(skbuff_ext_cache, GFP_ATOMIC);
|
|
|
|
if (!new)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
memcpy(new, old, old->chunks * SKB_EXT_ALIGN_VALUE);
|
|
|
|
refcount_set(&new->refcnt, 1);
|
|
|
|
|
2018-12-19 00:15:27 +08:00
|
|
|
#ifdef CONFIG_XFRM
|
|
|
|
if (old_active & (1 << SKB_EXT_SEC_PATH)) {
|
|
|
|
struct sec_path *sp = skb_ext_get_ptr(old, SKB_EXT_SEC_PATH);
|
|
|
|
unsigned int i;
|
|
|
|
|
|
|
|
for (i = 0; i < sp->len; i++)
|
|
|
|
xfrm_state_hold(sp->xvec[i]);
|
|
|
|
}
|
|
|
|
#endif
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
__skb_ext_put(old);
|
|
|
|
return new;
|
|
|
|
}
|
|
|
|
|
2020-01-09 23:59:24 +08:00
|
|
|
/**
|
|
|
|
* __skb_ext_set - attach the specified extension storage to this skb
|
|
|
|
* @skb: buffer
|
|
|
|
* @id: extension id
|
|
|
|
* @ext: extension storage previously allocated via __skb_ext_alloc()
|
|
|
|
*
|
|
|
|
* Existing extensions, if any, are cleared.
|
|
|
|
*
|
|
|
|
* Returns the pointer to the extension.
|
|
|
|
*/
|
|
|
|
void *__skb_ext_set(struct sk_buff *skb, enum skb_ext_id id,
|
|
|
|
struct skb_ext *ext)
|
|
|
|
{
|
|
|
|
unsigned int newlen, newoff = SKB_EXT_CHUNKSIZEOF(*ext);
|
|
|
|
|
|
|
|
skb_ext_put(skb);
|
|
|
|
newlen = newoff + skb_ext_type_len[id];
|
|
|
|
ext->chunks = newlen;
|
|
|
|
ext->offset[id] = newoff;
|
|
|
|
skb->extensions = ext;
|
|
|
|
skb->active_extensions = 1 << id;
|
|
|
|
return skb_ext_get_ptr(ext, id);
|
|
|
|
}
|
|
|
|
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
/**
|
|
|
|
* skb_ext_add - allocate space for given extension, COW if needed
|
|
|
|
* @skb: buffer
|
|
|
|
* @id: extension to allocate space for
|
|
|
|
*
|
|
|
|
* Allocates enough space for the given extension.
|
|
|
|
* If the extension is already present, a pointer to that extension
|
|
|
|
* is returned.
|
|
|
|
*
|
|
|
|
* If the skb was cloned, COW applies and the returned memory can be
|
|
|
|
* modified without changing the extension space of clones buffers.
|
|
|
|
*
|
|
|
|
* Returns pointer to the extension or NULL on allocation failure.
|
|
|
|
*/
|
|
|
|
void *skb_ext_add(struct sk_buff *skb, enum skb_ext_id id)
|
|
|
|
{
|
|
|
|
struct skb_ext *new, *old = NULL;
|
|
|
|
unsigned int newlen, newoff;
|
|
|
|
|
|
|
|
if (skb->active_extensions) {
|
|
|
|
old = skb->extensions;
|
|
|
|
|
2018-12-19 00:15:27 +08:00
|
|
|
new = skb_ext_maybe_cow(old, skb->active_extensions);
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
if (!new)
|
|
|
|
return NULL;
|
|
|
|
|
2018-12-22 02:03:15 +08:00
|
|
|
if (__skb_ext_exist(new, id))
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
goto set_active;
|
|
|
|
|
2018-12-22 02:03:13 +08:00
|
|
|
newoff = new->chunks;
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
} else {
|
|
|
|
newoff = SKB_EXT_CHUNKSIZEOF(*new);
|
|
|
|
|
2020-05-16 16:46:23 +08:00
|
|
|
new = __skb_ext_alloc(GFP_ATOMIC);
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
if (!new)
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
newlen = newoff + skb_ext_type_len[id];
|
|
|
|
new->chunks = newlen;
|
|
|
|
new->offset[id] = newoff;
|
|
|
|
set_active:
|
2021-07-29 00:24:01 +08:00
|
|
|
skb->slow_gro = 1;
|
2018-12-22 02:03:15 +08:00
|
|
|
skb->extensions = new;
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
skb->active_extensions |= 1 << id;
|
|
|
|
return skb_ext_get_ptr(new, id);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(skb_ext_add);
|
|
|
|
|
2018-12-19 00:15:27 +08:00
|
|
|
#ifdef CONFIG_XFRM
|
|
|
|
static void skb_ext_put_sp(struct sec_path *sp)
|
|
|
|
{
|
|
|
|
unsigned int i;
|
|
|
|
|
|
|
|
for (i = 0; i < sp->len; i++)
|
|
|
|
xfrm_state_put(sp->xvec[i]);
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2021-10-29 11:01:44 +08:00
|
|
|
#ifdef CONFIG_MCTP_FLOWS
|
|
|
|
static void skb_ext_put_mctp(struct mctp_flow *flow)
|
|
|
|
{
|
|
|
|
if (flow->key)
|
|
|
|
mctp_key_unref(flow->key);
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
void __skb_ext_del(struct sk_buff *skb, enum skb_ext_id id)
|
|
|
|
{
|
|
|
|
struct skb_ext *ext = skb->extensions;
|
|
|
|
|
|
|
|
skb->active_extensions &= ~(1 << id);
|
|
|
|
if (skb->active_extensions == 0) {
|
|
|
|
skb->extensions = NULL;
|
|
|
|
__skb_ext_put(ext);
|
2018-12-19 00:15:27 +08:00
|
|
|
#ifdef CONFIG_XFRM
|
|
|
|
} else if (id == SKB_EXT_SEC_PATH &&
|
|
|
|
refcount_read(&ext->refcnt) == 1) {
|
|
|
|
struct sec_path *sp = skb_ext_get_ptr(ext, SKB_EXT_SEC_PATH);
|
|
|
|
|
|
|
|
skb_ext_put_sp(sp);
|
|
|
|
sp->len = 0;
|
|
|
|
#endif
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(__skb_ext_del);
|
|
|
|
|
|
|
|
void __skb_ext_put(struct skb_ext *ext)
|
|
|
|
{
|
|
|
|
/* If this is last clone, nothing can increment
|
|
|
|
* it after check passes. Avoids one atomic op.
|
|
|
|
*/
|
|
|
|
if (refcount_read(&ext->refcnt) == 1)
|
|
|
|
goto free_now;
|
|
|
|
|
|
|
|
if (!refcount_dec_and_test(&ext->refcnt))
|
|
|
|
return;
|
|
|
|
free_now:
|
2018-12-19 00:15:27 +08:00
|
|
|
#ifdef CONFIG_XFRM
|
|
|
|
if (__skb_ext_exist(ext, SKB_EXT_SEC_PATH))
|
|
|
|
skb_ext_put_sp(skb_ext_get_ptr(ext, SKB_EXT_SEC_PATH));
|
|
|
|
#endif
|
2021-10-29 11:01:44 +08:00
|
|
|
#ifdef CONFIG_MCTP_FLOWS
|
|
|
|
if (__skb_ext_exist(ext, SKB_EXT_MCTP))
|
|
|
|
skb_ext_put_mctp(skb_ext_get_ptr(ext, SKB_EXT_MCTP));
|
|
|
|
#endif
|
2018-12-19 00:15:27 +08:00
|
|
|
|
sk_buff: add skb extension infrastructure
This adds an optional extension infrastructure, with ispec (xfrm) and
bridge netfilter as first users.
objdiff shows no changes if kernel is built without xfrm and br_netfilter
support.
The third (planned future) user is Multipath TCP which is still
out-of-tree.
MPTCP needs to map logical mptcp sequence numbers to the tcp sequence
numbers used by individual subflows.
This DSS mapping is read/written from tcp option space on receive and
written to tcp option space on transmitted tcp packets that are part of
and MPTCP connection.
Extending skb_shared_info or adding a private data field to skb fclones
doesn't work for incoming skb, so a different DSS propagation method would
be required for the receive side.
mptcp has same requirements as secpath/bridge netfilter:
1. extension memory is released when the sk_buff is free'd.
2. data is shared after cloning an skb (clone inherits extension)
3. adding extension to an skb will COW the extension buffer if needed.
The "MPTCP upstreaming" effort adds SKB_EXT_MPTCP extension to store the
mapping for tx and rx processing.
Two new members are added to sk_buff:
1. 'active_extensions' byte (filling a hole), telling which extensions
are available for this skb.
This has two purposes.
a) avoids the need to initialize the pointer.
b) allows to "delete" an extension by clearing its bit
value in ->active_extensions.
While it would be possible to store the active_extensions byte
in the extension struct instead of sk_buff, there is one problem
with this:
When an extension has to be disabled, we can always clear the
bit in skb->active_extensions. But in case it would be stored in the
extension buffer itself, we might have to COW it first, if
we are dealing with a cloned skb. On kmalloc failure we would
be unable to turn an extension off.
2. extension pointer, located at the end of the sk_buff.
If the active_extensions byte is 0, the pointer is undefined,
it is not initialized on skb allocation.
This adds extra code to skb clone and free paths (to deal with
refcount/free of extension area) but this replaces similar code that
manages skb->nf_bridge and skb->sp structs in the followup patches of
the series.
It is possible to add support for extensions that are not preseved on
clones/copies.
To do this, it would be needed to define a bitmask of all extensions that
need copy/cow semantics, and change __skb_ext_copy() to check
->active_extensions & SKB_EXT_PRESERVE_ON_CLONE, then just set
->active_extensions to 0 on the new clone.
This isn't done here because all extensions that get added here
need the copy/cow semantics.
v2:
Allocate entire extension space using kmem_cache.
Upside is that this allows better tracking of used memory,
downside is that we will allocate more space than strictly needed in
most cases (its unlikely that all extensions are active/needed at same
time for same skb).
The allocated memory (except the small extension header) is not cleared,
so no additonal overhead aside from memory usage.
Avoid atomic_dec_and_test operation on skb_ext_put()
by using similar trick as kfree_skbmem() does with fclone_ref:
If recount is 1, there is no concurrent user and we can free right away.
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-12-19 00:15:16 +08:00
|
|
|
kmem_cache_free(skbuff_ext_cache, ext);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(__skb_ext_put);
|
|
|
|
#endif /* CONFIG_SKB_EXTENSIONS */
|