2005-04-17 06:20:36 +08:00
|
|
|
/*
|
|
|
|
* UDP over IPv6
|
2007-02-09 22:24:49 +08:00
|
|
|
* Linux INET6 implementation
|
2005-04-17 06:20:36 +08:00
|
|
|
*
|
|
|
|
* Authors:
|
2007-02-09 22:24:49 +08:00
|
|
|
* Pedro Roque <roque@di.fc.ul.pt>
|
2005-04-17 06:20:36 +08:00
|
|
|
*
|
|
|
|
* Based on linux/ipv4/udp.c
|
|
|
|
*
|
|
|
|
* Fixes:
|
|
|
|
* Hideaki YOSHIFUJI : sin6_scope_id support
|
|
|
|
* YOSHIFUJI Hideaki @USAGI and: Support IPV6_V6ONLY socket option, which
|
|
|
|
* Alexey Kuznetsov allow both IPv4 and IPv6 sockets to bind
|
|
|
|
* a single port at the same time.
|
|
|
|
* Kazunori MIYAZAWA @USAGI: change process style to use ip6_append_data
|
|
|
|
* YOSHIFUJI Hideaki @USAGI: convert /proc/net/udp6 to seq_file.
|
|
|
|
*
|
|
|
|
* This program is free software; you can redistribute it and/or
|
|
|
|
* modify it under the terms of the GNU General Public License
|
|
|
|
* as published by the Free Software Foundation; either version
|
|
|
|
* 2 of the License, or (at your option) any later version.
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include <linux/errno.h>
|
|
|
|
#include <linux/types.h>
|
|
|
|
#include <linux/socket.h>
|
|
|
|
#include <linux/sockios.h>
|
|
|
|
#include <linux/net.h>
|
|
|
|
#include <linux/in6.h>
|
|
|
|
#include <linux/netdevice.h>
|
|
|
|
#include <linux/if_arp.h>
|
|
|
|
#include <linux/ipv6.h>
|
|
|
|
#include <linux/icmpv6.h>
|
|
|
|
#include <linux/init.h>
|
2007-12-12 03:30:32 +08:00
|
|
|
#include <linux/module.h>
|
2005-12-14 15:16:37 +08:00
|
|
|
#include <linux/skbuff.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 16:04:11 +08:00
|
|
|
#include <linux/slab.h>
|
2005-04-17 06:20:36 +08:00
|
|
|
#include <asm/uaccess.h>
|
|
|
|
|
|
|
|
#include <net/ndisc.h>
|
|
|
|
#include <net/protocol.h>
|
|
|
|
#include <net/transp_v6.h>
|
|
|
|
#include <net/ip6_route.h>
|
|
|
|
#include <net/raw.h>
|
2005-08-10 11:08:28 +08:00
|
|
|
#include <net/tcp_states.h>
|
2005-04-17 06:20:36 +08:00
|
|
|
#include <net/ip6_checksum.h>
|
|
|
|
#include <net/xfrm.h>
|
2013-01-22 17:50:44 +08:00
|
|
|
#include <net/inet6_hashtables.h>
|
2013-07-10 22:13:17 +08:00
|
|
|
#include <net/busy_poll.h>
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
#include <linux/proc_fs.h>
|
|
|
|
#include <linux/seq_file.h>
|
2012-06-27 08:23:44 +08:00
|
|
|
#include <trace/events/skb.h>
|
2006-11-28 03:10:57 +08:00
|
|
|
#include "udp_impl.h"
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2015-03-19 05:05:33 +08:00
|
|
|
static u32 udp6_ehashfn(const struct net *net,
|
|
|
|
const struct in6_addr *laddr,
|
|
|
|
const u16 lport,
|
|
|
|
const struct in6_addr *faddr,
|
|
|
|
const __be16 fport)
|
2013-10-20 03:48:52 +08:00
|
|
|
{
|
2013-10-20 03:48:57 +08:00
|
|
|
static u32 udp6_ehash_secret __read_mostly;
|
|
|
|
static u32 udp_ipv6_hash_secret __read_mostly;
|
|
|
|
|
|
|
|
u32 lhash, fhash;
|
|
|
|
|
|
|
|
net_get_random_once(&udp6_ehash_secret,
|
|
|
|
sizeof(udp6_ehash_secret));
|
|
|
|
net_get_random_once(&udp_ipv6_hash_secret,
|
|
|
|
sizeof(udp_ipv6_hash_secret));
|
|
|
|
|
|
|
|
lhash = (__force u32)laddr->s6_addr32[3];
|
|
|
|
fhash = __ipv6_addr_jhash(faddr, udp_ipv6_hash_secret);
|
|
|
|
|
2013-10-20 03:48:52 +08:00
|
|
|
return __inet6_ehashfn(lhash, lport, fhash, fport,
|
2013-10-20 03:48:57 +08:00
|
|
|
udp_ipv6_hash_secret + net_hash_mix(net));
|
2013-10-20 03:48:52 +08:00
|
|
|
}
|
|
|
|
|
2009-03-25 00:24:51 +08:00
|
|
|
int ipv6_rcv_saddr_equal(const struct sock *sk, const struct sock *sk2)
|
|
|
|
{
|
|
|
|
const struct in6_addr *sk2_rcv_saddr6 = inet6_rcv_saddr(sk2);
|
|
|
|
int sk2_ipv6only = inet_v6_ipv6only(sk2);
|
ipv6: make lookups simpler and faster
TCP listener refactoring, part 4 :
To speed up inet lookups, we moved IPv4 addresses from inet to struct
sock_common
Now is time to do the same for IPv6, because it permits us to have fast
lookups for all kind of sockets, including upcoming SYN_RECV.
Getting IPv6 addresses in TCP lookups currently requires two extra cache
lines, plus a dereference (and memory stall).
inet6_sk(sk) does the dereference of inet_sk(__sk)->pinet6
This patch is way bigger than its IPv4 counter part, because for IPv4,
we could add aliases (inet_daddr, inet_rcv_saddr), while on IPv6,
it's not doable easily.
inet6_sk(sk)->daddr becomes sk->sk_v6_daddr
inet6_sk(sk)->rcv_saddr becomes sk->sk_v6_rcv_saddr
And timewait socket also have tw->tw_v6_daddr & tw->tw_v6_rcv_saddr
at the same offset.
We get rid of INET6_TW_MATCH() as INET6_MATCH() is now the generic
macro.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-10-04 06:42:29 +08:00
|
|
|
int addr_type = ipv6_addr_type(&sk->sk_v6_rcv_saddr);
|
2009-03-25 00:24:51 +08:00
|
|
|
int addr_type2 = sk2_rcv_saddr6 ? ipv6_addr_type(sk2_rcv_saddr6) : IPV6_ADDR_MAPPED;
|
|
|
|
|
|
|
|
/* if both are mapped, treat as IPv4 */
|
|
|
|
if (addr_type == IPV6_ADDR_MAPPED && addr_type2 == IPV6_ADDR_MAPPED)
|
2009-04-10 01:37:33 +08:00
|
|
|
return (!sk2_ipv6only &&
|
inet: consolidate INET_TW_MATCH
TCP listener refactoring, part 2 :
We can use a generic lookup, sockets being in whatever state, if
we are sure all relevant fields are at the same place in all socket
types (ESTABLISH, TIME_WAIT, SYN_RECV)
This patch removes these macros :
inet_addrpair, inet_addrpair, tw_addrpair, tw_portpair
And adds :
sk_portpair, sk_addrpair, sk_daddr, sk_rcv_saddr
Then, INET_TW_MATCH() is really the same than INET_MATCH()
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-10-02 19:29:50 +08:00
|
|
|
(!sk->sk_rcv_saddr || !sk2->sk_rcv_saddr ||
|
|
|
|
sk->sk_rcv_saddr == sk2->sk_rcv_saddr));
|
2009-03-25 00:24:51 +08:00
|
|
|
|
|
|
|
if (addr_type2 == IPV6_ADDR_ANY &&
|
|
|
|
!(sk2_ipv6only && addr_type == IPV6_ADDR_MAPPED))
|
|
|
|
return 1;
|
|
|
|
|
|
|
|
if (addr_type == IPV6_ADDR_ANY &&
|
2014-06-27 23:36:16 +08:00
|
|
|
!(ipv6_only_sock(sk) && addr_type2 == IPV6_ADDR_MAPPED))
|
2009-03-25 00:24:51 +08:00
|
|
|
return 1;
|
|
|
|
|
|
|
|
if (sk2_rcv_saddr6 &&
|
ipv6: make lookups simpler and faster
TCP listener refactoring, part 4 :
To speed up inet lookups, we moved IPv4 addresses from inet to struct
sock_common
Now is time to do the same for IPv6, because it permits us to have fast
lookups for all kind of sockets, including upcoming SYN_RECV.
Getting IPv6 addresses in TCP lookups currently requires two extra cache
lines, plus a dereference (and memory stall).
inet6_sk(sk) does the dereference of inet_sk(__sk)->pinet6
This patch is way bigger than its IPv4 counter part, because for IPv4,
we could add aliases (inet_daddr, inet_rcv_saddr), while on IPv6,
it's not doable easily.
inet6_sk(sk)->daddr becomes sk->sk_v6_daddr
inet6_sk(sk)->rcv_saddr becomes sk->sk_v6_rcv_saddr
And timewait socket also have tw->tw_v6_daddr & tw->tw_v6_rcv_saddr
at the same offset.
We get rid of INET6_TW_MATCH() as INET6_MATCH() is now the generic
macro.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-10-04 06:42:29 +08:00
|
|
|
ipv6_addr_equal(&sk->sk_v6_rcv_saddr, sk2_rcv_saddr6))
|
2009-03-25 00:24:51 +08:00
|
|
|
return 1;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2015-03-19 05:05:33 +08:00
|
|
|
static u32 udp6_portaddr_hash(const struct net *net,
|
|
|
|
const struct in6_addr *addr6,
|
|
|
|
unsigned int port)
|
2009-11-08 18:17:30 +08:00
|
|
|
{
|
|
|
|
unsigned int hash, mix = net_hash_mix(net);
|
|
|
|
|
|
|
|
if (ipv6_addr_any(addr6))
|
|
|
|
hash = jhash_1word(0, mix);
|
2009-11-09 20:05:53 +08:00
|
|
|
else if (ipv6_addr_v4mapped(addr6))
|
2010-04-21 10:06:52 +08:00
|
|
|
hash = jhash_1word((__force u32)addr6->s6_addr32[3], mix);
|
2009-11-08 18:17:30 +08:00
|
|
|
else
|
2010-04-21 10:06:52 +08:00
|
|
|
hash = jhash2((__force u32 *)addr6->s6_addr32, 4, mix);
|
2009-11-08 18:17:30 +08:00
|
|
|
|
|
|
|
return hash ^ port;
|
|
|
|
}
|
|
|
|
|
2008-03-23 07:51:21 +08:00
|
|
|
int udp_v6_get_port(struct sock *sk, unsigned short snum)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2009-11-09 13:26:33 +08:00
|
|
|
unsigned int hash2_nulladdr =
|
|
|
|
udp6_portaddr_hash(sock_net(sk), &in6addr_any, snum);
|
2012-05-19 09:45:21 +08:00
|
|
|
unsigned int hash2_partial =
|
ipv6: make lookups simpler and faster
TCP listener refactoring, part 4 :
To speed up inet lookups, we moved IPv4 addresses from inet to struct
sock_common
Now is time to do the same for IPv6, because it permits us to have fast
lookups for all kind of sockets, including upcoming SYN_RECV.
Getting IPv6 addresses in TCP lookups currently requires two extra cache
lines, plus a dereference (and memory stall).
inet6_sk(sk) does the dereference of inet_sk(__sk)->pinet6
This patch is way bigger than its IPv4 counter part, because for IPv4,
we could add aliases (inet_daddr, inet_rcv_saddr), while on IPv6,
it's not doable easily.
inet6_sk(sk)->daddr becomes sk->sk_v6_daddr
inet6_sk(sk)->rcv_saddr becomes sk->sk_v6_rcv_saddr
And timewait socket also have tw->tw_v6_daddr & tw->tw_v6_rcv_saddr
at the same offset.
We get rid of INET6_TW_MATCH() as INET6_MATCH() is now the generic
macro.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-10-04 06:42:29 +08:00
|
|
|
udp6_portaddr_hash(sock_net(sk), &sk->sk_v6_rcv_saddr, 0);
|
2009-11-09 13:26:33 +08:00
|
|
|
|
2009-11-08 18:17:30 +08:00
|
|
|
/* precompute partial secondary hash */
|
2009-11-09 13:26:33 +08:00
|
|
|
udp_sk(sk)->udp_portaddr_hash = hash2_partial;
|
|
|
|
return udp_lib_get_port(sk, snum, ipv6_rcv_saddr_equal, hash2_nulladdr);
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
udp: add rehash on connect()
commit 30fff923 introduced in linux-2.6.33 (udp: bind() optimisation)
added a secondary hash on UDP, hashed on (local addr, local port).
Problem is that following sequence :
fd = socket(...)
connect(fd, &remote, ...)
not only selects remote end point (address and port), but also sets
local address, while UDP stack stored in secondary hash table the socket
while its local address was INADDR_ANY (or ipv6 equivalent)
Sequence is :
- autobind() : choose a random local port, insert socket in hash tables
[while local address is INADDR_ANY]
- connect() : set remote address and port, change local address to IP
given by a route lookup.
When an incoming UDP frame comes, if more than 10 sockets are found in
primary hash table, we switch to secondary table, and fail to find
socket because its local address changed.
One solution to this problem is to rehash datagram socket if needed.
We add a new rehash(struct socket *) method in "struct proto", and
implement this method for UDP v4 & v6, using a common helper.
This rehashing only takes care of secondary hash table, since primary
hash (based on local port only) is not changed.
Reported-by: Krzysztof Piotr Oledzki <ole@ans.pl>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Tested-by: Krzysztof Piotr Oledzki <ole@ans.pl>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-09-08 13:08:44 +08:00
|
|
|
static void udp_v6_rehash(struct sock *sk)
|
|
|
|
{
|
|
|
|
u16 new_hash = udp6_portaddr_hash(sock_net(sk),
|
ipv6: make lookups simpler and faster
TCP listener refactoring, part 4 :
To speed up inet lookups, we moved IPv4 addresses from inet to struct
sock_common
Now is time to do the same for IPv6, because it permits us to have fast
lookups for all kind of sockets, including upcoming SYN_RECV.
Getting IPv6 addresses in TCP lookups currently requires two extra cache
lines, plus a dereference (and memory stall).
inet6_sk(sk) does the dereference of inet_sk(__sk)->pinet6
This patch is way bigger than its IPv4 counter part, because for IPv4,
we could add aliases (inet_daddr, inet_rcv_saddr), while on IPv6,
it's not doable easily.
inet6_sk(sk)->daddr becomes sk->sk_v6_daddr
inet6_sk(sk)->rcv_saddr becomes sk->sk_v6_rcv_saddr
And timewait socket also have tw->tw_v6_daddr & tw->tw_v6_rcv_saddr
at the same offset.
We get rid of INET6_TW_MATCH() as INET6_MATCH() is now the generic
macro.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-10-04 06:42:29 +08:00
|
|
|
&sk->sk_v6_rcv_saddr,
|
udp: add rehash on connect()
commit 30fff923 introduced in linux-2.6.33 (udp: bind() optimisation)
added a secondary hash on UDP, hashed on (local addr, local port).
Problem is that following sequence :
fd = socket(...)
connect(fd, &remote, ...)
not only selects remote end point (address and port), but also sets
local address, while UDP stack stored in secondary hash table the socket
while its local address was INADDR_ANY (or ipv6 equivalent)
Sequence is :
- autobind() : choose a random local port, insert socket in hash tables
[while local address is INADDR_ANY]
- connect() : set remote address and port, change local address to IP
given by a route lookup.
When an incoming UDP frame comes, if more than 10 sockets are found in
primary hash table, we switch to secondary table, and fail to find
socket because its local address changed.
One solution to this problem is to rehash datagram socket if needed.
We add a new rehash(struct socket *) method in "struct proto", and
implement this method for UDP v4 & v6, using a common helper.
This rehashing only takes care of secondary hash table, since primary
hash (based on local port only) is not changed.
Reported-by: Krzysztof Piotr Oledzki <ole@ans.pl>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Tested-by: Krzysztof Piotr Oledzki <ole@ans.pl>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-09-08 13:08:44 +08:00
|
|
|
inet_sk(sk)->inet_num);
|
|
|
|
|
|
|
|
udp_lib_rehash(sk, new_hash);
|
|
|
|
}
|
|
|
|
|
2008-10-29 16:41:45 +08:00
|
|
|
static inline int compute_score(struct sock *sk, struct net *net,
|
|
|
|
unsigned short hnum,
|
2010-10-21 22:04:33 +08:00
|
|
|
const struct in6_addr *saddr, __be16 sport,
|
|
|
|
const struct in6_addr *daddr, __be16 dport,
|
2008-10-29 16:41:45 +08:00
|
|
|
int dif)
|
|
|
|
{
|
2014-12-02 12:29:06 +08:00
|
|
|
int score;
|
|
|
|
struct inet_sock *inet;
|
|
|
|
|
|
|
|
if (!net_eq(sock_net(sk), net) ||
|
|
|
|
udp_sk(sk)->udp_port_hash != hnum ||
|
|
|
|
sk->sk_family != PF_INET6)
|
|
|
|
return -1;
|
|
|
|
|
|
|
|
score = 0;
|
|
|
|
inet = inet_sk(sk);
|
|
|
|
|
|
|
|
if (inet->inet_dport) {
|
|
|
|
if (inet->inet_dport != sport)
|
|
|
|
return -1;
|
|
|
|
score++;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!ipv6_addr_any(&sk->sk_v6_rcv_saddr)) {
|
|
|
|
if (!ipv6_addr_equal(&sk->sk_v6_rcv_saddr, daddr))
|
|
|
|
return -1;
|
|
|
|
score++;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!ipv6_addr_any(&sk->sk_v6_daddr)) {
|
|
|
|
if (!ipv6_addr_equal(&sk->sk_v6_daddr, saddr))
|
|
|
|
return -1;
|
|
|
|
score++;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (sk->sk_bound_dev_if) {
|
|
|
|
if (sk->sk_bound_dev_if != dif)
|
|
|
|
return -1;
|
|
|
|
score++;
|
2008-10-29 16:41:45 +08:00
|
|
|
}
|
2014-12-02 12:29:06 +08:00
|
|
|
|
2008-10-29 16:41:45 +08:00
|
|
|
return score;
|
|
|
|
}
|
|
|
|
|
ipv6: udp: optimize unicast RX path
We first locate the (local port) hash chain head
If few sockets are in this chain, we proceed with previous lookup algo.
If too many sockets are listed, we take a look at the secondary
(port, address) hash chain.
We choose the shortest chain and proceed with a RCU lookup on the elected chain.
But, if we chose (port, address) chain, and fail to find a socket on given address,
we must try another lookup on (port, in6addr_any) chain to find sockets not bound
to a particular IP.
-> No extra cost for typical setups, where the first lookup will probabbly
be performed.
RCU lookups everywhere, we dont acquire spinlock.
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-11-08 18:18:30 +08:00
|
|
|
#define SCORE2_MAX (1 + 1 + 1)
|
|
|
|
static inline int compute_score2(struct sock *sk, struct net *net,
|
2014-12-02 12:29:06 +08:00
|
|
|
const struct in6_addr *saddr, __be16 sport,
|
|
|
|
const struct in6_addr *daddr,
|
|
|
|
unsigned short hnum, int dif)
|
ipv6: udp: optimize unicast RX path
We first locate the (local port) hash chain head
If few sockets are in this chain, we proceed with previous lookup algo.
If too many sockets are listed, we take a look at the secondary
(port, address) hash chain.
We choose the shortest chain and proceed with a RCU lookup on the elected chain.
But, if we chose (port, address) chain, and fail to find a socket on given address,
we must try another lookup on (port, in6addr_any) chain to find sockets not bound
to a particular IP.
-> No extra cost for typical setups, where the first lookup will probabbly
be performed.
RCU lookups everywhere, we dont acquire spinlock.
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-11-08 18:18:30 +08:00
|
|
|
{
|
2014-12-02 12:29:06 +08:00
|
|
|
int score;
|
|
|
|
struct inet_sock *inet;
|
ipv6: udp: optimize unicast RX path
We first locate the (local port) hash chain head
If few sockets are in this chain, we proceed with previous lookup algo.
If too many sockets are listed, we take a look at the secondary
(port, address) hash chain.
We choose the shortest chain and proceed with a RCU lookup on the elected chain.
But, if we chose (port, address) chain, and fail to find a socket on given address,
we must try another lookup on (port, in6addr_any) chain to find sockets not bound
to a particular IP.
-> No extra cost for typical setups, where the first lookup will probabbly
be performed.
RCU lookups everywhere, we dont acquire spinlock.
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-11-08 18:18:30 +08:00
|
|
|
|
2014-12-02 12:29:06 +08:00
|
|
|
if (!net_eq(sock_net(sk), net) ||
|
|
|
|
udp_sk(sk)->udp_port_hash != hnum ||
|
|
|
|
sk->sk_family != PF_INET6)
|
|
|
|
return -1;
|
ipv6: udp: optimize unicast RX path
We first locate the (local port) hash chain head
If few sockets are in this chain, we proceed with previous lookup algo.
If too many sockets are listed, we take a look at the secondary
(port, address) hash chain.
We choose the shortest chain and proceed with a RCU lookup on the elected chain.
But, if we chose (port, address) chain, and fail to find a socket on given address,
we must try another lookup on (port, in6addr_any) chain to find sockets not bound
to a particular IP.
-> No extra cost for typical setups, where the first lookup will probabbly
be performed.
RCU lookups everywhere, we dont acquire spinlock.
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-11-08 18:18:30 +08:00
|
|
|
|
2014-12-02 12:29:06 +08:00
|
|
|
if (!ipv6_addr_equal(&sk->sk_v6_rcv_saddr, daddr))
|
|
|
|
return -1;
|
|
|
|
|
|
|
|
score = 0;
|
|
|
|
inet = inet_sk(sk);
|
|
|
|
|
|
|
|
if (inet->inet_dport) {
|
|
|
|
if (inet->inet_dport != sport)
|
|
|
|
return -1;
|
|
|
|
score++;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!ipv6_addr_any(&sk->sk_v6_daddr)) {
|
|
|
|
if (!ipv6_addr_equal(&sk->sk_v6_daddr, saddr))
|
ipv6: udp: optimize unicast RX path
We first locate the (local port) hash chain head
If few sockets are in this chain, we proceed with previous lookup algo.
If too many sockets are listed, we take a look at the secondary
(port, address) hash chain.
We choose the shortest chain and proceed with a RCU lookup on the elected chain.
But, if we chose (port, address) chain, and fail to find a socket on given address,
we must try another lookup on (port, in6addr_any) chain to find sockets not bound
to a particular IP.
-> No extra cost for typical setups, where the first lookup will probabbly
be performed.
RCU lookups everywhere, we dont acquire spinlock.
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-11-08 18:18:30 +08:00
|
|
|
return -1;
|
2014-12-02 12:29:06 +08:00
|
|
|
score++;
|
ipv6: udp: optimize unicast RX path
We first locate the (local port) hash chain head
If few sockets are in this chain, we proceed with previous lookup algo.
If too many sockets are listed, we take a look at the secondary
(port, address) hash chain.
We choose the shortest chain and proceed with a RCU lookup on the elected chain.
But, if we chose (port, address) chain, and fail to find a socket on given address,
we must try another lookup on (port, in6addr_any) chain to find sockets not bound
to a particular IP.
-> No extra cost for typical setups, where the first lookup will probabbly
be performed.
RCU lookups everywhere, we dont acquire spinlock.
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-11-08 18:18:30 +08:00
|
|
|
}
|
2014-12-02 12:29:06 +08:00
|
|
|
|
|
|
|
if (sk->sk_bound_dev_if) {
|
|
|
|
if (sk->sk_bound_dev_if != dif)
|
|
|
|
return -1;
|
|
|
|
score++;
|
|
|
|
}
|
|
|
|
|
ipv6: udp: optimize unicast RX path
We first locate the (local port) hash chain head
If few sockets are in this chain, we proceed with previous lookup algo.
If too many sockets are listed, we take a look at the secondary
(port, address) hash chain.
We choose the shortest chain and proceed with a RCU lookup on the elected chain.
But, if we chose (port, address) chain, and fail to find a socket on given address,
we must try another lookup on (port, in6addr_any) chain to find sockets not bound
to a particular IP.
-> No extra cost for typical setups, where the first lookup will probabbly
be performed.
RCU lookups everywhere, we dont acquire spinlock.
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-11-08 18:18:30 +08:00
|
|
|
return score;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* called with read_rcu_lock() */
|
|
|
|
static struct sock *udp6_lib_lookup2(struct net *net,
|
|
|
|
const struct in6_addr *saddr, __be16 sport,
|
|
|
|
const struct in6_addr *daddr, unsigned int hnum, int dif,
|
|
|
|
struct udp_hslot *hslot2, unsigned int slot2)
|
|
|
|
{
|
|
|
|
struct sock *sk, *result;
|
|
|
|
struct hlist_nulls_node *node;
|
2013-01-22 17:50:44 +08:00
|
|
|
int score, badness, matches = 0, reuseport = 0;
|
|
|
|
u32 hash = 0;
|
ipv6: udp: optimize unicast RX path
We first locate the (local port) hash chain head
If few sockets are in this chain, we proceed with previous lookup algo.
If too many sockets are listed, we take a look at the secondary
(port, address) hash chain.
We choose the shortest chain and proceed with a RCU lookup on the elected chain.
But, if we chose (port, address) chain, and fail to find a socket on given address,
we must try another lookup on (port, in6addr_any) chain to find sockets not bound
to a particular IP.
-> No extra cost for typical setups, where the first lookup will probabbly
be performed.
RCU lookups everywhere, we dont acquire spinlock.
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-11-08 18:18:30 +08:00
|
|
|
|
|
|
|
begin:
|
|
|
|
result = NULL;
|
|
|
|
badness = -1;
|
|
|
|
udp_portaddr_for_each_entry_rcu(sk, node, &hslot2->head) {
|
|
|
|
score = compute_score2(sk, net, saddr, sport,
|
|
|
|
daddr, hnum, dif);
|
|
|
|
if (score > badness) {
|
|
|
|
result = sk;
|
|
|
|
badness = score;
|
2013-01-22 17:50:44 +08:00
|
|
|
reuseport = sk->sk_reuseport;
|
|
|
|
if (reuseport) {
|
2013-10-20 03:48:52 +08:00
|
|
|
hash = udp6_ehashfn(net, daddr, hnum,
|
|
|
|
saddr, sport);
|
2013-01-22 17:50:44 +08:00
|
|
|
matches = 1;
|
|
|
|
} else if (score == SCORE2_MAX)
|
ipv6: udp: optimize unicast RX path
We first locate the (local port) hash chain head
If few sockets are in this chain, we proceed with previous lookup algo.
If too many sockets are listed, we take a look at the secondary
(port, address) hash chain.
We choose the shortest chain and proceed with a RCU lookup on the elected chain.
But, if we chose (port, address) chain, and fail to find a socket on given address,
we must try another lookup on (port, in6addr_any) chain to find sockets not bound
to a particular IP.
-> No extra cost for typical setups, where the first lookup will probabbly
be performed.
RCU lookups everywhere, we dont acquire spinlock.
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-11-08 18:18:30 +08:00
|
|
|
goto exact_match;
|
2013-01-22 17:50:44 +08:00
|
|
|
} else if (score == badness && reuseport) {
|
|
|
|
matches++;
|
2014-08-24 02:58:54 +08:00
|
|
|
if (reciprocal_scale(hash, matches) == 0)
|
2013-01-22 17:50:44 +08:00
|
|
|
result = sk;
|
|
|
|
hash = next_pseudo_random32(hash);
|
ipv6: udp: optimize unicast RX path
We first locate the (local port) hash chain head
If few sockets are in this chain, we proceed with previous lookup algo.
If too many sockets are listed, we take a look at the secondary
(port, address) hash chain.
We choose the shortest chain and proceed with a RCU lookup on the elected chain.
But, if we chose (port, address) chain, and fail to find a socket on given address,
we must try another lookup on (port, in6addr_any) chain to find sockets not bound
to a particular IP.
-> No extra cost for typical setups, where the first lookup will probabbly
be performed.
RCU lookups everywhere, we dont acquire spinlock.
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-11-08 18:18:30 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
/*
|
|
|
|
* if the nulls value we got at the end of this lookup is
|
|
|
|
* not the expected one, we must restart lookup.
|
|
|
|
* We probably met an item that was moved to another chain.
|
|
|
|
*/
|
|
|
|
if (get_nulls_value(node) != slot2)
|
|
|
|
goto begin;
|
|
|
|
|
|
|
|
if (result) {
|
|
|
|
exact_match:
|
2010-11-16 03:58:26 +08:00
|
|
|
if (unlikely(!atomic_inc_not_zero_hint(&result->sk_refcnt, 2)))
|
ipv6: udp: optimize unicast RX path
We first locate the (local port) hash chain head
If few sockets are in this chain, we proceed with previous lookup algo.
If too many sockets are listed, we take a look at the secondary
(port, address) hash chain.
We choose the shortest chain and proceed with a RCU lookup on the elected chain.
But, if we chose (port, address) chain, and fail to find a socket on given address,
we must try another lookup on (port, in6addr_any) chain to find sockets not bound
to a particular IP.
-> No extra cost for typical setups, where the first lookup will probabbly
be performed.
RCU lookups everywhere, we dont acquire spinlock.
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-11-08 18:18:30 +08:00
|
|
|
result = NULL;
|
|
|
|
else if (unlikely(compute_score2(result, net, saddr, sport,
|
|
|
|
daddr, hnum, dif) < badness)) {
|
|
|
|
sock_put(result);
|
|
|
|
goto begin;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
2011-12-09 14:23:34 +08:00
|
|
|
struct sock *__udp6_lib_lookup(struct net *net,
|
2010-10-21 22:04:33 +08:00
|
|
|
const struct in6_addr *saddr, __be16 sport,
|
|
|
|
const struct in6_addr *daddr, __be16 dport,
|
2008-10-29 16:41:45 +08:00
|
|
|
int dif, struct udp_table *udptable)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
udp: RCU handling for Unicast packets.
Goals are :
1) Optimizing handling of incoming Unicast UDP frames, so that no memory
writes should happen in the fast path.
Note: Multicasts and broadcasts still will need to take a lock,
because doing a full lockless lookup in this case is difficult.
2) No expensive operations in the socket bind/unhash phases :
- No expensive synchronize_rcu() calls.
- No added rcu_head in socket structure, increasing memory needs,
but more important, forcing us to use call_rcu() calls,
that have the bad property of making sockets structure cold.
(rcu grace period between socket freeing and its potential reuse
make this socket being cold in CPU cache).
David did a previous patch using call_rcu() and noticed a 20%
impact on TCP connection rates.
Quoting Cristopher Lameter :
"Right. That results in cacheline cooldown. You'd want to recycle
the object as they are cache hot on a per cpu basis. That is screwed
up by the delayed regular rcu processing. We have seen multiple
regressions due to cacheline cooldown.
The only choice in cacheline hot sensitive areas is to deal with the
complexity that comes with SLAB_DESTROY_BY_RCU or give up on RCU."
- Because udp sockets are allocated from dedicated kmem_cache,
use of SLAB_DESTROY_BY_RCU can help here.
Theory of operation :
---------------------
As the lookup is lockfree (using rcu_read_lock()/rcu_read_unlock()),
special attention must be taken by readers and writers.
Use of SLAB_DESTROY_BY_RCU is tricky too, because a socket can be freed,
reused, inserted in a different chain or in worst case in the same chain
while readers could do lookups in the same time.
In order to avoid loops, a reader must check each socket found in a chain
really belongs to the chain the reader was traversing. If it finds a
mismatch, lookup must start again at the begining. This *restart* loop
is the reason we had to use rdlock for the multicast case, because
we dont want to send same message several times to the same socket.
We use RCU only for fast path.
Thus, /proc/net/udp still takes spinlocks.
Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-10-29 17:11:14 +08:00
|
|
|
struct sock *sk, *result;
|
2008-11-17 11:39:21 +08:00
|
|
|
struct hlist_nulls_node *node;
|
2005-04-17 06:20:36 +08:00
|
|
|
unsigned short hnum = ntohs(dport);
|
ipv6: udp: optimize unicast RX path
We first locate the (local port) hash chain head
If few sockets are in this chain, we proceed with previous lookup algo.
If too many sockets are listed, we take a look at the secondary
(port, address) hash chain.
We choose the shortest chain and proceed with a RCU lookup on the elected chain.
But, if we chose (port, address) chain, and fail to find a socket on given address,
we must try another lookup on (port, in6addr_any) chain to find sockets not bound
to a particular IP.
-> No extra cost for typical setups, where the first lookup will probabbly
be performed.
RCU lookups everywhere, we dont acquire spinlock.
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-11-08 18:18:30 +08:00
|
|
|
unsigned int hash2, slot2, slot = udp_hashfn(net, hnum, udptable->mask);
|
|
|
|
struct udp_hslot *hslot2, *hslot = &udptable->hash[slot];
|
2013-01-22 17:50:44 +08:00
|
|
|
int score, badness, matches = 0, reuseport = 0;
|
|
|
|
u32 hash = 0;
|
2008-10-29 16:41:45 +08:00
|
|
|
|
udp: RCU handling for Unicast packets.
Goals are :
1) Optimizing handling of incoming Unicast UDP frames, so that no memory
writes should happen in the fast path.
Note: Multicasts and broadcasts still will need to take a lock,
because doing a full lockless lookup in this case is difficult.
2) No expensive operations in the socket bind/unhash phases :
- No expensive synchronize_rcu() calls.
- No added rcu_head in socket structure, increasing memory needs,
but more important, forcing us to use call_rcu() calls,
that have the bad property of making sockets structure cold.
(rcu grace period between socket freeing and its potential reuse
make this socket being cold in CPU cache).
David did a previous patch using call_rcu() and noticed a 20%
impact on TCP connection rates.
Quoting Cristopher Lameter :
"Right. That results in cacheline cooldown. You'd want to recycle
the object as they are cache hot on a per cpu basis. That is screwed
up by the delayed regular rcu processing. We have seen multiple
regressions due to cacheline cooldown.
The only choice in cacheline hot sensitive areas is to deal with the
complexity that comes with SLAB_DESTROY_BY_RCU or give up on RCU."
- Because udp sockets are allocated from dedicated kmem_cache,
use of SLAB_DESTROY_BY_RCU can help here.
Theory of operation :
---------------------
As the lookup is lockfree (using rcu_read_lock()/rcu_read_unlock()),
special attention must be taken by readers and writers.
Use of SLAB_DESTROY_BY_RCU is tricky too, because a socket can be freed,
reused, inserted in a different chain or in worst case in the same chain
while readers could do lookups in the same time.
In order to avoid loops, a reader must check each socket found in a chain
really belongs to the chain the reader was traversing. If it finds a
mismatch, lookup must start again at the begining. This *restart* loop
is the reason we had to use rdlock for the multicast case, because
we dont want to send same message several times to the same socket.
We use RCU only for fast path.
Thus, /proc/net/udp still takes spinlocks.
Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-10-29 17:11:14 +08:00
|
|
|
rcu_read_lock();
|
ipv6: udp: optimize unicast RX path
We first locate the (local port) hash chain head
If few sockets are in this chain, we proceed with previous lookup algo.
If too many sockets are listed, we take a look at the secondary
(port, address) hash chain.
We choose the shortest chain and proceed with a RCU lookup on the elected chain.
But, if we chose (port, address) chain, and fail to find a socket on given address,
we must try another lookup on (port, in6addr_any) chain to find sockets not bound
to a particular IP.
-> No extra cost for typical setups, where the first lookup will probabbly
be performed.
RCU lookups everywhere, we dont acquire spinlock.
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-11-08 18:18:30 +08:00
|
|
|
if (hslot->count > 10) {
|
|
|
|
hash2 = udp6_portaddr_hash(net, daddr, hnum);
|
|
|
|
slot2 = hash2 & udptable->mask;
|
|
|
|
hslot2 = &udptable->hash2[slot2];
|
|
|
|
if (hslot->count < hslot2->count)
|
|
|
|
goto begin;
|
|
|
|
|
|
|
|
result = udp6_lib_lookup2(net, saddr, sport,
|
|
|
|
daddr, hnum, dif,
|
|
|
|
hslot2, slot2);
|
|
|
|
if (!result) {
|
|
|
|
hash2 = udp6_portaddr_hash(net, &in6addr_any, hnum);
|
|
|
|
slot2 = hash2 & udptable->mask;
|
|
|
|
hslot2 = &udptable->hash2[slot2];
|
|
|
|
if (hslot->count < hslot2->count)
|
|
|
|
goto begin;
|
|
|
|
|
2010-04-08 12:56:48 +08:00
|
|
|
result = udp6_lib_lookup2(net, saddr, sport,
|
|
|
|
&in6addr_any, hnum, dif,
|
ipv6: udp: optimize unicast RX path
We first locate the (local port) hash chain head
If few sockets are in this chain, we proceed with previous lookup algo.
If too many sockets are listed, we take a look at the secondary
(port, address) hash chain.
We choose the shortest chain and proceed with a RCU lookup on the elected chain.
But, if we chose (port, address) chain, and fail to find a socket on given address,
we must try another lookup on (port, in6addr_any) chain to find sockets not bound
to a particular IP.
-> No extra cost for typical setups, where the first lookup will probabbly
be performed.
RCU lookups everywhere, we dont acquire spinlock.
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-11-08 18:18:30 +08:00
|
|
|
hslot2, slot2);
|
|
|
|
}
|
|
|
|
rcu_read_unlock();
|
|
|
|
return result;
|
|
|
|
}
|
udp: RCU handling for Unicast packets.
Goals are :
1) Optimizing handling of incoming Unicast UDP frames, so that no memory
writes should happen in the fast path.
Note: Multicasts and broadcasts still will need to take a lock,
because doing a full lockless lookup in this case is difficult.
2) No expensive operations in the socket bind/unhash phases :
- No expensive synchronize_rcu() calls.
- No added rcu_head in socket structure, increasing memory needs,
but more important, forcing us to use call_rcu() calls,
that have the bad property of making sockets structure cold.
(rcu grace period between socket freeing and its potential reuse
make this socket being cold in CPU cache).
David did a previous patch using call_rcu() and noticed a 20%
impact on TCP connection rates.
Quoting Cristopher Lameter :
"Right. That results in cacheline cooldown. You'd want to recycle
the object as they are cache hot on a per cpu basis. That is screwed
up by the delayed regular rcu processing. We have seen multiple
regressions due to cacheline cooldown.
The only choice in cacheline hot sensitive areas is to deal with the
complexity that comes with SLAB_DESTROY_BY_RCU or give up on RCU."
- Because udp sockets are allocated from dedicated kmem_cache,
use of SLAB_DESTROY_BY_RCU can help here.
Theory of operation :
---------------------
As the lookup is lockfree (using rcu_read_lock()/rcu_read_unlock()),
special attention must be taken by readers and writers.
Use of SLAB_DESTROY_BY_RCU is tricky too, because a socket can be freed,
reused, inserted in a different chain or in worst case in the same chain
while readers could do lookups in the same time.
In order to avoid loops, a reader must check each socket found in a chain
really belongs to the chain the reader was traversing. If it finds a
mismatch, lookup must start again at the begining. This *restart* loop
is the reason we had to use rdlock for the multicast case, because
we dont want to send same message several times to the same socket.
We use RCU only for fast path.
Thus, /proc/net/udp still takes spinlocks.
Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-10-29 17:11:14 +08:00
|
|
|
begin:
|
|
|
|
result = NULL;
|
|
|
|
badness = -1;
|
2008-11-17 11:39:21 +08:00
|
|
|
sk_nulls_for_each_rcu(sk, node, &hslot->head) {
|
2008-10-29 16:41:45 +08:00
|
|
|
score = compute_score(sk, net, hnum, saddr, sport, daddr, dport, dif);
|
|
|
|
if (score > badness) {
|
|
|
|
result = sk;
|
|
|
|
badness = score;
|
2013-01-22 17:50:44 +08:00
|
|
|
reuseport = sk->sk_reuseport;
|
|
|
|
if (reuseport) {
|
2013-10-20 03:48:52 +08:00
|
|
|
hash = udp6_ehashfn(net, daddr, hnum,
|
|
|
|
saddr, sport);
|
2013-01-22 17:50:44 +08:00
|
|
|
matches = 1;
|
|
|
|
}
|
|
|
|
} else if (score == badness && reuseport) {
|
|
|
|
matches++;
|
2014-08-24 02:58:54 +08:00
|
|
|
if (reciprocal_scale(hash, matches) == 0)
|
2013-01-22 17:50:44 +08:00
|
|
|
result = sk;
|
|
|
|
hash = next_pseudo_random32(hash);
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
}
|
2008-11-17 11:39:21 +08:00
|
|
|
/*
|
|
|
|
* if the nulls value we got at the end of this lookup is
|
|
|
|
* not the expected one, we must restart lookup.
|
|
|
|
* We probably met an item that was moved to another chain.
|
|
|
|
*/
|
ipv6: udp: optimize unicast RX path
We first locate the (local port) hash chain head
If few sockets are in this chain, we proceed with previous lookup algo.
If too many sockets are listed, we take a look at the secondary
(port, address) hash chain.
We choose the shortest chain and proceed with a RCU lookup on the elected chain.
But, if we chose (port, address) chain, and fail to find a socket on given address,
we must try another lookup on (port, in6addr_any) chain to find sockets not bound
to a particular IP.
-> No extra cost for typical setups, where the first lookup will probabbly
be performed.
RCU lookups everywhere, we dont acquire spinlock.
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-11-08 18:18:30 +08:00
|
|
|
if (get_nulls_value(node) != slot)
|
2008-11-17 11:39:21 +08:00
|
|
|
goto begin;
|
|
|
|
|
udp: RCU handling for Unicast packets.
Goals are :
1) Optimizing handling of incoming Unicast UDP frames, so that no memory
writes should happen in the fast path.
Note: Multicasts and broadcasts still will need to take a lock,
because doing a full lockless lookup in this case is difficult.
2) No expensive operations in the socket bind/unhash phases :
- No expensive synchronize_rcu() calls.
- No added rcu_head in socket structure, increasing memory needs,
but more important, forcing us to use call_rcu() calls,
that have the bad property of making sockets structure cold.
(rcu grace period between socket freeing and its potential reuse
make this socket being cold in CPU cache).
David did a previous patch using call_rcu() and noticed a 20%
impact on TCP connection rates.
Quoting Cristopher Lameter :
"Right. That results in cacheline cooldown. You'd want to recycle
the object as they are cache hot on a per cpu basis. That is screwed
up by the delayed regular rcu processing. We have seen multiple
regressions due to cacheline cooldown.
The only choice in cacheline hot sensitive areas is to deal with the
complexity that comes with SLAB_DESTROY_BY_RCU or give up on RCU."
- Because udp sockets are allocated from dedicated kmem_cache,
use of SLAB_DESTROY_BY_RCU can help here.
Theory of operation :
---------------------
As the lookup is lockfree (using rcu_read_lock()/rcu_read_unlock()),
special attention must be taken by readers and writers.
Use of SLAB_DESTROY_BY_RCU is tricky too, because a socket can be freed,
reused, inserted in a different chain or in worst case in the same chain
while readers could do lookups in the same time.
In order to avoid loops, a reader must check each socket found in a chain
really belongs to the chain the reader was traversing. If it finds a
mismatch, lookup must start again at the begining. This *restart* loop
is the reason we had to use rdlock for the multicast case, because
we dont want to send same message several times to the same socket.
We use RCU only for fast path.
Thus, /proc/net/udp still takes spinlocks.
Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-10-29 17:11:14 +08:00
|
|
|
if (result) {
|
2010-11-16 03:58:26 +08:00
|
|
|
if (unlikely(!atomic_inc_not_zero_hint(&result->sk_refcnt, 2)))
|
udp: RCU handling for Unicast packets.
Goals are :
1) Optimizing handling of incoming Unicast UDP frames, so that no memory
writes should happen in the fast path.
Note: Multicasts and broadcasts still will need to take a lock,
because doing a full lockless lookup in this case is difficult.
2) No expensive operations in the socket bind/unhash phases :
- No expensive synchronize_rcu() calls.
- No added rcu_head in socket structure, increasing memory needs,
but more important, forcing us to use call_rcu() calls,
that have the bad property of making sockets structure cold.
(rcu grace period between socket freeing and its potential reuse
make this socket being cold in CPU cache).
David did a previous patch using call_rcu() and noticed a 20%
impact on TCP connection rates.
Quoting Cristopher Lameter :
"Right. That results in cacheline cooldown. You'd want to recycle
the object as they are cache hot on a per cpu basis. That is screwed
up by the delayed regular rcu processing. We have seen multiple
regressions due to cacheline cooldown.
The only choice in cacheline hot sensitive areas is to deal with the
complexity that comes with SLAB_DESTROY_BY_RCU or give up on RCU."
- Because udp sockets are allocated from dedicated kmem_cache,
use of SLAB_DESTROY_BY_RCU can help here.
Theory of operation :
---------------------
As the lookup is lockfree (using rcu_read_lock()/rcu_read_unlock()),
special attention must be taken by readers and writers.
Use of SLAB_DESTROY_BY_RCU is tricky too, because a socket can be freed,
reused, inserted in a different chain or in worst case in the same chain
while readers could do lookups in the same time.
In order to avoid loops, a reader must check each socket found in a chain
really belongs to the chain the reader was traversing. If it finds a
mismatch, lookup must start again at the begining. This *restart* loop
is the reason we had to use rdlock for the multicast case, because
we dont want to send same message several times to the same socket.
We use RCU only for fast path.
Thus, /proc/net/udp still takes spinlocks.
Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-10-29 17:11:14 +08:00
|
|
|
result = NULL;
|
|
|
|
else if (unlikely(compute_score(result, net, hnum, saddr, sport,
|
|
|
|
daddr, dport, dif) < badness)) {
|
|
|
|
sock_put(result);
|
|
|
|
goto begin;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
rcu_read_unlock();
|
2005-04-17 06:20:36 +08:00
|
|
|
return result;
|
|
|
|
}
|
2011-12-09 14:23:34 +08:00
|
|
|
EXPORT_SYMBOL_GPL(__udp6_lib_lookup);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2008-10-08 03:38:32 +08:00
|
|
|
static struct sock *__udp6_lib_lookup_skb(struct sk_buff *skb,
|
|
|
|
__be16 sport, __be16 dport,
|
2008-10-29 16:41:45 +08:00
|
|
|
struct udp_table *udptable)
|
2008-10-08 03:38:32 +08:00
|
|
|
{
|
2008-10-08 03:41:01 +08:00
|
|
|
struct sock *sk;
|
2011-04-22 12:53:02 +08:00
|
|
|
const struct ipv6hdr *iph = ipv6_hdr(skb);
|
2008-10-08 03:38:32 +08:00
|
|
|
|
2014-11-24 05:28:43 +08:00
|
|
|
sk = skb_steal_sock(skb);
|
|
|
|
if (unlikely(sk))
|
2008-10-08 03:41:01 +08:00
|
|
|
return sk;
|
2009-06-02 13:19:30 +08:00
|
|
|
return __udp6_lib_lookup(dev_net(skb_dst(skb)->dev), &iph->saddr, sport,
|
|
|
|
&iph->daddr, dport, inet6_iif(skb),
|
|
|
|
udptable);
|
2008-10-08 03:38:32 +08:00
|
|
|
}
|
|
|
|
|
2010-10-21 22:05:41 +08:00
|
|
|
struct sock *udp6_lib_lookup(struct net *net, const struct in6_addr *saddr, __be16 sport,
|
|
|
|
const struct in6_addr *daddr, __be16 dport, int dif)
|
|
|
|
{
|
|
|
|
return __udp6_lib_lookup(net, saddr, sport, daddr, dport, dif, &udp_table);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(udp6_lib_lookup);
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/*
|
2014-08-25 04:53:10 +08:00
|
|
|
* This should be easy, if there is something there we
|
|
|
|
* return it, otherwise we block.
|
2005-04-17 06:20:36 +08:00
|
|
|
*/
|
|
|
|
|
2015-03-02 15:37:48 +08:00
|
|
|
int udpv6_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
|
2005-04-17 06:20:36 +08:00
|
|
|
int noblock, int flags, int *addr_len)
|
|
|
|
{
|
|
|
|
struct ipv6_pinfo *np = inet6_sk(sk);
|
|
|
|
struct inet_sock *inet = inet_sk(sk);
|
2007-02-09 22:24:49 +08:00
|
|
|
struct sk_buff *skb;
|
2011-12-02 03:12:55 +08:00
|
|
|
unsigned int ulen, copied;
|
2012-02-21 15:30:58 +08:00
|
|
|
int peeked, off = 0;
|
2007-03-26 11:10:56 +08:00
|
|
|
int err;
|
|
|
|
int is_udplite = IS_UDPLITE(sk);
|
2008-11-03 00:11:01 +08:00
|
|
|
int is_udp4;
|
2010-05-27 03:20:18 +08:00
|
|
|
bool slow;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
if (flags & MSG_ERRQUEUE)
|
2013-11-23 07:46:12 +08:00
|
|
|
return ipv6_recv_error(sk, msg, len, addr_len);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2010-04-23 19:26:09 +08:00
|
|
|
if (np->rxpmtu && np->rxopt.bits.rxpmtu)
|
2013-11-23 07:46:12 +08:00
|
|
|
return ipv6_recv_rxpmtu(sk, msg, len, addr_len);
|
2010-04-23 19:26:09 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
try_again:
|
2007-12-05 17:53:40 +08:00
|
|
|
skb = __skb_recv_datagram(sk, flags | (noblock ? MSG_DONTWAIT : 0),
|
2012-02-21 15:30:58 +08:00
|
|
|
&peeked, &off, &err);
|
2005-04-17 06:20:36 +08:00
|
|
|
if (!skb)
|
|
|
|
goto out;
|
|
|
|
|
2007-03-26 11:10:56 +08:00
|
|
|
ulen = skb->len - sizeof(struct udphdr);
|
2011-12-02 03:12:55 +08:00
|
|
|
copied = len;
|
|
|
|
if (copied > ulen)
|
|
|
|
copied = ulen;
|
|
|
|
else if (copied < ulen)
|
2007-02-09 22:24:49 +08:00
|
|
|
msg->msg_flags |= MSG_TRUNC;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2008-11-03 00:11:01 +08:00
|
|
|
is_udp4 = (skb->protocol == htons(ETH_P_IP));
|
|
|
|
|
2006-11-28 03:10:57 +08:00
|
|
|
/*
|
2007-03-26 11:10:56 +08:00
|
|
|
* If checksum is needed at all, try to do it while copying the
|
|
|
|
* data. If the data is truncated, or if we only want a partial
|
|
|
|
* coverage checksum (UDP-Lite), do it before the copy.
|
2006-11-28 03:10:57 +08:00
|
|
|
*/
|
|
|
|
|
2011-12-02 03:12:55 +08:00
|
|
|
if (copied < ulen || UDP_SKB_CB(skb)->partial_cov) {
|
2007-03-26 11:10:56 +08:00
|
|
|
if (udp_lib_checksum_complete(skb))
|
2005-04-17 06:20:36 +08:00
|
|
|
goto csum_copy_err;
|
2006-11-28 03:10:57 +08:00
|
|
|
}
|
|
|
|
|
2007-04-10 02:59:39 +08:00
|
|
|
if (skb_csum_unnecessary(skb))
|
2014-11-06 05:46:40 +08:00
|
|
|
err = skb_copy_datagram_msg(skb, sizeof(struct udphdr),
|
|
|
|
msg, copied);
|
2006-11-28 03:10:57 +08:00
|
|
|
else {
|
2014-04-07 06:47:38 +08:00
|
|
|
err = skb_copy_and_csum_datagram_msg(skb, sizeof(struct udphdr), msg);
|
2005-04-17 06:20:36 +08:00
|
|
|
if (err == -EINVAL)
|
|
|
|
goto csum_copy_err;
|
|
|
|
}
|
2012-06-27 08:23:44 +08:00
|
|
|
if (unlikely(err)) {
|
|
|
|
trace_kfree_skb(skb, udpv6_recvmsg);
|
2012-09-06 07:34:44 +08:00
|
|
|
if (!peeked) {
|
|
|
|
atomic_inc(&sk->sk_drops);
|
|
|
|
if (is_udp4)
|
|
|
|
UDP_INC_STATS_USER(sock_net(sk),
|
|
|
|
UDP_MIB_INERRORS,
|
|
|
|
is_udplite);
|
|
|
|
else
|
|
|
|
UDP6_INC_STATS_USER(sock_net(sk),
|
|
|
|
UDP_MIB_INERRORS,
|
|
|
|
is_udplite);
|
|
|
|
}
|
2005-04-17 06:20:36 +08:00
|
|
|
goto out_free;
|
2012-06-27 08:23:44 +08:00
|
|
|
}
|
2008-11-03 00:11:01 +08:00
|
|
|
if (!peeked) {
|
|
|
|
if (is_udp4)
|
|
|
|
UDP_INC_STATS_USER(sock_net(sk),
|
|
|
|
UDP_MIB_INDATAGRAMS, is_udplite);
|
|
|
|
else
|
|
|
|
UDP6_INC_STATS_USER(sock_net(sk),
|
|
|
|
UDP_MIB_INDATAGRAMS, is_udplite);
|
|
|
|
}
|
2007-12-03 19:33:28 +08:00
|
|
|
|
net: Generalize socket rx gap / receive queue overflow cmsg
Create a new socket level option to report number of queue overflows
Recently I augmented the AF_PACKET protocol to report the number of frames lost
on the socket receive queue between any two enqueued frames. This value was
exported via a SOL_PACKET level cmsg. AFter I completed that work it was
requested that this feature be generalized so that any datagram oriented socket
could make use of this option. As such I've created this patch, It creates a
new SOL_SOCKET level option called SO_RXQ_OVFL, which when enabled exports a
SOL_SOCKET level cmsg that reports the nubmer of times the sk_receive_queue
overflowed between any two given frames. It also augments the AF_PACKET
protocol to take advantage of this new feature (as it previously did not touch
sk->sk_drops, which this patch uses to record the overflow count). Tested
successfully by me.
Notes:
1) Unlike my previous patch, this patch simply records the sk_drops value, which
is not a number of drops between packets, but rather a total number of drops.
Deltas must be computed in user space.
2) While this patch currently works with datagram oriented protocols, it will
also be accepted by non-datagram oriented protocols. I'm not sure if thats
agreeable to everyone, but my argument in favor of doing so is that, for those
protocols which aren't applicable to this option, sk_drops will always be zero,
and reporting no drops on a receive queue that isn't used for those
non-participating protocols seems reasonable to me. This also saves us having
to code in a per-protocol opt in mechanism.
3) This applies cleanly to net-next assuming that commit
977750076d98c7ff6cbda51858bb5a5894a9d9ab (my af packet cmsg patch) is reverted
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-10-13 04:26:31 +08:00
|
|
|
sock_recv_ts_and_drops(msg, sk, skb);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
/* Copy the address. */
|
|
|
|
if (msg->msg_name) {
|
2014-01-18 05:53:15 +08:00
|
|
|
DECLARE_SOCKADDR(struct sockaddr_in6 *, sin6, msg->msg_name);
|
2005-04-17 06:20:36 +08:00
|
|
|
sin6->sin6_family = AF_INET6;
|
2007-03-14 01:28:48 +08:00
|
|
|
sin6->sin6_port = udp_hdr(skb)->source;
|
2005-04-17 06:20:36 +08:00
|
|
|
sin6->sin6_flowinfo = 0;
|
|
|
|
|
2013-03-08 10:07:19 +08:00
|
|
|
if (is_udp4) {
|
2009-10-08 04:58:25 +08:00
|
|
|
ipv6_addr_set_v4mapped(ip_hdr(skb)->saddr,
|
|
|
|
&sin6->sin6_addr);
|
2013-03-08 10:07:19 +08:00
|
|
|
sin6->sin6_scope_id = 0;
|
|
|
|
} else {
|
2011-11-21 11:39:03 +08:00
|
|
|
sin6->sin6_addr = ipv6_hdr(skb)->saddr;
|
2013-03-08 10:07:19 +08:00
|
|
|
sin6->sin6_scope_id =
|
|
|
|
ipv6_iface_scope_id(&sin6->sin6_addr,
|
2014-08-01 09:52:58 +08:00
|
|
|
inet6_iif(skb));
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
2013-11-18 11:20:45 +08:00
|
|
|
*addr_len = sizeof(*sin6);
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
2014-01-20 10:43:08 +08:00
|
|
|
|
|
|
|
if (np->rxopt.all)
|
|
|
|
ip6_datagram_recv_common_ctl(sk, msg, skb);
|
|
|
|
|
2008-11-03 00:11:01 +08:00
|
|
|
if (is_udp4) {
|
2005-04-17 06:20:36 +08:00
|
|
|
if (inet->cmsg_flags)
|
|
|
|
ip_cmsg_recv(msg, skb);
|
|
|
|
} else {
|
|
|
|
if (np->rxopt.all)
|
2014-01-20 10:43:08 +08:00
|
|
|
ip6_datagram_recv_specific_ctl(sk, msg, skb);
|
2007-02-09 22:24:49 +08:00
|
|
|
}
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2011-12-02 03:12:55 +08:00
|
|
|
err = copied;
|
2005-04-17 06:20:36 +08:00
|
|
|
if (flags & MSG_TRUNC)
|
2007-03-26 11:10:56 +08:00
|
|
|
err = ulen;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
out_free:
|
2009-10-30 13:03:53 +08:00
|
|
|
skb_free_datagram_locked(sk, skb);
|
2005-04-17 06:20:36 +08:00
|
|
|
out:
|
|
|
|
return err;
|
|
|
|
|
|
|
|
csum_copy_err:
|
2010-05-27 03:20:18 +08:00
|
|
|
slow = lock_sock_fast(sk);
|
2008-11-03 00:14:27 +08:00
|
|
|
if (!skb_kill_datagram(sk, skb, flags)) {
|
2013-04-29 16:39:56 +08:00
|
|
|
if (is_udp4) {
|
|
|
|
UDP_INC_STATS_USER(sock_net(sk),
|
|
|
|
UDP_MIB_CSUMERRORS, is_udplite);
|
2008-11-03 00:14:27 +08:00
|
|
|
UDP_INC_STATS_USER(sock_net(sk),
|
|
|
|
UDP_MIB_INERRORS, is_udplite);
|
2013-04-29 16:39:56 +08:00
|
|
|
} else {
|
|
|
|
UDP6_INC_STATS_USER(sock_net(sk),
|
|
|
|
UDP_MIB_CSUMERRORS, is_udplite);
|
2008-11-03 00:14:27 +08:00
|
|
|
UDP6_INC_STATS_USER(sock_net(sk),
|
|
|
|
UDP_MIB_INERRORS, is_udplite);
|
2013-04-29 16:39:56 +08:00
|
|
|
}
|
2008-11-03 00:14:27 +08:00
|
|
|
}
|
2010-05-27 03:20:18 +08:00
|
|
|
unlock_sock_fast(sk, slow);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2015-05-31 00:16:53 +08:00
|
|
|
/* starting over for a new packet, but check if we need to yield */
|
|
|
|
cond_resched();
|
2011-06-21 18:43:40 +08:00
|
|
|
msg->msg_flags &= ~MSG_TRUNC;
|
2005-04-17 06:20:36 +08:00
|
|
|
goto try_again;
|
|
|
|
}
|
|
|
|
|
2006-11-28 03:10:57 +08:00
|
|
|
void __udp6_lib_err(struct sk_buff *skb, struct inet6_skb_parm *opt,
|
2009-06-23 19:31:07 +08:00
|
|
|
u8 type, u8 code, int offset, __be32 info,
|
2008-10-29 16:41:45 +08:00
|
|
|
struct udp_table *udptable)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
|
|
|
struct ipv6_pinfo *np;
|
2011-04-22 12:53:02 +08:00
|
|
|
const struct ipv6hdr *hdr = (const struct ipv6hdr *)skb->data;
|
|
|
|
const struct in6_addr *saddr = &hdr->saddr;
|
|
|
|
const struct in6_addr *daddr = &hdr->daddr;
|
2014-08-25 04:53:10 +08:00
|
|
|
struct udphdr *uh = (struct udphdr *)(skb->data+offset);
|
2005-04-17 06:20:36 +08:00
|
|
|
struct sock *sk;
|
|
|
|
int err;
|
2014-07-31 17:54:32 +08:00
|
|
|
struct net *net = dev_net(skb->dev);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2014-07-31 17:54:32 +08:00
|
|
|
sk = __udp6_lib_lookup(net, daddr, uh->dest,
|
2006-11-28 03:10:57 +08:00
|
|
|
saddr, uh->source, inet6_iif(skb), udptable);
|
2015-03-29 21:00:04 +08:00
|
|
|
if (!sk) {
|
2014-07-31 17:54:32 +08:00
|
|
|
ICMP6_INC_STATS_BH(net, __in6_dev_get(skb->dev),
|
|
|
|
ICMP6_MIB_INERRORS);
|
2005-04-17 06:20:36 +08:00
|
|
|
return;
|
2014-07-31 17:54:32 +08:00
|
|
|
}
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2013-12-15 10:41:14 +08:00
|
|
|
if (type == ICMPV6_PKT_TOOBIG) {
|
|
|
|
if (!ip6_sk_accept_pmtu(sk))
|
|
|
|
goto out;
|
2012-06-16 05:54:11 +08:00
|
|
|
ip6_sk_update_pmtu(skb, sk, info);
|
2013-12-15 10:41:14 +08:00
|
|
|
}
|
2013-09-20 18:20:28 +08:00
|
|
|
if (type == NDISC_REDIRECT) {
|
2012-07-12 15:25:15 +08:00
|
|
|
ip6_sk_redirect(skb, sk);
|
2013-09-20 18:20:28 +08:00
|
|
|
goto out;
|
|
|
|
}
|
2012-06-16 05:54:11 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
np = inet6_sk(sk);
|
|
|
|
|
|
|
|
if (!icmpv6_err_convert(type, code, &err) && !np->recverr)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
if (sk->sk_state != TCP_ESTABLISHED && !np->recverr)
|
|
|
|
goto out;
|
|
|
|
|
2010-06-01 14:44:05 +08:00
|
|
|
if (np->recverr)
|
2005-04-17 06:20:36 +08:00
|
|
|
ipv6_icmp_error(sk, skb, err, uh->dest, ntohl(info), (u8 *)(uh+1));
|
2010-06-01 14:44:05 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
sk->sk_err = err;
|
|
|
|
sk->sk_error_report(sk);
|
|
|
|
out:
|
|
|
|
sock_put(sk);
|
|
|
|
}
|
|
|
|
|
2012-04-27 16:23:21 +08:00
|
|
|
static int __udpv6_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
int rc;
|
|
|
|
|
ipv6: make lookups simpler and faster
TCP listener refactoring, part 4 :
To speed up inet lookups, we moved IPv4 addresses from inet to struct
sock_common
Now is time to do the same for IPv6, because it permits us to have fast
lookups for all kind of sockets, including upcoming SYN_RECV.
Getting IPv6 addresses in TCP lookups currently requires two extra cache
lines, plus a dereference (and memory stall).
inet6_sk(sk) does the dereference of inet_sk(__sk)->pinet6
This patch is way bigger than its IPv4 counter part, because for IPv4,
we could add aliases (inet_daddr, inet_rcv_saddr), while on IPv6,
it's not doable easily.
inet6_sk(sk)->daddr becomes sk->sk_v6_daddr
inet6_sk(sk)->rcv_saddr becomes sk->sk_v6_rcv_saddr
And timewait socket also have tw->tw_v6_daddr & tw->tw_v6_rcv_saddr
at the same offset.
We get rid of INET6_TW_MATCH() as INET6_MATCH() is now the generic
macro.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-10-04 06:42:29 +08:00
|
|
|
if (!ipv6_addr_any(&sk->sk_v6_daddr)) {
|
2012-04-27 16:23:21 +08:00
|
|
|
sock_rps_save_rxhash(sk, skb);
|
2013-10-08 00:01:38 +08:00
|
|
|
sk_mark_napi_id(sk, skb);
|
net: introduce SO_INCOMING_CPU
Alternative to RPS/RFS is to use hardware support for multiple
queues.
Then split a set of million of sockets into worker threads, each
one using epoll() to manage events on its own socket pool.
Ideally, we want one thread per RX/TX queue/cpu, but we have no way to
know after accept() or connect() on which queue/cpu a socket is managed.
We normally use one cpu per RX queue (IRQ smp_affinity being properly
set), so remembering on socket structure which cpu delivered last packet
is enough to solve the problem.
After accept(), connect(), or even file descriptor passing around
processes, applications can use :
int cpu;
socklen_t len = sizeof(cpu);
getsockopt(fd, SOL_SOCKET, SO_INCOMING_CPU, &cpu, &len);
And use this information to put the socket into the right silo
for optimal performance, as all networking stack should run
on the appropriate cpu, without need to send IPI (RPS/RFS).
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-11-11 21:54:28 +08:00
|
|
|
sk_incoming_cpu_update(sk);
|
2013-10-08 00:01:38 +08:00
|
|
|
}
|
2012-04-27 16:23:21 +08:00
|
|
|
|
|
|
|
rc = sock_queue_rcv_skb(sk, skb);
|
|
|
|
if (rc < 0) {
|
|
|
|
int is_udplite = IS_UDPLITE(sk);
|
|
|
|
|
|
|
|
/* Note that an ENOMEM error is charged twice */
|
|
|
|
if (rc == -ENOMEM)
|
|
|
|
UDP6_INC_STATS_BH(sock_net(sk),
|
|
|
|
UDP_MIB_RCVBUFERRORS, is_udplite);
|
|
|
|
UDP6_INC_STATS_BH(sock_net(sk), UDP_MIB_INERRORS, is_udplite);
|
|
|
|
kfree_skb(skb);
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2006-11-28 03:10:57 +08:00
|
|
|
static __inline__ void udpv6_err(struct sk_buff *skb,
|
2009-06-23 19:31:07 +08:00
|
|
|
struct inet6_skb_parm *opt, u8 type,
|
2014-08-25 04:53:10 +08:00
|
|
|
u8 code, int offset, __be32 info)
|
2006-11-28 03:10:57 +08:00
|
|
|
{
|
2008-10-29 16:41:45 +08:00
|
|
|
__udp6_lib_err(skb, opt, type, code, offset, info, &udp_table);
|
2006-11-28 03:10:57 +08:00
|
|
|
}
|
|
|
|
|
2012-04-27 16:24:08 +08:00
|
|
|
static struct static_key udpv6_encap_needed __read_mostly;
|
|
|
|
void udpv6_encap_enable(void)
|
|
|
|
{
|
|
|
|
if (!static_key_enabled(&udpv6_encap_needed))
|
|
|
|
static_key_slow_inc(&udpv6_encap_needed);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(udpv6_encap_enable);
|
|
|
|
|
2012-04-27 16:23:21 +08:00
|
|
|
int udpv6_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2006-11-28 03:10:57 +08:00
|
|
|
struct udp_sock *up = udp_sk(sk);
|
2006-08-15 15:00:09 +08:00
|
|
|
int rc;
|
2007-12-03 19:34:16 +08:00
|
|
|
int is_udplite = IS_UDPLITE(sk);
|
2006-08-15 15:00:09 +08:00
|
|
|
|
2006-11-28 03:10:57 +08:00
|
|
|
if (!xfrm6_policy_check(sk, XFRM_POLICY_IN, skb))
|
|
|
|
goto drop;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2012-04-27 16:24:08 +08:00
|
|
|
if (static_key_false(&udpv6_encap_needed) && up->encap_type) {
|
|
|
|
int (*encap_rcv)(struct sock *sk, struct sk_buff *skb);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This is an encapsulation socket so pass the skb to
|
|
|
|
* the socket's udp_encap_rcv() hook. Otherwise, just
|
|
|
|
* fall through and pass this up the UDP socket.
|
|
|
|
* up->encap_rcv() returns the following value:
|
|
|
|
* =0 if skb was successfully passed to the encap
|
|
|
|
* handler or was discarded by it.
|
|
|
|
* >0 if skb should be passed on to UDP.
|
|
|
|
* <0 if skb should be resubmitted as proto -N
|
|
|
|
*/
|
|
|
|
|
|
|
|
/* if we're overly short, let UDP handle it */
|
|
|
|
encap_rcv = ACCESS_ONCE(up->encap_rcv);
|
2015-03-29 21:00:05 +08:00
|
|
|
if (skb->len > sizeof(struct udphdr) && encap_rcv) {
|
2012-04-27 16:24:08 +08:00
|
|
|
int ret;
|
|
|
|
|
2014-05-08 07:52:39 +08:00
|
|
|
/* Verify checksum before giving to encap */
|
|
|
|
if (udp_lib_checksum_complete(skb))
|
|
|
|
goto csum_error;
|
|
|
|
|
2012-04-27 16:24:08 +08:00
|
|
|
ret = encap_rcv(sk, skb);
|
|
|
|
if (ret <= 0) {
|
|
|
|
UDP_INC_STATS_BH(sock_net(sk),
|
|
|
|
UDP_MIB_INDATAGRAMS,
|
|
|
|
is_udplite);
|
|
|
|
return -ret;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* FALLTHROUGH -- it's a UDP Packet */
|
|
|
|
}
|
|
|
|
|
2006-11-28 03:10:57 +08:00
|
|
|
/*
|
|
|
|
* UDP-Lite specific tests, ignored on UDP sockets (see net/ipv4/udp.c).
|
|
|
|
*/
|
2007-12-03 19:34:16 +08:00
|
|
|
if ((is_udplite & UDPLITE_RECV_CC) && UDP_SKB_CB(skb)->partial_cov) {
|
2006-11-28 03:10:57 +08:00
|
|
|
|
|
|
|
if (up->pcrlen == 0) { /* full coverage was set */
|
2014-11-12 02:59:17 +08:00
|
|
|
net_dbg_ratelimited("UDPLITE6: partial coverage %d while full coverage %d requested\n",
|
|
|
|
UDP_SKB_CB(skb)->cscov, skb->len);
|
2006-11-28 03:10:57 +08:00
|
|
|
goto drop;
|
|
|
|
}
|
|
|
|
if (UDP_SKB_CB(skb)->cscov < up->pcrlen) {
|
2014-11-12 02:59:17 +08:00
|
|
|
net_dbg_ratelimited("UDPLITE6: coverage %d too small, need min %d\n",
|
|
|
|
UDP_SKB_CB(skb)->cscov, up->pcrlen);
|
2006-11-28 03:10:57 +08:00
|
|
|
goto drop;
|
|
|
|
}
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
2011-08-12 03:30:52 +08:00
|
|
|
if (rcu_access_pointer(sk->sk_filter)) {
|
2007-03-07 12:29:58 +08:00
|
|
|
if (udp_lib_checksum_complete(skb))
|
2013-04-29 16:39:56 +08:00
|
|
|
goto csum_error;
|
2007-03-07 12:29:58 +08:00
|
|
|
}
|
2006-11-28 03:10:57 +08:00
|
|
|
|
2014-07-23 02:16:51 +08:00
|
|
|
if (sk_rcvqueues_full(sk, sk->sk_rcvbuf)) {
|
2014-06-26 05:38:13 +08:00
|
|
|
UDP6_INC_STATS_BH(sock_net(sk),
|
|
|
|
UDP_MIB_RCVBUFERRORS, is_udplite);
|
2012-04-27 16:23:59 +08:00
|
|
|
goto drop;
|
2014-06-26 05:38:13 +08:00
|
|
|
}
|
2012-04-27 16:23:59 +08:00
|
|
|
|
ipv4: PKTINFO doesnt need dst reference
Le lundi 07 novembre 2011 à 15:33 +0100, Eric Dumazet a écrit :
> At least, in recent kernels we dont change dst->refcnt in forwarding
> patch (usinf NOREF skb->dst)
>
> One particular point is the atomic_inc(dst->refcnt) we have to perform
> when queuing an UDP packet if socket asked PKTINFO stuff (for example a
> typical DNS server has to setup this option)
>
> I have one patch somewhere that stores the information in skb->cb[] and
> avoid the atomic_{inc|dec}(dst->refcnt).
>
OK I found it, I did some extra tests and believe its ready.
[PATCH net-next] ipv4: IP_PKTINFO doesnt need dst reference
When a socket uses IP_PKTINFO notifications, we currently force a dst
reference for each received skb. Reader has to access dst to get needed
information (rt_iif & rt_spec_dst) and must release dst reference.
We also forced a dst reference if skb was put in socket backlog, even
without IP_PKTINFO handling. This happens under stress/load.
We can instead store the needed information in skb->cb[], so that only
softirq handler really access dst, improving cache hit ratios.
This removes two atomic operations per packet, and false sharing as
well.
On a benchmark using a mono threaded receiver (doing only recvmsg()
calls), I can reach 720.000 pps instead of 570.000 pps.
IP_PKTINFO is typically used by DNS servers, and any multihomed aware
UDP application.
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2011-11-09 15:24:35 +08:00
|
|
|
skb_dst_drop(skb);
|
2007-12-03 19:33:28 +08:00
|
|
|
|
2012-04-27 16:23:59 +08:00
|
|
|
bh_lock_sock(sk);
|
|
|
|
rc = 0;
|
|
|
|
if (!sock_owned_by_user(sk))
|
|
|
|
rc = __udpv6_queue_rcv_skb(sk, skb);
|
|
|
|
else if (sk_add_backlog(sk, skb, sk->sk_rcvbuf)) {
|
|
|
|
bh_unlock_sock(sk);
|
|
|
|
goto drop;
|
|
|
|
}
|
|
|
|
bh_unlock_sock(sk);
|
2012-04-27 16:23:21 +08:00
|
|
|
|
|
|
|
return rc;
|
2014-06-26 05:38:13 +08:00
|
|
|
|
2013-04-29 16:39:56 +08:00
|
|
|
csum_error:
|
|
|
|
UDP6_INC_STATS_BH(sock_net(sk), UDP_MIB_CSUMERRORS, is_udplite);
|
2006-11-28 03:10:57 +08:00
|
|
|
drop:
|
2008-07-06 12:19:40 +08:00
|
|
|
UDP6_INC_STATS_BH(sock_net(sk), UDP_MIB_INERRORS, is_udplite);
|
2012-04-27 16:23:59 +08:00
|
|
|
atomic_inc(&sk->sk_drops);
|
2006-11-28 03:10:57 +08:00
|
|
|
kfree_skb(skb);
|
|
|
|
return -1;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
2014-07-16 11:28:31 +08:00
|
|
|
static bool __udp_v6_is_mcast_sock(struct net *net, struct sock *sk,
|
|
|
|
__be16 loc_port, const struct in6_addr *loc_addr,
|
|
|
|
__be16 rmt_port, const struct in6_addr *rmt_addr,
|
|
|
|
int dif, unsigned short hnum)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2014-07-16 11:28:31 +08:00
|
|
|
struct inet_sock *inet = inet_sk(sk);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2014-07-16 11:28:31 +08:00
|
|
|
if (!net_eq(sock_net(sk), net))
|
|
|
|
return false;
|
|
|
|
|
|
|
|
if (udp_sk(sk)->udp_port_hash != hnum ||
|
|
|
|
sk->sk_family != PF_INET6 ||
|
|
|
|
(inet->inet_dport && inet->inet_dport != rmt_port) ||
|
|
|
|
(!ipv6_addr_any(&sk->sk_v6_daddr) &&
|
|
|
|
!ipv6_addr_equal(&sk->sk_v6_daddr, rmt_addr)) ||
|
2015-05-19 03:08:49 +08:00
|
|
|
(sk->sk_bound_dev_if && sk->sk_bound_dev_if != dif) ||
|
|
|
|
(!ipv6_addr_any(&sk->sk_v6_rcv_saddr) &&
|
|
|
|
!ipv6_addr_equal(&sk->sk_v6_rcv_saddr, loc_addr)))
|
2014-07-16 11:28:31 +08:00
|
|
|
return false;
|
|
|
|
if (!inet6_mc_check(sk, loc_addr, rmt_addr))
|
|
|
|
return false;
|
|
|
|
return true;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
2009-11-08 18:18:52 +08:00
|
|
|
static void flush_stack(struct sock **stack, unsigned int count,
|
|
|
|
struct sk_buff *skb, unsigned int final)
|
|
|
|
{
|
2012-04-27 16:23:59 +08:00
|
|
|
struct sk_buff *skb1 = NULL;
|
2009-11-08 18:18:52 +08:00
|
|
|
struct sock *sk;
|
2012-04-27 16:23:59 +08:00
|
|
|
unsigned int i;
|
2009-11-08 18:18:52 +08:00
|
|
|
|
|
|
|
for (i = 0; i < count; i++) {
|
2009-11-08 18:20:19 +08:00
|
|
|
sk = stack[i];
|
2015-03-29 21:00:04 +08:00
|
|
|
if (likely(!skb1))
|
2012-04-27 16:23:59 +08:00
|
|
|
skb1 = (i == final) ? skb : skb_clone(skb, GFP_ATOMIC);
|
|
|
|
if (!skb1) {
|
|
|
|
atomic_inc(&sk->sk_drops);
|
|
|
|
UDP6_INC_STATS_BH(sock_net(sk), UDP_MIB_RCVBUFERRORS,
|
|
|
|
IS_UDPLITE(sk));
|
|
|
|
UDP6_INC_STATS_BH(sock_net(sk), UDP_MIB_INERRORS,
|
|
|
|
IS_UDPLITE(sk));
|
2009-11-08 18:18:52 +08:00
|
|
|
}
|
2012-04-27 16:23:59 +08:00
|
|
|
|
|
|
|
if (skb1 && udpv6_queue_rcv_skb(sk, skb1) <= 0)
|
|
|
|
skb1 = NULL;
|
2014-07-16 11:28:32 +08:00
|
|
|
sock_put(sk);
|
2009-11-08 18:18:52 +08:00
|
|
|
}
|
2012-04-27 16:23:59 +08:00
|
|
|
if (unlikely(skb1))
|
|
|
|
kfree_skb(skb1);
|
2009-11-08 18:18:52 +08:00
|
|
|
}
|
2014-05-03 07:29:58 +08:00
|
|
|
|
|
|
|
static void udp6_csum_zero_error(struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
/* RFC 2460 section 8.1 says that we SHOULD log
|
|
|
|
* this error. Well, it is reasonable.
|
|
|
|
*/
|
2014-11-12 02:59:17 +08:00
|
|
|
net_dbg_ratelimited("IPv6: udp checksum is 0 for [%pI6c]:%u->[%pI6c]:%u\n",
|
|
|
|
&ipv6_hdr(skb)->saddr, ntohs(udp_hdr(skb)->source),
|
|
|
|
&ipv6_hdr(skb)->daddr, ntohs(udp_hdr(skb)->dest));
|
2014-05-03 07:29:58 +08:00
|
|
|
}
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/*
|
|
|
|
* Note: called only from the BH handler context,
|
|
|
|
* so we don't need to lock the hashes.
|
|
|
|
*/
|
2008-06-17 08:12:11 +08:00
|
|
|
static int __udp6_lib_mcast_deliver(struct net *net, struct sk_buff *skb,
|
2011-04-22 12:53:02 +08:00
|
|
|
const struct in6_addr *saddr, const struct in6_addr *daddr,
|
2014-11-07 02:37:54 +08:00
|
|
|
struct udp_table *udptable, int proto)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2009-11-08 18:18:52 +08:00
|
|
|
struct sock *sk, *stack[256 / sizeof(struct sock *)];
|
2007-03-14 01:28:48 +08:00
|
|
|
const struct udphdr *uh = udp_hdr(skb);
|
2014-07-16 11:28:31 +08:00
|
|
|
struct hlist_nulls_node *node;
|
|
|
|
unsigned short hnum = ntohs(uh->dest);
|
|
|
|
struct udp_hslot *hslot = udp_hashslot(udptable, net, hnum);
|
|
|
|
int dif = inet6_iif(skb);
|
2014-07-16 11:28:32 +08:00
|
|
|
unsigned int count = 0, offset = offsetof(typeof(*sk), sk_nulls_node);
|
|
|
|
unsigned int hash2 = 0, hash2_any = 0, use_hash2 = (hslot->count > 10);
|
2014-11-07 02:37:54 +08:00
|
|
|
bool inner_flushed = false;
|
2014-07-16 11:28:32 +08:00
|
|
|
|
|
|
|
if (use_hash2) {
|
|
|
|
hash2_any = udp6_portaddr_hash(net, &in6addr_any, hnum) &
|
|
|
|
udp_table.mask;
|
|
|
|
hash2 = udp6_portaddr_hash(net, daddr, hnum) & udp_table.mask;
|
|
|
|
start_lookup:
|
|
|
|
hslot = &udp_table.hash2[hash2];
|
|
|
|
offset = offsetof(typeof(*sk), __sk_common.skc_portaddr_node);
|
|
|
|
}
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2008-10-29 16:41:45 +08:00
|
|
|
spin_lock(&hslot->lock);
|
2014-07-16 11:28:32 +08:00
|
|
|
sk_nulls_for_each_entry_offset(sk, node, &hslot->head, offset) {
|
2014-07-16 11:28:31 +08:00
|
|
|
if (__udp_v6_is_mcast_sock(net, sk,
|
|
|
|
uh->dest, daddr,
|
|
|
|
uh->source, saddr,
|
|
|
|
dif, hnum) &&
|
|
|
|
/* If zero checksum and no_check is not on for
|
|
|
|
* the socket then skip it.
|
|
|
|
*/
|
|
|
|
(uh->check || udp_sk(sk)->no_check6_rx)) {
|
|
|
|
if (unlikely(count == ARRAY_SIZE(stack))) {
|
|
|
|
flush_stack(stack, count, skb, ~0);
|
2014-11-07 02:37:54 +08:00
|
|
|
inner_flushed = true;
|
2014-07-16 11:28:31 +08:00
|
|
|
count = 0;
|
|
|
|
}
|
2014-05-03 07:29:58 +08:00
|
|
|
stack[count++] = sk;
|
2014-07-16 11:28:32 +08:00
|
|
|
sock_hold(sk);
|
2007-12-31 16:29:24 +08:00
|
|
|
}
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
2009-11-08 18:18:52 +08:00
|
|
|
|
2008-10-29 16:41:45 +08:00
|
|
|
spin_unlock(&hslot->lock);
|
2009-11-08 18:18:52 +08:00
|
|
|
|
2014-07-16 11:28:32 +08:00
|
|
|
/* Also lookup *:port if we are using hash2 and haven't done so yet. */
|
|
|
|
if (use_hash2 && hash2 != hash2_any) {
|
|
|
|
hash2 = hash2_any;
|
|
|
|
goto start_lookup;
|
|
|
|
}
|
|
|
|
|
2009-11-08 18:18:52 +08:00
|
|
|
if (count) {
|
|
|
|
flush_stack(stack, count, skb, count - 1);
|
|
|
|
} else {
|
2014-11-07 02:37:54 +08:00
|
|
|
if (!inner_flushed)
|
|
|
|
UDP_INC_STATS_BH(net, UDP_MIB_IGNOREDMULTI,
|
|
|
|
proto == IPPROTO_UDPLITE);
|
|
|
|
consume_skb(skb);
|
2009-11-08 18:18:52 +08:00
|
|
|
}
|
2006-11-28 03:10:57 +08:00
|
|
|
return 0;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
2008-10-29 16:41:45 +08:00
|
|
|
int __udp6_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,
|
2007-03-26 11:10:56 +08:00
|
|
|
int proto)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2010-02-18 16:25:24 +08:00
|
|
|
struct net *net = dev_net(skb->dev);
|
2005-04-17 06:20:36 +08:00
|
|
|
struct sock *sk;
|
2007-02-09 22:24:49 +08:00
|
|
|
struct udphdr *uh;
|
2011-04-22 12:53:02 +08:00
|
|
|
const struct in6_addr *saddr, *daddr;
|
2005-04-17 06:20:36 +08:00
|
|
|
u32 ulen = 0;
|
|
|
|
|
|
|
|
if (!pskb_may_pull(skb, sizeof(struct udphdr)))
|
2010-05-06 11:44:35 +08:00
|
|
|
goto discard;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2007-04-26 08:54:47 +08:00
|
|
|
saddr = &ipv6_hdr(skb)->saddr;
|
|
|
|
daddr = &ipv6_hdr(skb)->daddr;
|
2007-03-14 01:28:48 +08:00
|
|
|
uh = udp_hdr(skb);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
ulen = ntohs(uh->len);
|
2006-11-28 03:10:57 +08:00
|
|
|
if (ulen > skb->len)
|
|
|
|
goto short_packet;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2007-03-26 11:10:56 +08:00
|
|
|
if (proto == IPPROTO_UDP) {
|
|
|
|
/* UDP validates ulen. */
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2006-11-28 03:10:57 +08:00
|
|
|
/* Check for jumbo payload */
|
|
|
|
if (ulen == 0)
|
|
|
|
ulen = skb->len;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2006-11-28 03:10:57 +08:00
|
|
|
if (ulen < sizeof(*uh))
|
|
|
|
goto short_packet;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2006-11-28 03:10:57 +08:00
|
|
|
if (ulen < skb->len) {
|
|
|
|
if (pskb_trim_rcsum(skb, ulen))
|
|
|
|
goto short_packet;
|
2007-04-26 08:54:47 +08:00
|
|
|
saddr = &ipv6_hdr(skb)->saddr;
|
|
|
|
daddr = &ipv6_hdr(skb)->daddr;
|
2007-03-14 01:28:48 +08:00
|
|
|
uh = udp_hdr(skb);
|
2006-11-28 03:10:57 +08:00
|
|
|
}
|
|
|
|
}
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2007-03-26 11:10:56 +08:00
|
|
|
if (udp6_csum_init(skb, uh, proto))
|
2013-04-29 16:39:56 +08:00
|
|
|
goto csum_error;
|
2007-03-26 11:10:56 +08:00
|
|
|
|
2007-02-09 22:24:49 +08:00
|
|
|
/*
|
|
|
|
* Multicast receive code
|
2005-04-17 06:20:36 +08:00
|
|
|
*/
|
2006-11-28 03:10:57 +08:00
|
|
|
if (ipv6_addr_is_multicast(daddr))
|
2008-06-17 08:12:11 +08:00
|
|
|
return __udp6_lib_mcast_deliver(net, skb,
|
2014-11-07 02:37:54 +08:00
|
|
|
saddr, daddr, udptable, proto);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
/* Unicast */
|
|
|
|
|
2007-02-09 22:24:49 +08:00
|
|
|
/*
|
2005-04-17 06:20:36 +08:00
|
|
|
* check socket cache ... must talk to Alan about his plans
|
|
|
|
* for sock caches... i'll skip this for now.
|
|
|
|
*/
|
2008-10-08 03:38:32 +08:00
|
|
|
sk = __udp6_lib_lookup_skb(skb, uh->source, uh->dest, udptable);
|
2015-03-29 21:00:05 +08:00
|
|
|
if (sk) {
|
2013-06-10 16:40:00 +08:00
|
|
|
int ret;
|
|
|
|
|
2014-05-23 23:47:32 +08:00
|
|
|
if (!uh->check && !udp_sk(sk)->no_check6_rx) {
|
2014-05-06 07:43:58 +08:00
|
|
|
sock_put(sk);
|
2014-05-03 07:29:58 +08:00
|
|
|
udp6_csum_zero_error(skb);
|
|
|
|
goto csum_error;
|
|
|
|
}
|
|
|
|
|
2015-01-06 05:56:14 +08:00
|
|
|
if (inet_get_convert_csum(sk) && uh->check && !IS_UDPLITE(sk))
|
2014-09-01 06:12:43 +08:00
|
|
|
skb_checksum_try_convert(skb, IPPROTO_UDP, uh->check,
|
|
|
|
ip6_compute_pseudo);
|
|
|
|
|
2013-06-10 16:40:00 +08:00
|
|
|
ret = udpv6_queue_rcv_skb(sk, skb);
|
2012-04-27 16:23:59 +08:00
|
|
|
sock_put(sk);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2012-04-27 16:23:59 +08:00
|
|
|
/* a return value > 0 means to resubmit the input, but
|
|
|
|
* it wants the return to be -protocol, or 0
|
|
|
|
*/
|
|
|
|
if (ret > 0)
|
|
|
|
return -ret;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2007-03-09 12:42:35 +08:00
|
|
|
return 0;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
2007-02-09 22:24:49 +08:00
|
|
|
|
2014-05-03 07:29:58 +08:00
|
|
|
if (!uh->check) {
|
|
|
|
udp6_csum_zero_error(skb);
|
|
|
|
goto csum_error;
|
|
|
|
}
|
|
|
|
|
2012-04-27 16:23:59 +08:00
|
|
|
if (!xfrm6_policy_check(NULL, XFRM_POLICY_IN, skb))
|
2010-04-28 06:13:20 +08:00
|
|
|
goto discard;
|
2012-04-27 16:23:59 +08:00
|
|
|
|
|
|
|
if (udp_lib_checksum_complete(skb))
|
2013-04-29 16:39:56 +08:00
|
|
|
goto csum_error;
|
2012-04-27 16:23:59 +08:00
|
|
|
|
|
|
|
UDP6_INC_STATS_BH(net, UDP_MIB_NOPORTS, proto == IPPROTO_UDPLITE);
|
|
|
|
icmpv6_send(skb, ICMPV6_DEST_UNREACH, ICMPV6_PORT_UNREACH, 0);
|
|
|
|
|
|
|
|
kfree_skb(skb);
|
2007-03-09 12:42:35 +08:00
|
|
|
return 0;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2007-02-09 22:24:49 +08:00
|
|
|
short_packet:
|
2014-11-12 02:59:17 +08:00
|
|
|
net_dbg_ratelimited("UDP%sv6: short packet: From [%pI6c]:%u %d/%d to [%pI6c]:%u\n",
|
|
|
|
proto == IPPROTO_UDPLITE ? "-Lite" : "",
|
|
|
|
saddr, ntohs(uh->source),
|
|
|
|
ulen, skb->len,
|
|
|
|
daddr, ntohs(uh->dest));
|
2013-04-29 16:39:56 +08:00
|
|
|
goto discard;
|
|
|
|
csum_error:
|
|
|
|
UDP6_INC_STATS_BH(net, UDP_MIB_CSUMERRORS, proto == IPPROTO_UDPLITE);
|
2005-04-17 06:20:36 +08:00
|
|
|
discard:
|
2008-07-06 12:19:40 +08:00
|
|
|
UDP6_INC_STATS_BH(net, UDP_MIB_INERRORS, proto == IPPROTO_UDPLITE);
|
2005-04-17 06:20:36 +08:00
|
|
|
kfree_skb(skb);
|
2007-03-09 12:42:35 +08:00
|
|
|
return 0;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
2006-11-28 03:10:57 +08:00
|
|
|
|
2007-10-16 03:50:28 +08:00
|
|
|
static __inline__ int udpv6_rcv(struct sk_buff *skb)
|
2006-11-28 03:10:57 +08:00
|
|
|
{
|
2008-10-29 16:41:45 +08:00
|
|
|
return __udp6_lib_rcv(skb, &udp_table, IPPROTO_UDP);
|
2006-11-28 03:10:57 +08:00
|
|
|
}
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/*
|
|
|
|
* Throw away all pending data and cancel the corking. Socket is locked.
|
|
|
|
*/
|
|
|
|
static void udp_v6_flush_pending_frames(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct udp_sock *up = udp_sk(sk);
|
|
|
|
|
2008-06-04 19:49:07 +08:00
|
|
|
if (up->pending == AF_INET)
|
|
|
|
udp_flush_pending_frames(sk);
|
|
|
|
else if (up->pending) {
|
2005-04-17 06:20:36 +08:00
|
|
|
up->len = 0;
|
|
|
|
up->pending = 0;
|
|
|
|
ip6_flush_pending_frames(sk);
|
2007-02-09 22:24:49 +08:00
|
|
|
}
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
2009-07-09 16:09:54 +08:00
|
|
|
/**
|
2014-08-25 04:53:10 +08:00
|
|
|
* udp6_hwcsum_outgoing - handle outgoing HW checksumming
|
|
|
|
* @sk: socket we are sending on
|
|
|
|
* @skb: sk_buff containing the filled-in UDP header
|
|
|
|
* (checksum field must be zeroed out)
|
2009-07-09 16:09:54 +08:00
|
|
|
*/
|
|
|
|
static void udp6_hwcsum_outgoing(struct sock *sk, struct sk_buff *skb,
|
|
|
|
const struct in6_addr *saddr,
|
|
|
|
const struct in6_addr *daddr, int len)
|
|
|
|
{
|
|
|
|
unsigned int offset;
|
|
|
|
struct udphdr *uh = udp_hdr(skb);
|
2015-01-31 23:40:16 +08:00
|
|
|
struct sk_buff *frags = skb_shinfo(skb)->frag_list;
|
2009-07-09 16:09:54 +08:00
|
|
|
__wsum csum = 0;
|
|
|
|
|
2015-01-31 23:40:16 +08:00
|
|
|
if (!frags) {
|
2009-07-09 16:09:54 +08:00
|
|
|
/* Only one fragment on the socket. */
|
|
|
|
skb->csum_start = skb_transport_header(skb) - skb->head;
|
|
|
|
skb->csum_offset = offsetof(struct udphdr, check);
|
|
|
|
uh->check = ~csum_ipv6_magic(saddr, daddr, len, IPPROTO_UDP, 0);
|
|
|
|
} else {
|
|
|
|
/*
|
|
|
|
* HW-checksum won't work as there are two or more
|
|
|
|
* fragments on the socket so that all csums of sk_buffs
|
|
|
|
* should be together
|
|
|
|
*/
|
|
|
|
offset = skb_transport_offset(skb);
|
|
|
|
skb->csum = skb_checksum(skb, offset, skb->len - offset, 0);
|
|
|
|
|
|
|
|
skb->ip_summed = CHECKSUM_NONE;
|
|
|
|
|
2015-01-31 23:40:16 +08:00
|
|
|
do {
|
|
|
|
csum = csum_add(csum, frags->csum);
|
|
|
|
} while ((frags = frags->next));
|
2009-07-09 16:09:54 +08:00
|
|
|
|
|
|
|
uh->check = csum_ipv6_magic(saddr, daddr, len, IPPROTO_UDP,
|
|
|
|
csum);
|
|
|
|
if (uh->check == 0)
|
|
|
|
uh->check = CSUM_MANGLED_0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/*
|
|
|
|
* Sending
|
|
|
|
*/
|
|
|
|
|
2015-01-31 23:40:16 +08:00
|
|
|
static int udp_v6_send_skb(struct sk_buff *skb, struct flowi6 *fl6)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2015-01-31 23:40:16 +08:00
|
|
|
struct sock *sk = skb->sk;
|
2005-04-17 06:20:36 +08:00
|
|
|
struct udphdr *uh;
|
|
|
|
int err = 0;
|
2007-12-03 19:34:16 +08:00
|
|
|
int is_udplite = IS_UDPLITE(sk);
|
2006-11-15 13:35:48 +08:00
|
|
|
__wsum csum = 0;
|
2015-01-31 23:40:16 +08:00
|
|
|
int offset = skb_transport_offset(skb);
|
|
|
|
int len = skb->len - offset;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Create a UDP header
|
|
|
|
*/
|
2007-03-14 01:28:48 +08:00
|
|
|
uh = udp_hdr(skb);
|
2011-03-13 05:36:19 +08:00
|
|
|
uh->source = fl6->fl6_sport;
|
|
|
|
uh->dest = fl6->fl6_dport;
|
2015-01-31 23:40:16 +08:00
|
|
|
uh->len = htons(len);
|
2005-04-17 06:20:36 +08:00
|
|
|
uh->check = 0;
|
|
|
|
|
2007-12-03 19:34:16 +08:00
|
|
|
if (is_udplite)
|
2015-01-31 23:40:16 +08:00
|
|
|
csum = udplite_csum(skb);
|
|
|
|
else if (udp_sk(sk)->no_check6_tx) { /* UDP csum disabled */
|
2014-05-03 07:29:58 +08:00
|
|
|
skb->ip_summed = CHECKSUM_NONE;
|
|
|
|
goto send;
|
|
|
|
} else if (skb->ip_summed == CHECKSUM_PARTIAL) { /* UDP hardware csum */
|
2015-01-31 23:40:16 +08:00
|
|
|
udp6_hwcsum_outgoing(sk, skb, &fl6->saddr, &fl6->daddr, len);
|
2009-07-09 16:09:54 +08:00
|
|
|
goto send;
|
|
|
|
} else
|
2015-01-31 23:40:16 +08:00
|
|
|
csum = udp_csum(skb);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2006-11-28 03:10:57 +08:00
|
|
|
/* add protocol-dependent pseudo-header */
|
2011-03-13 05:22:43 +08:00
|
|
|
uh->check = csum_ipv6_magic(&fl6->saddr, &fl6->daddr,
|
2015-01-31 23:40:16 +08:00
|
|
|
len, fl6->flowi6_proto, csum);
|
2005-04-17 06:20:36 +08:00
|
|
|
if (uh->check == 0)
|
2006-11-16 18:36:50 +08:00
|
|
|
uh->check = CSUM_MANGLED_0;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2009-07-09 16:09:54 +08:00
|
|
|
send:
|
2015-01-31 23:40:16 +08:00
|
|
|
err = ip6_send_skb(skb);
|
ip: Report qdisc packet drops
Christoph Lameter pointed out that packet drops at qdisc level where not
accounted in SNMP counters. Only if application sets IP_RECVERR, drops
are reported to user (-ENOBUFS errors) and SNMP counters updated.
IP_RECVERR is used to enable extended reliable error message passing,
but these are not needed to update system wide SNMP stats.
This patch changes things a bit to allow SNMP counters to be updated,
regardless of IP_RECVERR being set or not on the socket.
Example after an UDP tx flood
# netstat -s
...
IP:
1487048 outgoing packets dropped
...
Udp:
...
SndbufErrors: 1487048
send() syscalls, do however still return an OK status, to not
break applications.
Note : send() manual page explicitly says for -ENOBUFS error :
"The output queue for a network interface was full.
This generally indicates that the interface has stopped sending,
but may be caused by transient congestion.
(Normally, this does not occur in Linux. Packets are just silently
dropped when a device queue overflows.) "
This is not true for IP_RECVERR enabled sockets : a send() syscall
that hit a qdisc drop returns an ENOBUFS error.
Many thanks to Christoph, David, and last but not least, Alexey !
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-09-03 09:05:33 +08:00
|
|
|
if (err) {
|
|
|
|
if (err == -ENOBUFS && !inet6_sk(sk)->recverr) {
|
|
|
|
UDP6_INC_STATS_USER(sock_net(sk),
|
|
|
|
UDP_MIB_SNDBUFERRORS, is_udplite);
|
|
|
|
err = 0;
|
|
|
|
}
|
|
|
|
} else
|
|
|
|
UDP6_INC_STATS_USER(sock_net(sk),
|
|
|
|
UDP_MIB_OUTDATAGRAMS, is_udplite);
|
2015-01-31 23:40:16 +08:00
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int udp_v6_push_pending_frames(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct sk_buff *skb;
|
|
|
|
struct udp_sock *up = udp_sk(sk);
|
|
|
|
struct flowi6 fl6;
|
|
|
|
int err = 0;
|
|
|
|
|
|
|
|
if (up->pending == AF_INET)
|
|
|
|
return udp_push_pending_frames(sk);
|
|
|
|
|
|
|
|
/* ip6_finish_skb will release the cork, so make a copy of
|
|
|
|
* fl6 here.
|
|
|
|
*/
|
|
|
|
fl6 = inet_sk(sk)->cork.fl.u.ip6;
|
|
|
|
|
|
|
|
skb = ip6_finish_skb(sk);
|
|
|
|
if (!skb)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
err = udp_v6_send_skb(skb, &fl6);
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
out:
|
|
|
|
up->len = 0;
|
|
|
|
up->pending = 0;
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2015-03-02 15:37:48 +08:00
|
|
|
int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
|
|
|
struct ipv6_txoptions opt_space;
|
|
|
|
struct udp_sock *up = udp_sk(sk);
|
|
|
|
struct inet_sock *inet = inet_sk(sk);
|
|
|
|
struct ipv6_pinfo *np = inet6_sk(sk);
|
2014-01-18 05:53:15 +08:00
|
|
|
DECLARE_SOCKADDR(struct sockaddr_in6 *, sin6, msg->msg_name);
|
2010-06-02 05:35:01 +08:00
|
|
|
struct in6_addr *daddr, *final_p, final;
|
2005-04-17 06:20:36 +08:00
|
|
|
struct ipv6_txoptions *opt = NULL;
|
|
|
|
struct ip6_flowlabel *flowlabel = NULL;
|
2011-03-13 05:22:43 +08:00
|
|
|
struct flowi6 fl6;
|
2005-04-17 06:20:36 +08:00
|
|
|
struct dst_entry *dst;
|
|
|
|
int addr_len = msg->msg_namelen;
|
|
|
|
int ulen = len;
|
|
|
|
int hlimit = -1;
|
2005-09-08 09:19:03 +08:00
|
|
|
int tclass = -1;
|
2010-04-23 19:26:08 +08:00
|
|
|
int dontfrag = -1;
|
2005-04-17 06:20:36 +08:00
|
|
|
int corkreq = up->corkflag || msg->msg_flags&MSG_MORE;
|
|
|
|
int err;
|
2005-09-18 15:30:08 +08:00
|
|
|
int connected = 0;
|
2007-12-03 19:34:16 +08:00
|
|
|
int is_udplite = IS_UDPLITE(sk);
|
2006-11-28 03:10:57 +08:00
|
|
|
int (*getfrag)(void *, char *, int, int, int, struct sk_buff *);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
/* destination address check */
|
|
|
|
if (sin6) {
|
|
|
|
if (addr_len < offsetof(struct sockaddr, sa_data))
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
switch (sin6->sin6_family) {
|
|
|
|
case AF_INET6:
|
|
|
|
if (addr_len < SIN6_LEN_RFC2133)
|
|
|
|
return -EINVAL;
|
|
|
|
daddr = &sin6->sin6_addr;
|
|
|
|
break;
|
|
|
|
case AF_INET:
|
|
|
|
goto do_udp_sendmsg;
|
|
|
|
case AF_UNSPEC:
|
|
|
|
msg->msg_name = sin6 = NULL;
|
|
|
|
msg->msg_namelen = addr_len = 0;
|
|
|
|
daddr = NULL;
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
} else if (!up->pending) {
|
|
|
|
if (sk->sk_state != TCP_ESTABLISHED)
|
|
|
|
return -EDESTADDRREQ;
|
ipv6: make lookups simpler and faster
TCP listener refactoring, part 4 :
To speed up inet lookups, we moved IPv4 addresses from inet to struct
sock_common
Now is time to do the same for IPv6, because it permits us to have fast
lookups for all kind of sockets, including upcoming SYN_RECV.
Getting IPv6 addresses in TCP lookups currently requires two extra cache
lines, plus a dereference (and memory stall).
inet6_sk(sk) does the dereference of inet_sk(__sk)->pinet6
This patch is way bigger than its IPv4 counter part, because for IPv4,
we could add aliases (inet_daddr, inet_rcv_saddr), while on IPv6,
it's not doable easily.
inet6_sk(sk)->daddr becomes sk->sk_v6_daddr
inet6_sk(sk)->rcv_saddr becomes sk->sk_v6_rcv_saddr
And timewait socket also have tw->tw_v6_daddr & tw->tw_v6_rcv_saddr
at the same offset.
We get rid of INET6_TW_MATCH() as INET6_MATCH() is now the generic
macro.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-10-04 06:42:29 +08:00
|
|
|
daddr = &sk->sk_v6_daddr;
|
2007-02-09 22:24:49 +08:00
|
|
|
} else
|
2005-04-17 06:20:36 +08:00
|
|
|
daddr = NULL;
|
|
|
|
|
|
|
|
if (daddr) {
|
2007-08-25 14:16:08 +08:00
|
|
|
if (ipv6_addr_v4mapped(daddr)) {
|
2005-04-17 06:20:36 +08:00
|
|
|
struct sockaddr_in sin;
|
|
|
|
sin.sin_family = AF_INET;
|
2009-10-15 14:30:45 +08:00
|
|
|
sin.sin_port = sin6 ? sin6->sin6_port : inet->inet_dport;
|
2005-04-17 06:20:36 +08:00
|
|
|
sin.sin_addr.s_addr = daddr->s6_addr32[3];
|
|
|
|
msg->msg_name = &sin;
|
|
|
|
msg->msg_namelen = sizeof(sin);
|
|
|
|
do_udp_sendmsg:
|
|
|
|
if (__ipv6_only_sock(sk))
|
|
|
|
return -ENETUNREACH;
|
2015-03-02 15:37:48 +08:00
|
|
|
return udp_sendmsg(sk, msg, len);
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (up->pending == AF_INET)
|
2015-03-02 15:37:48 +08:00
|
|
|
return udp_sendmsg(sk, msg, len);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
/* Rough check on arithmetic overflow,
|
[IPv6]: Fix incorrect length check in rawv6_sendmsg()
In article <20070329.142644.70222545.davem@davemloft.net> (at Thu, 29 Mar 2007 14:26:44 -0700 (PDT)), David Miller <davem@davemloft.net> says:
> From: Sridhar Samudrala <sri@us.ibm.com>
> Date: Thu, 29 Mar 2007 14:17:28 -0700
>
> > The check for length in rawv6_sendmsg() is incorrect.
> > As len is an unsigned int, (len < 0) will never be TRUE.
> > I think checking for IPV6_MAXPLEN(65535) is better.
> >
> > Is it possible to send ipv6 jumbo packets using raw
> > sockets? If so, we can remove this check.
>
> I don't see why such a limitation against jumbo would exist,
> does anyone else?
>
> Thanks for catching this Sridhar. A good compiler should simply
> fail to compile "if (x < 0)" when 'x' is an unsigned type, don't
> you think :-)
Dave, we use "int" for returning value,
so we should fix this anyway, IMHO;
we should not allow len > INT_MAX.
Signed-off-by: YOSHIFUJI Hideaki <yoshfuji@linux-ipv6.org>
Acked-by: Sridhar Samudrala <sri@us.ibm.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-03-31 05:45:35 +08:00
|
|
|
better check is made in ip6_append_data().
|
2005-04-17 06:20:36 +08:00
|
|
|
*/
|
|
|
|
if (len > INT_MAX - sizeof(struct udphdr))
|
|
|
|
return -EMSGSIZE;
|
2007-02-09 22:24:49 +08:00
|
|
|
|
2015-01-31 23:40:17 +08:00
|
|
|
getfrag = is_udplite ? udplite_getfrag : ip_generic_getfrag;
|
2005-04-17 06:20:36 +08:00
|
|
|
if (up->pending) {
|
|
|
|
/*
|
|
|
|
* There are pending frames.
|
|
|
|
* The socket lock must be held while it's corked.
|
|
|
|
*/
|
|
|
|
lock_sock(sk);
|
|
|
|
if (likely(up->pending)) {
|
|
|
|
if (unlikely(up->pending != AF_INET6)) {
|
|
|
|
release_sock(sk);
|
|
|
|
return -EAFNOSUPPORT;
|
|
|
|
}
|
|
|
|
dst = NULL;
|
|
|
|
goto do_append_data;
|
|
|
|
}
|
|
|
|
release_sock(sk);
|
|
|
|
}
|
|
|
|
ulen += sizeof(struct udphdr);
|
|
|
|
|
2011-03-13 05:22:43 +08:00
|
|
|
memset(&fl6, 0, sizeof(fl6));
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
if (sin6) {
|
|
|
|
if (sin6->sin6_port == 0)
|
|
|
|
return -EINVAL;
|
|
|
|
|
2011-03-13 05:36:19 +08:00
|
|
|
fl6.fl6_dport = sin6->sin6_port;
|
2005-04-17 06:20:36 +08:00
|
|
|
daddr = &sin6->sin6_addr;
|
|
|
|
|
|
|
|
if (np->sndflow) {
|
2011-03-13 05:22:43 +08:00
|
|
|
fl6.flowlabel = sin6->sin6_flowinfo&IPV6_FLOWINFO_MASK;
|
|
|
|
if (fl6.flowlabel&IPV6_FLOWLABEL_MASK) {
|
|
|
|
flowlabel = fl6_sock_lookup(sk, fl6.flowlabel);
|
2015-03-29 21:00:04 +08:00
|
|
|
if (!flowlabel)
|
2005-04-17 06:20:36 +08:00
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Otherwise it will be difficult to maintain
|
|
|
|
* sk->sk_dst_cache.
|
|
|
|
*/
|
|
|
|
if (sk->sk_state == TCP_ESTABLISHED &&
|
ipv6: make lookups simpler and faster
TCP listener refactoring, part 4 :
To speed up inet lookups, we moved IPv4 addresses from inet to struct
sock_common
Now is time to do the same for IPv6, because it permits us to have fast
lookups for all kind of sockets, including upcoming SYN_RECV.
Getting IPv6 addresses in TCP lookups currently requires two extra cache
lines, plus a dereference (and memory stall).
inet6_sk(sk) does the dereference of inet_sk(__sk)->pinet6
This patch is way bigger than its IPv4 counter part, because for IPv4,
we could add aliases (inet_daddr, inet_rcv_saddr), while on IPv6,
it's not doable easily.
inet6_sk(sk)->daddr becomes sk->sk_v6_daddr
inet6_sk(sk)->rcv_saddr becomes sk->sk_v6_rcv_saddr
And timewait socket also have tw->tw_v6_daddr & tw->tw_v6_rcv_saddr
at the same offset.
We get rid of INET6_TW_MATCH() as INET6_MATCH() is now the generic
macro.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-10-04 06:42:29 +08:00
|
|
|
ipv6_addr_equal(daddr, &sk->sk_v6_daddr))
|
|
|
|
daddr = &sk->sk_v6_daddr;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
|
|
|
if (addr_len >= sizeof(struct sockaddr_in6) &&
|
|
|
|
sin6->sin6_scope_id &&
|
2013-03-08 10:07:19 +08:00
|
|
|
__ipv6_addr_needs_scope_id(__ipv6_addr_type(daddr)))
|
2011-03-13 05:22:43 +08:00
|
|
|
fl6.flowi6_oif = sin6->sin6_scope_id;
|
2005-04-17 06:20:36 +08:00
|
|
|
} else {
|
|
|
|
if (sk->sk_state != TCP_ESTABLISHED)
|
|
|
|
return -EDESTADDRREQ;
|
|
|
|
|
2011-03-13 05:36:19 +08:00
|
|
|
fl6.fl6_dport = inet->inet_dport;
|
ipv6: make lookups simpler and faster
TCP listener refactoring, part 4 :
To speed up inet lookups, we moved IPv4 addresses from inet to struct
sock_common
Now is time to do the same for IPv6, because it permits us to have fast
lookups for all kind of sockets, including upcoming SYN_RECV.
Getting IPv6 addresses in TCP lookups currently requires two extra cache
lines, plus a dereference (and memory stall).
inet6_sk(sk) does the dereference of inet_sk(__sk)->pinet6
This patch is way bigger than its IPv4 counter part, because for IPv4,
we could add aliases (inet_daddr, inet_rcv_saddr), while on IPv6,
it's not doable easily.
inet6_sk(sk)->daddr becomes sk->sk_v6_daddr
inet6_sk(sk)->rcv_saddr becomes sk->sk_v6_rcv_saddr
And timewait socket also have tw->tw_v6_daddr & tw->tw_v6_rcv_saddr
at the same offset.
We get rid of INET6_TW_MATCH() as INET6_MATCH() is now the generic
macro.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-10-04 06:42:29 +08:00
|
|
|
daddr = &sk->sk_v6_daddr;
|
2011-03-13 05:22:43 +08:00
|
|
|
fl6.flowlabel = np->flow_label;
|
2005-09-18 15:30:08 +08:00
|
|
|
connected = 1;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
2011-03-13 05:22:43 +08:00
|
|
|
if (!fl6.flowi6_oif)
|
|
|
|
fl6.flowi6_oif = sk->sk_bound_dev_if;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2011-03-13 05:22:43 +08:00
|
|
|
if (!fl6.flowi6_oif)
|
|
|
|
fl6.flowi6_oif = np->sticky_pktinfo.ipi6_ifindex;
|
2008-12-16 18:08:29 +08:00
|
|
|
|
2011-03-13 05:22:43 +08:00
|
|
|
fl6.flowi6_mark = sk->sk_mark;
|
2009-10-05 16:24:16 +08:00
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
if (msg->msg_controllen) {
|
|
|
|
opt = &opt_space;
|
|
|
|
memset(opt, 0, sizeof(struct ipv6_txoptions));
|
|
|
|
opt->tot_len = sizeof(*opt);
|
|
|
|
|
2013-01-31 09:02:24 +08:00
|
|
|
err = ip6_datagram_send_ctl(sock_net(sk), sk, msg, &fl6, opt,
|
|
|
|
&hlimit, &tclass, &dontfrag);
|
2005-04-17 06:20:36 +08:00
|
|
|
if (err < 0) {
|
|
|
|
fl6_sock_release(flowlabel);
|
|
|
|
return err;
|
|
|
|
}
|
2011-03-13 05:22:43 +08:00
|
|
|
if ((fl6.flowlabel&IPV6_FLOWLABEL_MASK) && !flowlabel) {
|
|
|
|
flowlabel = fl6_sock_lookup(sk, fl6.flowlabel);
|
2015-03-29 21:00:04 +08:00
|
|
|
if (!flowlabel)
|
2005-04-17 06:20:36 +08:00
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
if (!(opt->opt_nflen|opt->opt_flen))
|
|
|
|
opt = NULL;
|
2005-09-18 15:30:08 +08:00
|
|
|
connected = 0;
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
2015-03-29 21:00:04 +08:00
|
|
|
if (!opt)
|
2005-04-17 06:20:36 +08:00
|
|
|
opt = np->opt;
|
2005-11-20 11:23:18 +08:00
|
|
|
if (flowlabel)
|
|
|
|
opt = fl6_merge_options(&opt_space, flowlabel, opt);
|
|
|
|
opt = ipv6_fixup_options(&opt_space, opt);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2011-03-13 05:22:43 +08:00
|
|
|
fl6.flowi6_proto = sk->sk_protocol;
|
2008-04-11 12:38:24 +08:00
|
|
|
if (!ipv6_addr_any(daddr))
|
2011-11-21 11:39:03 +08:00
|
|
|
fl6.daddr = *daddr;
|
2008-04-11 12:38:24 +08:00
|
|
|
else
|
2011-03-13 05:22:43 +08:00
|
|
|
fl6.daddr.s6_addr[15] = 0x1; /* :: means loopback (BSD'ism) */
|
|
|
|
if (ipv6_addr_any(&fl6.saddr) && !ipv6_addr_any(&np->saddr))
|
2011-11-21 11:39:03 +08:00
|
|
|
fl6.saddr = np->saddr;
|
2011-03-13 05:36:19 +08:00
|
|
|
fl6.fl6_sport = inet->inet_sport;
|
2007-02-09 22:24:49 +08:00
|
|
|
|
2011-03-13 05:22:43 +08:00
|
|
|
final_p = fl6_update_dst(&fl6, opt, &final);
|
2010-06-02 05:35:01 +08:00
|
|
|
if (final_p)
|
2005-09-18 15:30:08 +08:00
|
|
|
connected = 0;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2011-03-13 05:22:43 +08:00
|
|
|
if (!fl6.flowi6_oif && ipv6_addr_is_multicast(&fl6.daddr)) {
|
|
|
|
fl6.flowi6_oif = np->mcast_oif;
|
2005-09-18 15:30:08 +08:00
|
|
|
connected = 0;
|
2012-02-08 17:11:08 +08:00
|
|
|
} else if (!fl6.flowi6_oif)
|
|
|
|
fl6.flowi6_oif = np->ucast_oif;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2011-03-13 05:22:43 +08:00
|
|
|
security_sk_classify_flow(sk, flowi6_to_flowi(&fl6));
|
2006-08-05 14:12:42 +08:00
|
|
|
|
2013-08-28 14:04:14 +08:00
|
|
|
dst = ip6_sk_dst_lookup_flow(sk, &fl6, final_p);
|
2011-03-02 05:19:07 +08:00
|
|
|
if (IS_ERR(dst)) {
|
|
|
|
err = PTR_ERR(dst);
|
|
|
|
dst = NULL;
|
2005-04-17 06:20:36 +08:00
|
|
|
goto out;
|
2007-05-25 09:17:54 +08:00
|
|
|
}
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2014-04-29 10:57:34 +08:00
|
|
|
if (hlimit < 0)
|
|
|
|
hlimit = ip6_sk_dst_hoplimit(np, &fl6, dst);
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2009-08-09 16:12:48 +08:00
|
|
|
if (tclass < 0)
|
2005-09-08 09:19:03 +08:00
|
|
|
tclass = np->tclass;
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
if (msg->msg_flags&MSG_CONFIRM)
|
|
|
|
goto do_confirm;
|
|
|
|
back_from_confirm:
|
|
|
|
|
2015-01-31 23:40:17 +08:00
|
|
|
/* Lockless fast path for the non-corking case */
|
|
|
|
if (!corkreq) {
|
|
|
|
struct sk_buff *skb;
|
|
|
|
|
|
|
|
skb = ip6_make_skb(sk, getfrag, msg, ulen,
|
|
|
|
sizeof(struct udphdr), hlimit, tclass, opt,
|
|
|
|
&fl6, (struct rt6_info *)dst,
|
|
|
|
msg->msg_flags, dontfrag);
|
|
|
|
err = PTR_ERR(skb);
|
|
|
|
if (!IS_ERR_OR_NULL(skb))
|
|
|
|
err = udp_v6_send_skb(skb, &fl6);
|
|
|
|
goto release_dst;
|
|
|
|
}
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
lock_sock(sk);
|
|
|
|
if (unlikely(up->pending)) {
|
|
|
|
/* The socket is already corked while preparing it. */
|
|
|
|
/* ... which is an evident application bug. --ANK */
|
|
|
|
release_sock(sk);
|
|
|
|
|
2014-11-12 02:59:17 +08:00
|
|
|
net_dbg_ratelimited("udp cork app bug 2\n");
|
2005-04-17 06:20:36 +08:00
|
|
|
err = -EINVAL;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
up->pending = AF_INET6;
|
|
|
|
|
|
|
|
do_append_data:
|
2013-10-19 18:29:15 +08:00
|
|
|
if (dontfrag < 0)
|
|
|
|
dontfrag = np->dontfrag;
|
2005-04-17 06:20:36 +08:00
|
|
|
up->len += ulen;
|
2014-11-25 02:23:40 +08:00
|
|
|
err = ip6_append_data(sk, getfrag, msg, ulen,
|
2011-03-13 05:22:43 +08:00
|
|
|
sizeof(struct udphdr), hlimit, tclass, opt, &fl6,
|
2014-08-25 04:53:10 +08:00
|
|
|
(struct rt6_info *)dst,
|
2010-04-23 19:26:08 +08:00
|
|
|
corkreq ? msg->msg_flags|MSG_MORE : msg->msg_flags, dontfrag);
|
2005-04-17 06:20:36 +08:00
|
|
|
if (err)
|
|
|
|
udp_v6_flush_pending_frames(sk);
|
|
|
|
else if (!corkreq)
|
2006-11-28 01:29:59 +08:00
|
|
|
err = udp_v6_push_pending_frames(sk);
|
2006-10-04 05:35:49 +08:00
|
|
|
else if (unlikely(skb_queue_empty(&sk->sk_write_queue)))
|
|
|
|
up->pending = 0;
|
2005-04-17 06:20:36 +08:00
|
|
|
|
2015-01-31 23:40:17 +08:00
|
|
|
if (err > 0)
|
|
|
|
err = np->recverr ? net_xmit_errno(err) : 0;
|
|
|
|
release_sock(sk);
|
|
|
|
|
|
|
|
release_dst:
|
2005-10-04 05:21:58 +08:00
|
|
|
if (dst) {
|
|
|
|
if (connected) {
|
|
|
|
ip6_dst_store(sk, dst,
|
ipv6: make lookups simpler and faster
TCP listener refactoring, part 4 :
To speed up inet lookups, we moved IPv4 addresses from inet to struct
sock_common
Now is time to do the same for IPv6, because it permits us to have fast
lookups for all kind of sockets, including upcoming SYN_RECV.
Getting IPv6 addresses in TCP lookups currently requires two extra cache
lines, plus a dereference (and memory stall).
inet6_sk(sk) does the dereference of inet_sk(__sk)->pinet6
This patch is way bigger than its IPv4 counter part, because for IPv4,
we could add aliases (inet_daddr, inet_rcv_saddr), while on IPv6,
it's not doable easily.
inet6_sk(sk)->daddr becomes sk->sk_v6_daddr
inet6_sk(sk)->rcv_saddr becomes sk->sk_v6_rcv_saddr
And timewait socket also have tw->tw_v6_daddr & tw->tw_v6_rcv_saddr
at the same offset.
We get rid of INET6_TW_MATCH() as INET6_MATCH() is now the generic
macro.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-10-04 06:42:29 +08:00
|
|
|
ipv6_addr_equal(&fl6.daddr, &sk->sk_v6_daddr) ?
|
|
|
|
&sk->sk_v6_daddr : NULL,
|
2006-08-30 08:15:09 +08:00
|
|
|
#ifdef CONFIG_IPV6_SUBTREES
|
2011-03-13 05:22:43 +08:00
|
|
|
ipv6_addr_equal(&fl6.saddr, &np->saddr) ?
|
2006-08-30 08:15:09 +08:00
|
|
|
&np->saddr :
|
|
|
|
#endif
|
|
|
|
NULL);
|
2005-10-04 05:21:58 +08:00
|
|
|
} else {
|
|
|
|
dst_release(dst);
|
|
|
|
}
|
2008-06-04 00:30:25 +08:00
|
|
|
dst = NULL;
|
2005-10-04 05:21:58 +08:00
|
|
|
}
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
out:
|
2008-06-04 00:30:25 +08:00
|
|
|
dst_release(dst);
|
2005-04-17 06:20:36 +08:00
|
|
|
fl6_sock_release(flowlabel);
|
2007-09-15 08:15:01 +08:00
|
|
|
if (!err)
|
2005-04-17 06:20:36 +08:00
|
|
|
return len;
|
2006-08-15 15:00:09 +08:00
|
|
|
/*
|
|
|
|
* ENOBUFS = no kernel mem, SOCK_NOSPACE = no sndbuf space. Reporting
|
|
|
|
* ENOBUFS might not be good (it's not tunable per se), but otherwise
|
|
|
|
* we don't have a good statistic (IpOutDiscards but it can be too many
|
|
|
|
* things). We could add another new stat but at least for now that
|
|
|
|
* seems like overkill.
|
|
|
|
*/
|
|
|
|
if (err == -ENOBUFS || test_bit(SOCK_NOSPACE, &sk->sk_socket->flags)) {
|
2008-07-06 12:19:20 +08:00
|
|
|
UDP6_INC_STATS_USER(sock_net(sk),
|
|
|
|
UDP_MIB_SNDBUFERRORS, is_udplite);
|
2006-08-15 15:00:09 +08:00
|
|
|
}
|
2005-04-17 06:20:36 +08:00
|
|
|
return err;
|
|
|
|
|
|
|
|
do_confirm:
|
|
|
|
dst_confirm(dst);
|
|
|
|
if (!(msg->msg_flags&MSG_PROBE) || len)
|
|
|
|
goto back_from_confirm;
|
|
|
|
err = 0;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
2008-06-15 08:04:49 +08:00
|
|
|
void udpv6_destroy_sock(struct sock *sk)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2013-03-19 14:11:12 +08:00
|
|
|
struct udp_sock *up = udp_sk(sk);
|
2005-04-17 06:20:36 +08:00
|
|
|
lock_sock(sk);
|
|
|
|
udp_v6_flush_pending_frames(sk);
|
|
|
|
release_sock(sk);
|
|
|
|
|
2013-03-19 14:11:12 +08:00
|
|
|
if (static_key_false(&udpv6_encap_needed) && up->encap_type) {
|
|
|
|
void (*encap_destroy)(struct sock *sk);
|
|
|
|
encap_destroy = ACCESS_ONCE(up->encap_destroy);
|
|
|
|
if (encap_destroy)
|
|
|
|
encap_destroy(sk);
|
|
|
|
}
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
inet6_destroy_sock(sk);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Socket option code for UDP
|
|
|
|
*/
|
2006-11-28 03:10:57 +08:00
|
|
|
int udpv6_setsockopt(struct sock *sk, int level, int optname,
|
2009-10-01 07:12:20 +08:00
|
|
|
char __user *optval, unsigned int optlen)
|
2006-03-21 14:45:21 +08:00
|
|
|
{
|
2008-03-07 08:22:02 +08:00
|
|
|
if (level == SOL_UDP || level == SOL_UDPLITE)
|
2006-11-28 01:29:59 +08:00
|
|
|
return udp_lib_setsockopt(sk, level, optname, optval, optlen,
|
|
|
|
udp_v6_push_pending_frames);
|
2006-11-28 03:10:57 +08:00
|
|
|
return ipv6_setsockopt(sk, level, optname, optval, optlen);
|
2006-03-21 14:45:21 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
#ifdef CONFIG_COMPAT
|
2006-11-28 03:10:57 +08:00
|
|
|
int compat_udpv6_setsockopt(struct sock *sk, int level, int optname,
|
2009-10-01 07:12:20 +08:00
|
|
|
char __user *optval, unsigned int optlen)
|
2006-03-21 14:45:21 +08:00
|
|
|
{
|
2008-03-07 08:22:02 +08:00
|
|
|
if (level == SOL_UDP || level == SOL_UDPLITE)
|
2006-11-28 01:29:59 +08:00
|
|
|
return udp_lib_setsockopt(sk, level, optname, optval, optlen,
|
|
|
|
udp_v6_push_pending_frames);
|
2006-11-28 03:10:57 +08:00
|
|
|
return compat_ipv6_setsockopt(sk, level, optname, optval, optlen);
|
2006-03-21 14:45:21 +08:00
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2006-11-28 03:10:57 +08:00
|
|
|
int udpv6_getsockopt(struct sock *sk, int level, int optname,
|
|
|
|
char __user *optval, int __user *optlen)
|
2006-03-21 14:45:21 +08:00
|
|
|
{
|
2008-03-07 08:22:02 +08:00
|
|
|
if (level == SOL_UDP || level == SOL_UDPLITE)
|
2006-11-28 01:29:59 +08:00
|
|
|
return udp_lib_getsockopt(sk, level, optname, optval, optlen);
|
2006-11-28 03:10:57 +08:00
|
|
|
return ipv6_getsockopt(sk, level, optname, optval, optlen);
|
2006-03-21 14:45:21 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
#ifdef CONFIG_COMPAT
|
2006-11-28 03:10:57 +08:00
|
|
|
int compat_udpv6_getsockopt(struct sock *sk, int level, int optname,
|
|
|
|
char __user *optval, int __user *optlen)
|
2006-03-21 14:45:21 +08:00
|
|
|
{
|
2008-03-07 08:22:02 +08:00
|
|
|
if (level == SOL_UDP || level == SOL_UDPLITE)
|
2006-11-28 01:29:59 +08:00
|
|
|
return udp_lib_getsockopt(sk, level, optname, optval, optlen);
|
2006-11-28 03:10:57 +08:00
|
|
|
return compat_ipv6_getsockopt(sk, level, optname, optval, optlen);
|
2006-03-21 14:45:21 +08:00
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2009-09-14 20:22:28 +08:00
|
|
|
static const struct inet6_protocol udpv6_protocol = {
|
2005-04-17 06:20:36 +08:00
|
|
|
.handler = udpv6_rcv,
|
|
|
|
.err_handler = udpv6_err,
|
|
|
|
.flags = INET6_PROTO_NOPOLICY|INET6_PROTO_FINAL,
|
|
|
|
};
|
|
|
|
|
|
|
|
/* ------------------------------------------------------------------------ */
|
|
|
|
#ifdef CONFIG_PROC_FS
|
2006-11-28 03:10:57 +08:00
|
|
|
int udp6_seq_show(struct seq_file *seq, void *v)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2013-05-31 23:05:48 +08:00
|
|
|
if (v == SEQ_START_TOKEN) {
|
|
|
|
seq_puts(seq, IPV6_SEQ_DGRAM_HEADER);
|
|
|
|
} else {
|
|
|
|
int bucket = ((struct udp_iter_state *)seq->private)->bucket;
|
|
|
|
struct inet_sock *inet = inet_sk(v);
|
|
|
|
__u16 srcp = ntohs(inet->inet_sport);
|
|
|
|
__u16 destp = ntohs(inet->inet_dport);
|
|
|
|
ip6_dgram_sock_seq_show(seq, v, srcp, destp, bucket);
|
|
|
|
}
|
2005-04-17 06:20:36 +08:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2011-10-30 14:46:30 +08:00
|
|
|
static const struct file_operations udp6_afinfo_seq_fops = {
|
|
|
|
.owner = THIS_MODULE,
|
|
|
|
.open = udp_seq_open,
|
|
|
|
.read = seq_read,
|
|
|
|
.llseek = seq_lseek,
|
|
|
|
.release = seq_release_net
|
|
|
|
};
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
static struct udp_seq_afinfo udp6_seq_afinfo = {
|
|
|
|
.name = "udp6",
|
|
|
|
.family = AF_INET6,
|
2008-10-29 16:41:45 +08:00
|
|
|
.udp_table = &udp_table,
|
2011-10-30 14:46:30 +08:00
|
|
|
.seq_fops = &udp6_afinfo_seq_fops,
|
2008-03-29 09:24:26 +08:00
|
|
|
.seq_ops = {
|
|
|
|
.show = udp6_seq_show,
|
|
|
|
},
|
2005-04-17 06:20:36 +08:00
|
|
|
};
|
|
|
|
|
2010-01-17 11:35:32 +08:00
|
|
|
int __net_init udp6_proc_init(struct net *net)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2008-03-21 19:14:17 +08:00
|
|
|
return udp_proc_register(net, &udp6_seq_afinfo);
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
|
2008-03-21 19:14:17 +08:00
|
|
|
void udp6_proc_exit(struct net *net) {
|
|
|
|
udp_proc_unregister(net, &udp6_seq_afinfo);
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|
|
|
|
#endif /* CONFIG_PROC_FS */
|
|
|
|
|
2013-05-09 18:28:16 +08:00
|
|
|
void udp_v6_clear_sk(struct sock *sk, int size)
|
|
|
|
{
|
|
|
|
struct inet_sock *inet = inet_sk(sk);
|
|
|
|
|
|
|
|
/* we do not want to clear pinet6 field, because of RCU lookups */
|
|
|
|
sk_prot_clear_portaddr_nulls(sk, offsetof(struct inet_sock, pinet6));
|
|
|
|
|
|
|
|
size -= offsetof(struct inet_sock, pinet6) + sizeof(inet->pinet6);
|
|
|
|
memset(&inet->pinet6 + 1, 0, size);
|
|
|
|
}
|
|
|
|
|
2005-04-17 06:20:36 +08:00
|
|
|
/* ------------------------------------------------------------------------ */
|
|
|
|
|
|
|
|
struct proto udpv6_prot = {
|
2006-03-21 14:48:35 +08:00
|
|
|
.name = "UDPv6",
|
|
|
|
.owner = THIS_MODULE,
|
2006-11-28 03:10:57 +08:00
|
|
|
.close = udp_lib_close,
|
2006-03-21 14:48:35 +08:00
|
|
|
.connect = ip6_datagram_connect,
|
|
|
|
.disconnect = udp_disconnect,
|
|
|
|
.ioctl = udp_ioctl,
|
|
|
|
.destroy = udpv6_destroy_sock,
|
|
|
|
.setsockopt = udpv6_setsockopt,
|
|
|
|
.getsockopt = udpv6_getsockopt,
|
|
|
|
.sendmsg = udpv6_sendmsg,
|
|
|
|
.recvmsg = udpv6_recvmsg,
|
2012-04-27 16:23:21 +08:00
|
|
|
.backlog_rcv = __udpv6_queue_rcv_skb,
|
2006-11-28 03:10:57 +08:00
|
|
|
.hash = udp_lib_hash,
|
|
|
|
.unhash = udp_lib_unhash,
|
udp: add rehash on connect()
commit 30fff923 introduced in linux-2.6.33 (udp: bind() optimisation)
added a secondary hash on UDP, hashed on (local addr, local port).
Problem is that following sequence :
fd = socket(...)
connect(fd, &remote, ...)
not only selects remote end point (address and port), but also sets
local address, while UDP stack stored in secondary hash table the socket
while its local address was INADDR_ANY (or ipv6 equivalent)
Sequence is :
- autobind() : choose a random local port, insert socket in hash tables
[while local address is INADDR_ANY]
- connect() : set remote address and port, change local address to IP
given by a route lookup.
When an incoming UDP frame comes, if more than 10 sockets are found in
primary hash table, we switch to secondary table, and fail to find
socket because its local address changed.
One solution to this problem is to rehash datagram socket if needed.
We add a new rehash(struct socket *) method in "struct proto", and
implement this method for UDP v4 & v6, using a common helper.
This rehashing only takes care of secondary hash table, since primary
hash (based on local port only) is not changed.
Reported-by: Krzysztof Piotr Oledzki <ole@ans.pl>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Tested-by: Krzysztof Piotr Oledzki <ole@ans.pl>
Signed-off-by: David S. Miller <davem@davemloft.net>
2010-09-08 13:08:44 +08:00
|
|
|
.rehash = udp_v6_rehash,
|
2006-03-21 14:48:35 +08:00
|
|
|
.get_port = udp_v6_get_port,
|
2007-12-31 16:29:24 +08:00
|
|
|
.memory_allocated = &udp_memory_allocated,
|
|
|
|
.sysctl_mem = sysctl_udp_mem,
|
|
|
|
.sysctl_wmem = &sysctl_udp_wmem_min,
|
|
|
|
.sysctl_rmem = &sysctl_udp_rmem_min,
|
2006-03-21 14:48:35 +08:00
|
|
|
.obj_size = sizeof(struct udp6_sock),
|
udp: RCU handling for Unicast packets.
Goals are :
1) Optimizing handling of incoming Unicast UDP frames, so that no memory
writes should happen in the fast path.
Note: Multicasts and broadcasts still will need to take a lock,
because doing a full lockless lookup in this case is difficult.
2) No expensive operations in the socket bind/unhash phases :
- No expensive synchronize_rcu() calls.
- No added rcu_head in socket structure, increasing memory needs,
but more important, forcing us to use call_rcu() calls,
that have the bad property of making sockets structure cold.
(rcu grace period between socket freeing and its potential reuse
make this socket being cold in CPU cache).
David did a previous patch using call_rcu() and noticed a 20%
impact on TCP connection rates.
Quoting Cristopher Lameter :
"Right. That results in cacheline cooldown. You'd want to recycle
the object as they are cache hot on a per cpu basis. That is screwed
up by the delayed regular rcu processing. We have seen multiple
regressions due to cacheline cooldown.
The only choice in cacheline hot sensitive areas is to deal with the
complexity that comes with SLAB_DESTROY_BY_RCU or give up on RCU."
- Because udp sockets are allocated from dedicated kmem_cache,
use of SLAB_DESTROY_BY_RCU can help here.
Theory of operation :
---------------------
As the lookup is lockfree (using rcu_read_lock()/rcu_read_unlock()),
special attention must be taken by readers and writers.
Use of SLAB_DESTROY_BY_RCU is tricky too, because a socket can be freed,
reused, inserted in a different chain or in worst case in the same chain
while readers could do lookups in the same time.
In order to avoid loops, a reader must check each socket found in a chain
really belongs to the chain the reader was traversing. If it finds a
mismatch, lookup must start again at the begining. This *restart* loop
is the reason we had to use rdlock for the multicast case, because
we dont want to send same message several times to the same socket.
We use RCU only for fast path.
Thus, /proc/net/udp still takes spinlocks.
Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-10-29 17:11:14 +08:00
|
|
|
.slab_flags = SLAB_DESTROY_BY_RCU,
|
2008-10-29 16:41:45 +08:00
|
|
|
.h.udp_table = &udp_table,
|
2006-03-21 14:45:21 +08:00
|
|
|
#ifdef CONFIG_COMPAT
|
2006-03-21 14:48:35 +08:00
|
|
|
.compat_setsockopt = compat_udpv6_setsockopt,
|
|
|
|
.compat_getsockopt = compat_udpv6_getsockopt,
|
2006-03-21 14:45:21 +08:00
|
|
|
#endif
|
2013-05-09 18:28:16 +08:00
|
|
|
.clear_sk = udp_v6_clear_sk,
|
2005-04-17 06:20:36 +08:00
|
|
|
};
|
|
|
|
|
|
|
|
static struct inet_protosw udpv6_protosw = {
|
|
|
|
.type = SOCK_DGRAM,
|
|
|
|
.protocol = IPPROTO_UDP,
|
|
|
|
.prot = &udpv6_prot,
|
|
|
|
.ops = &inet6_dgram_ops,
|
|
|
|
.flags = INET_PROTOSW_PERMANENT,
|
|
|
|
};
|
|
|
|
|
2007-12-11 18:25:35 +08:00
|
|
|
int __init udpv6_init(void)
|
2005-04-17 06:20:36 +08:00
|
|
|
{
|
2007-12-11 18:25:35 +08:00
|
|
|
int ret;
|
|
|
|
|
2012-11-15 16:49:15 +08:00
|
|
|
ret = inet6_add_protocol(&udpv6_protocol, IPPROTO_UDP);
|
|
|
|
if (ret)
|
2012-11-15 16:49:22 +08:00
|
|
|
goto out;
|
2012-11-15 16:49:15 +08:00
|
|
|
|
2007-12-11 18:25:35 +08:00
|
|
|
ret = inet6_register_protosw(&udpv6_protosw);
|
|
|
|
if (ret)
|
|
|
|
goto out_udpv6_protocol;
|
|
|
|
out:
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
out_udpv6_protocol:
|
|
|
|
inet6_del_protocol(&udpv6_protocol, IPPROTO_UDP);
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
2007-12-13 21:34:58 +08:00
|
|
|
void udpv6_exit(void)
|
2007-12-11 18:25:35 +08:00
|
|
|
{
|
|
|
|
inet6_unregister_protosw(&udpv6_protosw);
|
|
|
|
inet6_del_protocol(&udpv6_protocol, IPPROTO_UDP);
|
2005-04-17 06:20:36 +08:00
|
|
|
}
|