UbiquitousOS
/
OpenCloudOS-Kernel
mirror of https://gitee.com/OpenCloudOS/OpenCloudOS-Kernel.git
11
0
Fork 0
Code Issues Projects Releases Wiki Activity
OpenCloudOS-Kernel/net/packet/internal.h

126 lines
2.7 KiB
C
Raw Normal View History

packet: Introduce net/packet/internal.h header The diag module will need to access some private packet_sock data, so move it to a header in advance. This file will be shared between the af_packet.c and the diag.c Signed-off-by: Pavel Emelyanov <xemul@parallels.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-13 13:49:39 +08:00
#ifndef __PACKET_INTERNAL_H__
#define __PACKET_INTERNAL_H__
struct packet_mclist {
struct packet_mclist *next;
int ifindex;
int count;
unsigned short type;
unsigned short alen;
unsigned char addr[MAX_ADDR_LEN];
};
/* kbdq - kernel block descriptor queue */
struct tpacket_kbdq_core {
struct pgv *pkbdq;
unsigned int feature_req_word;
unsigned int hdrlen;
unsigned char reset_pending_on_curr_blk;
unsigned char delete_blk_timer;
unsigned short kactive_blk_num;
unsigned short blk_sizeof_priv;
/* last_kactive_blk_num:
* trick to see if user-space has caught up
* in order to avoid refreshing timer when every single pkt arrives.
*/
unsigned short last_kactive_blk_num;
char *pkblk_start;
char *pkblk_end;
int kblk_size;
unsigned int knum_blocks;
uint64_t knxt_seq_num;
char *prev;
char *nxt_offset;
struct sk_buff *skb;
atomic_t blk_fill_in_prog;
/* Default is set to 8ms */
#define DEFAULT_PRB_RETIRE_TOV (8)
unsigned short retire_blk_tov;
unsigned short version;
unsigned long tov_in_jiffies;
/* timer to retire an outstanding block */
struct timer_list retire_blk_timer;
};
struct pgv {
char *buffer;
};
struct packet_ring_buffer {
struct pgv *pg_vec;
packet: reorder a member in packet_ring_buffer There's a 4 byte hole in packet_ring_buffer structure before prb_bdqc, that can be filled with 'pending' member, thus we can reduce the overall structure size from 224 bytes to 216 bytes. This also has the side-effect, that in struct packet_sock 2*4 byte holes after the embedded packet_ring_buffer members are removed, and overall, packet_sock can be reduced by 1 cacheline: Before: size: 1344, cachelines: 21, members: 24 After: size: 1280, cachelines: 20, members: 24 Signed-off-by: Daniel Borkmann <dborkman@redhat.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2013-04-19 14:12:28 +08:00
packet: Introduce net/packet/internal.h header The diag module will need to access some private packet_sock data, so move it to a header in advance. This file will be shared between the af_packet.c and the diag.c Signed-off-by: Pavel Emelyanov <xemul@parallels.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-13 13:49:39 +08:00
unsigned int head;
unsigned int frames_per_block;
unsigned int frame_size;
unsigned int frame_max;
unsigned int pg_vec_order;
unsigned int pg_vec_pages;
unsigned int pg_vec_len;
atomic_t pending;
packet: reorder a member in packet_ring_buffer There's a 4 byte hole in packet_ring_buffer structure before prb_bdqc, that can be filled with 'pending' member, thus we can reduce the overall structure size from 224 bytes to 216 bytes. This also has the side-effect, that in struct packet_sock 2*4 byte holes after the embedded packet_ring_buffer members are removed, and overall, packet_sock can be reduced by 1 cacheline: Before: size: 1344, cachelines: 21, members: 24 After: size: 1280, cachelines: 20, members: 24 Signed-off-by: Daniel Borkmann <dborkman@redhat.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2013-04-19 14:12:28 +08:00
struct tpacket_kbdq_core prb_bdqc;
packet: Introduce net/packet/internal.h header The diag module will need to access some private packet_sock data, so move it to a header in advance. This file will be shared between the af_packet.c and the diag.c Signed-off-by: Pavel Emelyanov <xemul@parallels.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-13 13:49:39 +08:00
};
packet: Report fanout status via diag engine Reported value is the same reported by the FANOUT getsockoption, but unlike it, the absent fanout setup results in absent nlattr, rather than in nlattr with zero value. This is done so, since zero fanout report may mean both -- no fanout, and fanout with both id and type zero. Signed-off-by: Pavel Emelyanov <xemul@parallels.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-16 13:36:48 +08:00
extern struct mutex fanout_mutex;
#define PACKET_FANOUT_MAX 256
struct packet_fanout {
#ifdef CONFIG_NET_NS
struct net *net;
#endif
unsigned int num_members;
u16 id;
u8 type;
packet: packet fanout rollover during socket overload Changes: v3->v2: rebase (no other changes) passes selftest v2->v1: read f->num_members only once fix bug: test rollover mode + flag Minimize packet drop in a fanout group. If one socket is full, roll over packets to another from the group. Maintain flow affinity during normal load using an rxhash fanout policy, while dispersing unexpected traffic storms that hit a single cpu, such as spoofed-source DoS flows. Rollover breaks affinity for flows arriving at saturated sockets during those conditions. The patch adds a fanout policy ROLLOVER that rotates between sockets, filling each socket before moving to the next. It also adds a fanout flag ROLLOVER. If passed along with any other fanout policy, the primary policy is applied until the chosen socket is full. Then, rollover selects another socket, to delay packet drop until the entire system is saturated. Probing sockets is not free. Selecting the last used socket, as rollover does, is a greedy approach that maximizes chance of success, at the cost of extreme load imbalance. In practice, with sufficiently long queues to absorb bursts, sockets are drained in parallel and load balance looks uniform in `top`. To avoid contention, scales counters with number of sockets and accesses them lockfree. Values are bounds checked to ensure correctness. Tested using an application with 9 threads pinned to CPUs, one socket per thread and sufficient busywork per packet operation to limits each thread to handling 32 Kpps. When sent 500 Kpps single UDP stream packets, a FANOUT_CPU setup processes 32 Kpps in total without this patch, 270 Kpps with the patch. Tested with read() and with a packet ring (V1). Also, passes psock_fanout.c unit test added to selftests. Signed-off-by: Willem de Bruijn <willemb@google.com> Reviewed-by: Eric Dumazet <edumazet@google.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2013-03-19 18:18:11 +08:00
u8 flags;
packet: Report fanout status via diag engine Reported value is the same reported by the FANOUT getsockoption, but unlike it, the absent fanout setup results in absent nlattr, rather than in nlattr with zero value. This is done so, since zero fanout report may mean both -- no fanout, and fanout with both id and type zero. Signed-off-by: Pavel Emelyanov <xemul@parallels.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-16 13:36:48 +08:00
atomic_t rr_cur;
struct list_head list;
struct sock *arr[PACKET_FANOUT_MAX];
packet: packet fanout rollover during socket overload Changes: v3->v2: rebase (no other changes) passes selftest v2->v1: read f->num_members only once fix bug: test rollover mode + flag Minimize packet drop in a fanout group. If one socket is full, roll over packets to another from the group. Maintain flow affinity during normal load using an rxhash fanout policy, while dispersing unexpected traffic storms that hit a single cpu, such as spoofed-source DoS flows. Rollover breaks affinity for flows arriving at saturated sockets during those conditions. The patch adds a fanout policy ROLLOVER that rotates between sockets, filling each socket before moving to the next. It also adds a fanout flag ROLLOVER. If passed along with any other fanout policy, the primary policy is applied until the chosen socket is full. Then, rollover selects another socket, to delay packet drop until the entire system is saturated. Probing sockets is not free. Selecting the last used socket, as rollover does, is a greedy approach that maximizes chance of success, at the cost of extreme load imbalance. In practice, with sufficiently long queues to absorb bursts, sockets are drained in parallel and load balance looks uniform in `top`. To avoid contention, scales counters with number of sockets and accesses them lockfree. Values are bounds checked to ensure correctness. Tested using an application with 9 threads pinned to CPUs, one socket per thread and sufficient busywork per packet operation to limits each thread to handling 32 Kpps. When sent 500 Kpps single UDP stream packets, a FANOUT_CPU setup processes 32 Kpps in total without this patch, 270 Kpps with the patch. Tested with read() and with a packet ring (V1). Also, passes psock_fanout.c unit test added to selftests. Signed-off-by: Willem de Bruijn <willemb@google.com> Reviewed-by: Eric Dumazet <edumazet@google.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2013-03-19 18:18:11 +08:00
int next[PACKET_FANOUT_MAX];
packet: Report fanout status via diag engine Reported value is the same reported by the FANOUT getsockoption, but unlike it, the absent fanout setup results in absent nlattr, rather than in nlattr with zero value. This is done so, since zero fanout report may mean both -- no fanout, and fanout with both id and type zero. Signed-off-by: Pavel Emelyanov <xemul@parallels.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-16 13:36:48 +08:00
spinlock_t lock;
atomic_t sk_ref;
struct packet_type prot_hook ____cacheline_aligned_in_smp;
};
packet: Introduce net/packet/internal.h header The diag module will need to access some private packet_sock data, so move it to a header in advance. This file will be shared between the af_packet.c and the diag.c Signed-off-by: Pavel Emelyanov <xemul@parallels.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-13 13:49:39 +08:00
struct packet_sock {
/* struct sock has to be the first member of packet_sock */
struct sock sk;
struct packet_fanout *fanout;
packet: account statistics only in tpacket_stats_u Currently, packet_sock has a struct tpacket_stats stats member for TPACKET_V1 and TPACKET_V2 statistic accounting, and with TPACKET_V3 ``union tpacket_stats_u stats_u'' was introduced, where however only statistics for TPACKET_V3 are held, and when copied to user space, TPACKET_V3 does some hackery and access also tpacket_stats' stats, although everything could have been done within the union itself. Unify accounting within the tpacket_stats_u union so that we can remove 8 bytes from packet_sock that are there unnecessary. Note that even if we switch to TPACKET_V3 and would use non mmap(2)ed option, this still works due to the union with same types + offsets, that are exposed to the user space. Signed-off-by: Daniel Borkmann <dborkman@redhat.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2013-04-19 14:12:29 +08:00
union tpacket_stats_u stats;
packet: Introduce net/packet/internal.h header The diag module will need to access some private packet_sock data, so move it to a header in advance. This file will be shared between the af_packet.c and the diag.c Signed-off-by: Pavel Emelyanov <xemul@parallels.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-13 13:49:39 +08:00
struct packet_ring_buffer rx_ring;
struct packet_ring_buffer tx_ring;
int copy_thresh;
spinlock_t bind_lock;
struct mutex pg_vec_lock;
unsigned int running:1, /* prot_hook is attached*/
auxdata:1,
origdev:1,
has_vnet_hdr:1;
int ifindex; /* bound device */
__be16 num;
struct packet_mclist *mclist;
atomic_t mapped;
enum tpacket_versions tp_version;
unsigned int tp_hdrlen;
unsigned int tp_reserve;
unsigned int tp_loss:1;
packet: tx_ring: allow the user to choose tx data offset The tx data offset of packet mmap tx ring used to be : (TPACKET2_HDRLEN - sizeof(struct sockaddr_ll)) The problem is that, with SOCK_RAW socket, the payload (14 bytes after the beginning of the user data) is misaligned. This patch allows to let the user gives an offset for it's tx data if he desires. Set sock option PACKET_TX_HAS_OFF to 1, then specify in each frame of your tx ring tp_net for SOCK_DGRAM, or tp_mac for SOCK_RAW. Signed-off-by: Paul Chavent <paul.chavent@onera.fr> Signed-off-by: David S. Miller <davem@davemloft.net>
2012-11-07 07:10:47 +08:00
unsigned int tp_tx_has_off:1;
packet: Introduce net/packet/internal.h header The diag module will need to access some private packet_sock data, so move it to a header in advance. This file will be shared between the af_packet.c and the diag.c Signed-off-by: Pavel Emelyanov <xemul@parallels.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-13 13:49:39 +08:00
unsigned int tp_tstamp;
packet: fix use after free race in send path when dev is released Salam reported a use after free bug in PF_PACKET that occurs when we're sending out frames on a socket bound device and suddenly the net device is being unregistered. It appears that commit 827d9780 introduced a possible race condition between {t,}packet_snd() and packet_notifier(). In the case of a bound socket, packet_notifier() can drop the last reference to the net_device and {t,}packet_snd() might end up suddenly sending a packet over a freed net_device. To avoid reverting 827d9780 and thus introducing a performance regression compared to the current state of things, we decided to hold a cached RCU protected pointer to the net device and maintain it on write side via bind spin_lock protected register_prot_hook() and __unregister_prot_hook() calls. In {t,}packet_snd() path, we access this pointer under rcu_read_lock through packet_cached_dev_get() that holds reference to the device to prevent it from being freed through packet_notifier() while we're in send path. This is okay to do as dev_put()/dev_hold() are per-cpu counters, so this should not be a performance issue. Also, the code simplifies a bit as we don't need need_rls_dev anymore. Fixes: 827d978037d7 ("af-packet: Use existing netdev reference for bound sockets.") Reported-by: Salam Noureddine <noureddine@aristanetworks.com> Signed-off-by: Daniel Borkmann <dborkman@redhat.com> Signed-off-by: Salam Noureddine <noureddine@aristanetworks.com> Cc: Ben Greear <greearb@candelatech.com> Cc: Eric Dumazet <eric.dumazet@gmail.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2013-11-21 23:50:58 +08:00
struct net_device __rcu *cached_dev;
packet: Introduce net/packet/internal.h header The diag module will need to access some private packet_sock data, so move it to a header in advance. This file will be shared between the af_packet.c and the diag.c Signed-off-by: Pavel Emelyanov <xemul@parallels.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2012-08-13 13:49:39 +08:00
struct packet_type prot_hook ____cacheline_aligned_in_smp;
};
static struct packet_sock *pkt_sk(struct sock *sk)
{
return (struct packet_sock *)sk;
}
#endif