395 lines
22 KiB
Plaintext
395 lines
22 KiB
Plaintext
Ethernet switch device driver model (switchdev)
|
||
===============================================
|
||
Copyright (c) 2014 Jiri Pirko <jiri@resnulli.us>
|
||
Copyright (c) 2014-2015 Scott Feldman <sfeldma@gmail.com>
|
||
|
||
|
||
The Ethernet switch device driver model (switchdev) is an in-kernel driver
|
||
model for switch devices which offload the forwarding (data) plane from the
|
||
kernel.
|
||
|
||
Figure 1 is a block diagram showing the components of the switchdev model for
|
||
an example setup using a data-center-class switch ASIC chip. Other setups
|
||
with SR-IOV or soft switches, such as OVS, are possible.
|
||
|
||
|
||
User-space tools
|
||
|
||
user space |
|
||
+-------------------------------------------------------------------+
|
||
kernel | Netlink
|
||
|
|
||
+--------------+-------------------------------+
|
||
| Network stack |
|
||
| (Linux) |
|
||
| |
|
||
+----------------------------------------------+
|
||
|
||
sw1p2 sw1p4 sw1p6
|
||
sw1p1 + sw1p3 + sw1p5 + eth1
|
||
+ | + | + | +
|
||
| | | | | | |
|
||
+--+----+----+----+-+--+----+---+ +-----+-----+
|
||
| Switch driver | | mgmt |
|
||
| (this document) | | driver |
|
||
| | | |
|
||
+--------------+----------------+ +-----------+
|
||
|
|
||
kernel | HW bus (eg PCI)
|
||
+-------------------------------------------------------------------+
|
||
hardware |
|
||
+--------------+---+------------+
|
||
| Switch device (sw1) |
|
||
| +----+ +--------+
|
||
| | v offloaded data path | mgmt port
|
||
| | | |
|
||
+--|----|----+----+----+----+---+
|
||
| | | | | |
|
||
+ + + + + +
|
||
p1 p2 p3 p4 p5 p6
|
||
|
||
front-panel ports
|
||
|
||
|
||
Fig 1.
|
||
|
||
|
||
Include Files
|
||
-------------
|
||
|
||
#include <linux/netdevice.h>
|
||
#include <net/switchdev.h>
|
||
|
||
|
||
Configuration
|
||
-------------
|
||
|
||
Use "depends NET_SWITCHDEV" in driver's Kconfig to ensure switchdev model
|
||
support is built for driver.
|
||
|
||
|
||
Switch Ports
|
||
------------
|
||
|
||
On switchdev driver initialization, the driver will allocate and register a
|
||
struct net_device (using register_netdev()) for each enumerated physical switch
|
||
port, called the port netdev. A port netdev is the software representation of
|
||
the physical port and provides a conduit for control traffic to/from the
|
||
controller (the kernel) and the network, as well as an anchor point for higher
|
||
level constructs such as bridges, bonds, VLANs, tunnels, and L3 routers. Using
|
||
standard netdev tools (iproute2, ethtool, etc), the port netdev can also
|
||
provide to the user access to the physical properties of the switch port such
|
||
as PHY link state and I/O statistics.
|
||
|
||
There is (currently) no higher-level kernel object for the switch beyond the
|
||
port netdevs. All of the switchdev driver ops are netdev ops or switchdev ops.
|
||
|
||
A switch management port is outside the scope of the switchdev driver model.
|
||
Typically, the management port is not participating in offloaded data plane and
|
||
is loaded with a different driver, such as a NIC driver, on the management port
|
||
device.
|
||
|
||
Switch ID
|
||
^^^^^^^^^
|
||
|
||
The switchdev driver must implement the switchdev op switchdev_port_attr_get
|
||
for SWITCHDEV_ATTR_ID_PORT_PARENT_ID for each port netdev, returning the same
|
||
physical ID for each port of a switch. The ID must be unique between switches
|
||
on the same system. The ID does not need to be unique between switches on
|
||
different systems.
|
||
|
||
The switch ID is used to locate ports on a switch and to know if aggregated
|
||
ports belong to the same switch.
|
||
|
||
Port Netdev Naming
|
||
^^^^^^^^^^^^^^^^^^
|
||
|
||
Udev rules should be used for port netdev naming, using some unique attribute
|
||
of the port as a key, for example the port MAC address or the port PHYS name.
|
||
Hard-coding of kernel netdev names within the driver is discouraged; let the
|
||
kernel pick the default netdev name, and let udev set the final name based on a
|
||
port attribute.
|
||
|
||
Using port PHYS name (ndo_get_phys_port_name) for the key is particularly
|
||
useful for dynamically-named ports where the device names its ports based on
|
||
external configuration. For example, if a physical 40G port is split logically
|
||
into 4 10G ports, resulting in 4 port netdevs, the device can give a unique
|
||
name for each port using port PHYS name. The udev rule would be:
|
||
|
||
SUBSYSTEM=="net", ACTION=="add", ATTR{phys_switch_id}=="<phys_switch_id>", \
|
||
ATTR{phys_port_name}!="", NAME="swX$attr{phys_port_name}"
|
||
|
||
Suggested naming convention is "swXpYsZ", where X is the switch name or ID, Y
|
||
is the port name or ID, and Z is the sub-port name or ID. For example, sw1p1s0
|
||
would be sub-port 0 on port 1 on switch 1.
|
||
|
||
Port Features
|
||
^^^^^^^^^^^^^
|
||
|
||
NETIF_F_NETNS_LOCAL
|
||
|
||
If the switchdev driver (and device) only supports offloading of the default
|
||
network namespace (netns), the driver should set this feature flag to prevent
|
||
the port netdev from being moved out of the default netns. A netns-aware
|
||
driver/device would not set this flag and be responsible for partitioning
|
||
hardware to preserve netns containment. This means hardware cannot forward
|
||
traffic from a port in one namespace to another port in another namespace.
|
||
|
||
Port Topology
|
||
^^^^^^^^^^^^^
|
||
|
||
The port netdevs representing the physical switch ports can be organized into
|
||
higher-level switching constructs. The default construct is a standalone
|
||
router port, used to offload L3 forwarding. Two or more ports can be bonded
|
||
together to form a LAG. Two or more ports (or LAGs) can be bridged to bridge
|
||
L2 networks. VLANs can be applied to sub-divide L2 networks. L2-over-L3
|
||
tunnels can be built on ports. These constructs are built using standard Linux
|
||
tools such as the bridge driver, the bonding/team drivers, and netlink-based
|
||
tools such as iproute2.
|
||
|
||
The switchdev driver can know a particular port's position in the topology by
|
||
monitoring NETDEV_CHANGEUPPER notifications. For example, a port moved into a
|
||
bond will see it's upper master change. If that bond is moved into a bridge,
|
||
the bond's upper master will change. And so on. The driver will track such
|
||
movements to know what position a port is in in the overall topology by
|
||
registering for netdevice events and acting on NETDEV_CHANGEUPPER.
|
||
|
||
L2 Forwarding Offload
|
||
---------------------
|
||
|
||
The idea is to offload the L2 data forwarding (switching) path from the kernel
|
||
to the switchdev device by mirroring bridge FDB entries down to the device. An
|
||
FDB entry is the {port, MAC, VLAN} tuple forwarding destination.
|
||
|
||
To offloading L2 bridging, the switchdev driver/device should support:
|
||
|
||
- Static FDB entries installed on a bridge port
|
||
- Notification of learned/forgotten src mac/vlans from device
|
||
- STP state changes on the port
|
||
- VLAN flooding of multicast/broadcast and unknown unicast packets
|
||
|
||
Static FDB Entries
|
||
^^^^^^^^^^^^^^^^^^
|
||
|
||
The switchdev driver should implement ndo_fdb_add, ndo_fdb_del and ndo_fdb_dump
|
||
to support static FDB entries installed to the device. Static bridge FDB
|
||
entries are installed, for example, using iproute2 bridge cmd:
|
||
|
||
bridge fdb add ADDR dev DEV [vlan VID] [self]
|
||
|
||
The driver should use the helper switchdev_port_fdb_xxx ops for ndo_fdb_xxx
|
||
ops, and handle add/delete/dump of SWITCHDEV_OBJ_ID_PORT_FDB object using
|
||
switchdev_port_obj_xxx ops.
|
||
|
||
XXX: what should be done if offloading this rule to hardware fails (for
|
||
example, due to full capacity in hardware tables) ?
|
||
|
||
Note: by default, the bridge does not filter on VLAN and only bridges untagged
|
||
traffic. To enable VLAN support, turn on VLAN filtering:
|
||
|
||
echo 1 >/sys/class/net/<bridge>/bridge/vlan_filtering
|
||
|
||
Notification of Learned/Forgotten Source MAC/VLANs
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
||
The switch device will learn/forget source MAC address/VLAN on ingress packets
|
||
and notify the switch driver of the mac/vlan/port tuples. The switch driver,
|
||
in turn, will notify the bridge driver using the switchdev notifier call:
|
||
|
||
err = call_switchdev_notifiers(val, dev, info);
|
||
|
||
Where val is SWITCHDEV_FDB_ADD when learning and SWITCHDEV_FDB_DEL when
|
||
forgetting, and info points to a struct switchdev_notifier_fdb_info. On
|
||
SWITCHDEV_FDB_ADD, the bridge driver will install the FDB entry into the
|
||
bridge's FDB and mark the entry as NTF_EXT_LEARNED. The iproute2 bridge
|
||
command will label these entries "offload":
|
||
|
||
$ bridge fdb
|
||
52:54:00:12:35:01 dev sw1p1 master br0 permanent
|
||
00:02:00:00:02:00 dev sw1p1 master br0 offload
|
||
00:02:00:00:02:00 dev sw1p1 self
|
||
52:54:00:12:35:02 dev sw1p2 master br0 permanent
|
||
00:02:00:00:03:00 dev sw1p2 master br0 offload
|
||
00:02:00:00:03:00 dev sw1p2 self
|
||
33:33:00:00:00:01 dev eth0 self permanent
|
||
01:00:5e:00:00:01 dev eth0 self permanent
|
||
33:33:ff:00:00:00 dev eth0 self permanent
|
||
01:80:c2:00:00:0e dev eth0 self permanent
|
||
33:33:00:00:00:01 dev br0 self permanent
|
||
01:00:5e:00:00:01 dev br0 self permanent
|
||
33:33:ff:12:35:01 dev br0 self permanent
|
||
|
||
Learning on the port should be disabled on the bridge using the bridge command:
|
||
|
||
bridge link set dev DEV learning off
|
||
|
||
Learning on the device port should be enabled, as well as learning_sync:
|
||
|
||
bridge link set dev DEV learning on self
|
||
bridge link set dev DEV learning_sync on self
|
||
|
||
Learning_sync attribute enables syncing of the learned/forgotton FDB entry to
|
||
the bridge's FDB. It's possible, but not optimal, to enable learning on the
|
||
device port and on the bridge port, and disable learning_sync.
|
||
|
||
To support learning and learning_sync port attributes, the driver implements
|
||
switchdev op switchdev_port_attr_get/set for
|
||
SWITCHDEV_ATTR_PORT_ID_BRIDGE_FLAGS. The driver should initialize the attributes
|
||
to the hardware defaults.
|
||
|
||
FDB Ageing
|
||
^^^^^^^^^^
|
||
|
||
The bridge will skip ageing FDB entries marked with NTF_EXT_LEARNED and it is
|
||
the responsibility of the port driver/device to age out these entries. If the
|
||
port device supports ageing, when the FDB entry expires, it will notify the
|
||
driver which in turn will notify the bridge with SWITCHDEV_FDB_DEL. If the
|
||
device does not support ageing, the driver can simulate ageing using a
|
||
garbage collection timer to monitor FBD entries. Expired entries will be
|
||
notified to the bridge using SWITCHDEV_FDB_DEL. See rocker driver for
|
||
example of driver running ageing timer.
|
||
|
||
To keep an NTF_EXT_LEARNED entry "alive", the driver should refresh the FDB
|
||
entry by calling call_switchdev_notifiers(SWITCHDEV_FDB_ADD, ...). The
|
||
notification will reset the FDB entry's last-used time to now. The driver
|
||
should rate limit refresh notifications, for example, no more than once a
|
||
second. (The last-used time is visible using the bridge -s fdb option).
|
||
|
||
STP State Change on Port
|
||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
||
Internally or with a third-party STP protocol implementation (e.g. mstpd), the
|
||
bridge driver maintains the STP state for ports, and will notify the switch
|
||
driver of STP state change on a port using the switchdev op
|
||
switchdev_attr_port_set for SWITCHDEV_ATTR_PORT_ID_STP_UPDATE.
|
||
|
||
State is one of BR_STATE_*. The switch driver can use STP state updates to
|
||
update ingress packet filter list for the port. For example, if port is
|
||
DISABLED, no packets should pass, but if port moves to BLOCKED, then STP BPDUs
|
||
and other IEEE 01:80:c2:xx:xx:xx link-local multicast packets can pass.
|
||
|
||
Note that STP BDPUs are untagged and STP state applies to all VLANs on the port
|
||
so packet filters should be applied consistently across untagged and tagged
|
||
VLANs on the port.
|
||
|
||
Flooding L2 domain
|
||
^^^^^^^^^^^^^^^^^^
|
||
|
||
For a given L2 VLAN domain, the switch device should flood multicast/broadcast
|
||
and unknown unicast packets to all ports in domain, if allowed by port's
|
||
current STP state. The switch driver, knowing which ports are within which
|
||
vlan L2 domain, can program the switch device for flooding. The packet may
|
||
be sent to the port netdev for processing by the bridge driver. The
|
||
bridge should not reflood the packet to the same ports the device flooded,
|
||
otherwise there will be duplicate packets on the wire.
|
||
|
||
To avoid duplicate packets, the switch driver should mark a packet as already
|
||
forwarded by setting the skb->offload_fwd_mark bit. The bridge driver will mark
|
||
the skb using the ingress bridge port's mark and prevent it from being forwarded
|
||
through any bridge port with the same mark.
|
||
|
||
It is possible for the switch device to not handle flooding and push the
|
||
packets up to the bridge driver for flooding. This is not ideal as the number
|
||
of ports scale in the L2 domain as the device is much more efficient at
|
||
flooding packets that software.
|
||
|
||
If supported by the device, flood control can be offloaded to it, preventing
|
||
certain netdevs from flooding unicast traffic for which there is no FDB entry.
|
||
|
||
IGMP Snooping
|
||
^^^^^^^^^^^^^
|
||
|
||
In order to support IGMP snooping, the port netdevs should trap to the bridge
|
||
driver all IGMP join and leave messages.
|
||
The bridge multicast module will notify port netdevs on every multicast group
|
||
changed whether it is static configured or dynamically joined/leave.
|
||
The hardware implementation should be forwarding all registered multicast
|
||
traffic groups only to the configured ports.
|
||
|
||
L3 Routing Offload
|
||
------------------
|
||
|
||
Offloading L3 routing requires that device be programmed with FIB entries from
|
||
the kernel, with the device doing the FIB lookup and forwarding. The device
|
||
does a longest prefix match (LPM) on FIB entries matching route prefix and
|
||
forwards the packet to the matching FIB entry's nexthop(s) egress ports.
|
||
|
||
To program the device, the driver has to register a FIB notifier handler
|
||
using register_fib_notifier. The following events are available:
|
||
FIB_EVENT_ENTRY_ADD: used for both adding a new FIB entry to the device,
|
||
or modifying an existing entry on the device.
|
||
FIB_EVENT_ENTRY_DEL: used for removing a FIB entry
|
||
FIB_EVENT_RULE_ADD, FIB_EVENT_RULE_DEL: used to propagate FIB rule changes
|
||
|
||
FIB_EVENT_ENTRY_ADD and FIB_EVENT_ENTRY_DEL events pass:
|
||
|
||
struct fib_entry_notifier_info {
|
||
struct fib_notifier_info info; /* must be first */
|
||
u32 dst;
|
||
int dst_len;
|
||
struct fib_info *fi;
|
||
u8 tos;
|
||
u8 type;
|
||
u32 tb_id;
|
||
u32 nlflags;
|
||
};
|
||
|
||
to add/modify/delete IPv4 dst/dest_len prefix on table tb_id. The *fi
|
||
structure holds details on the route and route's nexthops. *dev is one of the
|
||
port netdevs mentioned in the route's next hop list.
|
||
|
||
Routes offloaded to the device are labeled with "offload" in the ip route
|
||
listing:
|
||
|
||
$ ip route show
|
||
default via 192.168.0.2 dev eth0
|
||
11.0.0.0/30 dev sw1p1 proto kernel scope link src 11.0.0.2 offload
|
||
11.0.0.4/30 via 11.0.0.1 dev sw1p1 proto zebra metric 20 offload
|
||
11.0.0.8/30 dev sw1p2 proto kernel scope link src 11.0.0.10 offload
|
||
11.0.0.12/30 via 11.0.0.9 dev sw1p2 proto zebra metric 20 offload
|
||
12.0.0.2 proto zebra metric 30 offload
|
||
nexthop via 11.0.0.1 dev sw1p1 weight 1
|
||
nexthop via 11.0.0.9 dev sw1p2 weight 1
|
||
12.0.0.3 via 11.0.0.1 dev sw1p1 proto zebra metric 20 offload
|
||
12.0.0.4 via 11.0.0.9 dev sw1p2 proto zebra metric 20 offload
|
||
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.15
|
||
|
||
The "offload" flag is set in case at least one device offloads the FIB entry.
|
||
|
||
XXX: add/mod/del IPv6 FIB API
|
||
|
||
Nexthop Resolution
|
||
^^^^^^^^^^^^^^^^^^
|
||
|
||
The FIB entry's nexthop list contains the nexthop tuple (gateway, dev), but for
|
||
the switch device to forward the packet with the correct dst mac address, the
|
||
nexthop gateways must be resolved to the neighbor's mac address. Neighbor mac
|
||
address discovery comes via the ARP (or ND) process and is available via the
|
||
arp_tbl neighbor table. To resolve the routes nexthop gateways, the driver
|
||
should trigger the kernel's neighbor resolution process. See the rocker
|
||
driver's rocker_port_ipv4_resolve() for an example.
|
||
|
||
The driver can monitor for updates to arp_tbl using the netevent notifier
|
||
NETEVENT_NEIGH_UPDATE. The device can be programmed with resolved nexthops
|
||
for the routes as arp_tbl updates. The driver implements ndo_neigh_destroy
|
||
to know when arp_tbl neighbor entries are purged from the port.
|
||
|
||
Transaction item queue
|
||
^^^^^^^^^^^^^^^^^^^^^^
|
||
|
||
For switchdev ops attr_set and obj_add, there is a 2 phase transaction model
|
||
used. First phase is to "prepare" anything needed, including various checks,
|
||
memory allocation, etc. The goal is to handle the stuff that is not unlikely
|
||
to fail here. The second phase is to "commit" the actual changes.
|
||
|
||
Switchdev provides an infrastructure for sharing items (for example memory
|
||
allocations) between the two phases.
|
||
|
||
The object created by a driver in "prepare" phase and it is queued up by:
|
||
switchdev_trans_item_enqueue()
|
||
During the "commit" phase, the driver gets the object by:
|
||
switchdev_trans_item_dequeue()
|
||
|
||
If a transaction is aborted during "prepare" phase, switchdev code will handle
|
||
cleanup of the queued-up objects.
|