diff options
Diffstat (limited to 'Documentation/networking/dsa/configuration.rst')
| -rw-r--r-- | Documentation/networking/dsa/configuration.rst | 510 |
1 files changed, 338 insertions, 172 deletions
diff --git a/Documentation/networking/dsa/configuration.rst b/Documentation/networking/dsa/configuration.rst index 11bd5e6108c0..6cc4ded3cc23 100644 --- a/Documentation/networking/dsa/configuration.rst +++ b/Documentation/networking/dsa/configuration.rst @@ -5,7 +5,7 @@ DSA switch configuration from userspace ======================================= The DSA switch configuration is not integrated into the main userspace -network configuration suites by now and has to be performed manualy. +network configuration suites by now and has to be performed manually. .. _dsa-config-showcases: @@ -31,28 +31,38 @@ at https://www.kernel.org/pub/linux/utils/net/iproute2/ Through DSA every port of a switch is handled like a normal linux Ethernet interface. The CPU port is the switch port connected to an Ethernet MAC chip. -The corresponding linux Ethernet interface is called the master interface. -All other corresponding linux interfaces are called slave interfaces. +The corresponding linux Ethernet interface is called the conduit interface. +All other corresponding linux interfaces are called user interfaces. -The slave interfaces depend on the master interface. They can only brought up, -when the master interface is up. +The user interfaces depend on the conduit interface being up in order for them +to send or receive traffic. Prior to kernel v5.12, the state of the conduit +interface had to be managed explicitly by the user. Starting with kernel v5.12, +the behavior is as follows: + +- when a DSA user interface is brought up, the conduit interface is + automatically brought up. +- when the conduit interface is brought down, all DSA user interfaces are + automatically brought down. In this documentation the following Ethernet interfaces are used: *eth0* - the master interface + the conduit interface + +*eth1* + another conduit interface *lan1* - a slave interface + a user interface *lan2* - another slave interface + another user interface *lan3* - a third slave interface + a third user interface *wan* - A slave interface dedicated for upstream traffic + A user interface dedicated for upstream traffic Further Ethernet interfaces can be configured similar. The configured IPs and networks are: @@ -78,79 +88,76 @@ The tagging based configuration is desired and supported by the majority of DSA switches. These switches are capable to tag incoming and outgoing traffic without using a VLAN based configuration. -single port -~~~~~~~~~~~ - -.. code-block:: sh - - # configure each interface - ip addr add 192.0.2.1/30 dev lan1 - ip addr add 192.0.2.5/30 dev lan2 - ip addr add 192.0.2.9/30 dev lan3 - - # The master interface needs to be brought up before the slave ports. - ip link set eth0 up +*single port* + .. code-block:: sh - # bring up the slave interfaces - ip link set lan1 up - ip link set lan2 up - ip link set lan3 up + # configure each interface + ip addr add 192.0.2.1/30 dev lan1 + ip addr add 192.0.2.5/30 dev lan2 + ip addr add 192.0.2.9/30 dev lan3 -bridge -~~~~~~ + # For kernels earlier than v5.12, the conduit interface needs to be + # brought up manually before the user ports. + ip link set eth0 up -.. code-block:: sh + # bring up the user interfaces + ip link set lan1 up + ip link set lan2 up + ip link set lan3 up - # The master interface needs to be brought up before the slave ports. - ip link set eth0 up +*bridge* + .. code-block:: sh - # bring up the slave interfaces - ip link set lan1 up - ip link set lan2 up - ip link set lan3 up + # For kernels earlier than v5.12, the conduit interface needs to be + # brought up manually before the user ports. + ip link set eth0 up - # create bridge - ip link add name br0 type bridge + # bring up the user interfaces + ip link set lan1 up + ip link set lan2 up + ip link set lan3 up - # add ports to bridge - ip link set dev lan1 master br0 - ip link set dev lan2 master br0 - ip link set dev lan3 master br0 + # create bridge + ip link add name br0 type bridge - # configure the bridge - ip addr add 192.0.2.129/25 dev br0 + # add ports to bridge + ip link set dev lan1 master br0 + ip link set dev lan2 master br0 + ip link set dev lan3 master br0 - # bring up the bridge - ip link set dev br0 up + # configure the bridge + ip addr add 192.0.2.129/25 dev br0 -gateway -~~~~~~~ + # bring up the bridge + ip link set dev br0 up -.. code-block:: sh +*gateway* + .. code-block:: sh - # The master interface needs to be brought up before the slave ports. - ip link set eth0 up + # For kernels earlier than v5.12, the conduit interface needs to be + # brought up manually before the user ports. + ip link set eth0 up - # bring up the slave interfaces - ip link set wan up - ip link set lan1 up - ip link set lan2 up + # bring up the user interfaces + ip link set wan up + ip link set lan1 up + ip link set lan2 up - # configure the upstream port - ip addr add 192.0.2.1/30 dev wan + # configure the upstream port + ip addr add 192.0.2.1/30 dev wan - # create bridge - ip link add name br0 type bridge + # create bridge + ip link add name br0 type bridge - # add ports to bridge - ip link set dev lan1 master br0 - ip link set dev lan2 master br0 + # add ports to bridge + ip link set dev lan1 master br0 + ip link set dev lan2 master br0 - # configure the bridge - ip addr add 192.0.2.129/25 dev br0 + # configure the bridge + ip addr add 192.0.2.129/25 dev br0 - # bring up the bridge - ip link set dev br0 up + # bring up the bridge + ip link set dev br0 up .. _dsa-vlan-configuration: @@ -161,132 +168,291 @@ A minority of switches are not capable to use a taging protocol (DSA_TAG_PROTO_NONE). These switches can be configured by a VLAN based configuration. -single port -~~~~~~~~~~~ -The configuration can only be set up via VLAN tagging and bridge setup. - -.. code-block:: sh - - # tag traffic on CPU port - ip link add link eth0 name eth0.1 type vlan id 1 - ip link add link eth0 name eth0.2 type vlan id 2 - ip link add link eth0 name eth0.3 type vlan id 3 - - # The master interface needs to be brought up before the slave ports. - ip link set eth0 up - ip link set eth0.1 up - ip link set eth0.2 up - ip link set eth0.3 up - - # bring up the slave interfaces - ip link set lan1 up - ip link set lan2 up - ip link set lan3 up - - # create bridge - ip link add name br0 type bridge - - # activate VLAN filtering - ip link set dev br0 type bridge vlan_filtering 1 - - # add ports to bridges - ip link set dev lan1 master br0 - ip link set dev lan2 master br0 - ip link set dev lan3 master br0 - - # tag traffic on ports - bridge vlan add dev lan1 vid 1 pvid untagged - bridge vlan add dev lan2 vid 2 pvid untagged - bridge vlan add dev lan3 vid 3 pvid untagged - - # configure the VLANs - ip addr add 192.0.2.1/30 dev eth0.1 - ip addr add 192.0.2.5/30 dev eth0.2 - ip addr add 192.0.2.9/30 dev eth0.3 - - # bring up the bridge devices - ip link set br0 up - +*single port* + The configuration can only be set up via VLAN tagging and bridge setup. -bridge -~~~~~~ + .. code-block:: sh -.. code-block:: sh + # tag traffic on CPU port + ip link add link eth0 name eth0.1 type vlan id 1 + ip link add link eth0 name eth0.2 type vlan id 2 + ip link add link eth0 name eth0.3 type vlan id 3 - # tag traffic on CPU port - ip link add link eth0 name eth0.1 type vlan id 1 + # For kernels earlier than v5.12, the conduit interface needs to be + # brought up manually before the user ports. + ip link set eth0 up + ip link set eth0.1 up + ip link set eth0.2 up + ip link set eth0.3 up - # The master interface needs to be brought up before the slave ports. - ip link set eth0 up - ip link set eth0.1 up + # bring up the user interfaces + ip link set lan1 up + ip link set lan2 up + ip link set lan3 up - # bring up the slave interfaces - ip link set lan1 up - ip link set lan2 up - ip link set lan3 up + # create bridge + ip link add name br0 type bridge - # create bridge - ip link add name br0 type bridge + # activate VLAN filtering + ip link set dev br0 type bridge vlan_filtering 1 - # activate VLAN filtering - ip link set dev br0 type bridge vlan_filtering 1 + # add ports to bridges + ip link set dev lan1 master br0 + ip link set dev lan2 master br0 + ip link set dev lan3 master br0 - # add ports to bridge - ip link set dev lan1 master br0 - ip link set dev lan2 master br0 - ip link set dev lan3 master br0 - ip link set eth0.1 master br0 + # tag traffic on ports + bridge vlan add dev lan1 vid 1 pvid untagged + bridge vlan add dev lan2 vid 2 pvid untagged + bridge vlan add dev lan3 vid 3 pvid untagged - # tag traffic on ports - bridge vlan add dev lan1 vid 1 pvid untagged - bridge vlan add dev lan2 vid 1 pvid untagged - bridge vlan add dev lan3 vid 1 pvid untagged + # configure the VLANs + ip addr add 192.0.2.1/30 dev eth0.1 + ip addr add 192.0.2.5/30 dev eth0.2 + ip addr add 192.0.2.9/30 dev eth0.3 - # configure the bridge - ip addr add 192.0.2.129/25 dev br0 + # bring up the bridge devices + ip link set br0 up - # bring up the bridge - ip link set dev br0 up -gateway -~~~~~~~ +*bridge* + .. code-block:: sh -.. code-block:: sh + # tag traffic on CPU port + ip link add link eth0 name eth0.1 type vlan id 1 - # tag traffic on CPU port - ip link add link eth0 name eth0.1 type vlan id 1 - ip link add link eth0 name eth0.2 type vlan id 2 + # For kernels earlier than v5.12, the conduit interface needs to be + # brought up manually before the user ports. + ip link set eth0 up + ip link set eth0.1 up - # The master interface needs to be brought up before the slave ports. - ip link set eth0 up - ip link set eth0.1 up - ip link set eth0.2 up + # bring up the user interfaces + ip link set lan1 up + ip link set lan2 up + ip link set lan3 up - # bring up the slave interfaces - ip link set wan up - ip link set lan1 up - ip link set lan2 up + # create bridge + ip link add name br0 type bridge - # create bridge - ip link add name br0 type bridge + # activate VLAN filtering + ip link set dev br0 type bridge vlan_filtering 1 - # activate VLAN filtering - ip link set dev br0 type bridge vlan_filtering 1 + # add ports to bridge + ip link set dev lan1 master br0 + ip link set dev lan2 master br0 + ip link set dev lan3 master br0 + ip link set eth0.1 master br0 - # add ports to bridges - ip link set dev wan master br0 - ip link set eth0.1 master br0 - ip link set dev lan1 master br0 - ip link set dev lan2 master br0 + # tag traffic on ports + bridge vlan add dev lan1 vid 1 pvid untagged + bridge vlan add dev lan2 vid 1 pvid untagged + bridge vlan add dev lan3 vid 1 pvid untagged - # tag traffic on ports - bridge vlan add dev lan1 vid 1 pvid untagged - bridge vlan add dev lan2 vid 1 pvid untagged - bridge vlan add dev wan vid 2 pvid untagged + # configure the bridge + ip addr add 192.0.2.129/25 dev br0 - # configure the VLANs - ip addr add 192.0.2.1/30 dev eth0.2 - ip addr add 192.0.2.129/25 dev br0 + # bring up the bridge + ip link set dev br0 up - # bring up the bridge devices - ip link set br0 up +*gateway* + .. code-block:: sh + + # tag traffic on CPU port + ip link add link eth0 name eth0.1 type vlan id 1 + ip link add link eth0 name eth0.2 type vlan id 2 + + # For kernels earlier than v5.12, the conduit interface needs to be + # brought up manually before the user ports. + ip link set eth0 up + ip link set eth0.1 up + ip link set eth0.2 up + + # bring up the user interfaces + ip link set wan up + ip link set lan1 up + ip link set lan2 up + + # create bridge + ip link add name br0 type bridge + + # activate VLAN filtering + ip link set dev br0 type bridge vlan_filtering 1 + + # add ports to bridges + ip link set dev wan master br0 + ip link set eth0.1 master br0 + ip link set dev lan1 master br0 + ip link set dev lan2 master br0 + + # tag traffic on ports + bridge vlan add dev lan1 vid 1 pvid untagged + bridge vlan add dev lan2 vid 1 pvid untagged + bridge vlan add dev wan vid 2 pvid untagged + + # configure the VLANs + ip addr add 192.0.2.1/30 dev eth0.2 + ip addr add 192.0.2.129/25 dev br0 + + # bring up the bridge devices + ip link set br0 up + +Forwarding database (FDB) management +------------------------------------ + +The existing DSA switches do not have the necessary hardware support to keep +the software FDB of the bridge in sync with the hardware tables, so the two +tables are managed separately (``bridge fdb show`` queries both, and depending +on whether the ``self`` or ``master`` flags are being used, a ``bridge fdb +add`` or ``bridge fdb del`` command acts upon entries from one or both tables). + +Up until kernel v4.14, DSA only supported user space management of bridge FDB +entries using the bridge bypass operations (which do not update the software +FDB, just the hardware one) using the ``self`` flag (which is optional and can +be omitted). + + .. code-block:: sh + + bridge fdb add dev swp0 00:01:02:03:04:05 self static + # or shorthand + bridge fdb add dev swp0 00:01:02:03:04:05 static + +Due to a bug, the bridge bypass FDB implementation provided by DSA did not +distinguish between ``static`` and ``local`` FDB entries (``static`` are meant +to be forwarded, while ``local`` are meant to be locally terminated, i.e. sent +to the host port). Instead, all FDB entries with the ``self`` flag (implicit or +explicit) are treated by DSA as ``static`` even if they are ``local``. + + .. code-block:: sh + + # This command: + bridge fdb add dev swp0 00:01:02:03:04:05 static + # behaves the same for DSA as this command: + bridge fdb add dev swp0 00:01:02:03:04:05 local + # or shorthand, because the 'local' flag is implicit if 'static' is not + # specified, it also behaves the same as: + bridge fdb add dev swp0 00:01:02:03:04:05 + +The last command is an incorrect way of adding a static bridge FDB entry to a +DSA switch using the bridge bypass operations, and works by mistake. Other +drivers will treat an FDB entry added by the same command as ``local`` and as +such, will not forward it, as opposed to DSA. + +Between kernel v4.14 and v5.14, DSA has supported in parallel two modes of +adding a bridge FDB entry to the switch: the bridge bypass discussed above, as +well as a new mode using the ``master`` flag which installs FDB entries in the +software bridge too. + + .. code-block:: sh + + bridge fdb add dev swp0 00:01:02:03:04:05 master static + +Since kernel v5.14, DSA has gained stronger integration with the bridge's +software FDB, and the support for its bridge bypass FDB implementation (using +the ``self`` flag) has been removed. This results in the following changes: + + .. code-block:: sh + + # This is the only valid way of adding an FDB entry that is supported, + # compatible with v4.14 kernels and later: + bridge fdb add dev swp0 00:01:02:03:04:05 master static + # This command is no longer buggy and the entry is properly treated as + # 'local' instead of being forwarded: + bridge fdb add dev swp0 00:01:02:03:04:05 + # This command no longer installs a static FDB entry to hardware: + bridge fdb add dev swp0 00:01:02:03:04:05 static + +Script writers are therefore encouraged to use the ``master static`` set of +flags when working with bridge FDB entries on DSA switch interfaces. + +Affinity of user ports to CPU ports +----------------------------------- + +Typically, DSA switches are attached to the host via a single Ethernet +interface, but in cases where the switch chip is discrete, the hardware design +may permit the use of 2 or more ports connected to the host, for an increase in +termination throughput. + +DSA can make use of multiple CPU ports in two ways. First, it is possible to +statically assign the termination traffic associated with a certain user port +to be processed by a certain CPU port. This way, user space can implement +custom policies of static load balancing between user ports, by spreading the +affinities according to the available CPU ports. + +Secondly, it is possible to perform load balancing between CPU ports on a per +packet basis, rather than statically assigning user ports to CPU ports. +This can be achieved by placing the DSA conduits under a LAG interface (bonding +or team). DSA monitors this operation and creates a mirror of this software LAG +on the CPU ports facing the physical DSA conduits that constitute the LAG slave +devices. + +To make use of multiple CPU ports, the firmware (device tree) description of +the switch must mark all the links between CPU ports and their DSA conduits +using the ``ethernet`` reference/phandle. At startup, only a single CPU port +and DSA conduit will be used - the numerically first port from the firmware +description which has an ``ethernet`` property. It is up to the user to +configure the system for the switch to use other conduits. + +DSA uses the ``rtnl_link_ops`` mechanism (with a "dsa" ``kind``) to allow +changing the DSA conduit of a user port. The ``IFLA_DSA_CONDUIT`` u32 netlink +attribute contains the ifindex of the conduit device that handles each user +device. The DSA conduit must be a valid candidate based on firmware node +information, or a LAG interface which contains only slaves which are valid +candidates. + +Using iproute2, the following manipulations are possible: + + .. code-block:: sh + + # See the DSA conduit in current use + ip -d link show dev swp0 + (...) + dsa master eth0 + + # Static CPU port distribution + ip link set swp0 type dsa master eth1 + ip link set swp1 type dsa master eth0 + ip link set swp2 type dsa master eth1 + ip link set swp3 type dsa master eth0 + + # CPU ports in LAG, using explicit assignment of the DSA conduit + ip link add bond0 type bond mode balance-xor && ip link set bond0 up + ip link set eth1 down && ip link set eth1 master bond0 + ip link set swp0 type dsa master bond0 + ip link set swp1 type dsa master bond0 + ip link set swp2 type dsa master bond0 + ip link set swp3 type dsa master bond0 + ip link set eth0 down && ip link set eth0 master bond0 + ip -d link show dev swp0 + (...) + dsa master bond0 + + # CPU ports in LAG, relying on implicit migration of the DSA conduit + ip link add bond0 type bond mode balance-xor && ip link set bond0 up + ip link set eth0 down && ip link set eth0 master bond0 + ip link set eth1 down && ip link set eth1 master bond0 + ip -d link show dev swp0 + (...) + dsa master bond0 + +Notice that in the case of CPU ports under a LAG, the use of the +``IFLA_DSA_CONDUIT`` netlink attribute is not strictly needed, but rather, DSA +reacts to the ``IFLA_MASTER`` attribute change of its present conduit (``eth0``) +and migrates all user ports to the new upper of ``eth0``, ``bond0``. Similarly, +when ``bond0`` is destroyed using ``RTM_DELLINK``, DSA migrates the user ports +that were assigned to this interface to the first physical DSA conduit which is +eligible, based on the firmware description (it effectively reverts to the +startup configuration). + +In a setup with more than 2 physical CPU ports, it is therefore possible to mix +static user to CPU port assignment with LAG between DSA conduits. It is not +possible to statically assign a user port towards a DSA conduit that has any +upper interfaces (this includes LAG devices - the conduit must always be the LAG +in this case). + +Live changing of the DSA conduit (and thus CPU port) affinity of a user port is +permitted, in order to allow dynamic redistribution in response to traffic. + +Physical DSA conduits are allowed to join and leave at any time a LAG interface +used as a DSA conduit; however, DSA will reject a LAG interface as a valid +candidate for being a DSA conduit unless it has at least one physical DSA conduit +as a slave device. |