Description of problem: Bond types (rr, xor, tlb and alb) don`t work correctly: * traffic losses * no traffic * no traffic in failover scenarios Version-Release number of selected component (if applicable): How reproducible: 100% Steps to Reproduce: 1.Create VFs via sriov-network and sriov-node policies 2.Create Bond Network attachment definition (bond type balance-alb) 3.Create 2 pods on different workes with bond interface which includes VFs from different PFs 4. run ICMP traffic between pods 5. shut down active VF on one pod Actual results: No traffic Expected results: Traffic should pass towards bond interfaces Additional info: Bod MAC is not changed as expected in balance-alb failover scenario
Could you please provide detailed logs? How was the bond CNI and the bond itself in RHEL configured? Franck pointed me to "man ip link" (https://man7.org/linux/man-pages/man8/ip-link.8.html): state auto|enable|disable - set the virtual link state as seen by the specified VF. Setting to auto means a reflection of the PF link state, enable lets the VF to communicate with other VFs on this host even if the PF link state is down, disable causes the HW to drop any packets sent by the VF. Marcin, perhaps the VF state is set to "enable". Per Franck's recommendation, we want it defaulted to "auto". A user-configurable option would be nice but not a must-have for OCP 4.10.
"auto" is a default value of "link-state": 12: ens1f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 0c:42:a1:bc:f7:b1 brd ff:ff:ff:ff:ff:ff vf 0 link/ether b6:79:17:6b:0a:b0 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 1 link/ether 62:6b:a6:cf:56:f8 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 2 link/ether d6:5d:93:4e:c5:bd brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 3 link/ether 9e:8c:a5:54:75:0f brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 4 link/ether 52:a3:c0:2d:93:1d brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off Even more sriovnetwork doesn`t cahnge "link-state: Spec: Link State: disable Network Namespace: bond-test Resource Name: three Spoof Chk: off 4: ens8f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 40:a6:b7:38:b4:e0 brd ff:ff:ff:ff:ff:ff vf 0 link/ether fa:ab:bb:63:76:87 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off vf 1 link/ether ca:7d:c1:7f:4e:aa brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off vf 2 link/ether 1a:57:27:b9:fa:9e brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off vf 3 link/ether 06:34:c4:5d:70:d2 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off vf 4 link/ether 96:07:cc:c4:b7:79 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
last my statement "Even more sriovnetwork doesn`t cahnge "link-state" is not correct
Evgeny, thanks! Could you please file a new BZ against RHEL reporting this issue and set a dependency with this one? Marcin, it is unlikely to expect the RHEL bond issue to be fixed in time before OCP 4.10 GA. The release notes and/or bond CNI documentation should highlight this known issue (Doc Type field).
Today, I did a basic test with mode 1,2,4,5,6 bonds over SR-IOV VFs, and didn't see any issue. ping could succeed even after failover. So, for BZ #2037214 and https://bugzilla.redhat.com/show_bug.cgi?id=2039755 , if the containers are connected by OVS/OVN virtual network, please make sure the icmp traffic is not blocked by OVS/OVN rules after failover as the src mac will change after failover. kernel: 4.18.0-305.el8.x86_64 NIC: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
(In reply to LiLiang from comment #6) > Today, I did a basic test with mode 1,2,4,5,6 bonds over SR-IOV VFs, and > didn't see any issue. ping could succeed even after failover. > > So, for BZ #2037214 and https://bugzilla.redhat.com/show_bug.cgi?id=2039755 > , if the containers are connected by OVS/OVN virtual network, please make > sure the icmp traffic is not blocked by OVS/OVN rules after failover as the > src mac will change after failover. > > kernel: 4.18.0-305.el8.x86_64 > NIC: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] I did those tests with VM, not container.
This VF configuration must be set via the "linkState" parameter in SR-IOV: https://docs.openshift.com/container-platform/4.10/networking/hardware_networks/configuring-sriov-net-attach.html#nw-sriov-network-object_configuring-sriov-net-attach This needs to be added as a requirement to the bond CNI documentation page that Kevin Quinn is working on in https://github.com/openshift/openshift-docs/pull/47172
Carlos, "linkState" is auto by default. "trust on" fix the issue https://docs.google.com/presentation/d/1GWLNMZl7oaVDCT7jmFl6qOsRfVR7CSoknluKP8JKqlc/edit#slide=id.g255339b51f_0_890
What if the VF is set to non-auto for a different workload, moved back to the VF pool and later added to a bond? Is the link state reset to "auto" or left unchanged? If left unchanged, the requirement that the link state must be "auto" is valid and should be documented.
Evgeny `trust on` is needed to allow the bond inside the pod to change the mac address of the VF Carlos every time we move a vf back from a pod we restore its default mode. but agree with you better to document it
Good, we are all in agreement. The doc request is to note that the SR-IOV VF link state must not be changed from the default "auto" value.
This was replaced by the following bugs, relating to the specific modes: alb: https://bugzilla.redhat.com/show_bug.cgi?id=2109123 tlb: https://bugzilla.redhat.com/show_bug.cgi?id=2106906 *** This bug has been marked as a duplicate of bug 2109123 ***