Description of problem: Version-Release number of selected component (if applicable): OCP 4.10.4 CNI: SDN Multus using whereabouts on bonded interface vlan. How reproducible: Every time (on customer platform) -- difficult to reproduce internally (not reproduced yet). Steps to Reproduce: **SEE FIRST COMMENT BELOW PROBLEM DESCRIPTION FOR INTERNAL REPRODUCER STEPS** 1. deploy vlan on target host node 2. deploy networkattachmentdefinition specifying sriov interface net1,net2 and bond0 interface linking both. 3.create pod1 in target hostname on nodeA -- observe multus create bonded interface with ip: 10.0.0.16 4. create pod2 in secondary namespace on nodeA -- observe multus create bonded interface with ip: 10.0.0.16 (dup entry). observational statement: We are seeing that the reconciler is running every 15 minutes and is notating that the ip's are cleaned, at which point we believe the IP is released again into the pool as a viable assignment. Observed also that the bond0 interface is not providing the IP addresses in the network-status, which may be part of the problem with the reconciler -- may be that it does not detect that this IP is allocated and therefore clears it again for re-assignment after it runs. Actual results: Pods deploy with the same secondary interface at bond0 (duplicated IP). Same host node, same networkattachmentdefinition and same vlan tied in. NETATTACHDEF defines a HUGE whereabouts range .0|254 allocation and is not restrictive by restrictions annotation -- should be no conflict or reason for reuse on these pods. Expected results: Pods should deploy every time with a new IP unless host pod is removed, clearing entry for IP allocation. Additional info: Please see linked case for specific deployment information, including: - must-gather - network attachment definition yamls - pod deployment describe outs - pod creation yamls for reproduction Available to gather any additional information required here. Have been working with Doug in Forum multus here: see ongoing discussion: https://coreos.slack.com/archives/CFFSAHWHF/p1651758745821409
Looks like we've also found that the bond-cni isn't returning an IP address result, from the multus CNI cache ``` sh-4.4# cat /var/lib/cni/multus/results/bond-net1-823c50be23bc812099fe3adb88f1eb8e45c33cf70e2026925442795a3d34275d-net3 {"kind":"cniCacheV1","containerId":"823c50be23bc812099fe3adb88f1eb8e45c33cf70e2026925442795a3d34275d","config":"eyAidHlwZSI6ICJib25kIiwgImNuaVZlcnNpb24iOiAiMC4zLjEiLCAibmFtZSI6ICJib25kLW5ldDEiLCAiaWZuYW1lIjogImJvbmQwIiwgIm1vZGUiOiAiYWN0aXZlLWJhY2t1cCIsICJmYWlsT3Zlck1hYyI6IDEsICJsaW5rc0luQ29udGFpbmVyIjogdHJ1ZSwgIm1paW1vbiI6ICIxMDAiLCAibXR1IjogMTUwMCwgImxpbmtzIjogWyB7Im5hbWUiOiAibmV0MSJ9LCB7Im5hbWUiOiAibmV0MiJ9IF0sICJpcGFtIjogeyAidHlwZSI6ICJ3aGVyZWFib3V0cyIsICJyYW5nZSI6ICIxOTIuMC4yLjAvMjQiIH0gfQ==","ifName":"net3","networkName":"bond-net1","cniArgs":[["IgnoreUnknown","true"],["K8S_POD_NAMESPACE","default"],["K8S_POD_NAME","singlepod"],["K8S_POD_INFRA_CONTAINER_ID","823c50be23bc812099fe3adb88f1eb8e45c33cf70e2026925442795a3d34275d"],["K8S_POD_UID","d196f7ab-2d9d-4ae0-8319-93b991670b66"]],"result":{"cniVersion":"0.3.1","dns":{},"interfaces":[{"mac":"36:d7:d2:7f:13:4a","name":"bond0","sandbox":"/var/run/netns/4f5cdec4-6958-4402-9193-8ef6c608d231"}]}} ``` Even though the interface shows it's been assigned when exec'ing into the pod: ``` 6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000 link/ether 36:d7:d2:7f:13:4a brd ff:ff:ff:ff:ff:ff inet 192.0.2.1/24 brd 192.0.2.255 scope global bond0 ``` (documentation IP address used)
Have a proposed upstream fix: https://github.com/k8snetworkplumbingwg/bond-cni/pull/34
Further analysis: * The Whereabouts ip reconciler deletes the IP address allocation because it is not shown in the */network-status annotation * The bond cni is not returning a proper IP address result * The bond cni is using CNI 1.0.0 * There appears to be CNI version incompatibility between bond-cni @ CNI v1.0.0 and Multus at 0.3.2. In brief: It appears this problem occurs because of a CNI version incompatibility between Multus CNI and Bond-CNI.
Just an additional update to note that it looks like we can solve for this inconsistency between CNI versions with Multus CNI. We've got a change merged upstream, and we're looking into bumping up a dependency. Additionally, we're planning on backporting this fix to 4.10.
QE can install bonding interface cluster in vsphere only. The vsphere cluster installation is blocked by https://bugzilla.redhat.com/show_bug.cgi?id=2092129
Tested and verified in 4.11.0-0.nightly-2022-06-01-200905 by following steps in https://gist.github.com/dougbtv/bea4214382f1bbb1e820457e14a46eca [weliang@weliang ~]$ oc get pod NAME READY STATUS RESTARTS AGE singlepod 1/1 Running 0 7s [weliang@weliang ~]$ [weliang@weliang ~]$ [weliang@weliang ~]$ [weliang@weliang ~]$ oc describe pod singlepod | grep network-status -A35 Annotations: k8s.v1.cni.cncf.io/network-status: [{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "10.129.2.7" ], "default": true, "dns": {} },{ "name": "test1/firstnet", "interface": "net1", "ips": [ "10.10.0.1" ], "mac": "5a:bc:70:4e:ec:5e", "dns": {} },{ "name": "test1/secondnet", "interface": "net2", "ips": [ "10.10.0.2" ], "mac": "52:ad:6d:73:a8:8f", "dns": {} },{ "name": "test1/bond-net", "interface": "net3", "ips": [ "192.0.2.1" ], "mac": "5a:bc:70:4e:ec:5e", "dns": {} }] k8s.v1.cni.cncf.io/networks: firstnet@net1,secondnet@net2,bond-net k8s.v1.cni.cncf.io/networks-status: [weliang@weliang ~]$ oc describe pod singlepod | grep network-status -A35 | grep -P "network.status|192.0" Annotations: k8s.v1.cni.cncf.io/network-status: "192.0.2.1" [weliang@weliang ~]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.0-0.nightly-2022-06-01-200905 True False 16m Cluster version is 4.11.0-0.nightly-2022-06-01-200905 [weliang@weliang ~]$
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069