Bug 2082360 - OCP 4.10.4, CNI: SDN; Whereabouts IPAM: Duplicate IP address with bond-cni
Summary: OCP 4.10.4, CNI: SDN; Whereabouts IPAM: Duplicate IP address with bond-cni
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.10
Hardware: x86_64
OS: Linux
Target Milestone: ---
: 4.11.0
Assignee: Douglas Smith
QA Contact: Weibin Liang
Depends On:
Blocks: 2084289
TreeView+ depends on / blocked
Reported: 2022-05-05 21:14 UTC by Will Russell
Modified: 2022-08-10 11:11 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Bond CNI version was at 1.0 CNI libraries but Multus CNI was not compatible with the CNI 1.0 format. Consequence: Bond CNI IPAM results were not properly populated in the network-status annotation. This caused Whereabouts IP reconciliation to erroneously clean up these addresses, as it relies on the network-status annotation to know if an IP address is in use during reconciliation. Fix: Implement net-attach-def client library fix to support CNI 1.0 IP addressing format (in pre-1.0 releases) Result: Whereabouts IPAM can be used properly with Bond CNI.
Clone Of:
: 2084289 (view as bug list)
Last Closed: 2022-08-10 11:10:37 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift multus-cni pull 127 0 None open Bug 2082360: Bumps net-attach-def client library (for CNI v1.0 IP compatibility) 2022-05-11 19:32:45 UTC
Red Hat Product Errata RHSA-2022:5069 0 None None None 2022-08-10 11:11:11 UTC

Description Will Russell 2022-05-05 21:14:50 UTC
Description of problem:

Version-Release number of selected component (if applicable):
OCP 4.10.4
Multus using whereabouts on bonded interface vlan.

How reproducible:
Every time (on customer platform) -- difficult to reproduce internally (not reproduced yet). 

Steps to Reproduce:

1. deploy vlan on target host node
2. deploy networkattachmentdefinition specifying sriov interface net1,net2 and bond0 interface linking both.
3.create pod1 in target hostname on nodeA -- observe multus create bonded interface with ip:
4. create pod2 in secondary namespace on nodeA -- observe multus create bonded interface with ip: (dup entry).

observational statement:
We are seeing that the reconciler is running every 15 minutes and is notating that the ip's are cleaned, at which point we believe the IP is released again into the pool as a viable assignment.

Observed also that the bond0 interface is not providing the IP addresses in the network-status, which may be part of the problem with the reconciler -- may be that it does not detect that this IP is allocated and therefore clears it again for re-assignment after it runs.

Actual results:
Pods deploy with the same secondary interface at bond0 (duplicated IP). Same host node, same networkattachmentdefinition and same vlan tied in.

NETATTACHDEF defines a HUGE whereabouts range .0|254 allocation and is not restrictive by restrictions annotation -- should be no conflict or reason for reuse on these pods.

Expected results:
Pods should deploy every time with a new IP unless host pod is removed, clearing entry for IP allocation.

Additional info:
Please see linked case for specific deployment information, including:
- must-gather
- network attachment definition yamls
- pod deployment describe outs
- pod creation yamls for reproduction

Available to gather any additional information required here.

Have been working with Doug in Forum multus here:
see ongoing discussion:

Comment 5 Douglas Smith 2022-05-06 13:43:52 UTC
Looks like we've also found that the bond-cni isn't returning an IP address result, from the multus CNI cache

sh-4.4# cat /var/lib/cni/multus/results/bond-net1-823c50be23bc812099fe3adb88f1eb8e45c33cf70e2026925442795a3d34275d-net3

Even though the interface shows it's been assigned when exec'ing into the pod:

6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 36:d7:d2:7f:13:4a brd ff:ff:ff:ff:ff:ff
    inet brd scope global bond0

(documentation IP address used)

Comment 6 Douglas Smith 2022-05-06 14:38:35 UTC
Have a proposed upstream fix: https://github.com/k8snetworkplumbingwg/bond-cni/pull/34

Comment 7 Douglas Smith 2022-05-06 19:11:47 UTC
Further analysis:

* The Whereabouts ip reconciler deletes the IP address allocation because it is not shown in the */network-status annotation
* The bond cni is not returning a proper IP address result
* The bond cni is using CNI 1.0.0
* There appears to be CNI version incompatibility between bond-cni @ CNI v1.0.0 and Multus at 0.3.2.

In brief: It appears this problem occurs because of a CNI version incompatibility between Multus CNI and Bond-CNI.

Comment 8 Douglas Smith 2022-05-11 18:10:28 UTC
Just an additional update to note that it looks like we can solve for this inconsistency between CNI versions with Multus CNI. We've got a change merged upstream, and we're looking into bumping up a dependency.

Additionally, we're planning on backporting this fix to 4.10.

Comment 13 Weibin Liang 2022-05-31 22:00:56 UTC
QE can install bonding interface cluster in vsphere only. 
The vsphere cluster installation is blocked by https://bugzilla.redhat.com/show_bug.cgi?id=2092129

Comment 14 Weibin Liang 2022-06-02 14:12:17 UTC
Tested and verified in 4.11.0-0.nightly-2022-06-01-200905 by following steps in https://gist.github.com/dougbtv/bea4214382f1bbb1e820457e14a46eca

[weliang@weliang ~]$ oc get pod
singlepod   1/1     Running   0          7s
[weliang@weliang ~]$ 
[weliang@weliang ~]$ 
[weliang@weliang ~]$ 
[weliang@weliang ~]$ oc describe pod singlepod | grep network-status -A35 
Annotations:  k8s.v1.cni.cncf.io/network-status:
                    "name": "openshift-sdn",
                    "interface": "eth0",
                    "ips": [
                    "default": true,
                    "dns": {}
                    "name": "test1/firstnet",
                    "interface": "net1",
                    "ips": [
                    "mac": "5a:bc:70:4e:ec:5e",
                    "dns": {}
                    "name": "test1/secondnet",
                    "interface": "net2",
                    "ips": [
                    "mac": "52:ad:6d:73:a8:8f",
                    "dns": {}
                    "name": "test1/bond-net",
                    "interface": "net3",
                    "ips": [
                    "mac": "5a:bc:70:4e:ec:5e",
                    "dns": {}
              k8s.v1.cni.cncf.io/networks: firstnet@net1,secondnet@net2,bond-net
[weliang@weliang ~]$ oc describe pod singlepod | grep network-status -A35 | grep -P "network.status|192.0"
Annotations:  k8s.v1.cni.cncf.io/network-status:
[weliang@weliang ~]$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-06-01-200905   True        False         16m     Cluster version is 4.11.0-0.nightly-2022-06-01-200905
[weliang@weliang ~]$

Comment 16 errata-xmlrpc 2022-08-10 11:10:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.