Bug 2082360

Looks like we've also found that the bond-cni isn't returning an IP address result, from the multus CNI cache

```
sh-4.4# cat /var/lib/cni/multus/results/bond-net1-823c50be23bc812099fe3adb88f1eb8e45c33cf70e2026925442795a3d34275d-net3
{"kind":"cniCacheV1","containerId":"823c50be23bc812099fe3adb88f1eb8e45c33cf70e2026925442795a3d34275d","config":"eyAidHlwZSI6ICJib25kIiwgImNuaVZlcnNpb24iOiAiMC4zLjEiLCAibmFtZSI6ICJib25kLW5ldDEiLCAiaWZuYW1lIjogImJvbmQwIiwgIm1vZGUiOiAiYWN0aXZlLWJhY2t1cCIsICJmYWlsT3Zlck1hYyI6IDEsICJsaW5rc0luQ29udGFpbmVyIjogdHJ1ZSwgIm1paW1vbiI6ICIxMDAiLCAibXR1IjogMTUwMCwgImxpbmtzIjogWyB7Im5hbWUiOiAibmV0MSJ9LCB7Im5hbWUiOiAibmV0MiJ9IF0sICJpcGFtIjogeyAidHlwZSI6ICJ3aGVyZWFib3V0cyIsICJyYW5nZSI6ICIxOTIuMC4yLjAvMjQiIH0gfQ==","ifName":"net3","networkName":"bond-net1","cniArgs":[["IgnoreUnknown","true"],["K8S_POD_NAMESPACE","default"],["K8S_POD_NAME","singlepod"],["K8S_POD_INFRA_CONTAINER_ID","823c50be23bc812099fe3adb88f1eb8e45c33cf70e2026925442795a3d34275d"],["K8S_POD_UID","d196f7ab-2d9d-4ae0-8319-93b991670b66"]],"result":{"cniVersion":"0.3.1","dns":{},"interfaces":[{"mac":"36:d7:d2:7f:13:4a","name":"bond0","sandbox":"/var/run/netns/4f5cdec4-6958-4402-9193-8ef6c608d231"}]}}
```

Even though the interface shows it's been assigned when exec'ing into the pod:

```
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 36:d7:d2:7f:13:4a brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.1/24 brd 192.0.2.255 scope global bond0
```

(documentation IP address used)

Have a proposed upstream fix: https://github.com/k8snetworkplumbingwg/bond-cni/pull/34

Further analysis:

* The Whereabouts ip reconciler deletes the IP address allocation because it is not shown in the */network-status annotation
* The bond cni is not returning a proper IP address result
* The bond cni is using CNI 1.0.0
* There appears to be CNI version incompatibility between bond-cni @ CNI v1.0.0 and Multus at 0.3.2.

In brief: It appears this problem occurs because of a CNI version incompatibility between Multus CNI and Bond-CNI.

Just an additional update to note that it looks like we can solve for this inconsistency between CNI versions with Multus CNI. We've got a change merged upstream, and we're looking into bumping up a dependency.

Additionally, we're planning on backporting this fix to 4.10.

QE can install bonding interface cluster in vsphere only. 
The vsphere cluster installation is blocked by https://bugzilla.redhat.com/show_bug.cgi?id=2092129

Tested and verified in 4.11.0-0.nightly-2022-06-01-200905 by following steps in https://gist.github.com/dougbtv/bea4214382f1bbb1e820457e14a46eca

[weliang@weliang ~]$ oc get pod
NAME        READY   STATUS    RESTARTS   AGE
singlepod   1/1     Running   0          7s
[weliang@weliang ~]$ 
[weliang@weliang ~]$ 
[weliang@weliang ~]$ 
[weliang@weliang ~]$ oc describe pod singlepod | grep network-status -A35 
Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "openshift-sdn",
                    "interface": "eth0",
                    "ips": [
                        "10.129.2.7"
                    ],
                    "default": true,
                    "dns": {}
                },{
                    "name": "test1/firstnet",
                    "interface": "net1",
                    "ips": [
                        "10.10.0.1"
                    ],
                    "mac": "5a:bc:70:4e:ec:5e",
                    "dns": {}
                },{
                    "name": "test1/secondnet",
                    "interface": "net2",
                    "ips": [
                        "10.10.0.2"
                    ],
                    "mac": "52:ad:6d:73:a8:8f",
                    "dns": {}
                },{
                    "name": "test1/bond-net",
                    "interface": "net3",
                    "ips": [
                        "192.0.2.1"
                    ],
                    "mac": "5a:bc:70:4e:ec:5e",
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks: firstnet@net1,secondnet@net2,bond-net
              k8s.v1.cni.cncf.io/networks-status:
[weliang@weliang ~]$ oc describe pod singlepod | grep network-status -A35 | grep -P "network.status|192.0"
Annotations:  k8s.v1.cni.cncf.io/network-status:
                        "192.0.2.1"
[weliang@weliang ~]$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-06-01-200905   True        False         16m     Cluster version is 4.11.0-0.nightly-2022-06-01-200905
[weliang@weliang ~]$

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069