Bug 2063801

Description of problem:

kubevirt-hyperconverged-operator.v4.8.4 was installed in an OCP 4.8.34 cluster. Then, when checking if the operator was installed correctly, it failed because of the well-known issue related to nmstate reported here: https://access.redhat.com/solutions/5794421.


Version-Release number of selected component (if applicable):

OCP 4.8.34
kubevirt-hyperconverged-operator.v4.8.4

How reproducible:

(Note that it is not happening all the times, and when happening, it may affect to different nodes to the ones reported in this BZ)

Steps to Reproduce:

1. Deploy OCP 4.8.34 in a cluster composed by 3 master nodes and 5 worker nodes, using IPI installation and Ansible playbooks from baremetal-deployment.

2. Install hyperconverged-operator

3. Confirm the installation of hyperconverged-operator

Actual results:

We have used DCI in order to deploy the cluster. You can see the DCI job here: https://www.distributed-ci.io/jobs/5ffaa4a3-e880-47c3-bf53-255e4fc78407/jobStates, and you can check the must-gather related to this installation in this URL: https://www.distributed-ci.io/jobs/5ffaa4a3-e880-47c3-bf53-255e4fc78407/files > then download must_gather.tar.gz file and use cnv-must-gather data.

Checking the must-gather, we can see all the nodes are in ready status:

$ omg get nodes
NAME      STATUS  ROLES   AGE    VERSION
master-1  Ready   master  1h41m  v1.21.8+8a3bf4a
master-2  Ready   master  1h44m  v1.21.8+8a3bf4a
worker-4  Ready   master  1h44m  v1.21.8+8a3bf4a
master-0  Ready   worker  1h4m   v1.21.8+8a3bf4a
worker-0  Ready   worker  1h8m   v1.21.8+8a3bf4a
worker-1  Ready   worker  1h9m   v1.21.8+8a3bf4a
worker-2  Ready   worker  1h6m   v1.21.8+8a3bf4a
worker-3  Ready   worker  1h8m   v1.21.8+8a3bf4a

Then, following the same approach commented on https://access.redhat.com/solutions/5794421, we can see a nmstate-handler pod that is not working properly. In fact, all the pods under the nmstate-handler daemonset are not running and there's one that is not ready:

$ omg get daemonset -o wide -n openshift-cnv
NAME                          DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE SELECTOR                  AGE  CONTAINERS       IMAGES
nmstate-handler               8        8        7      8           7          beta.kubernetes.io/arch=amd64  36m  nmstate-handler  registry.redhat.io/container-native-virtualization/kubernetes-nmstate-handler-rhel8@sha256:e8f4f93e02643e13e146b194e349726adbce231f0f6aa39f2748b3a0b74d29b5

$ omg get pods -o wide -n openshift-cnv
nmstate-handler-5h87k                                1/1    Running  0         36m  192.168.42.12  master-1
nmstate-handler-6t2jr                                1/1    Running  0         36m  192.168.42.21  worker-1
nmstate-handler-76bfd                                1/1    Running  0         36m  192.168.42.22  worker-2
nmstate-handler-dxqcg                                1/1    Running  0         36m  192.168.42.24  worker-4
nmstate-handler-q8wfp                                1/1    Running  0         36m  192.168.42.20  worker-0
nmstate-handler-wjg2s                                1/1    Running  0         36m  192.168.42.11  master-0
nmstate-handler-wkskc                                0/1    Running  11        36m  192.168.42.13  master-2
nmstate-handler-xkr66                                1/1    Running  0         36m  192.168.42.23  worker-3

If we check the logs of the problematic pod, we will see the error message related to the interface conversion:

$ omg logs -n openshift-cnv nmstate-handler-wkskc
...
2022-03-12T02:27:56.390752856Z E0312 02:27:56.390655       1 runtime.go:78] Observed a panic: &runtime.TypeAssertionError{_interface:(*runtime._type)(0x1725380), concrete:(*runtime._type)(0x16bff20), asserted:(*runtime._type)(0x16ea6e0), missingMethod:""} (interface conversion: interface {} is float64, not string)
...

And effectively, if we check master-2/cluster4 interfaces, we will see that one of the interfaces created during OCP installation has a name that can be interpreted as float:

# checking in cnv-must-gather/nodes/master-2/master-2_logs_NetworkManager
...
Mar 12 00:53:17.852193 master-2 NetworkManager[2658]: <info>  [1647046397.8521] device (veth14e7ad7b): interface index 35 renamed iface from 'veth14e7ad7b' to '229e55633017480'
...


Expected results:

The hyperconverged operator should be installed correctly, regardless of the interface naming.

Additional info:

We are aware of the solution provided on https://access.redhat.com/solutions/5794421, but we cannot rely on a workaround here because the OCP installation is managed with an automated tool. Moreover, we have also seen some other BZs opened with regards to this topic, such as https://bugzilla.redhat.com/show_bug.cgi?id=2055786, https://bugzilla.redhat.com/show_bug.cgi?id=2055304 or https://bugzilla.redhat.com/show_bug.cgi?id=2051461, but we've not seen an issue related to OCP 4.8, so that we are opening this.

Note that, in theory, with this announcement: https://access.redhat.com/errata/RHSA-2021:2920, from Openshift Virtualization 4.8.0 in advance, the issue should be fixed, but here we are observing the same problem with the HCO v4.8.4.