Bug 2077900

Description of problem:

After deploying successfully OCP 4.11 in a cluster, we are dealing with a lot of failures during subsequent deployments of different operators and workloads due to the following issue which involves ovn-kubernetes:

"error adding pod XXX to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [openshift-network-diagnostics/network-check-target-ncvhd/3bb3b05e-1ef5-496e-b67b-88a6aba42cbf]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition"

We use Ansible community.kubernetes.k8s module to automate the creation of the resources, and running twice or more causes this behavior. We use DCI to automate all the process.

The intention of this Bugzilla is to report this strange behavior, so that it can be quickly fixed in next OCP 4.11 versions released.

Version-Release number of selected component (if applicable):

OCP 4.11.0, starting from the version released the 2022-04-07 until 2022-04-16, which is the latest one tested.

How reproducible:

(Note that it is not happening all the times, and when happening, it may affect to different operators and/or workloads provided after the OCP installation. In Actual results and Additional information sections, I will extend the cases observed)

Steps to Reproduce:

1. Deploy OCP 4.11 in a cluster composed by 3 master nodes and 3 worker nodes, using IPI installation and Ansible playbooks from baremetal-deployment. For this purpose, DCI automation too has been used.

2. Deploy the operators and/or workloads to be used in the OCP cluster.

Actual results:

We are finding a lot of cases in which we can see pods that are having failures when being created due to the failure reported above: "error adding pod XXX to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [openshift-network-diagnostics/network-check-target-ncvhd/3bb3b05e-1ef5-496e-b67b-88a6aba42cbf]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition", and also, that there are cases in which several ovnkube-node pods remains in CrashLoopBackOff status:

The examples in which it has been happening this case are the following, all related to DCI jobs executing OCP 4.11:

- Installation of SRIOV operator: https://www.distributed-ci.io/jobs/9a0e5274-e1ca-407f-8d75-ff23ee75ce39/jobStates
- Application of a PerformanceProfile, waiting for MCP to be in the correct status: https://www.distributed-ci.io/jobs/7a658a2a-4e85-4e9e-969b-a1f69351a5fc/jobStates
- Deploying example-cnf, based on testpmd operators, even implying different operators: https://www.distributed-ci.io/jobs/1b6dcd1a-4496-4baa-89ca-46b00563d8f4/jobStates or https://www.distributed-ci.io/jobs/a66344a0-af2a-4420-8d7b-e15131ccde4e/jobStates
- Deploying a simple deployment based on a couple of pods: https://www.distributed-ci.io/jobs/b4aae3e1-2e37-480c-91f3-3f095b4292a9/jobStates
- Deploying an example of a basic operator: https://www.distributed-ci.io/jobs/07703e22-f6a6-49c9-8f9c-d1cf9276b028/jobStates

If you check all cases, you will observe that there are pods reporting the problem specified in the title. In Additional information, it is explained how to check the useful information to get to that point.

Expected results:

OCP cluster should be installed correctly and without OVN-related problems.

Additional info:

We were using Distributed CI (DCI) to do the tests. If you enter to the URLs provided, you will see the jobs executed to deploy OCP 4.11 with different operators and workloads.

If you move to the Files section, you can see the following files in order to get the information related to the failed pods:

- events.txt: it registers all the events related to the pods present in all namespaces. If you filter these files by the error message provided, you will see the high amount of pods that are affected by it.
- must_gather.tar.gz: complete must-gather copy of each OCP deployment, so that it is possible to retrieve more information regarding the cluster, namespaces, operators, etc.