Bug 2059812
| Summary: | [OVN-Kubernetes] [RHOCP 4.9.13] failed to configure pod interface: timed out waiting for OVS port binding - database connection failed | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux Fast Datapath | Reporter: | Vincent Lours <vlours> |
| Component: | OVN | Assignee: | OVN Team <ovnteam> |
| Status: | CLOSED DUPLICATE | QA Contact: | Jianlin Shi <jishi> |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | FDP 22.L | CC: | akaris, anbhat, ctrautma, jiji |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-03-25 09:24:50 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Hi Andreas, Thanks a lot for checking it this week. And no worries :) As written, the CU is now waiting for us to provide the RCA. I hope you will get enough logs to find something in there. Cheers, Vincent Hi Andreas, Yes, it does suffice for the RCA. I will share the detail with the customer. Thanks again for all the details that you provide. It's helping me to provide great feedback to the customer, but also give a better understanding of OVN-kubernetes. Always learning something when working with you on a BZ. Cheers, Vincent Great. Then hopefully it's really 2046274 :-) Marking this as a duplicate and closing this out. - Andreas *** This bug has been marked as a duplicate of bug 2046274 *** O.k, too happy, too soon: ovn-2021-21.12.0-15.el8fdp is the RPM with the fix. 4.9.23 is still on ovn-21.12.0-25.el8fdp ovn21.12 vs ovn-2021-21.12. It turns out that they're different packages. ovn-2021-21.12 is the one currently supported and maintained by the ovn team. ovn21.12 started as a pre-release stream (before upstream 21.12.0) to enable us to provide early performance related features/changes to OCP in 4.9. That switch to ovn-2021 was only added with https://github.com/openshift/ovn-kubernetes/commit/e4046189e8a1d08d0e5d0797b1aeda602389e4c7 , and that's not yet available to customers, but it will be with the next release. Sorry for the confusion! - Andreas |
Description of problem: The customer is running his cluster in RHOCP 4.9.13, and is facing the following issue in OVS ~~~ 2022-03-01T06:02:56.445260207Z I0301 06:02:56.445206 992951 cni.go:190] [openshift-operator-lifecycle-manager/collect-profiles-27427335--1-bghsp 5a61c327b8d5596b432e808cc1cd37df7d5edc335731a4a60b20442facc3f27d] ADD finished CNI request [openshift-operator-lifecycle-manager/collect-profiles-27427335--1-bghsp 5a61c327b8d5596b432e808cc1cd37df7d5edc335731a4a60b20442facc3f27d], result "", err failed to configure pod interface: failure in plugging pod interface: failed to run 'ovs-vsctl --timeout=30 add-port br-int 5a61c327b8d5596 -- set interface 5a61c327b8d5596 external_ids:attached_mac=0a:58:0a:82:04:62 external_ids:iface-id=openshift-operator-lifecycle-manager_collect-profiles-27427335--1-bghsp external_ids:iface-id-ver=78cc308f-c47c-4b1f-a092-927448c05945 external_ids:ip_addresses=10.130.4.98/23 external_ids:sandbox=5a61c327b8d5596b432e808cc1cd37df7d5edc335731a4a60b20442facc3f27d': exit status 1 2022-03-01T06:02:56.445260207Z "ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (Protocol error)\n" 2022-03-01T06:02:56.445260207Z "" 2022-03-01T06:02:56.449913235Z W0301 06:02:56.449859 992951 helper_linux.go:426] Failed to delete OVS port e34691f06486c2e: failed to run 'ovs-vsctl --timeout=30 del-port br-int e34691f06486c2e': exit status 1 2022-03-01T06:02:56.449913235Z "ovs-vsctl: no port named e34691f06486c2e\n" 2022-03-01T06:02:56.449913235Z "" ~~~ Version-Release number of selected component (if applicable): RHOCP 4.9.13 How reproducible: N/A Actual results: All PODs are Pending waiting on: ~~~ $ oc get event -n ${namespace:-default} LAST SEEN TYPE REASON OBJECT MESSAGE 5m30s Warning FailedCreatePodSandBox pod/etcd-operator-c7bdd6b66-z4vxd (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_etcd-operator-c7bdd6b66-z4vxd_openshift-etcd-operator_e875fc23-f68e-414b-96cb-c8f34e06f6aa_0(9f75b66059883bad4f5ff5216c1c02fe8dc83e6f6600df7ce516a0dc74203ca5): error adding pod openshift-etcd-operator_etcd-operator-c7bdd6b66-z4vxd to CNI network "multus-cni-network": [openshift-etcd-operator/etcd-operator-c7bdd6b66-z4vxd/e875fc23-f68e-414b-96cb-c8f34e06f6aa:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[openshift-etcd-operator/etcd-operator-c7bdd6b66-z4vxd 9f75b66059883bad4f5ff5216c1c02fe8dc83e6f6600df7ce516a0dc74203ca5] [openshift-etcd-operator/etcd-operator-c7bdd6b66-z4vxd 9f75b66059883bad4f5ff5216c1c02fe8dc83e6f6600df7ce516a0dc74203ca5] failed to configure pod interface: timed out waiting for OVS port binding (ovn-installed) for 0a:58:xx:xx:xx:xx [10.xx.xx.xx/23] ~~~ Expected results: OVN should not block the POD deployment. Additional info: - Look similar to the Bug 2005598 [1], but the PODs are in **Pending** state, and not in **ContainerCreating** state. [1] https://bugzilla.redhat.com/show_bug.cgi?id=2005598