Bug 1838556
| Summary: | [ovn]failed to configure pod interface | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | zhaozhanqi <zzhao> | ||||
| Component: | Networking | Assignee: | Ben Bennett <bbennett> | ||||
| Networking sub component: | ovn-kubernetes | QA Contact: | zhaozhanqi <zzhao> | ||||
| Status: | CLOSED ERRATA | Docs Contact: | |||||
| Severity: | urgent | ||||||
| Priority: | urgent | CC: | bbennett, rbrattai, weliang | ||||
| Version: | 4.5 | ||||||
| Target Milestone: | --- | ||||||
| Target Release: | 4.6.0 | ||||||
| Hardware: | All | ||||||
| OS: | All | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2020-10-27 16:00:22 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
seems same logs with bug https://bugzilla.redhat.com/show_bug.cgi?id=1828343 Moving to 4.6 because of the lack of information to debug yet. Do not meet this issue in recent build. guess this issue should be fixed by the ovn version. Move this issue to 'verified' I also saw this in some pods on a new replacement master when testing https://bugzilla.redhat.com/show_bug.cgi?id=1854072. After replacing a master and observing the ovn_db cleanup fix a few pods were stuck in ContainerCreating with similar errors Warning FailedCreatePodSandBox 32s kubelet, ip-10-0-206-120.ca-central-1.compute.internal Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_apiserver-6cbbf6f5b6-b8ljn_openshift-apiserver_651dc152-d461-4b28-99a1-05f06d512134_0(e8271dd1cb8bb0811ce74359c1498cdd617941bf44bdc374471dc364197a7226): Multus: [openshift-apiserver/apiserver-6cbbf6f5b6-b8ljn]: error adding container to network "ovn-kubernetes": delegateAdd: error invoking confAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: '[openshift-apiserver/apiserver-6cbbf6f5b6-b8ljn] failed to configure pod interface: timed out dumping br-int flow entries for sandbox: timed out waiting for the condition ' Warning FailedCreatePodSandBox 8s kubelet, ip-10-0-206-120.ca-central-1.compute.internal Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_apiserver-6cbbf6f5b6-b8ljn_openshift-apiserver_651dc152-d461-4b28-99a1-05f06d512134_0(252261aa02688bd3446b50f073d28468cab9cac005e858c961576a889ac48de0): Multus: [openshift-apiserver/apiserver-6cbbf6f5b6-b8ljn]: error adding container to network "ovn-kubernetes": delegateAdd: error invoking confAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: '[openshift-apiserver/apiserver-6cbbf6f5b6-b8ljn] failed to configure pod interface: timed out dumping br-int flow entries for sandbox: timed out waiting for the condition Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196 |
Created attachment 1690600 [details] ovn logs Description of problem: when upgrade from 4.4.4 to 4.5.0-0.nightly-2020-05-20-203028 with OVN network. openshift-apiserver pod canot be running due to error : kubelet, ip-10-0-48-198.us-east-2.compute.internal (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_apiserver-6b48fc8cb-4qkld_openshift-apiserver_82ac84f9-0f14-4859-a6a6-bd3f360336c5_0(a17089b837aac14341a07ff07096a1f9e0fd53c6307d618726ae75cb20ad308a): Multus: [openshift-apiserver/apiserver-6b48fc8cb-4qkld]: error adding container to network "ovn-kubernetes": delegateAdd: error invoking confAdd - "ovn-k8s-cni-overlay": error in getting result from AddNetwork: CNI request failed with status 400: '[openshift-apiserver/apiserver-6b48fc8cb-4qkld] failed to configure pod interface: timed out dumping br-int flow entries for sandbox: timed out waiting for the condition Version-Release number of selected component (if applicable): 4.4.4 to 4.5.0-0.nightly-2020-05-20-203028 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: oc get pod -A -o wide | grep "ip-10-0-48-198" openshift-apiserver apiserver-6b48fc8cb-4qkld 0/1 Init:0/1 0 8h <none> ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-cluster-node-tuning-operator tuned-4j7bx 1/1 Running 0 8h 10.0.48.198 ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-controller-manager controller-manager-t2xjf 1/1 Running 0 8h 10.130.0.15 ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-dns dns-default-q4r4n 2/3 CrashLoopBackOff 113 8h 10.130.0.4 ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-etcd etcd-ip-10-0-48-198.us-east-2.compute.internal 3/3 Running 0 8h 10.0.48.198 ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-etcd revision-pruner-4-ip-10-0-48-198.us-east-2.compute.internal 0/1 Completed 0 8h 10.130.0.7 ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-image-registry node-ca-bgs7x 1/1 Running 0 8h 10.130.0.17 ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-kube-apiserver kube-apiserver-ip-10-0-48-198.us-east-2.compute.internal 4/4 Running 0 8h 10.0.48.198 ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-kube-apiserver revision-pruner-9-ip-10-0-48-198.us-east-2.compute.internal 0/1 Completed 0 8h 10.130.0.6 ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-kube-controller-manager kube-controller-manager-ip-10-0-48-198.us-east-2.compute.internal 4/4 Running 0 8h 10.0.48.198 ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-kube-controller-manager revision-pruner-11-ip-10-0-48-198.us-east-2.compute.internal 0/1 Completed 0 8h 10.130.0.5 ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-kube-scheduler openshift-kube-scheduler-ip-10-0-48-198.us-east-2.compute.internal 2/2 Running 0 8h 10.0.48.198 ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-kube-scheduler revision-pruner-8-ip-10-0-48-198.us-east-2.compute.internal 0/1 ContainerCreating 0 8h <none> ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-machine-config-operator etcd-quorum-guard-8574dc4788-sgwt8 1/1 Running 0 8h 10.0.48.198 ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-machine-config-operator machine-config-daemon-nm57v 2/2 Running 0 8h 10.0.48.198 ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-machine-config-operator machine-config-server-44j6r 1/1 Running 0 8h 10.0.48.198 ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-monitoring node-exporter-zd8fg 2/2 Running 0 8h 10.0.48.198 ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-multus multus-admission-controller-wlgmg 2/2 Running 0 8h 10.130.0.3 ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-multus multus-fqmgm 1/1 Running 0 8h 10.0.48.198 ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-ovn-kubernetes ovnkube-master-48fwt 4/4 Running 0 8h 10.0.48.198 ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-ovn-kubernetes ovnkube-node-x74dv 2/2 Running 0 8h 10.0.48.198 ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-ovn-kubernetes ovs-node-wqb5k 1/1 Running 0 8h 10.0.48.198 ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-service-catalog-apiserver apiserver-lgm8p 0/1 ContainerCreating 0 4h16m <none> ip-10-0-48-198.us-east-2.compute.internal <none> <none> openshift-service-catalog-controller-manager controller-manager-pgsnl 0/1 CrashLoopBackOff 141 9h 10.130.0.37 ip-10-0-48-198.us-east-2.compute.internal <none> <none> Expected results: Additional info: