Bug 1973286
| Summary: | getting timeout error while waiting for OVS port binding when ds pod recreated | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Shelly Miron <smiron> |
| Component: | Networking | Assignee: | Tim Rozet <trozet> |
| Networking sub component: | ovn-kubernetes | QA Contact: | Shelly Miron <smiron> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | medium | CC: | abeekhof, anbhat, bbennett, dcbw, dosmith, trozet, vpickard, weliang |
| Version: | 4.8 | Flags: | trozet:
needinfo-
|
| Target Milestone: | --- | ||
| Target Release: | 4.10.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-03-16 11:12:07 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1969958 | ||
|
Comment 3
Tim Rozet
2021-06-23 20:56:00 UTC
Thanks Shelly for providing a live setup. I see what's going on now. So when we get the delete event we tear down everything related to that node, this includes the OVN logical switch and the switch ports for all the containers associated with it. However, in this test the node comes back as a node ADD, and the containers still exist on the node. In other words there are no new pod adds happening here. Due to the fact that the ports have been deleted, and we do not get any new pod Add/Update events for the pods on this node, ovnkube-master never recreates the ports for these pods. If we look in nbdb for worker-0-2, we see it barely has any ports for pods:
switch a6476b33-71b9-4e3c-9fa5-9729b284ba50 (worker-0-2)
port k8s-worker-0-2
addresses: ["aa:16:3f:21:e1:69 10.129.2.2"]
port stor-worker-0-2
type: router
addresses: ["0a:58:0a:81:02:01"]
router-port: rtos-worker-0-2
port openshift-monitoring_prometheus-adapter-7d9df9b7f6-k7rrw
addresses: ["0a:58:0a:81:02:04 10.129.2.4"]
This is a sign that ovnkube-master is the root cause because it should have created ports for containers that are trying to launch via CNI on the node. If I restart the active ovnkube-master, all the missing ports are added and the pods go to running state:
[root@master-0-1 ~]# ovn-nbctl --no-leader-only show worker-0-2
switch a6476b33-71b9-4e3c-9fa5-9729b284ba50 (worker-0-2)
port openshift-ingress-canary_ingress-canary-pkm9s
addresses: ["0a:58:0a:81:02:06 10.129.2.6"]
port openshift-multus_network-metrics-daemon-258kt
addresses: ["0a:58:0a:81:02:03 10.129.2.3"]
port openshift-monitoring_thanos-querier-6794c57947-2pwtv
addresses: ["0a:58:0a:81:02:20 10.129.2.32"]
port openshift-monitoring_alertmanager-main-1
addresses: ["0a:58:0a:81:02:22 10.129.2.34"]
port openshift-dns_dns-default-cmj9p
addresses: ["0a:58:0a:81:02:05 10.129.2.5"]
port openshift-image-registry_image-registry-678cc4657-45jqd
addresses: ["0a:58:0a:81:02:1f 10.129.2.31"]
port openshift-operators_poison-pill-ds-xzlps
addresses: ["0a:58:0a:81:02:23 10.129.2.35"]
port k8s-worker-0-2
addresses: ["aa:16:3f:21:e1:69 10.129.2.2"]
port stor-worker-0-2
type: router
addresses: ["0a:58:0a:81:02:01"]
router-port: rtos-worker-0-2
port openshift-monitoring_prometheus-k8s-1
addresses: ["0a:58:0a:81:02:21 10.129.2.33"]
port openshift-network-diagnostics_network-check-target-8q9q6
addresses: ["0a:58:0a:81:02:2b 10.129.2.43"]
port openshift-monitoring_prometheus-adapter-7d9df9b7f6-k7rrw
addresses: ["0a:58:0a:81:02:04 10.129.2.4"]
This is because an ovnkube-master restart will cause the informers to replay all of the pod events, forcing us to create ports that are missing in nbdb.
tl;dr we need to be able to reconcile missing pods during node add it looks like to handle this case. Shelly, is there a way I can reproduce this state of a node being deleted, but its pods still existing? I want to add an upstream test for this as well, but I cannot exhaust the storage or memory of a node as we use kubernetes in docker (KIND) and a very small VM to handle our upstream tests.
Tim, thanks for the detailed analysis. can we have a fix in 4.8 timeframe? is there a way to workaround this from an operator? e.g. can we modify some CR to trigger the replay of all pod events? if that helps to reproduce, the flow is basically: 1. marking node as unschedudlable (by setting node.Spec.Unschedulable to true) 2. keeping a copy of the node object (can be simulated by oc get node <nodeName> -oyaml > backup.yaml) 3. the node itself runs a software reboot command 4. delete the node (oc delete node <nodeName>) 5. restore the node (can be simulated with oc apply -f backup.yaml) I don't have ovn cluster to check if this reproduces, I will try next week. note that when we restore the node, we also restore all the OVN annotations. we can avoid that if this would help I was able to reproduce using the steps you listed Nir. It looks like if you delete a node, there is some grace period before pods are evicted and a new pod will be created when the node is re-added. If you re-add the node withing the grace period, the existing containers stay put. I have a fix here: https://github.com/ovn-org/ovn-kubernetes/pull/2281 Will start a downstream merge for it. *** Bug 1969958 has been marked as a duplicate of this bug. *** Surya, found a bug in the logic to add retry pods during node add: https://github.com/openshift/ovn-kubernetes/pull/926 Without this fix the pod retry cache could become corrupted and the pod would never be retried correctly. *** Bug 2047334 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.10.4 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:0811 |