Bug 1859766
| Summary: | ovnkube-master failed annotating worker node | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Nabeel Cocker <ncocker> |
| Component: | Networking | Assignee: | Aniket Bhat <anbhat> |
| Networking sub component: | ovn-kubernetes | QA Contact: | Anurag saxena <anusaxen> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | unspecified | CC: | anbhat |
| Version: | 4.4 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.6.0 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-10-27 16:16:44 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Moving this to verified based on above comments. @Nabeel, feel free to re-open if your observations are otherwise Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |
Description of problem: During worker node deployment one of the worker nodes was in a NotReady state. Looking at logs we see the following: (ovnkune-node logs of the worker) Last State: Terminated Reason: Error Message: start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation I0722 21:14:47.959715 37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation I0722 21:14:48.460001 37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation I0722 21:14:48.959313 37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation I0722 21:14:49.459165 37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation I0722 21:14:49.960128 37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation I0722 21:14:50.459020 37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation I0722 21:14:50.959003 37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation I0722 21:14:51.459291 37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation I0722 21:14:51.958763 37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation I0722 21:14:52.458584 37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation I0722 21:14:52.460116 37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation F0722 21:14:52.460156 37446 ovnkube.go:104] timed out waiting for node's: "worker-5" logical switch: timed out waiting for the condition Exit Code: 1 Started: Wed, 22 Jul 2020 21:09:52 +0000 Finished: Wed, 22 Jul 2020 21:14:52 +0000 Ready: False Restart Count: 4 And then on the ovnkube-master corona@bastion welktxews-kub1.vzwops.com]$ oc logs ovnkube-master-bdpwj -n openshift-ovn-kubernetes -c ovnkube-master | grep worker-5 I0722 20:53:43.137049 1 master.go:628] Allocated node worker-5 HostSubnet 192.168.6.0/24 I0722 20:53:43.182163 1 kube.go:72] Setting annotations map[k8s.ovn.org/node-subnets:{"default":"192.168.6.0/24"}] on node worker-5 E0722 20:53:43.184080 1 kube.go:81] Error in setting annotation on node worker-5: Patch https://api-int.welktxews-kub1.vzwops.com:6443/api/v1/nodes/worker-5: dial tcp 172.19.0.6:6443: connect: connection refused E0722 20:53:43.184106 1 ovn.go:578] error creating subnet for node worker-5: failed to set node-subnets annotation on node worker-5: Patch https://api-int.welktxews-kub1.vzwops.com:6443/api/v1/nodes/worker-5: dial tcp 172.19.0.6:6443: connect: connection refused E0722 20:53:43.184127 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 20:53:43.184135 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 20:53:53.150079 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 20:53:53.150100 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 20:53:58.832196 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 20:53:58.832215 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 20:54:23.163969 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 20:54:23.163999 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 20:54:53.180213 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 20:54:53.180235 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 20:58:37.282257 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 20:58:37.282277 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 20:58:37.370918 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 20:58:37.370937 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 20:59:53.300565 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 20:59:53.300590 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 21:01:53.347398 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 21:01:53.347423 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 21:03:57.998167 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 21:03:57.998197 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 21:04:08.008862 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 21:04:08.008908 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 21:06:57.110819 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 21:06:57.110838 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 21:06:58.080967 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 21:06:58.080999 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 21:11:58.199586 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 21:11:58.199602 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 21:16:58.316356 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 21:16:58.316386 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 21:21:58.481074 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" E0722 21:21:58.481096 1 node_annotations.go:227] macAddress annotation not found for node "worker-5" Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: On failure to reach the API to set annotations, Ovnkube-master attempts this once. It appears not to retry. Expected results: The expectation is that the controller will retry to set annotations on the node. Additional info: To recover from this: find the active master pod in openshift-ovn-kubernetes oc logs <active master pod> -n openshift-ovn-kubernetes -c ovnkube-master oc delete pod/<active master pod> oc logs <active master pod> -n openshift-ovn-kubernetes -c ovnkube-master (verify annotation applied to worker in NotReady state) oc get nodes (validate worker is now Ready) oc get node/<NotReady worker> -o yaml (validate annotations are now applied to node)