Bug 1859766 - ovnkube-master failed annotating worker node [NEEDINFO]
Summary: ovnkube-master failed annotating worker node
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.4
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: 4.6.0
Assignee: Aniket Bhat
QA Contact: Anurag saxena
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-22 21:39 UTC by Nabeel Cocker
Modified: 2020-10-27 16:17 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 16:16:44 UTC
Target Upstream Version:
anbhat: needinfo? (ncocker)
anbhat: needinfo? (ncocker)


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift ovn-kubernetes pull 232 0 None closed Merge upstream - 20200807 2021-01-22 04:37:42 UTC
Github openshift ovn-kubernetes pull 234 0 None closed Bug 1868082: 8 14 2020 merge 2021-01-22 04:38:23 UTC
Github ovn-org ovn-kubernetes pull 1563 0 None closed Add a retry mechanism for setting node annotations 2021-01-22 04:37:42 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:17:06 UTC

Description Nabeel Cocker 2020-07-22 21:39:56 UTC
Description of problem:

During worker node deployment one of the worker nodes was in a NotReady state.  Looking at logs we see the following: (ovnkune-node logs of the worker)

Last State:  Terminated
      Reason:    Error
      Message:    start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation
I0722 21:14:47.959715   37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation
I0722 21:14:48.460001   37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation
I0722 21:14:48.959313   37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation
I0722 21:14:49.459165   37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation
I0722 21:14:49.960128   37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation
I0722 21:14:50.459020   37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation
I0722 21:14:50.959003   37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation
I0722 21:14:51.459291   37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation
I0722 21:14:51.958763   37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation
I0722 21:14:52.458584   37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation
I0722 21:14:52.460116   37446 node.go:186] waiting for node worker-5 to start, no annotation found on node for subnet: node "worker-5" has no host subnet annotation
F0722 21:14:52.460156   37446 ovnkube.go:104] timed out waiting for node's: "worker-5" logical switch: timed out waiting for the condition
      Exit Code:    1
      Started:      Wed, 22 Jul 2020 21:09:52 +0000
      Finished:     Wed, 22 Jul 2020 21:14:52 +0000
    Ready:          False
    Restart Count:  4



And then on the ovnkube-master
corona@bastion welktxews-kub1.vzwops.com]$ oc logs ovnkube-master-bdpwj -n openshift-ovn-kubernetes -c ovnkube-master | grep worker-5
I0722 20:53:43.137049       1 master.go:628] Allocated node worker-5 HostSubnet 192.168.6.0/24
I0722 20:53:43.182163       1 kube.go:72] Setting annotations map[k8s.ovn.org/node-subnets:{"default":"192.168.6.0/24"}] on node worker-5
E0722 20:53:43.184080       1 kube.go:81] Error in setting annotation on node worker-5: Patch https://api-int.welktxews-kub1.vzwops.com:6443/api/v1/nodes/worker-5: dial tcp 172.19.0.6:6443: connect: connection refused
E0722 20:53:43.184106       1 ovn.go:578] error creating subnet for node worker-5: failed to set node-subnets annotation on node worker-5: Patch https://api-int.welktxews-kub1.vzwops.com:6443/api/v1/nodes/worker-5: dial tcp 172.19.0.6:6443: connect: connection refused
E0722 20:53:43.184127       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 20:53:43.184135       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 20:53:53.150079       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 20:53:53.150100       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 20:53:58.832196       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 20:53:58.832215       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 20:54:23.163969       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 20:54:23.163999       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 20:54:53.180213       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 20:54:53.180235       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 20:58:37.282257       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 20:58:37.282277       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 20:58:37.370918       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 20:58:37.370937       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 20:59:53.300565       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 20:59:53.300590       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 21:01:53.347398       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 21:01:53.347423       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 21:03:57.998167       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 21:03:57.998197       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 21:04:08.008862       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 21:04:08.008908       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 21:06:57.110819       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 21:06:57.110838       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 21:06:58.080967       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 21:06:58.080999       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 21:11:58.199586       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 21:11:58.199602       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 21:16:58.316356       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 21:16:58.316386       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 21:21:58.481074       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"
E0722 21:21:58.481096       1 node_annotations.go:227] macAddress annotation not found for node "worker-5"








Version-Release number of selected component (if applicable):



How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:
On failure to reach the API to set annotations, Ovnkube-master attempts this once.  It appears not to retry.  


Expected results:


The expectation is that the controller will retry to set annotations on the node.


Additional info:

To recover from this:
find the active master pod in openshift-ovn-kubernetes
oc logs <active master pod> -n openshift-ovn-kubernetes -c ovnkube-master
oc delete pod/<active master pod>
oc logs <active master pod> -n openshift-ovn-kubernetes -c ovnkube-master (verify annotation applied to worker in NotReady state)
oc get nodes (validate worker is now Ready)
oc get node/<NotReady worker> -o yaml (validate annotations are now applied to node)

Comment 11 Anurag saxena 2020-09-23 15:13:45 UTC
Moving this to verified based on above comments. @Nabeel, feel free to re-open if your observations are otherwise

Comment 13 errata-xmlrpc 2020-10-27 16:16:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.