Bug 2094903
| Summary: | Node fails to join cluster - No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started? | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Yurii Prokulevych <yprokule> |
| Component: | Networking | Assignee: | Surya Seetharaman <surya> |
| Networking sub component: | ovn-kubernetes | QA Contact: | Anurag saxena <anusaxen> |
| Status: | CLOSED DUPLICATE | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | achernet, dcbw, fbaudin, mzamot, ncocker, surya |
| Version: | 4.10 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.11.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-06-09 07:41:38 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
E0608 08:18:01.017172 1 ovn.go:1254] error creating gateway for node openshift-master-0: failed to init shared interface gateway: failed to add logical switch port: br-ex1_openshift-master-0 to switch exgw-ext_openshift-master-0, err: unable to update model, err: table Logical_Switch_Port, column tag_request: failed to generate ovs element. ovsset supports only go slice/string/numbers/uuid or pointers to those types thus the worker-0's ovn-controller doesn't have the flows to init the exgw ports: I0608 16:04:01.591290 193800 ovs.go:204] exec(41): /usr/bin/ovs-vsctl --timeout=15 --if-exists get interface patch-br-ex1_openshift-worker-0-to-br-int ofport I0608 16:04:01.594669 193800 ovs.go:207] exec(41): stdout: "" I0608 16:04:01.594675 193800 ovs.go:208] exec(41): stderr: "" and ovnkube-node waits forever at startup for that to get an ofport, thus never fully initializes. Thanks to the good team work from Dan, Tim, Jaime, Numan, we solved the bug on why this specific node was having issues: 1) We first created all the gateway setup for worker0 2) Then we have deleted and re-added worker0 as evident from the logs: I0608 13:54:50.007631 1 master.go:1200] Deleted HostSubnet 10.128.2.0/23 for node openshift-worker-0 3) Because of a bug in the gatewayCleanup code: https://github.com/openshift/ovn-kubernetes/blob/d4d93cc3bc7dc7f06eacf84fd9f638963076372f/go-controller/pkg/ovn/gateway_cleanup.go#L89 we didn't delete the old exgw switch 4) So when the node was re-added, this was in turn an update and not an insert, so we hit the ovsset bug: https://bugzilla.redhat.com/show_bug.cgi?id=2092937 5) Thus the new LRP was never created and patch port wasn't created and hence the ovnkube-node pod was stuck in CLBO. https://github.com/ovn-org/ovn-kubernetes/pull/3020/commits/6beef933d7bb75716b5b10c788efbf69a90af0e5 should fix the problem seen in comment#4 Workaround was to: 1) Delete the exgw-ext_openshift-worker-0 switch from NBDB using ovn-nbctl ls-del exgw-ext_openshift-worker-0 2) Restart the ovnk master leader pod 3) Kick out the stuck ovnkube-node pod That cleaned up ovnkue-node, network operator and node became ready since now ovnkube-node came up fine and was able to copy the binary required to pass the CNI healthy check. NAME STATUS ROLES AGE VERSION [40/1803] openshift-master-0 Ready master 24h v1.23.5+3afdacb openshift-master-1 Ready master 24h v1.23.5+3afdacb openshift-master-2 Ready master 24h v1.23.5+3afdacb openshift-worker-0 Ready worker 16h v1.23.5+3afdacb openshift-worker-1 Ready worker,workercnf 23h v1.23.5+3afdacb openshift-worker-2 Ready loadbalancer,worker 23h v1.23.5+3afdacb openshift-worker-3 Ready loadbalancer,worker 23h v1.23.5+3afdacb Marking this as dupe of https://bugzilla.redhat.com/show_bug.cgi?id=2094088 because the fix is included in the PR that has being posted to fix that bug. *** This bug has been marked as a duplicate of bug 2094088 *** |
Description of problem: ======================= After node reprovisioning it fails to join cluster: NAME STATUS ROLES AGE VERSION openshift-master-0 Ready master 7h47m v1.23.5+3afdacb openshift-master-1 Ready master 7h48m v1.23.5+3afdacb openshift-master-2 Ready master 7h48m v1.23.5+3afdacb openshift-worker-0 NotReady worker 27m v1.23.5+3afdacb openshift-worker-1 Ready worker,workercnf 7h19m v1.23.5+3afdacb openshift-worker-2 Ready loadbalancer,worker 7h18m v1.23.5+3afdacb openshift-worker-3 Ready loadbalancer,worker 7h18m v1.23.5+3afdacb oc describe node openshift-worker-0 Name: openshift-worker-0 Roles: worker Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=openshift-worker-0 kubernetes.io/os=linux node-role.kubernetes.io/worker= node.openshift.io/os_id=rhcos Annotations: k8s.ovn.org/hybrid-overlay-distributed-router-gateway-mac: 0a:58:0a:80:02:03 k8s.ovn.org/l3-gateway-config: {"default":{"mode":"shared","interface-id":"br-ex_openshift-worker-0","mac-address":"0c:42:a1:ee:7e:12","ip-addresses":["10.1.208.23/24"],... k8s.ovn.org/node-chassis-id: 1815e298-a467-4f92-868e-a7a8ce32a19a k8s.ovn.org/node-mgmt-port-mac-address: fe:af:82:db:db:e7 k8s.ovn.org/node-primary-ifaddr: {"ipv4":"10.1.208.23/24"} k8s.ovn.org/node-subnets: {"default":"10.128.2.0/23"} machine.openshift.io/machine: openshift-machine-api/kni-qe-4-x6w7m-worker-0-w5rcj machineconfiguration.openshift.io/controlPlaneTopology: HighlyAvailable volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Wed, 08 Jun 2022 10:31:00 -0400 Taints: node.kubernetes.io/not-ready:NoExecute node.kubernetes.io/not-ready:NoSchedule UpdateInProgress:PreferNoSchedule Unschedulable: false Lease: HolderIdentity: openshift-worker-0 AcquireTime: <unset> RenewTime: Wed, 08 Jun 2022 10:52:25 -0400 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Wed, 08 Jun 2022 10:50:24 -0400 Wed, 08 Jun 2022 10:31:00 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Wed, 08 Jun 2022 10:50:24 -0400 Wed, 08 Jun 2022 10:31:00 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Wed, 08 Jun 2022 10:50:24 -0400 Wed, 08 Jun 2022 10:31:00 -0400 KubeletHasSufficientPID kubelet has sufficient PID available Ready False Wed, 08 Jun 2022 10:50:24 -0400 Wed, 08 Jun 2022 10:31:00 -0400 KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started? ... Version-Release number of selected component (if applicable): ============================================================= 4.10.17 Steps to Reproduce: =================== 1. Deploy and configure baremetal cluster - 3 masters + 4 workers 2. Split nodes in 2 different MCPs 3. Perform migration to 2nd interface on both masters and workers 4. Scale down cluster and remove BMH(Baremetal-host CR) 5. Add BMH and respective secret 6. Wait for introspection to finish 7. Scale up cluster Actual results: =============== Node is added but is in 'NotReady' state Expected results: ================= Node successfully joins the cluster