Bug 1892243

Summary:	When scaling down the status of the node is stuck on deleting
Product:	OpenShift Container Platform	Reporter:	Polina Rabinovich <prabinov>
Component:	Installer	Assignee:	Andrea Fasano <afasano>
Installer sub component:	OpenShift on Bare Metal IPI	QA Contact:	Amit Ugol <augol>
Status:	CLOSED DUPLICATE	Docs Contact:
Severity:	medium
Priority:	medium	CC:	kiran, smiron
Version:	4.7	Keywords:	TestBlocker, Triaged, UpcomingSprint
Target Milestone:	---
Target Release:	4.7.0
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-11-04 12:21:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Polina Rabinovich 2020-10-28 09:51:03 UTC

Version:
Cluster version is 4.7.0-0.nightly-2020-10-27-051128
-----------

Setup:
Provisioning_net_IPv6, Baremetal_net_IPv4, disconnected install
-----------

Platform:

libvirt
IPI (automated install with `openshift-baremetal-install`) 
-----------

Description:
-------------
When trying to remove a node the status of the node is stuck on deleting state after oc scale machineset -n openshift-machine-api ocp-edge-cluster-0-worker-0 --replicas=<N-1> although it should have been removed from nodes and machines lists ($ oc get nodes $ oc get machine -o wide -n openshift-machine-api).

[kni@provisionhost-0-0 ~]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2020-10-27-051128   True        False         15h     Cluster version is 4.7.0-0.nightly-2020-10-27-051128
[kni@provisionhost-0-0 ~]$ oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
cloud-credential                           4.7.0-0.nightly-2020-10-27-051128   True        False         False      16h
cluster-autoscaler                         4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
config-operator                            4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
console                                    4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
csi-snapshot-controller                    4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
dns                                        4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
etcd                                       4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
image-registry                             4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
ingress                                    4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
insights                                   4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
kube-apiserver                             4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
kube-controller-manager                    4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
kube-scheduler                             4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
kube-storage-version-migrator              4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
machine-api                                4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
machine-approver                           4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
machine-config                             4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
marketplace                                4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
monitoring                                 4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
network                                    4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
node-tuning                                4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
openshift-apiserver                        4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
openshift-controller-manager               4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
openshift-samples                          4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
operator-lifecycle-manager                 4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
operator-lifecycle-manager-catalog         4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
operator-lifecycle-manager-packageserver   4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
service-ca                                 4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h
storage                                    4.7.0-0.nightly-2020-10-27-051128   True        False         False      15h

Steps:
-----------

[kni@provisionhost-0-0 ~]$ oc annotate machine ocp-edge-cluster-0-hhhnh-worker-0-snbck machine.openshift.io/cluster-api-delete-machine=yes -n openshift-machine-api
machine.machine.openshift.io/ocp-edge-cluster-0-hhhnh-worker-0-snbck annotated

[kni@provisionhost-0-0 ~]$ oc scale machineset -n openshift-machine-api ocp-edge-cluster-0-hhhnh-worker-0 --replicas=1
machineset.machine.openshift.io/ocp-edge-cluster-0-hhhnh-worker-0 scaled

[kni@provisionhost-0-0 ~]$ oc get nodes
NAME         STATUS                     ROLES    AGE   VERSION
master-0-0   Ready                      master   16h   v1.19.0+e67f5dc
master-0-1   Ready                      master   16h   v1.19.0+e67f5dc
master-0-2   Ready                      master   16h   v1.19.0+e67f5dc
worker-0-0   Ready,SchedulingDisabled   worker   15h   v1.19.0+e67f5dc
worker-0-1   Ready                      worker   15h   v1.19.0+e67f5dc
[kni@provisionhost-0-0 ~]$ oc get machine -o wide -n openshift-machine-api 
NAME                                      PHASE      TYPE   REGION   ZONE   AGE   NODE         PROVIDERID                                                    STATE
ocp-edge-cluster-0-hhhnh-master-0         Running                           16h   master-0-0   baremetalhost:///openshift-machine-api/openshift-master-0-0   
ocp-edge-cluster-0-hhhnh-master-1         Running                           16h   master-0-1   baremetalhost:///openshift-machine-api/openshift-master-0-1   
ocp-edge-cluster-0-hhhnh-master-2         Running                           16h   master-0-2   baremetalhost:///openshift-machine-api/openshift-master-0-2   
ocp-edge-cluster-0-hhhnh-worker-0-kl7zs   Running                           15h   worker-0-1   baremetalhost:///openshift-machine-api/openshift-worker-0-1   
ocp-edge-cluster-0-hhhnh-worker-0-snbck   Deleting                          15h   worker-0-0   baremetalhost:///openshift-machine-api/openshift-worker-0-0   
-------------

Must-gather:
-------------------------------------
https://drive.google.com/drive/folders/1xGIPBB04xvKeGOjwcYwayrfLJUaswxid?usp=sharing


Expected results:
-------------------------------------
Deleted node not listed in nodes and machines lists

Actual results:
-------------------------------------
Status of the node is stuck on deleting state


How to reproduce:
-------------------------------------
1.Match the bmh name <name-CONSUMER> to machine (CONSUMER column)
  $ oc get bmh -n openshift-machine-api

2.$ oc annotate machine <name-CONSUMER> machine.openshift.io/cluster-api- 
  delete-machine=yes -n openshift-machine-api

3.Scale down the replicas number
  $ oc scale machineset -n openshift-machine-api ocp-edge-cluster-0-worker-0 -- 
  replicas=<N-1>

4.Ensure the deleted node not listed
  $ oc get nodes 
  $ oc get machine -o wide -n openshift-machine-api

Comment 2 Andrea Fasano 2020-11-04 12:21:24 UTC


*** This bug has been marked as a duplicate of bug 1893832 ***