Description of problem: Machine Deletion Hook cannot work on IBMCloud, lifecycleHooks field will be ignored in machine, and machine can be deleted successfully Version-Release number of selected component (if applicable): 4.10.0-0.nightly-2022-01-17-223655 How reproducible: Always Steps to Reproduce: 1.Create a machineset with Machine Deletion Hook, for example spec: lifecycleHooks: preTerminate: - name: MigrateImportantApp owner: etcd 2.Machine created but without Machine Deletion Hook liuhuali@Lius-MacBook-Pro huali-test % oc create -f ms4.yaml machineset.machine.openshift.io/huliu-033-tnd6f-test2 created liuhuali@Lius-MacBook-Pro huali-test % oc get machine huliu-033-tnd6f-test2-tvfn4 -o yaml apiVersion: machine.openshift.io/v1beta1 kind: Machine metadata: annotations: machine.openshift.io/instance-state: running creationTimestamp: "2022-01-19T09:40:01Z" finalizers: - machine.machine.openshift.io generateName: huliu-033-tnd6f-test2- generation: 3 labels: machine.openshift.io/cluster-api-cluster: huliu-033-tnd6f machine.openshift.io/cluster-api-machine-role: worker machine.openshift.io/cluster-api-machine-type: worker machine.openshift.io/cluster-api-machineset: huliu-033-tnd6f-test2 machine.openshift.io/instance-type: bx2d-4x16 machine.openshift.io/region: eu-gb machine.openshift.io/zone: eu-gb-3 name: huliu-033-tnd6f-test2-tvfn4 namespace: openshift-machine-api ownerReferences: - apiVersion: machine.openshift.io/v1beta1 blockOwnerDeletion: true controller: true kind: MachineSet name: huliu-033-tnd6f-test2 uid: 535c8f21-def7-423a-af42-03972d1c17e6 resourceVersion: "557348" uid: b7e2b967-8889-4bfe-b4cf-bc070cfcf62d spec: metadata: {} providerID: ibmvpc://huliu-033-tnd6f/eu-gb-3/huliu-033-tnd6f-test2-tvfn4 providerSpec: value: apiVersion: ibmcloudproviderconfig.openshift.io/v1beta1 credentialsSecret: name: ibmcloud-credentials image: huliu-033-tnd6f-rhcos kind: IBMCloudMachineProviderSpec metadata: creationTimestamp: null primaryNetworkInterface: securityGroups: - huliu-033-tnd6f-sg-cluster-wide - huliu-033-tnd6f-sg-openshift-net subnet: huliu-033-tnd6f-subnet-compute-eu-gb-3 profile: bx2d-4x16 region: eu-gb resourceGroup: huliu-033-tnd6f userDataSecret: name: worker-user-data vpc: huliu-033-tnd6f-vpc zone: eu-gb-3 status: addresses: - address: huliu-033-tnd6f-test2-tvfn4 type: InternalDNS - address: 10.242.129.25 type: InternalIP conditions: - lastTransitionTime: "2022-01-19T09:41:29Z" status: "True" type: InstanceExists lastUpdated: "2022-01-19T09:43:36Z" nodeRef: kind: Node name: huliu-033-tnd6f-test2-tvfn4 uid: 752bfd5a-c2a4-48d4-a5bb-6e04dfc09cba phase: Running providerStatus: conditions: - lastProbeTime: "2022-01-19T09:40:19Z" lastTransitionTime: "2022-01-19T09:40:19Z" message: Machine successfully created reason: MachineCreationSucceeded status: "True" type: MachineCreated instanceId: 07a7_57cf19d7-d3b6-460d-9999-e920540b1732 instanceState: running liuhuali@Lius-MacBook-Pro huali-test % 3.Machine can be deleted successfully liuhuali@Lius-MacBook-Pro huali-test % oc delete machine huliu-033-tnd6f-test2-tvfn4 machine.machine.openshift.io "huliu-033-tnd6f-test2-tvfn4" deleted liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-033-tnd6f-master-0 Running bx2d-4x16 eu-gb eu-gb-1 28h huliu-033-tnd6f-master-1 Running bx2d-4x16 eu-gb eu-gb-2 28h huliu-033-tnd6f-master-2 Running bx2d-4x16 eu-gb eu-gb-3 28h huliu-033-tnd6f-test-brvps Running bx2d-4x16 eu-gb eu-gb-3 4h15m huliu-033-tnd6f-test2-g2k5m Provisioned bx2d-4x16 eu-gb eu-gb-3 100s huliu-033-tnd6f-worker-1-7v4bl Running bx2d-4x16 eu-gb eu-gb-1 26h huliu-033-tnd6f-worker-1-btr7t Deleting bx2d-4x16 eu-gb eu-gb-1 28h huliu-033-tnd6f-worker-2-x68w8 Running bx2d-4x16 eu-gb eu-gb-2 28h huliu-033-tnd6f-worker-3-9dqqp Running bx2d-4x16 eu-gb eu-gb-3 28h liuhuali@Lius-MacBook-Pro huali-test % Actual results: Machine without Machine Deletion Hook, machine deleted successfully Expected results: Machine with Machine Deletion Hook, Machine stuck in deleting phase Additional info: https://issues.redhat.com/browse/OCPCLOUD-1348
Verified on 4.10.0-0.nightly-2022-01-21-074618, Machine Deletion Hook works and machine stuck in deleting phase now. Steps: 1.Create a machineset with Machine Deletion Hook, for example spec: lifecycleHooks: preTerminate: - name: MigrateImportantApp owner: etcd liuhuali@Lius-MacBook-Pro huali-test % oc create -f ms1.yaml machineset.machine.openshift.io/huliu-045-nvsdk-terminate created 2.Machine created with Machine Deletion Hook liuhuali@Lius-MacBook-Pro huali-test % oc get machine huliu-045-nvsdk-terminate-9n7wp -o yaml apiVersion: machine.openshift.io/v1beta1 kind: Machine metadata: annotations: machine.openshift.io/instance-state: running creationTimestamp: "2022-01-24T04:53:10Z" finalizers: - machine.machine.openshift.io generateName: huliu-045-nvsdk-terminate- generation: 2 labels: machine.openshift.io/cluster-api-cluster: huliu-045-nvsdk machine.openshift.io/cluster-api-machine-role: worker machine.openshift.io/cluster-api-machine-type: worker machine.openshift.io/cluster-api-machineset: huliu-045-nvsdk-terminate machine.openshift.io/instance-type: bx2d-4x16 machine.openshift.io/region: eu-gb machine.openshift.io/zone: eu-gb-1 name: huliu-045-nvsdk-terminate-9n7wp namespace: openshift-machine-api ownerReferences: - apiVersion: machine.openshift.io/v1beta1 blockOwnerDeletion: true controller: true kind: MachineSet name: huliu-045-nvsdk-terminate uid: b7f8b192-5cdb-4c37-8fd6-f8f9c1f10275 resourceVersion: "84874" uid: 07e55608-679d-49c2-a2fd-919e12a23825 spec: lifecycleHooks: preTerminate: - name: MigrateImportantApp owner: etcd metadata: {} providerID: ibmvpc://huliu-045-nvsdk/eu-gb-1/huliu-045-nvsdk-terminate-9n7wp providerSpec: value: apiVersion: ibmcloudproviderconfig.openshift.io/v1beta1 credentialsSecret: name: ibmcloud-credentials image: huliu-045-nvsdk-rhcos kind: IBMCloudMachineProviderSpec metadata: creationTimestamp: null primaryNetworkInterface: securityGroups: - huliu-045-nvsdk-sg-cluster-wide - huliu-045-nvsdk-sg-openshift-net subnet: huliu-045-nvsdk-subnet-compute-eu-gb-1 profile: bx2d-4x16 region: eu-gb resourceGroup: huliu-045-nvsdk userDataSecret: name: worker-user-data vpc: huliu-045-nvsdk-vpc zone: eu-gb-1 status: addresses: - address: huliu-045-nvsdk-terminate-9n7wp type: InternalDNS - address: 10.242.1.6 type: InternalIP conditions: - lastTransitionTime: "2022-01-24T04:53:19Z" status: "True" type: Drainable - lastTransitionTime: "2022-01-24T04:54:45Z" status: "True" type: InstanceExists - lastTransitionTime: "2022-01-24T04:53:19Z" message: 'Terminate operation currently blocked by: [{Name:MigrateImportantApp Owner:etcd}]' reason: HookPresent severity: Warning status: "False" type: Terminable lastUpdated: "2022-01-24T04:56:25Z" nodeRef: kind: Node name: huliu-045-nvsdk-terminate-9n7wp uid: 5da810a7-509f-4e9f-87b7-5f80441af87d phase: Running providerStatus: conditions: - lastProbeTime: "2022-01-24T04:53:39Z" lastTransitionTime: "2022-01-24T04:53:39Z" message: Machine successfully created reason: MachineCreationSucceeded status: "True" type: MachineCreated instanceId: 0787_302172ca-ec0a-4f79-a5dd-d3f0391d9473 instanceState: running liuhuali@Lius-MacBook-Pro huali-test % 3. delete machine and machine stuck in deleting phase liuhuali@Lius-MacBook-Pro huali-test % oc delete machine huliu-045-nvsdk-terminate-9n7wp machine.machine.openshift.io "huliu-045-nvsdk-terminate-9n7wp" deleted ^C liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-045-nvsdk-drain-m8rzt Running bx2d-4x16 eu-gb eu-gb-1 11m huliu-045-nvsdk-invalid-4wv85 Provisioned bx2d-4x16 eu-gb eu-gb-1 2m41s huliu-045-nvsdk-master-0 Running bx2d-4x16 eu-gb eu-gb-1 3h44m huliu-045-nvsdk-master-1 Running bx2d-4x16 eu-gb eu-gb-2 3h44m huliu-045-nvsdk-master-2 Running bx2d-4x16 eu-gb eu-gb-3 3h45m huliu-045-nvsdk-terminate-9n7wp Deleting bx2d-4x16 eu-gb eu-gb-1 14m huliu-045-nvsdk-terminate-g45mg Running bx2d-4x16 eu-gb eu-gb-1 7m29s huliu-045-nvsdk-worker-1-2ws4h Running bx2d-4x16 eu-gb eu-gb-1 3h36m huliu-045-nvsdk-worker-2-wgf7q Running bx2d-4x16 eu-gb eu-gb-2 3h36m huliu-045-nvsdk-worker-3-nk74c Running bx2d-4x16 eu-gb eu-gb-3 3h36m liuhuali@Lius-MacBook-Pro huali-test %
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056