Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2042711

Summary: [IBMCloud] Machine Deletion Hook cannot work on IBMCloud
Product: OpenShift Container Platform Reporter: Huali Liu <huliu>
Component: Cloud ComputeAssignee: Joel Speed <jspeed>
Cloud Compute sub component: Other Providers QA Contact: Huali Liu <huliu>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: unspecified    
Version: 4.10   
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-10 16:40:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Huali Liu 2022-01-20 01:39:23 UTC
Description of problem:
Machine Deletion Hook cannot work on IBMCloud, lifecycleHooks field will be ignored in machine, and machine can be deleted successfully

Version-Release number of selected component (if applicable):
4.10.0-0.nightly-2022-01-17-223655

How reproducible:
Always

Steps to Reproduce:
1.Create a machineset with Machine Deletion Hook, for example
    spec:
      lifecycleHooks:
        preTerminate:
        - name: MigrateImportantApp
          owner: etcd

2.Machine created but without Machine Deletion Hook
liuhuali@Lius-MacBook-Pro huali-test % oc create -f ms4.yaml 
machineset.machine.openshift.io/huliu-033-tnd6f-test2 created

liuhuali@Lius-MacBook-Pro huali-test % oc get machine huliu-033-tnd6f-test2-tvfn4 -o yaml
apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
  annotations:
    machine.openshift.io/instance-state: running
  creationTimestamp: "2022-01-19T09:40:01Z"
  finalizers:
  - machine.machine.openshift.io
  generateName: huliu-033-tnd6f-test2-
  generation: 3
  labels:
    machine.openshift.io/cluster-api-cluster: huliu-033-tnd6f
    machine.openshift.io/cluster-api-machine-role: worker
    machine.openshift.io/cluster-api-machine-type: worker
    machine.openshift.io/cluster-api-machineset: huliu-033-tnd6f-test2
    machine.openshift.io/instance-type: bx2d-4x16
    machine.openshift.io/region: eu-gb
    machine.openshift.io/zone: eu-gb-3
  name: huliu-033-tnd6f-test2-tvfn4
  namespace: openshift-machine-api
  ownerReferences:
  - apiVersion: machine.openshift.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: MachineSet
    name: huliu-033-tnd6f-test2
    uid: 535c8f21-def7-423a-af42-03972d1c17e6
  resourceVersion: "557348"
  uid: b7e2b967-8889-4bfe-b4cf-bc070cfcf62d
spec:
  metadata: {}
  providerID: ibmvpc://huliu-033-tnd6f/eu-gb-3/huliu-033-tnd6f-test2-tvfn4
  providerSpec:
    value:
      apiVersion: ibmcloudproviderconfig.openshift.io/v1beta1
      credentialsSecret:
        name: ibmcloud-credentials
      image: huliu-033-tnd6f-rhcos
      kind: IBMCloudMachineProviderSpec
      metadata:
        creationTimestamp: null
      primaryNetworkInterface:
        securityGroups:
        - huliu-033-tnd6f-sg-cluster-wide
        - huliu-033-tnd6f-sg-openshift-net
        subnet: huliu-033-tnd6f-subnet-compute-eu-gb-3
      profile: bx2d-4x16
      region: eu-gb
      resourceGroup: huliu-033-tnd6f
      userDataSecret:
        name: worker-user-data
      vpc: huliu-033-tnd6f-vpc
      zone: eu-gb-3
status:
  addresses:
  - address: huliu-033-tnd6f-test2-tvfn4
    type: InternalDNS
  - address: 10.242.129.25
    type: InternalIP
  conditions:
  - lastTransitionTime: "2022-01-19T09:41:29Z"
    status: "True"
    type: InstanceExists
  lastUpdated: "2022-01-19T09:43:36Z"
  nodeRef:
    kind: Node
    name: huliu-033-tnd6f-test2-tvfn4
    uid: 752bfd5a-c2a4-48d4-a5bb-6e04dfc09cba
  phase: Running
  providerStatus:
    conditions:
    - lastProbeTime: "2022-01-19T09:40:19Z"
      lastTransitionTime: "2022-01-19T09:40:19Z"
      message: Machine successfully created
      reason: MachineCreationSucceeded
      status: "True"
      type: MachineCreated
    instanceId: 07a7_57cf19d7-d3b6-460d-9999-e920540b1732
    instanceState: running
liuhuali@Lius-MacBook-Pro huali-test % 

3.Machine can be deleted successfully
liuhuali@Lius-MacBook-Pro huali-test % oc delete machine huliu-033-tnd6f-test2-tvfn4
machine.machine.openshift.io "huliu-033-tnd6f-test2-tvfn4" deleted
liuhuali@Lius-MacBook-Pro huali-test % oc get machine
NAME                             PHASE         TYPE        REGION   ZONE      AGE
huliu-033-tnd6f-master-0         Running       bx2d-4x16   eu-gb    eu-gb-1   28h
huliu-033-tnd6f-master-1         Running       bx2d-4x16   eu-gb    eu-gb-2   28h
huliu-033-tnd6f-master-2         Running       bx2d-4x16   eu-gb    eu-gb-3   28h
huliu-033-tnd6f-test-brvps       Running       bx2d-4x16   eu-gb    eu-gb-3   4h15m
huliu-033-tnd6f-test2-g2k5m      Provisioned   bx2d-4x16   eu-gb    eu-gb-3   100s
huliu-033-tnd6f-worker-1-7v4bl   Running       bx2d-4x16   eu-gb    eu-gb-1   26h
huliu-033-tnd6f-worker-1-btr7t   Deleting      bx2d-4x16   eu-gb    eu-gb-1   28h
huliu-033-tnd6f-worker-2-x68w8   Running       bx2d-4x16   eu-gb    eu-gb-2   28h
huliu-033-tnd6f-worker-3-9dqqp   Running       bx2d-4x16   eu-gb    eu-gb-3   28h
liuhuali@Lius-MacBook-Pro huali-test % 

Actual results:
Machine without Machine Deletion Hook, machine deleted successfully

Expected results:
Machine with Machine Deletion Hook, Machine stuck in deleting phase

Additional info:
https://issues.redhat.com/browse/OCPCLOUD-1348

Comment 3 Huali Liu 2022-01-24 05:10:02 UTC
Verified on 4.10.0-0.nightly-2022-01-21-074618, Machine Deletion Hook works and machine stuck in deleting phase now.

Steps:
1.Create a machineset with Machine Deletion Hook, for example
    spec:
      lifecycleHooks:
        preTerminate:
        - name: MigrateImportantApp
          owner: etcd
liuhuali@Lius-MacBook-Pro huali-test % oc create -f ms1.yaml 
machineset.machine.openshift.io/huliu-045-nvsdk-terminate created

2.Machine created with Machine Deletion Hook
liuhuali@Lius-MacBook-Pro huali-test % oc get machine huliu-045-nvsdk-terminate-9n7wp -o yaml
apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
  annotations:
    machine.openshift.io/instance-state: running
  creationTimestamp: "2022-01-24T04:53:10Z"
  finalizers:
  - machine.machine.openshift.io
  generateName: huliu-045-nvsdk-terminate-
  generation: 2
  labels:
    machine.openshift.io/cluster-api-cluster: huliu-045-nvsdk
    machine.openshift.io/cluster-api-machine-role: worker
    machine.openshift.io/cluster-api-machine-type: worker
    machine.openshift.io/cluster-api-machineset: huliu-045-nvsdk-terminate
    machine.openshift.io/instance-type: bx2d-4x16
    machine.openshift.io/region: eu-gb
    machine.openshift.io/zone: eu-gb-1
  name: huliu-045-nvsdk-terminate-9n7wp
  namespace: openshift-machine-api
  ownerReferences:
  - apiVersion: machine.openshift.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: MachineSet
    name: huliu-045-nvsdk-terminate
    uid: b7f8b192-5cdb-4c37-8fd6-f8f9c1f10275
  resourceVersion: "84874"
  uid: 07e55608-679d-49c2-a2fd-919e12a23825
spec:
  lifecycleHooks:
    preTerminate:
    - name: MigrateImportantApp
      owner: etcd
  metadata: {}
  providerID: ibmvpc://huliu-045-nvsdk/eu-gb-1/huliu-045-nvsdk-terminate-9n7wp
  providerSpec:
    value:
      apiVersion: ibmcloudproviderconfig.openshift.io/v1beta1
      credentialsSecret:
        name: ibmcloud-credentials
      image: huliu-045-nvsdk-rhcos
      kind: IBMCloudMachineProviderSpec
      metadata:
        creationTimestamp: null
      primaryNetworkInterface:
        securityGroups:
        - huliu-045-nvsdk-sg-cluster-wide
        - huliu-045-nvsdk-sg-openshift-net
        subnet: huliu-045-nvsdk-subnet-compute-eu-gb-1
      profile: bx2d-4x16
      region: eu-gb
      resourceGroup: huliu-045-nvsdk
      userDataSecret:
        name: worker-user-data
      vpc: huliu-045-nvsdk-vpc
      zone: eu-gb-1
status:
  addresses:
  - address: huliu-045-nvsdk-terminate-9n7wp
    type: InternalDNS
  - address: 10.242.1.6
    type: InternalIP
  conditions:
  - lastTransitionTime: "2022-01-24T04:53:19Z"
    status: "True"
    type: Drainable
  - lastTransitionTime: "2022-01-24T04:54:45Z"
    status: "True"
    type: InstanceExists
  - lastTransitionTime: "2022-01-24T04:53:19Z"
    message: 'Terminate operation currently blocked by: [{Name:MigrateImportantApp
      Owner:etcd}]'
    reason: HookPresent
    severity: Warning
    status: "False"
    type: Terminable
  lastUpdated: "2022-01-24T04:56:25Z"
  nodeRef:
    kind: Node
    name: huliu-045-nvsdk-terminate-9n7wp
    uid: 5da810a7-509f-4e9f-87b7-5f80441af87d
  phase: Running
  providerStatus:
    conditions:
    - lastProbeTime: "2022-01-24T04:53:39Z"
      lastTransitionTime: "2022-01-24T04:53:39Z"
      message: Machine successfully created
      reason: MachineCreationSucceeded
      status: "True"
      type: MachineCreated
    instanceId: 0787_302172ca-ec0a-4f79-a5dd-d3f0391d9473
    instanceState: running
liuhuali@Lius-MacBook-Pro huali-test % 

3. delete machine and machine stuck in deleting phase 

liuhuali@Lius-MacBook-Pro huali-test % oc delete machine huliu-045-nvsdk-terminate-9n7wp
machine.machine.openshift.io "huliu-045-nvsdk-terminate-9n7wp" deleted
^C
liuhuali@Lius-MacBook-Pro huali-test % oc get machine                                   
NAME                              PHASE         TYPE        REGION   ZONE      AGE
huliu-045-nvsdk-drain-m8rzt       Running       bx2d-4x16   eu-gb    eu-gb-1   11m
huliu-045-nvsdk-invalid-4wv85     Provisioned   bx2d-4x16   eu-gb    eu-gb-1   2m41s
huliu-045-nvsdk-master-0          Running       bx2d-4x16   eu-gb    eu-gb-1   3h44m
huliu-045-nvsdk-master-1          Running       bx2d-4x16   eu-gb    eu-gb-2   3h44m
huliu-045-nvsdk-master-2          Running       bx2d-4x16   eu-gb    eu-gb-3   3h45m
huliu-045-nvsdk-terminate-9n7wp   Deleting      bx2d-4x16   eu-gb    eu-gb-1   14m
huliu-045-nvsdk-terminate-g45mg   Running       bx2d-4x16   eu-gb    eu-gb-1   7m29s
huliu-045-nvsdk-worker-1-2ws4h    Running       bx2d-4x16   eu-gb    eu-gb-1   3h36m
huliu-045-nvsdk-worker-2-wgf7q    Running       bx2d-4x16   eu-gb    eu-gb-2   3h36m
huliu-045-nvsdk-worker-3-nk74c    Running       bx2d-4x16   eu-gb    eu-gb-3   3h36m
liuhuali@Lius-MacBook-Pro huali-test %

Comment 6 errata-xmlrpc 2022-03-10 16:40:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056