Bug 2042711 - [IBMCloud] Machine Deletion Hook cannot work on IBMCloud
Summary: [IBMCloud] Machine Deletion Hook cannot work on IBMCloud
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Compute
Version: 4.10
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.10.0
Assignee: Joel Speed
QA Contact: Huali Liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-20 01:39 UTC by Huali Liu
Modified: 2022-03-10 16:41 UTC (History)
0 users

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-10 16:40:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-api-provider-ibmcloud pull 16 0 None open Bug 2042711: [OCPCLOUD-1268] Migrate Machine API to openshift/api 2022-01-20 09:20:21 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:41:09 UTC

Description Huali Liu 2022-01-20 01:39:23 UTC
Description of problem:
Machine Deletion Hook cannot work on IBMCloud, lifecycleHooks field will be ignored in machine, and machine can be deleted successfully

Version-Release number of selected component (if applicable):
4.10.0-0.nightly-2022-01-17-223655

How reproducible:
Always

Steps to Reproduce:
1.Create a machineset with Machine Deletion Hook, for example
    spec:
      lifecycleHooks:
        preTerminate:
        - name: MigrateImportantApp
          owner: etcd

2.Machine created but without Machine Deletion Hook
liuhuali@Lius-MacBook-Pro huali-test % oc create -f ms4.yaml 
machineset.machine.openshift.io/huliu-033-tnd6f-test2 created

liuhuali@Lius-MacBook-Pro huali-test % oc get machine huliu-033-tnd6f-test2-tvfn4 -o yaml
apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
  annotations:
    machine.openshift.io/instance-state: running
  creationTimestamp: "2022-01-19T09:40:01Z"
  finalizers:
  - machine.machine.openshift.io
  generateName: huliu-033-tnd6f-test2-
  generation: 3
  labels:
    machine.openshift.io/cluster-api-cluster: huliu-033-tnd6f
    machine.openshift.io/cluster-api-machine-role: worker
    machine.openshift.io/cluster-api-machine-type: worker
    machine.openshift.io/cluster-api-machineset: huliu-033-tnd6f-test2
    machine.openshift.io/instance-type: bx2d-4x16
    machine.openshift.io/region: eu-gb
    machine.openshift.io/zone: eu-gb-3
  name: huliu-033-tnd6f-test2-tvfn4
  namespace: openshift-machine-api
  ownerReferences:
  - apiVersion: machine.openshift.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: MachineSet
    name: huliu-033-tnd6f-test2
    uid: 535c8f21-def7-423a-af42-03972d1c17e6
  resourceVersion: "557348"
  uid: b7e2b967-8889-4bfe-b4cf-bc070cfcf62d
spec:
  metadata: {}
  providerID: ibmvpc://huliu-033-tnd6f/eu-gb-3/huliu-033-tnd6f-test2-tvfn4
  providerSpec:
    value:
      apiVersion: ibmcloudproviderconfig.openshift.io/v1beta1
      credentialsSecret:
        name: ibmcloud-credentials
      image: huliu-033-tnd6f-rhcos
      kind: IBMCloudMachineProviderSpec
      metadata:
        creationTimestamp: null
      primaryNetworkInterface:
        securityGroups:
        - huliu-033-tnd6f-sg-cluster-wide
        - huliu-033-tnd6f-sg-openshift-net
        subnet: huliu-033-tnd6f-subnet-compute-eu-gb-3
      profile: bx2d-4x16
      region: eu-gb
      resourceGroup: huliu-033-tnd6f
      userDataSecret:
        name: worker-user-data
      vpc: huliu-033-tnd6f-vpc
      zone: eu-gb-3
status:
  addresses:
  - address: huliu-033-tnd6f-test2-tvfn4
    type: InternalDNS
  - address: 10.242.129.25
    type: InternalIP
  conditions:
  - lastTransitionTime: "2022-01-19T09:41:29Z"
    status: "True"
    type: InstanceExists
  lastUpdated: "2022-01-19T09:43:36Z"
  nodeRef:
    kind: Node
    name: huliu-033-tnd6f-test2-tvfn4
    uid: 752bfd5a-c2a4-48d4-a5bb-6e04dfc09cba
  phase: Running
  providerStatus:
    conditions:
    - lastProbeTime: "2022-01-19T09:40:19Z"
      lastTransitionTime: "2022-01-19T09:40:19Z"
      message: Machine successfully created
      reason: MachineCreationSucceeded
      status: "True"
      type: MachineCreated
    instanceId: 07a7_57cf19d7-d3b6-460d-9999-e920540b1732
    instanceState: running
liuhuali@Lius-MacBook-Pro huali-test % 

3.Machine can be deleted successfully
liuhuali@Lius-MacBook-Pro huali-test % oc delete machine huliu-033-tnd6f-test2-tvfn4
machine.machine.openshift.io "huliu-033-tnd6f-test2-tvfn4" deleted
liuhuali@Lius-MacBook-Pro huali-test % oc get machine
NAME                             PHASE         TYPE        REGION   ZONE      AGE
huliu-033-tnd6f-master-0         Running       bx2d-4x16   eu-gb    eu-gb-1   28h
huliu-033-tnd6f-master-1         Running       bx2d-4x16   eu-gb    eu-gb-2   28h
huliu-033-tnd6f-master-2         Running       bx2d-4x16   eu-gb    eu-gb-3   28h
huliu-033-tnd6f-test-brvps       Running       bx2d-4x16   eu-gb    eu-gb-3   4h15m
huliu-033-tnd6f-test2-g2k5m      Provisioned   bx2d-4x16   eu-gb    eu-gb-3   100s
huliu-033-tnd6f-worker-1-7v4bl   Running       bx2d-4x16   eu-gb    eu-gb-1   26h
huliu-033-tnd6f-worker-1-btr7t   Deleting      bx2d-4x16   eu-gb    eu-gb-1   28h
huliu-033-tnd6f-worker-2-x68w8   Running       bx2d-4x16   eu-gb    eu-gb-2   28h
huliu-033-tnd6f-worker-3-9dqqp   Running       bx2d-4x16   eu-gb    eu-gb-3   28h
liuhuali@Lius-MacBook-Pro huali-test % 

Actual results:
Machine without Machine Deletion Hook, machine deleted successfully

Expected results:
Machine with Machine Deletion Hook, Machine stuck in deleting phase

Additional info:
https://issues.redhat.com/browse/OCPCLOUD-1348

Comment 3 Huali Liu 2022-01-24 05:10:02 UTC
Verified on 4.10.0-0.nightly-2022-01-21-074618, Machine Deletion Hook works and machine stuck in deleting phase now.

Steps:
1.Create a machineset with Machine Deletion Hook, for example
    spec:
      lifecycleHooks:
        preTerminate:
        - name: MigrateImportantApp
          owner: etcd
liuhuali@Lius-MacBook-Pro huali-test % oc create -f ms1.yaml 
machineset.machine.openshift.io/huliu-045-nvsdk-terminate created

2.Machine created with Machine Deletion Hook
liuhuali@Lius-MacBook-Pro huali-test % oc get machine huliu-045-nvsdk-terminate-9n7wp -o yaml
apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
  annotations:
    machine.openshift.io/instance-state: running
  creationTimestamp: "2022-01-24T04:53:10Z"
  finalizers:
  - machine.machine.openshift.io
  generateName: huliu-045-nvsdk-terminate-
  generation: 2
  labels:
    machine.openshift.io/cluster-api-cluster: huliu-045-nvsdk
    machine.openshift.io/cluster-api-machine-role: worker
    machine.openshift.io/cluster-api-machine-type: worker
    machine.openshift.io/cluster-api-machineset: huliu-045-nvsdk-terminate
    machine.openshift.io/instance-type: bx2d-4x16
    machine.openshift.io/region: eu-gb
    machine.openshift.io/zone: eu-gb-1
  name: huliu-045-nvsdk-terminate-9n7wp
  namespace: openshift-machine-api
  ownerReferences:
  - apiVersion: machine.openshift.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: MachineSet
    name: huliu-045-nvsdk-terminate
    uid: b7f8b192-5cdb-4c37-8fd6-f8f9c1f10275
  resourceVersion: "84874"
  uid: 07e55608-679d-49c2-a2fd-919e12a23825
spec:
  lifecycleHooks:
    preTerminate:
    - name: MigrateImportantApp
      owner: etcd
  metadata: {}
  providerID: ibmvpc://huliu-045-nvsdk/eu-gb-1/huliu-045-nvsdk-terminate-9n7wp
  providerSpec:
    value:
      apiVersion: ibmcloudproviderconfig.openshift.io/v1beta1
      credentialsSecret:
        name: ibmcloud-credentials
      image: huliu-045-nvsdk-rhcos
      kind: IBMCloudMachineProviderSpec
      metadata:
        creationTimestamp: null
      primaryNetworkInterface:
        securityGroups:
        - huliu-045-nvsdk-sg-cluster-wide
        - huliu-045-nvsdk-sg-openshift-net
        subnet: huliu-045-nvsdk-subnet-compute-eu-gb-1
      profile: bx2d-4x16
      region: eu-gb
      resourceGroup: huliu-045-nvsdk
      userDataSecret:
        name: worker-user-data
      vpc: huliu-045-nvsdk-vpc
      zone: eu-gb-1
status:
  addresses:
  - address: huliu-045-nvsdk-terminate-9n7wp
    type: InternalDNS
  - address: 10.242.1.6
    type: InternalIP
  conditions:
  - lastTransitionTime: "2022-01-24T04:53:19Z"
    status: "True"
    type: Drainable
  - lastTransitionTime: "2022-01-24T04:54:45Z"
    status: "True"
    type: InstanceExists
  - lastTransitionTime: "2022-01-24T04:53:19Z"
    message: 'Terminate operation currently blocked by: [{Name:MigrateImportantApp
      Owner:etcd}]'
    reason: HookPresent
    severity: Warning
    status: "False"
    type: Terminable
  lastUpdated: "2022-01-24T04:56:25Z"
  nodeRef:
    kind: Node
    name: huliu-045-nvsdk-terminate-9n7wp
    uid: 5da810a7-509f-4e9f-87b7-5f80441af87d
  phase: Running
  providerStatus:
    conditions:
    - lastProbeTime: "2022-01-24T04:53:39Z"
      lastTransitionTime: "2022-01-24T04:53:39Z"
      message: Machine successfully created
      reason: MachineCreationSucceeded
      status: "True"
      type: MachineCreated
    instanceId: 0787_302172ca-ec0a-4f79-a5dd-d3f0391d9473
    instanceState: running
liuhuali@Lius-MacBook-Pro huali-test % 

3. delete machine and machine stuck in deleting phase 

liuhuali@Lius-MacBook-Pro huali-test % oc delete machine huliu-045-nvsdk-terminate-9n7wp
machine.machine.openshift.io "huliu-045-nvsdk-terminate-9n7wp" deleted
^C
liuhuali@Lius-MacBook-Pro huali-test % oc get machine                                   
NAME                              PHASE         TYPE        REGION   ZONE      AGE
huliu-045-nvsdk-drain-m8rzt       Running       bx2d-4x16   eu-gb    eu-gb-1   11m
huliu-045-nvsdk-invalid-4wv85     Provisioned   bx2d-4x16   eu-gb    eu-gb-1   2m41s
huliu-045-nvsdk-master-0          Running       bx2d-4x16   eu-gb    eu-gb-1   3h44m
huliu-045-nvsdk-master-1          Running       bx2d-4x16   eu-gb    eu-gb-2   3h44m
huliu-045-nvsdk-master-2          Running       bx2d-4x16   eu-gb    eu-gb-3   3h45m
huliu-045-nvsdk-terminate-9n7wp   Deleting      bx2d-4x16   eu-gb    eu-gb-1   14m
huliu-045-nvsdk-terminate-g45mg   Running       bx2d-4x16   eu-gb    eu-gb-1   7m29s
huliu-045-nvsdk-worker-1-2ws4h    Running       bx2d-4x16   eu-gb    eu-gb-1   3h36m
huliu-045-nvsdk-worker-2-wgf7q    Running       bx2d-4x16   eu-gb    eu-gb-2   3h36m
huliu-045-nvsdk-worker-3-nk74c    Running       bx2d-4x16   eu-gb    eu-gb-3   3h36m
liuhuali@Lius-MacBook-Pro huali-test %

Comment 6 errata-xmlrpc 2022-03-10 16:40:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.