Bug 2042715 - [AliCloud] Machine Deletion Hook cannot work on AliCloud
Summary: [AliCloud] Machine Deletion Hook cannot work on AliCloud
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Compute
Version: 4.10
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.10.0
Assignee: Joel Speed
QA Contact: Huali Liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-20 01:47 UTC by Huali Liu
Modified: 2022-03-10 16:41 UTC (History)
0 users

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-10 16:40:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-api-provider-alibaba pull 27 0 None Waiting on Red Hat Unable to install a host with bond 2022-06-24 14:25:09 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:41:09 UTC

Description Huali Liu 2022-01-20 01:47:45 UTC
Description of problem:
Machine Deletion Hook cannot work on AliCloud, lifecycleHooks field exists in machine, but machine can be deleted successfully

Version-Release number of selected component (if applicable):
4.10.0-0.nightly-2022-01-17-223655

How reproducible:
Always

Steps to Reproduce:
1.Create a machineset with Machine Deletion Hook, for example
    spec:
      lifecycleHooks:
        preTerminate:
        - name: MigrateImportantApp
          owner: etcd

2.Machine created with Machine Deletion Hook

liuhuali@Lius-MacBook-Pro huali-test % oc create -f ms3.yaml 
machineset.machine.openshift.io/huliu-038-fsbld-test created

liuhuali@Lius-MacBook-Pro huali-test % oc get machine huliu-038-fsbld-test-b9jwz -o yaml
apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
  annotations:
    machine.openshift.io/instance-state: Running
  creationTimestamp: "2022-01-19T09:41:00Z"
  finalizers:
  - machine.machine.openshift.io
  generateName: huliu-038-fsbld-test-
  generation: 2
  labels:
    machine.openshift.io/cluster-api-cluster: huliu-038-fsbld
    machine.openshift.io/cluster-api-machine-role: worker
    machine.openshift.io/cluster-api-machine-type: worker
    machine.openshift.io/cluster-api-machineset: huliu-038-fsbld-test
    machine.openshift.io/instance-type: ecs.g6.large
    machine.openshift.io/region: us-east-1
    machine.openshift.io/zone: us-east-1a
  name: huliu-038-fsbld-test-b9jwz
  namespace: openshift-machine-api
  ownerReferences:
  - apiVersion: machine.openshift.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: MachineSet
    name: huliu-038-fsbld-test
    uid: 504e2f1f-7cf3-4bb8-b0ac-407b912dab19
  resourceVersion: "97854"
  uid: ebe33d06-26a0-4890-848c-36bf2dfcd2f5
spec:
  lifecycleHooks:
    preTerminate:
    - name: MigrateImportantApp
      owner: etcd
  metadata: {}
  providerID: alicloud://us-east-1.i-0xih4n0v86yytlhs9giu
  providerSpec:
    value:
      apiVersion: alibabacloudmachineproviderconfig.openshift.io/v1beta1
      bandwidth: {}
      credentialsSecret:
        name: alibabacloud-credentials
      dedicatedHostId: ""
      imageId: m-0xi5sh1y90lsbo8ayl0a
      instanceType: ecs.g6.large
      kind: AlibabaCloudMachineProviderConfig
      metadata:
        creationTimestamp: null
      ramRoleName: huliu-038-fsbld-role-worker
      regionId: us-east-1
      securityGroups:
      - tags:
        - Key: kubernetes.io/cluster/huliu-038-fsbld
          Value: owned
        - Key: GISV
          Value: ocp
        - Key: sigs.k8s.io/cloud-provider-alibaba/origin
          Value: ocp
        - Key: Name
          Value: huliu-038-fsbld-sg-worker
      subscription: {}
      systemDisk:
        category: cloud_essd
        size: 120
      tags:
      - Key: kubernetes.io/cluster/huliu-038-fsbld
        Value: owned
      - Key: GISV
        Value: ocp
      - Key: sigs.k8s.io/cloud-provider-alibaba/origin
        Value: ocp
      userDataSecret:
        name: worker-user-data
      vSwitch:
        tags:
        - Key: kubernetes.io/cluster/huliu-038-fsbld
          Value: owned
        - Key: GISV
          Value: ocp
        - Key: sigs.k8s.io/cloud-provider-alibaba/origin
          Value: ocp
        - Key: Name
          Value: huliu-038-fsbld-vswitch-us-east-1a
      zoneId: us-east-1a
status:
  addresses:
  - address: 10.0.31.75
    type: InternalIP
  - address: huliu-038-fsbld-test-b9jwz
    type: InternalDNS
  conditions:
  - lastTransitionTime: "2022-01-19T09:42:39Z"
    status: "True"
    type: InstanceExists
  lastUpdated: "2022-01-19T09:46:06Z"
  nodeRef:
    kind: Node
    name: huliu-038-fsbld-test-b9jwz
    uid: 47359d20-c1b8-48e5-b41f-11698c7bd832
  phase: Running
  providerStatus:
    conditions:
    - lastTransitionTime: "2022-01-19T09:42:33Z"
      message: Machine successfully created
      reason: MachineCreationSucceeded
      status: "True"
      type: MachineCreation
    instanceId: i-0xih4n0v86yytlhs9giu
    instanceState: Running
    metadata: {}
liuhuali@Lius-MacBook-Pro huali-test % 

3. Machine can be deleted successfully

liuhuali@Lius-MacBook-Pro huali-test % oc delete machine huliu-038-fsbld-test-b9jwz  
machine.machine.openshift.io "huliu-038-fsbld-test-b9jwz" deleted
liuhuali@Lius-MacBook-Pro huali-test % oc get machine
NAME                                      PHASE         TYPE            REGION      ZONE         AGE
huliu-038-fsbld-master-0                  Running       ecs.g6.xlarge   us-east-1   us-east-1b   5h
huliu-038-fsbld-master-1                  Running       ecs.g6.xlarge   us-east-1   us-east-1a   5h
huliu-038-fsbld-master-2                  Running       ecs.g6.xlarge   us-east-1   us-east-1b   5h
huliu-038-fsbld-test-p95qb                Provisioned   ecs.g6.large    us-east-1   us-east-1a   2m16s
huliu-038-fsbld-worker-us-east-1a-t7kqp   Running       ecs.g6.large    us-east-1   us-east-1a   4h53m
huliu-038-fsbld-worker-us-east-1b-gttws   Running       ecs.g6.large    us-east-1   us-east-1b   4h53m
huliu-038-fsbld-worker-us-east-1b-xx4pt   Running       ecs.g6.large    us-east-1   us-east-1b   4h53m
liuhuali@Lius-MacBook-Pro huali-test % 

Actual results:
Machine deleted successfully

Expected results:
Machine stuck in deleting phase

Additional info:
https://issues.redhat.com/browse/OCPCLOUD-1348

Comment 3 Huali Liu 2022-01-24 02:23:51 UTC
Verified on 4.10.0-0.nightly-2022-01-21-074618, Machine Deletion Hook works and machine stuck in deleting phase now.

Steps:
1.Create a machineset with Machine Deletion Hook, for example
    spec:
      lifecycleHooks:
        preTerminate:
        - name: MigrateImportantApp
          owner: etcd
liuhuali@Lius-MacBook-Pro huali-test % oc create -f ms1.yaml 
machineset.machine.openshift.io/huliu-046-kxqvn-test created

2.Machine created with Machine Deletion Hook
liuhuali@Lius-MacBook-Pro huali-test % oc get machine huliu-046-kxqvn-test-9c6lv -o yaml
apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
  annotations:
    machine.openshift.io/instance-state: Running
  creationTimestamp: "2022-01-24T02:04:00Z"
  finalizers:
  - machine.machine.openshift.io
  generateName: huliu-046-kxqvn-test-
  generation: 2
  labels:
    machine.openshift.io/cluster-api-cluster: huliu-046-kxqvn
    machine.openshift.io/cluster-api-machine-role: worker
    machine.openshift.io/cluster-api-machine-type: worker
    machine.openshift.io/cluster-api-machineset: huliu-046-kxqvn-test
    machine.openshift.io/instance-type: ecs.g6.large
    machine.openshift.io/region: us-east-1
    machine.openshift.io/zone: us-east-1a
  name: huliu-046-kxqvn-test-9c6lv
  namespace: openshift-machine-api
  ownerReferences:
  - apiVersion: machine.openshift.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: MachineSet
    name: huliu-046-kxqvn-test
    uid: 1e62b9e0-63bf-4460-a28a-e9cf2f9b90ec
  resourceVersion: "41915"
  uid: 1cd0da4f-816a-4ea2-8d42-2b0da9bd896c
spec:
  lifecycleHooks:
    preTerminate:
    - name: MigrateImportantApp
      owner: etcd
  metadata: {}
  providerID: alicloud://us-east-1.i-0xi5mkzdhi83bmvep8ql
  providerSpec:
    value:
      apiVersion: alibabacloudmachineproviderconfig.openshift.io/v1beta1
      bandwidth: {}
      credentialsSecret:
        name: alibabacloud-credentials
      dedicatedHostId: ""
      imageId: m-0xi5sh1y90lsbo8ayl0a
      instanceType: ecs.g6.large
      kind: AlibabaCloudMachineProviderConfig
      metadata:
        creationTimestamp: null
      ramRoleName: huliu-046-kxqvn-role-worker
      regionId: us-east-1
      securityGroups:
      - tags:
        - Key: kubernetes.io/cluster/huliu-046-kxqvn
          Value: owned
        - Key: GISV
          Value: ocp
        - Key: sigs.k8s.io/cloud-provider-alibaba/origin
          Value: ocp
        - Key: Name
          Value: huliu-046-kxqvn-sg-worker
      subscription: {}
      systemDisk:
        category: cloud_essd
        size: 120
      tags:
      - Key: kubernetes.io/cluster/huliu-046-kxqvn
        Value: owned
      - Key: GISV
        Value: ocp
      - Key: sigs.k8s.io/cloud-provider-alibaba/origin
        Value: ocp
      userDataSecret:
        name: worker-user-data
      vSwitch:
        tags:
        - Key: kubernetes.io/cluster/huliu-046-kxqvn
          Value: owned
        - Key: GISV
          Value: ocp
        - Key: sigs.k8s.io/cloud-provider-alibaba/origin
          Value: ocp
        - Key: Name
          Value: huliu-046-kxqvn-vswitch-us-east-1a
      zoneId: us-east-1a
status:
  addresses:
  - address: 10.0.28.183
    type: InternalIP
  - address: huliu-046-kxqvn-test-9c6lv
    type: InternalDNS
  conditions:
  - lastTransitionTime: "2022-01-24T02:04:01Z"
    status: "True"
    type: Drainable
  - lastTransitionTime: "2022-01-24T02:04:58Z"
    status: "True"
    type: InstanceExists
  - lastTransitionTime: "2022-01-24T02:04:01Z"
    message: 'Terminate operation currently blocked by: [{Name:MigrateImportantApp
      Owner:etcd}]'
    reason: HookPresent
    severity: Warning
    status: "False"
    type: Terminable
  lastUpdated: "2022-01-24T02:08:14Z"
  nodeRef:
    kind: Node
    name: huliu-046-kxqvn-test-9c6lv
    uid: 991f86d5-7b7f-4099-b9e2-1218085ddc73
  phase: Running
  providerStatus:
    conditions:
    - lastTransitionTime: "2022-01-24T02:04:58Z"
      message: Machine successfully created
      reason: MachineCreationSucceeded
      status: "True"
      type: MachineCreation
    instanceId: i-0xi5mkzdhi83bmvep8ql
    instanceState: Running
    metadata: {}
liuhuali@Lius-MacBook-Pro huali-test % 

3. delete machine and machine stuck in deleting phase 

liuhuali@Lius-MacBook-Pro huali-test % oc delete machine huliu-046-kxqvn-test-9c6lv
machine.machine.openshift.io "huliu-046-kxqvn-test-9c6lv" deleted
^C
liuhuali@Lius-MacBook-Pro huali-test % oc get machine                              
NAME                                      PHASE      TYPE            REGION      ZONE         AGE
huliu-046-kxqvn-drain-4nsb5               Running    ecs.g6.large    us-east-1   us-east-1a   7m50s
huliu-046-kxqvn-master-0                  Running    ecs.g6.xlarge   us-east-1   us-east-1b   63m
huliu-046-kxqvn-master-1                  Running    ecs.g6.xlarge   us-east-1   us-east-1a   63m
huliu-046-kxqvn-master-2                  Running    ecs.g6.xlarge   us-east-1   us-east-1b   63m
huliu-046-kxqvn-test-9c6lv                Deleting   ecs.g6.large    us-east-1   us-east-1a   14m
huliu-046-kxqvn-test-mx4mh                Running    ecs.g6.large    us-east-1   us-east-1a   6m17s
huliu-046-kxqvn-worker-us-east-1a-qbpck   Running    ecs.g6.large    us-east-1   us-east-1a   57m
huliu-046-kxqvn-worker-us-east-1b-8bp24   Running    ecs.g6.large    us-east-1   us-east-1b   57m
huliu-046-kxqvn-worker-us-east-1b-f2fnt   Running    ecs.g6.large    us-east-1   us-east-1b   57m
liuhuali@Lius-MacBook-Pro huali-test %

Comment 6 errata-xmlrpc 2022-03-10 16:40:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.