Because Cinder CSI driver currently doesn't support adding cluster ID to volume snapshot metadata we had to introduce a workaround PR to delete snapshots: https://github.com/openshift/installer/pull/4962 It should be reverted when the upstream fix is merged: https://github.com/kubernetes/cloud-provider-openstack/pull/1544
Removing the Triaged keyword because: * the target release value is missing * the QE automation assessment (flag qe_test_coverage) is missing
Verified in 4.11.0-0.nightly-2022-06-15-222801 on top of OSP 16.1.6. Cluster destroy deletes now the volumes and snapshots in OSP created by the cluster. Verification steps: > Create a PVC: $ cat <<EOF | oc apply -f - --- apiVersion: project.openshift.io/v1 kind: Project metadata: name: cinderpvc labels: kubernetes.io/metadata.name: cinderpvc --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: "pvc-1" namespace: "cinderpvc" spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi #storageClassName: (default) --- apiVersion: apps/v1 kind: Deployment metadata: name: demo-1 namespace: "cinderpvc" spec: replicas: 1 selector: matchLabels: app: demo-1 template: metadata: labels: app: demo-1 spec: containers: - name: demo image: quay.io/kuryr/demo ports: - containerPort: 80 protocol: TCP volumeMounts: - mountPath: /var/lib/www/data name: mydata volumes: - name: mydata persistentVolumeClaim: claimName: pvc-1 readOnly: false EOF $ oc -n cinderpvc get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES demo-1-d9b4c79b6-f58lm 1/1 Running 0 53s 10.131.0.18 ostest-b5mzn-worker-0-2hhl5 <none> <none> $ oc -n cinderpvc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-1 Bound pvc-5c1fc842-ae70-4397-a4f9-7a0130739444 1Gi RWO standard-csi 36s $ openstack volume list +--------------------------------------+------------------------------------------+--------+------+------------------------------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+------------------------------------------+--------+------+------------------------------------------------------+ | 2134abf0-9b56-4c81-9b4c-8fb75e037b81 | pvc-5c1fc842-ae70-4397-a4f9-7a0130739444 | in-use | 1 | Attached to ostest-b5mzn-worker-0-2hhl5 on /dev/vdb | +--------------------------------------+------------------------------------------+--------+------+------------------------------------------------------+ > Write some data in the volume: $ oc -n cinderpvc rsh demo-1-d9b4c79b6-f58lm /bin/sh /home/kuryr $ echo “hello from pvc-5c1fc842-ae70-4397-a4f9-7a0130739444 volume” > /var/lib/www/data/test.txt /home/kuryr $ cat /var/lib/www/data/test.txt “hello from pvc-5c1fc842-ae70-4397-a4f9-7a0130739444 volume” /home/kuryr $ > Create a VolumeSnapshotClass: $ cat <<EOF | oc apply -f - --- apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: cinderpvc-snapclass namespace: cinderpvc driver: cinder.csi.openstack.org deletionPolicy: Delete parameters: force-create: "true" EOF $ oc get VolumeSnapshotClass NAME DRIVER DELETIONPOLICY AGE cinderpvc-snapclass cinder.csi.openstack.org Delete 28s standard-csi cinder.csi.openstack.org Delete 127m > Create a VolumeSnapshot: $ cat <<EOF | oc apply -f - --- apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: pvc-1-snap namespace: cinderpvc spec: volumeSnapshotClassName: cinderpvc-snapclass source: persistentVolumeClaimName: pvc-1 EOF $ oc -n cinderpvc get volumesnapshot NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE pvc-1-snap true pvc-1 1Gi cinderpvc-snapclass snapcontent-77dc9462-3910-4a76-b004-1c45f628d140 111s 114s $ oc -n cinderpvc get volumesnapshotcontent NAME READYTOUSE RESTORESIZE DELETIONPOLICY DRIVER VOLUMESNAPSHOTCLASS VOLUMESNAPSHOT VOLUMESNAPSHOTNAMESPACE AGE snapcontent-77dc9462-3910-4a76-b004-1c45f628d140 true 1073741824 Delete cinder.csi.openstack.org cinderpvc-snapclass pvc-1-snap cinderpvc 2m12s $ oc -n cinderpvc describe volumesnapshot pvc-1-snap ame: pvc-1-snap Namespace: cinderpvc Labels: <none> Annotations: <none> API Version: snapshot.storage.k8s.io/v1 Kind: VolumeSnapshot Metadata: Creation Timestamp: 2022-06-21T11:04:39Z Finalizers: snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection snapshot.storage.kubernetes.io/volumesnapshot-bound-protection Generation: 1 Managed Fields: API Version: snapshot.storage.k8s.io/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubectl.kubernetes.io/last-applied-configuration: f:spec: .: f:source: .: f:persistentVolumeClaimName: f:volumeSnapshotClassName: Manager: kubectl-client-side-apply Operation: Update Time: 2022-06-21T11:04:39Z API Version: snapshot.storage.k8s.io/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:finalizers: .: v:"snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection": v:"snapshot.storage.kubernetes.io/volumesnapshot-bound-protection": Manager: snapshot-controller Operation: Update Time: 2022-06-21T11:04:39Z API Version: snapshot.storage.k8s.io/v1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:boundVolumeSnapshotContentName: f:creationTime: f:readyToUse: f:restoreSize: Manager: snapshot-controller Operation: Update Subresource: status Time: 2022-06-21T11:04:44Z Resource Version: 70668 UID: 77dc9462-3910-4a76-b004-1c45f628d140 Spec: Source: Persistent Volume Claim Name: pvc-1 Volume Snapshot Class Name: cinderpvc-snapclass Status: Bound Volume Snapshot Content Name: snapcontent-77dc9462-3910-4a76-b004-1c45f628d140 Creation Time: 2022-06-21T11:04:42Z Ready To Use: true Restore Size: 1Gi Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal CreatingSnapshot 2m29s snapshot-controller Waiting for a snapshot cinderpvc/pvc-1-snap to be created by the CSI driver. Normal SnapshotCreated 2m24s snapshot-controller Snapshot cinderpvc/pvc-1-snap was successfully created by the CSI driver. Normal SnapshotReady 2m24s snapshot-controller Snapshot cinderpvc/pvc-1-snap is ready to use. > Add data to the original pvc-1 volume: $ oc -n cinderpvc rsh demo-1-d9b4c79b6-f58lm /bin/sh /home/kuryr $ cat /var/lib/www/data/test.txt “hello from pvc-5c1fc842-ae70-4397-a4f9-7a0130739444 volume” /home/kuryr $ echo “new data after pvc-1-snap creation” >> /var/lib/www/data/test.txt /home/kuryr $ cat /var/lib/www/data/test.txt “hello from pvc-5c1fc842-ae70-4397-a4f9-7a0130739444 volume” “new data after pvc-1-snap creation” /home/kuryr $ > Create a new pvc from pvc-1-snap snapshot: $ cat <<EOF | oc apply -f - --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-1-restore namespace: cinderpvc spec: #storageClassName: (default) dataSource: name: pvc-1-snap kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io accessModes: - ReadWriteOnce resources: requests: storage: 1Gi EOF $ oc -n cinderpvc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-1 Bound pvc-5c1fc842-ae70-4397-a4f9-7a0130739444 1Gi RWO standard-csi 13m pvc-1-restore Pending standard-csi 16s $ openstack volume list +--------------------------------------+------------------------------------------+--------+------+------------------------------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+------------------------------------------+--------+------+------------------------------------------------------+ | 2134abf0-9b56-4c81-9b4c-8fb75e037b81 | pvc-5c1fc842-ae70-4397-a4f9-7a0130739444 | in-use | 1 | Attached to ostest-b5mzn-worker-0-2hhl5 on /dev/vdb | +--------------------------------------+------------------------------------------+--------+------+------------------------------------------------------+ > Attach to new pvc pvc-1-restore to the deployment, and dettach the previous one pvc-1: $ oc -n cinderpvc edit deployment.apps/demo-1 Replace: volumes: - name: mydata persistentVolumeClaim: claimName: pvc-1 by: volumes: - name: mydata persistentVolumeClaim: claimName: pvc-1-restore $ oc -n cinderpvc get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES demo-1-d5bfbf74f-8xlwd 1/1 Running 0 2m14s 10.128.2.29 ostest-b5mzn-worker-0-gpcz7 <none> <none> $ oc -n cinderpvc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-1 Bound pvc-5c1fc842-ae70-4397-a4f9-7a0130739444 1Gi RWO standard-csi 17m pvc-1-restore Bound pvc-376c25ed-c9f5-4655-801a-ef1cc155e945 1Gi RWO standard-csi 4m20s $ openstack volume list +--------------------------------------+------------------------------------------+-----------+------+------------------------------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+------------------------------------------+-----------+------+------------------------------------------------------+ | eeda7488-1906-47a8-85f4-8b9211cb3d2e | pvc-376c25ed-c9f5-4655-801a-ef1cc155e945 | in-use | 1 | Attached to ostest-b5mzn-worker-0-gpcz7 on /dev/vdb | | 2134abf0-9b56-4c81-9b4c-8fb75e037b81 | pvc-5c1fc842-ae70-4397-a4f9-7a0130739444 | available | 1 | | +--------------------------------------+------------------------------------------+-----------+------+------------------------------------------------------+ > Check the content (original content from pvc-1 volume): $ oc -n cinderpvc rsh demo-1-d5bfbf74f-8xlwd /bin/sh /home/kuryr $ cat /var/lib/www/data/test.txt “hello from pvc-5c1fc842-ae70-4397-a4f9-7a0130739444 volume” > Additional data: $ openstack volume snapshot list +--------------------------------------+-----------------------------------------------+----------------------------------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+-----------------------------------------------+----------------------------------------+-----------+------+ | bb87eb75-66f0-4390-836b-6f298639825f | snapshot-77dc9462-3910-4a76-b004-1c45f628d140 | Created by OpenStack Cinder CSI driver | available | 1 | +--------------------------------------+-----------------------------------------------+----------------------------------------+-----------+------+ $ openstack volume snapshot show snapshot-77dc9462-3910-4a76-b004-1c45f628d140 +--------------------------------------------+-------------------------------------------------+ | Field | Value | +--------------------------------------------+-------------------------------------------------+ | created_at | 2022-06-21T11:04:42.000000 | | description | Created by OpenStack Cinder CSI driver | | id | bb87eb75-66f0-4390-836b-6f298639825f | | name | snapshot-77dc9462-3910-4a76-b004-1c45f628d140 | | os-extended-snapshot-attributes:progress | 100% | | os-extended-snapshot-attributes:project_id | 42a001f17da442e9a0e894a1a7052603 | | properties | cinder.csi.openstack.org/cluster='ostest-b5mzn' | | size | 1 | | status | available | | updated_at | 2022-06-21T11:04:43.000000 | | volume_id | 2134abf0-9b56-4c81-9b4c-8fb75e037b81 | +--------------------------------------------+-------------------------------------------------+ $ openstack volume show pvc-376c25ed-c9f5-4655-801a-ef1cc155e945 -f value -c properties {'cinder.csi.openstack.org/cluster': 'ostest-b5mzn'} $ openstack volume show pvc-5c1fc842-ae70-4397-a4f9-7a0130739444 -f value -c properties {'cinder.csi.openstack.org/cluster': 'ostest-b5mzn'} $ openstack volume show eeda7488-1906-47a8-85f4-8b9211cb3d2e +------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | attachments | [{'id': 'eeda7488-1906-47a8-85f4-8b9211cb3d2e', 'attachment_id': '12a4f496-a9c6-4ed6-aa5e-7116d92db0de', 'volume_id': 'eeda7488-1906-47a8-85f4-8b9211cb3d2e', 'server_id': '4c51eda8-c9a3-4c75-bc8a-c8b02e5410f6', 'host_name': 'compute-1.redhat.local', 'device': '/dev/vdb', 'attached_at': '2022-06-21T11:15:23.000000'}] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2022-06-21T11:15:17.000000 | | description | Created by OpenStack Cinder CSI driver | | encrypted | False | | id | eeda7488-1906-47a8-85f4-8b9211cb3d2e | | multiattach | False | | name | pvc-376c25ed-c9f5-4655-801a-ef1cc155e945 | | os-vol-tenant-attr:tenant_id | 42a001f17da442e9a0e894a1a7052603 | | properties | cinder.csi.openstack.org/cluster='ostest-b5mzn' | | replication_status | None | | size | 1 | | snapshot_id | bb87eb75-66f0-4390-836b-6f298639825f | | source_volid | None | | status | in-use | | type | tripleo | | updated_at | 2022-06-21T11:15:24.000000 | | user_id | 550df7bf64db4a20850d477c9f889a43 | +------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ $ openstack volume show 2134abf0-9b56-4c81-9b4c-8fb75e037b81 +------------------------------+-------------------------------------------------+ | Field | Value | +------------------------------+-------------------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | consistencygroup_id | None | | created_at | 2022-06-21T10:57:31.000000 | | description | Created by OpenStack Cinder CSI driver | | encrypted | False | | id | 2134abf0-9b56-4c81-9b4c-8fb75e037b81 | | multiattach | False | | name | pvc-5c1fc842-ae70-4397-a4f9-7a0130739444 | | os-vol-tenant-attr:tenant_id | 42a001f17da442e9a0e894a1a7052603 | | properties | cinder.csi.openstack.org/cluster='ostest-b5mzn' | | replication_status | None | | size | 1 | | snapshot_id | None | | source_volid | None | | status | available | | type | tripleo | | updated_at | 2022-06-21T11:15:40.000000 | | user_id | 550df7bf64db4a20850d477c9f889a43 | +------------------------------+-------------------------------------------------+ > Destroy the cluster and check the volumes and snapshot were deleted: [...] time="2022-06-21T07:32:20-04:00" level=debug msg="Deleting volume snapshot \"bb87eb75-66f0-4390-836b-6f298639825f\"" time="2022-06-21T07:32:21-04:00" level=debug msg="Deleting volume \"eeda7488-1906-47a8-85f4-8b9211cb3d2e\"" time="2022-06-21T07:32:21-04:00" level=debug msg="Deleting volume \"2134abf0-9b56-4c81-9b4c-8fb75e037b81\"" [...] $ openstack volume snapshot list $ $ openstack volume list $
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069