2223361 – PVCs prime are not cleaned up and stay in the 'Lost ' Phase

Bug 2223361 - PVCs prime are not cleaned up and stay in the 'Lost ' Phase

Summary: PVCs prime are not cleaned up and stay in the 'Lost ' Phase

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Container Native Virtualization (CNV)
Classification:	Red Hat
Component:	Storage
Sub Component:
Version:	4.14.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	4.14.0
Assignee:	Álvaro Romero
QA Contact:	Jenia Peimer
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2023-07-17 13:28 UTC by Jenia Peimer
Modified:	2023-11-08 14:06 UTC (History)
CC List:	3 users (show)
Fixed In Version:	v4.14.0.rhel9-1557
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2023-11-08 14:05:58 UTC
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	kubevirt containerized-data-importer pull 2821	None	Merged	Fix race condition in populators so we clean up PVC' after population succeeds	2023-08-02 13:09:48 UTC
Github	kubevirt containerized-data-importer pull 2833	None	open	[release-v1.57] Manual cherrypick of #2821	2023-08-02 13:09:39 UTC
Red Hat Issue Tracker	CNV-31071	None	None	None	2023-07-17 13:29:27 UTC
Red Hat Product Errata	RHSA-2023:6817	None	None	None	2023-11-08 14:06:08 UTC

Description Jenia Peimer 2023-07-17 13:28:24 UTC

Description of problem:
PVCs prime in 'Lost' phase in openshift-virtualization-os-images namespace

Version-Release number of selected component (if applicable):
CNV v4.14.0.rhel9-1259

How reproducible:
Always

Steps to Reproduce:
0. Install CNV, set 'hostpath-csi-basic' storage as default (CSI, RWO, FS), wait for boot source images to import

1.  $ oc get pvc -A
NAMESPACE                            NAME                                                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
openshift-cnv                        hpp-pool-hpp-csi-pvc-block-c01-jp414-qfjfs-worker-0-f6qmc   Bound    local-pv-fc7f621c                          70Gi       RWO            local-block-hpp      31m
openshift-cnv                        hpp-pool-hpp-csi-pvc-block-c01-jp414-qfjfs-worker-0-tw5x9   Bound    local-pv-26adfea1                          70Gi       RWO            local-block-hpp      31m
openshift-cnv                        hpp-pool-hpp-csi-pvc-block-c01-jp414-qfjfs-worker-0-w2m6l   Bound    local-pv-33924ef7                          70Gi       RWO            local-block-hpp      31m
openshift-storage                    ocs-deviceset-0-data-052fbd                                 Bound    local-pv-7e381611                          70Gi       RWO            local-block-ocs      38m
openshift-storage                    ocs-deviceset-0-data-1cv2lb                                 Bound    local-pv-5ab0734f                          70Gi       RWO            local-block-ocs      38m
openshift-storage                    ocs-deviceset-0-data-2rb2bj                                 Bound    local-pv-4b950316                          70Gi       RWO            local-block-ocs      38m
openshift-virtualization-os-images   centos-stream8-b9b768dcd73b                                 Bound    pvc-2656c6b9-fafd-4ca5-bfd0-d6dd5a3cec5e   149Gi      RWO            hostpath-csi-basic   30m
openshift-virtualization-os-images   centos-stream9-49f10670e687                                 Bound    pvc-e15c2615-ea72-4ff2-9da4-5251a25390da   149Gi      RWO            hostpath-csi-basic   30m
openshift-virtualization-os-images   centos7-680e9b4e0fba                                        Bound    pvc-fd6804cf-4bd6-496d-85c9-ec3976371cae   149Gi      RWO            hostpath-csi-basic   30m
openshift-virtualization-os-images   fedora-f7cc15256f08                                         Bound    pvc-11529545-a22b-4f74-b17b-019fe34a8f53   149Gi      RWO            hostpath-csi-basic   30m
openshift-virtualization-os-images   prime-4f58de2b-e099-439f-9d99-d1ebe4dfedf7                  Lost     pvc-e15c2615-ea72-4ff2-9da4-5251a25390da   0                         hostpath-csi-basic   30m
openshift-virtualization-os-images   prime-915f7304-1f52-4fa5-9eca-4fdaff8beba9                  Lost     pvc-fd6804cf-4bd6-496d-85c9-ec3976371cae   0                         hostpath-csi-basic   30m
openshift-virtualization-os-images   prime-b3c897af-46da-4385-8a39-edec802e99d6                  Lost     pvc-c630f451-1d7c-41b7-a080-04a42d2de8c1   0                         hostpath-csi-basic   30m
openshift-virtualization-os-images   rhel8-0da894200daa                                          Bound    pvc-eac206e9-6793-4bc1-bcdb-acd5ccaa6ffd   149Gi      RWO            hostpath-csi-basic   30m
openshift-virtualization-os-images   rhel9-b006ef7856b6                                          Bound    pvc-c630f451-1d7c-41b7-a080-04a42d2de8c1   149Gi      RWO            hostpath-csi-basic   30m


2. $ oc get dv -A
NAMESPACE                            NAME                          PHASE       PROGRESS   RESTARTS   AGE
openshift-virtualization-os-images   centos-stream8-b9b768dcd73b   Succeeded   100.0%                30m
openshift-virtualization-os-images   centos-stream9-49f10670e687   Succeeded   100.0%                30m
openshift-virtualization-os-images   centos7-680e9b4e0fba          Succeeded   100.0%                30m
openshift-virtualization-os-images   fedora-f7cc15256f08           Succeeded   100.0%                30m
openshift-virtualization-os-images   rhel8-0da894200daa            Succeeded   100.0%                30m
openshift-virtualization-os-images   rhel9-b006ef7856b6            Succeeded   100.0%                30m


Actual results:
3 'Lost' prime PVCs
openshift-virtualization-os-images   prime-4f58de2b-e099-439f-9d99-d1ebe4dfedf7                  Lost     pvc-e15c2615-ea72-4ff2-9da4-5251a25390da   0                         hostpath-csi-basic   30m
openshift-virtualization-os-images   prime-915f7304-1f52-4fa5-9eca-4fdaff8beba9                  Lost     pvc-fd6804cf-4bd6-496d-85c9-ec3976371cae   0                         hostpath-csi-basic   30m
openshift-virtualization-os-images   prime-b3c897af-46da-4385-8a39-edec802e99d6                  Lost     pvc-c630f451-1d7c-41b7-a080-04a42d2de8c1   0                         hostpath-csi-basic   30m


Expected results:
No prime PVCs


Additional info:

[cloud-user@ocp-psi-executor jenia]$ oc get pvc -n openshift-virtualization-os-images prime-4f58de2b-e099-439f-9d99-d1ebe4dfedf7 -oyaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    cdi.kubevirt.io/storage.bind.immediate.requested: ""
    cdi.kubevirt.io/storage.condition.running: "false"
    cdi.kubevirt.io/storage.condition.running.message: Import Complete
    cdi.kubevirt.io/storage.condition.running.reason: Completed
    cdi.kubevirt.io/storage.contentType: kubevirt
    cdi.kubevirt.io/storage.import.endpoint: docker://quay.io/containerdisks/centos-stream@sha256:49f10670e687e20b29f0dbc30e23bb784942e58573cf4d62017dbeabca5eb02f
    cdi.kubevirt.io/storage.import.importPodName: importer-prime-4f58de2b-e099-439f-9d99-d1ebe4dfedf7
    cdi.kubevirt.io/storage.import.registryImportMethod: node
    cdi.kubevirt.io/storage.import.source: registry
    cdi.kubevirt.io/storage.pod.phase: Succeeded
    cdi.kubevirt.io/storage.pod.restarts: "0"
    cdi.kubevirt.io/storage.populator.kind: VolumeImportSource
    cdi.kubevirt.io/storage.preallocation.requested: "false"
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    sidecar.istio.io/inject: "false"
    volume.beta.kubernetes.io/storage-provisioner: kubevirt.io.hostpath-provisioner
    volume.kubernetes.io/selected-node: c01-jp414-qfjfs-worker-0-f6qmc
    volume.kubernetes.io/storage-provisioner: kubevirt.io.hostpath-provisioner
  creationTimestamp: "2023-07-17T12:44:13Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: containerized-data-importer
    app.kubernetes.io/component: storage
    app.kubernetes.io/managed-by: cdi-controller
    app.kubernetes.io/part-of: hyperconverged-cluster
    app.kubernetes.io/version: 4.14.0
  name: prime-4f58de2b-e099-439f-9d99-d1ebe4dfedf7
  namespace: openshift-virtualization-os-images
  ownerReferences:
  - apiVersion: v1
    blockOwnerDeletion: true
    controller: true
    kind: PersistentVolumeClaim
    name: centos-stream9-49f10670e687
    uid: 4f58de2b-e099-439f-9d99-d1ebe4dfedf7
  resourceVersion: "75947"
  uid: e15c2615-ea72-4ff2-9da4-5251a25390da
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: "34087042032"
  storageClassName: hostpath-csi-basic
  volumeMode: Filesystem
  volumeName: pvc-e15c2615-ea72-4ff2-9da4-5251a25390da
status:
  phase: Lost

Comment 1 Jenia Peimer 2023-07-17 20:27:26 UTC

It can be reproduced with no regard to the boot source images
And can also be reproduced on 'ocs-storagecluster-ceph-rbd' storage
When I created 9 DVs - 4 PVC primes stayed 'Lost':

$ oc get dv
dv-source-ocs-1          Succeeded   100.0%                5m9s
dv-source-ocs-2          Succeeded   100.0%                4m52s
dv-source-ocs-3          Succeeded   100.0%                4m43s
dv-source-ocs-4          Succeeded   100.0%                4m21s
dv-source-ocs-5          Succeeded   100.0%                4m15s
dv-source-ocs-6          Succeeded   100.0%                4m5s
dv-source-ocs-7          Succeeded   100.0%                3m58s
dv-source-ocs-8          Succeeded   100.0%                3m12s
dv-source-ocs-9          Succeeded   100.0%                2m57s


$ oc get pvc
dv-source-ocs-1                              Bound    pvc-52266e25-8ce8-41f4-b0c2-2436512cdc37   1Gi        RWX            ocs-storagecluster-ceph-rbd   5m12s
dv-source-ocs-2                              Bound    pvc-22beca5e-7ac2-4948-994e-04eef4460eb3   1Gi        RWX            ocs-storagecluster-ceph-rbd   4m55s
dv-source-ocs-3                              Bound    pvc-17d7b3b3-d650-48a4-b795-07a5127aba10   1Gi        RWX            ocs-storagecluster-ceph-rbd   4m46s
dv-source-ocs-4                              Bound    pvc-0802f98e-a6b7-4020-9176-c000b121b6e6   1Gi        RWX            ocs-storagecluster-ceph-rbd   4m24s
dv-source-ocs-5                              Bound    pvc-190bfade-bdda-4eed-88ac-1673b08d8616   1Gi        RWX            ocs-storagecluster-ceph-rbd   4m18s
dv-source-ocs-6                              Bound    pvc-020c6236-d252-4eb4-a7e0-8f8985556545   1Gi        RWX            ocs-storagecluster-ceph-rbd   4m8s
dv-source-ocs-7                              Bound    pvc-3019fbfe-5a18-401d-be0f-75c67ec41cb0   1Gi        RWX            ocs-storagecluster-ceph-rbd   4m1s
dv-source-ocs-8                              Bound    pvc-e1ee58c2-9d86-4f60-9671-9b378b0fd14a   1Gi        RWX            ocs-storagecluster-ceph-rbd   3m15s
dv-source-ocs-9                              Bound    pvc-e9493805-5227-427a-96bf-33d68d8e07a9   1Gi        RWX            ocs-storagecluster-ceph-rbd   3m
prime-08b1ee81-2df8-473a-b188-a47cc46d06b5   Lost     pvc-e1ee58c2-9d86-4f60-9671-9b378b0fd14a   0                         ocs-storagecluster-ceph-rbd   3m15s
prime-54464bb9-1e13-4c4c-ae31-6ec5fb8d4e86   Lost     pvc-0802f98e-a6b7-4020-9176-c000b121b6e6   0                         ocs-storagecluster-ceph-rbd   4m24s
prime-59f176b5-1b6b-460c-ab44-497ab4796872   Lost     pvc-190bfade-bdda-4eed-88ac-1673b08d8616   0                         ocs-storagecluster-ceph-rbd   4m18s
prime-cbb46102-a47b-4f4c-ad2f-8180bafea0cd   Lost     pvc-3019fbfe-5a18-401d-be0f-75c67ec41cb0   0                         ocs-storagecluster-ceph-rbd   3m41s


$ cat dv-ocs.yaml 
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: dv-source-ocs-9
spec:
  source:
      http:
         url: "http://<link>/cirros-0.4.0-x86_64-disk.qcow2"
  storage:
    resources:
      requests:
        storage: 1Gi
    storageClassName: ocs-storagecluster-ceph-rbd

Comment 3 Jenia Peimer 2023-07-26 13:08:44 UTC

This is also seen with Upload:

[cloud-user@ocp-psi-executor ~]$ oc get pvc 
NAME                                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
cirros-dv-ocs                                Bound    pvc-71ff3390-0298-400f-b3a7-61d2f0fa0d70   1Gi        RWX            ocs-storagecluster-ceph-rbd   33s
cirros-dv-ocs-2                              Bound    pvc-c2f4a4f2-13ea-4cdf-83f2-c3d4b6156fd6   1Gi        RWX            ocs-storagecluster-ceph-rbd   30s
prime-6842bddb-3144-4108-8258-de45c4da49a7   Lost     pvc-c2f4a4f2-13ea-4cdf-83f2-c3d4b6156fd6   0                         ocs-storagecluster-ceph-rbd   30s
[cloud-user@ocp-psi-executor ~]$ 
[cloud-user@ocp-psi-executor ~]$ oc get pvc prime-6842bddb-3144-4108-8258-de45c4da49a7 -oyaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    cdi.kubevirt.io/storage.bind.immediate.requested: ""
    cdi.kubevirt.io/storage.condition.bound: "true"
    cdi.kubevirt.io/storage.condition.bound.message: ""
    cdi.kubevirt.io/storage.condition.bound.reason: ""
    cdi.kubevirt.io/storage.condition.running: "false"
    cdi.kubevirt.io/storage.condition.running.message: Upload Complete
    cdi.kubevirt.io/storage.condition.running.reason: Completed
    cdi.kubevirt.io/storage.contentType: kubevirt
    cdi.kubevirt.io/storage.pod.phase: Succeeded
    cdi.kubevirt.io/storage.pod.ready: "false"
    cdi.kubevirt.io/storage.pod.restarts: "0"
    cdi.kubevirt.io/storage.populator.kind: VolumeUploadSource
    cdi.kubevirt.io/storage.preallocation.requested: "false"
    cdi.kubevirt.io/storage.upload.target: ""
    cdi.kubevirt.io/storage.uploadPodName: cdi-upload-prime-6842bddb-3144-4108-8258-de45c4da49a7
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    sidecar.istio.io/inject: "false"
    volume.beta.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
    volume.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
  creationTimestamp: "2023-07-26T13:04:13Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: containerized-data-importer
    app.kubernetes.io/component: storage
    app.kubernetes.io/managed-by: cdi-controller
    app.kubernetes.io/part-of: hyperconverged-cluster
    app.kubernetes.io/version: 4.14.0
  name: prime-6842bddb-3144-4108-8258-de45c4da49a7
  namespace: default
  ownerReferences:
  - apiVersion: v1
    blockOwnerDeletion: true
    controller: true
    kind: PersistentVolumeClaim
    name: cirros-dv-ocs-2
    uid: 6842bddb-3144-4108-8258-de45c4da49a7
  resourceVersion: "13655925"
  uid: c2f4a4f2-13ea-4cdf-83f2-c3d4b6156fd6
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: "1073741824"
  storageClassName: ocs-storagecluster-ceph-rbd
  volumeMode: Block
  volumeName: pvc-c2f4a4f2-13ea-4cdf-83f2-c3d4b6156fd6
status:
  phase: Lost

Comment 4 Jenia Peimer 2023-07-26 14:17:38 UTC

And here's the PVC 'cirros-dv-ocs-2', whose prime got lost:

[cloud-user@ocp-psi-executor jenia]$ oc get pvc cirros-dv-ocs-2 -oyaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    cdi.kubevirt.io/storage.condition.running: "false"
    cdi.kubevirt.io/storage.condition.running.message: Upload Complete
    cdi.kubevirt.io/storage.condition.running.reason: Completed
    cdi.kubevirt.io/storage.contentType: kubevirt
    cdi.kubevirt.io/storage.pod.phase: Succeeded
    cdi.kubevirt.io/storage.pod.ready: "false"
    cdi.kubevirt.io/storage.pod.restarts: "0"
    cdi.kubevirt.io/storage.preallocation.requested: "false"
    cdi.kubevirt.io/storage.usePopulator: "true"
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
    volume.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
  creationTimestamp: "2023-07-26T13:04:13Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: containerized-data-importer
    app.kubernetes.io/component: storage
    app.kubernetes.io/managed-by: cdi-controller
    app.kubernetes.io/part-of: hyperconverged-cluster
    app.kubernetes.io/version: 4.14.0
  name: cirros-dv-ocs-2
  namespace: default
  ownerReferences:
  - apiVersion: cdi.kubevirt.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: DataVolume
    name: cirros-dv-ocs-2
    uid: 8c40f20a-a96f-41c0-a4ed-4cba490da76b
  resourceVersion: "13655753"
  uid: 6842bddb-3144-4108-8258-de45c4da49a7
spec:
  accessModes:
  - ReadWriteMany
  dataSource:
    apiGroup: cdi.kubevirt.io
    kind: VolumeUploadSource
    name: volume-upload-source-8c40f20a-a96f-41c0-a4ed-4cba490da76b
  dataSourceRef:
    apiGroup: cdi.kubevirt.io
    kind: VolumeUploadSource
    name: volume-upload-source-8c40f20a-a96f-41c0-a4ed-4cba490da76b
  resources:
    requests:
      storage: "1073741824"
  storageClassName: ocs-storagecluster-ceph-rbd
  volumeMode: Block
  volumeName: pvc-c2f4a4f2-13ea-4cdf-83f2-c3d4b6156fd6
status:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 1Gi
  phase: Bound

Comment 5 Jenia Peimer 2023-08-13 11:36:09 UTC

Verified with import and upload on CNV v4.14.0.rhel9-1576

Comment 7 errata-xmlrpc 2023-11-08 14:05:58 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Virtualization 4.14.0 Images security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6817

Note You need to log in before you can comment on or make changes to this bug.