Bug 2223361
| Summary: | PVCs prime are not cleaned up and stay in the 'Lost ' Phase | ||
|---|---|---|---|
| Product: | Container Native Virtualization (CNV) | Reporter: | Jenia Peimer <jpeimer> |
| Component: | Storage | Assignee: | Álvaro Romero <alromero> |
| Status: | VERIFIED --- | QA Contact: | Jenia Peimer <jpeimer> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 4.14.0 | CC: | alitke, alromero, mperetz |
| Target Milestone: | --- | ||
| Target Release: | 4.14.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | v4.14.0.rhel9-1557 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | Type: | Bug | |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
It can be reproduced with no regard to the boot source images
And can also be reproduced on 'ocs-storagecluster-ceph-rbd' storage
When I created 9 DVs - 4 PVC primes stayed 'Lost':
$ oc get dv
dv-source-ocs-1 Succeeded 100.0% 5m9s
dv-source-ocs-2 Succeeded 100.0% 4m52s
dv-source-ocs-3 Succeeded 100.0% 4m43s
dv-source-ocs-4 Succeeded 100.0% 4m21s
dv-source-ocs-5 Succeeded 100.0% 4m15s
dv-source-ocs-6 Succeeded 100.0% 4m5s
dv-source-ocs-7 Succeeded 100.0% 3m58s
dv-source-ocs-8 Succeeded 100.0% 3m12s
dv-source-ocs-9 Succeeded 100.0% 2m57s
$ oc get pvc
dv-source-ocs-1 Bound pvc-52266e25-8ce8-41f4-b0c2-2436512cdc37 1Gi RWX ocs-storagecluster-ceph-rbd 5m12s
dv-source-ocs-2 Bound pvc-22beca5e-7ac2-4948-994e-04eef4460eb3 1Gi RWX ocs-storagecluster-ceph-rbd 4m55s
dv-source-ocs-3 Bound pvc-17d7b3b3-d650-48a4-b795-07a5127aba10 1Gi RWX ocs-storagecluster-ceph-rbd 4m46s
dv-source-ocs-4 Bound pvc-0802f98e-a6b7-4020-9176-c000b121b6e6 1Gi RWX ocs-storagecluster-ceph-rbd 4m24s
dv-source-ocs-5 Bound pvc-190bfade-bdda-4eed-88ac-1673b08d8616 1Gi RWX ocs-storagecluster-ceph-rbd 4m18s
dv-source-ocs-6 Bound pvc-020c6236-d252-4eb4-a7e0-8f8985556545 1Gi RWX ocs-storagecluster-ceph-rbd 4m8s
dv-source-ocs-7 Bound pvc-3019fbfe-5a18-401d-be0f-75c67ec41cb0 1Gi RWX ocs-storagecluster-ceph-rbd 4m1s
dv-source-ocs-8 Bound pvc-e1ee58c2-9d86-4f60-9671-9b378b0fd14a 1Gi RWX ocs-storagecluster-ceph-rbd 3m15s
dv-source-ocs-9 Bound pvc-e9493805-5227-427a-96bf-33d68d8e07a9 1Gi RWX ocs-storagecluster-ceph-rbd 3m
prime-08b1ee81-2df8-473a-b188-a47cc46d06b5 Lost pvc-e1ee58c2-9d86-4f60-9671-9b378b0fd14a 0 ocs-storagecluster-ceph-rbd 3m15s
prime-54464bb9-1e13-4c4c-ae31-6ec5fb8d4e86 Lost pvc-0802f98e-a6b7-4020-9176-c000b121b6e6 0 ocs-storagecluster-ceph-rbd 4m24s
prime-59f176b5-1b6b-460c-ab44-497ab4796872 Lost pvc-190bfade-bdda-4eed-88ac-1673b08d8616 0 ocs-storagecluster-ceph-rbd 4m18s
prime-cbb46102-a47b-4f4c-ad2f-8180bafea0cd Lost pvc-3019fbfe-5a18-401d-be0f-75c67ec41cb0 0 ocs-storagecluster-ceph-rbd 3m41s
$ cat dv-ocs.yaml
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
name: dv-source-ocs-9
spec:
source:
http:
url: "http://<link>/cirros-0.4.0-x86_64-disk.qcow2"
storage:
resources:
requests:
storage: 1Gi
storageClassName: ocs-storagecluster-ceph-rbd
This is also seen with Upload:
[cloud-user@ocp-psi-executor ~]$ oc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cirros-dv-ocs Bound pvc-71ff3390-0298-400f-b3a7-61d2f0fa0d70 1Gi RWX ocs-storagecluster-ceph-rbd 33s
cirros-dv-ocs-2 Bound pvc-c2f4a4f2-13ea-4cdf-83f2-c3d4b6156fd6 1Gi RWX ocs-storagecluster-ceph-rbd 30s
prime-6842bddb-3144-4108-8258-de45c4da49a7 Lost pvc-c2f4a4f2-13ea-4cdf-83f2-c3d4b6156fd6 0 ocs-storagecluster-ceph-rbd 30s
[cloud-user@ocp-psi-executor ~]$
[cloud-user@ocp-psi-executor ~]$ oc get pvc prime-6842bddb-3144-4108-8258-de45c4da49a7 -oyaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
cdi.kubevirt.io/storage.bind.immediate.requested: ""
cdi.kubevirt.io/storage.condition.bound: "true"
cdi.kubevirt.io/storage.condition.bound.message: ""
cdi.kubevirt.io/storage.condition.bound.reason: ""
cdi.kubevirt.io/storage.condition.running: "false"
cdi.kubevirt.io/storage.condition.running.message: Upload Complete
cdi.kubevirt.io/storage.condition.running.reason: Completed
cdi.kubevirt.io/storage.contentType: kubevirt
cdi.kubevirt.io/storage.pod.phase: Succeeded
cdi.kubevirt.io/storage.pod.ready: "false"
cdi.kubevirt.io/storage.pod.restarts: "0"
cdi.kubevirt.io/storage.populator.kind: VolumeUploadSource
cdi.kubevirt.io/storage.preallocation.requested: "false"
cdi.kubevirt.io/storage.upload.target: ""
cdi.kubevirt.io/storage.uploadPodName: cdi-upload-prime-6842bddb-3144-4108-8258-de45c4da49a7
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
sidecar.istio.io/inject: "false"
volume.beta.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
volume.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
creationTimestamp: "2023-07-26T13:04:13Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
app: containerized-data-importer
app.kubernetes.io/component: storage
app.kubernetes.io/managed-by: cdi-controller
app.kubernetes.io/part-of: hyperconverged-cluster
app.kubernetes.io/version: 4.14.0
name: prime-6842bddb-3144-4108-8258-de45c4da49a7
namespace: default
ownerReferences:
- apiVersion: v1
blockOwnerDeletion: true
controller: true
kind: PersistentVolumeClaim
name: cirros-dv-ocs-2
uid: 6842bddb-3144-4108-8258-de45c4da49a7
resourceVersion: "13655925"
uid: c2f4a4f2-13ea-4cdf-83f2-c3d4b6156fd6
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: "1073741824"
storageClassName: ocs-storagecluster-ceph-rbd
volumeMode: Block
volumeName: pvc-c2f4a4f2-13ea-4cdf-83f2-c3d4b6156fd6
status:
phase: Lost
And here's the PVC 'cirros-dv-ocs-2', whose prime got lost:
[cloud-user@ocp-psi-executor jenia]$ oc get pvc cirros-dv-ocs-2 -oyaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
cdi.kubevirt.io/storage.condition.running: "false"
cdi.kubevirt.io/storage.condition.running.message: Upload Complete
cdi.kubevirt.io/storage.condition.running.reason: Completed
cdi.kubevirt.io/storage.contentType: kubevirt
cdi.kubevirt.io/storage.pod.phase: Succeeded
cdi.kubevirt.io/storage.pod.ready: "false"
cdi.kubevirt.io/storage.pod.restarts: "0"
cdi.kubevirt.io/storage.preallocation.requested: "false"
cdi.kubevirt.io/storage.usePopulator: "true"
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
volume.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
creationTimestamp: "2023-07-26T13:04:13Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
app: containerized-data-importer
app.kubernetes.io/component: storage
app.kubernetes.io/managed-by: cdi-controller
app.kubernetes.io/part-of: hyperconverged-cluster
app.kubernetes.io/version: 4.14.0
name: cirros-dv-ocs-2
namespace: default
ownerReferences:
- apiVersion: cdi.kubevirt.io/v1beta1
blockOwnerDeletion: true
controller: true
kind: DataVolume
name: cirros-dv-ocs-2
uid: 8c40f20a-a96f-41c0-a4ed-4cba490da76b
resourceVersion: "13655753"
uid: 6842bddb-3144-4108-8258-de45c4da49a7
spec:
accessModes:
- ReadWriteMany
dataSource:
apiGroup: cdi.kubevirt.io
kind: VolumeUploadSource
name: volume-upload-source-8c40f20a-a96f-41c0-a4ed-4cba490da76b
dataSourceRef:
apiGroup: cdi.kubevirt.io
kind: VolumeUploadSource
name: volume-upload-source-8c40f20a-a96f-41c0-a4ed-4cba490da76b
resources:
requests:
storage: "1073741824"
storageClassName: ocs-storagecluster-ceph-rbd
volumeMode: Block
volumeName: pvc-c2f4a4f2-13ea-4cdf-83f2-c3d4b6156fd6
status:
accessModes:
- ReadWriteMany
capacity:
storage: 1Gi
phase: Bound
Verified with import and upload on CNV v4.14.0.rhel9-1576 |
Description of problem: PVCs prime in 'Lost' phase in openshift-virtualization-os-images namespace Version-Release number of selected component (if applicable): CNV v4.14.0.rhel9-1259 How reproducible: Always Steps to Reproduce: 0. Install CNV, set 'hostpath-csi-basic' storage as default (CSI, RWO, FS), wait for boot source images to import 1. $ oc get pvc -A NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE openshift-cnv hpp-pool-hpp-csi-pvc-block-c01-jp414-qfjfs-worker-0-f6qmc Bound local-pv-fc7f621c 70Gi RWO local-block-hpp 31m openshift-cnv hpp-pool-hpp-csi-pvc-block-c01-jp414-qfjfs-worker-0-tw5x9 Bound local-pv-26adfea1 70Gi RWO local-block-hpp 31m openshift-cnv hpp-pool-hpp-csi-pvc-block-c01-jp414-qfjfs-worker-0-w2m6l Bound local-pv-33924ef7 70Gi RWO local-block-hpp 31m openshift-storage ocs-deviceset-0-data-052fbd Bound local-pv-7e381611 70Gi RWO local-block-ocs 38m openshift-storage ocs-deviceset-0-data-1cv2lb Bound local-pv-5ab0734f 70Gi RWO local-block-ocs 38m openshift-storage ocs-deviceset-0-data-2rb2bj Bound local-pv-4b950316 70Gi RWO local-block-ocs 38m openshift-virtualization-os-images centos-stream8-b9b768dcd73b Bound pvc-2656c6b9-fafd-4ca5-bfd0-d6dd5a3cec5e 149Gi RWO hostpath-csi-basic 30m openshift-virtualization-os-images centos-stream9-49f10670e687 Bound pvc-e15c2615-ea72-4ff2-9da4-5251a25390da 149Gi RWO hostpath-csi-basic 30m openshift-virtualization-os-images centos7-680e9b4e0fba Bound pvc-fd6804cf-4bd6-496d-85c9-ec3976371cae 149Gi RWO hostpath-csi-basic 30m openshift-virtualization-os-images fedora-f7cc15256f08 Bound pvc-11529545-a22b-4f74-b17b-019fe34a8f53 149Gi RWO hostpath-csi-basic 30m openshift-virtualization-os-images prime-4f58de2b-e099-439f-9d99-d1ebe4dfedf7 Lost pvc-e15c2615-ea72-4ff2-9da4-5251a25390da 0 hostpath-csi-basic 30m openshift-virtualization-os-images prime-915f7304-1f52-4fa5-9eca-4fdaff8beba9 Lost pvc-fd6804cf-4bd6-496d-85c9-ec3976371cae 0 hostpath-csi-basic 30m openshift-virtualization-os-images prime-b3c897af-46da-4385-8a39-edec802e99d6 Lost pvc-c630f451-1d7c-41b7-a080-04a42d2de8c1 0 hostpath-csi-basic 30m openshift-virtualization-os-images rhel8-0da894200daa Bound pvc-eac206e9-6793-4bc1-bcdb-acd5ccaa6ffd 149Gi RWO hostpath-csi-basic 30m openshift-virtualization-os-images rhel9-b006ef7856b6 Bound pvc-c630f451-1d7c-41b7-a080-04a42d2de8c1 149Gi RWO hostpath-csi-basic 30m 2. $ oc get dv -A NAMESPACE NAME PHASE PROGRESS RESTARTS AGE openshift-virtualization-os-images centos-stream8-b9b768dcd73b Succeeded 100.0% 30m openshift-virtualization-os-images centos-stream9-49f10670e687 Succeeded 100.0% 30m openshift-virtualization-os-images centos7-680e9b4e0fba Succeeded 100.0% 30m openshift-virtualization-os-images fedora-f7cc15256f08 Succeeded 100.0% 30m openshift-virtualization-os-images rhel8-0da894200daa Succeeded 100.0% 30m openshift-virtualization-os-images rhel9-b006ef7856b6 Succeeded 100.0% 30m Actual results: 3 'Lost' prime PVCs openshift-virtualization-os-images prime-4f58de2b-e099-439f-9d99-d1ebe4dfedf7 Lost pvc-e15c2615-ea72-4ff2-9da4-5251a25390da 0 hostpath-csi-basic 30m openshift-virtualization-os-images prime-915f7304-1f52-4fa5-9eca-4fdaff8beba9 Lost pvc-fd6804cf-4bd6-496d-85c9-ec3976371cae 0 hostpath-csi-basic 30m openshift-virtualization-os-images prime-b3c897af-46da-4385-8a39-edec802e99d6 Lost pvc-c630f451-1d7c-41b7-a080-04a42d2de8c1 0 hostpath-csi-basic 30m Expected results: No prime PVCs Additional info: [cloud-user@ocp-psi-executor jenia]$ oc get pvc -n openshift-virtualization-os-images prime-4f58de2b-e099-439f-9d99-d1ebe4dfedf7 -oyaml apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: cdi.kubevirt.io/storage.bind.immediate.requested: "" cdi.kubevirt.io/storage.condition.running: "false" cdi.kubevirt.io/storage.condition.running.message: Import Complete cdi.kubevirt.io/storage.condition.running.reason: Completed cdi.kubevirt.io/storage.contentType: kubevirt cdi.kubevirt.io/storage.import.endpoint: docker://quay.io/containerdisks/centos-stream@sha256:49f10670e687e20b29f0dbc30e23bb784942e58573cf4d62017dbeabca5eb02f cdi.kubevirt.io/storage.import.importPodName: importer-prime-4f58de2b-e099-439f-9d99-d1ebe4dfedf7 cdi.kubevirt.io/storage.import.registryImportMethod: node cdi.kubevirt.io/storage.import.source: registry cdi.kubevirt.io/storage.pod.phase: Succeeded cdi.kubevirt.io/storage.pod.restarts: "0" cdi.kubevirt.io/storage.populator.kind: VolumeImportSource cdi.kubevirt.io/storage.preallocation.requested: "false" pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" sidecar.istio.io/inject: "false" volume.beta.kubernetes.io/storage-provisioner: kubevirt.io.hostpath-provisioner volume.kubernetes.io/selected-node: c01-jp414-qfjfs-worker-0-f6qmc volume.kubernetes.io/storage-provisioner: kubevirt.io.hostpath-provisioner creationTimestamp: "2023-07-17T12:44:13Z" finalizers: - kubernetes.io/pvc-protection labels: app: containerized-data-importer app.kubernetes.io/component: storage app.kubernetes.io/managed-by: cdi-controller app.kubernetes.io/part-of: hyperconverged-cluster app.kubernetes.io/version: 4.14.0 name: prime-4f58de2b-e099-439f-9d99-d1ebe4dfedf7 namespace: openshift-virtualization-os-images ownerReferences: - apiVersion: v1 blockOwnerDeletion: true controller: true kind: PersistentVolumeClaim name: centos-stream9-49f10670e687 uid: 4f58de2b-e099-439f-9d99-d1ebe4dfedf7 resourceVersion: "75947" uid: e15c2615-ea72-4ff2-9da4-5251a25390da spec: accessModes: - ReadWriteOnce resources: requests: storage: "34087042032" storageClassName: hostpath-csi-basic volumeMode: Filesystem volumeName: pvc-e15c2615-ea72-4ff2-9da4-5251a25390da status: phase: Lost