Bug 1673683
| Summary: | CDI Importer logic prevents the deletion of PVCs | ||
|---|---|---|---|
| Product: | Container Native Virtualization (CNV) | Reporter: | Natalie Gavrielov <ngavrilo> |
| Component: | Storage | Assignee: | John Griffith <jgriffith> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Natalie Gavrielov <ngavrilo> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 1.4 | CC: | alitke, cnv-qe-bugs, fsimonce, igoihman, jgriffith, ncredi, qixuan.wang, vparekh, ycui |
| Target Milestone: | --- | ||
| Target Release: | 2.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | 1.5 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-07-29 07:24:09 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
We should fix this by having the CDI controller delete any importer/cloner/uploader pods when a PVC is deleted. As a workaround, the pod can be manually deleted. *** Bug 1669116 has been marked as a duplicate of this bug. *** Should we document this for 1.4? If so can you link here the relevant documentation BZ? Created doc bug here: https://bugzilla.redhat.com/show_bug.cgi?id=1678412 A change was added in 1.5 that detects the PVC delete and kills the pod retry. It's possible that we're hitting another corner case (or introduced a new one) but it was verified with DataVolumes and Import PVCs here (https://github.com/kubevirt/containerized-data-importer/commit/30c0e1b8140103502710985ffe308ee54b1155fc). Still need the docs update for prior versions, but just noting that change merged with a functional test. Just noticed a fix was targeted for 2.0; I think 2.0 target is to rework the retry strategy entirely. Tested with CDI v2.0.0-5, the bug has been fixed. [root@dell-r640-005 ~]# oc delete pvc pvc-invalid-url persistentvolumeclaim "pvc-invalid-url" deleted [root@dell-r640-005 ~]# oc get pvc No resources found. Didn't verify on 1.4 |
Description of problem: When creating a PVC with a wrong URL (endpoint), the PVC cannot be deleted because the importer pod keeps retrying. Version-Release number of selected component: virt-cdi-importer:v1.4.0 virt-cdi-cloner:v1.4.0 virt-cdi-uploadserver:v1.4.0 virt-cdi-controller:v1.4.0-6 How reproducible: 100% Steps to Reproduce: 1. Crate a PVC with an invalid http endpoint. 2. Wait until importer pod is in retrying. 3. Delete the PVC. Actual results: 1. Delete operation does not finish (CLI is stuck waiting for a response). 2. PVC is stuck in Terminating. 3. Importer pod continues retrying. Expected results: 1. For the importer pod to stop retrying, and terminate. 2. For the PVC to delete. Additional info: Notes: 1. Simple (yet not very elegant) workaround is to delete the pod (after trying to delete the PVC). 2. This is not a regression bug. The yaml used: --- apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: cdi.kubevirt.io/storage.contentType: kubevirt cdi.kubevirt.io/storage.import.endpoint: "https://distro.ibiblio.org/tinycorelinux/10.x/x86/release/TinyCore-current-wrong.iso" cdi.kubevirt.io/storage.import.secretName: "" cdi.kubevirt.io/storage.import.source: http name: pvc-with-wrong-url-gluster-1 spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi selector: ~ storageClassName: glusterfs-storage Output: $ oc create -f test-pvc.yaml persistentvolumeclaim/pvc-with-wrong-url-gluster-1 created $ oc get pods -w NAME READY STATUS RESTARTS AGE cdi-upload-upload-pvc 1/1 Running 0 3d cdi-upload-upload-test 1/1 Running 0 3d docker-registry-1-zrbss 1/1 Running 0 11d importer-pvc-with-wrong-url-gluster-1-q4wn8 0/1 Pending 0 7s local-volume-provisioner-7xrj9 1/1 Running 0 9d local-volume-provisioner-p6f2k 1/1 Running 0 9d local-volume-provisioner-znfhs 1/1 Running 0 9d registry-console-1-gkscv 1/1 Running 0 11d router-1-hsp6f 1/1 Running 0 11d virt-launcher-vmi-fedora-47b5v 1/1 Running 0 36m importer-pvc-with-wrong-url-gluster-1-q4wn8 0/1 Pending 0 11s importer-pvc-with-wrong-url-gluster-1-q4wn8 0/1 ContainerCreating 0 11s importer-pvc-with-wrong-url-gluster-1-q4wn8 0/1 ContainerCreating 0 19s importer-pvc-with-wrong-url-gluster-1-q4wn8 1/1 Running 0 19s importer-pvc-with-wrong-url-gluster-1-q4wn8 0/1 Error 0 20s importer-pvc-with-wrong-url-gluster-1-q4wn8 0/1 Error 1 21s importer-pvc-with-wrong-url-gluster-1-q4wn8 0/1 CrashLoopBackOff 1 22s importer-pvc-with-wrong-url-gluster-1-q4wn8 1/1 Running 2 36s importer-pvc-with-wrong-url-gluster-1-q4wn8 0/1 Error 2 37s importer-pvc-with-wrong-url-gluster-1-q4wn8 0/1 CrashLoopBackOff 2 50s importer-pvc-with-wrong-url-gluster-1-q4wn8 1/1 Running 3 1m importer-pvc-with-wrong-url-gluster-1-q4wn8 0/1 Error 3 1m $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE dv-with-secret-httpd-fedora-15-gluster Bound pvc-82922563-2afe-11e9-bea2-fa163e625adc 15Gi RWO glusterfs-storage 57m local-pvc-2 Bound local-pv-2 1Gi RWO manual 2d pvc-on-hdd-for-vmi-rhel Bound local-pv-c204d9ce 6521Mi RWO hdd 1d pvc-with-wrong-url-gluster-1 Bound pvc-55157405-2b06-11e9-bea2-fa163e625adc 1Gi RWO glusterfs-storage 1m upload-pvc Bound pvc-8a361be8-2881-11e9-bea2-fa163e625adc 1Gi RWO glusterfs-storage 3d upload-test Bound pvc-739007b6-2884-11e9-bea2-fa163e625adc 1Gi RWO glusterfs-storage 3d $ oc delete pvc pvc-with-wrong-url-gluster-1 persistentvolumeclaim "pvc-with-wrong-url-gluster-1" deleted ^C $ oc get pods NAME READY STATUS RESTARTS AGE cdi-upload-upload-pvc 1/1 Running 0 3d cdi-upload-upload-test 1/1 Running 0 3d docker-registry-1-zrbss 1/1 Running 0 11d importer-pvc-with-wrong-url-gluster-1-q4wn8 0/1 CrashLoopBackOff 4 2m local-volume-provisioner-7xrj9 1/1 Running 0 9d local-volume-provisioner-p6f2k 1/1 Running 0 9d local-volume-provisioner-znfhs 1/1 Running 0 9d registry-console-1-gkscv 1/1 Running 0 11d router-1-hsp6f 1/1 Running 0 11d virt-launcher-vmi-fedora-47b5v 1/1 Running 0 38m $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE dv-with-secret-httpd-fedora-15-gluster Bound pvc-82922563-2afe-11e9-bea2-fa163e625adc 15Gi RWO glusterfs-storage 58m local-pvc-2 Bound local-pv-2 1Gi RWO manual 2d pvc-on-hdd-for-vmi-rhel Bound local-pv-c204d9ce 6521Mi RWO hdd 1d pvc-with-wrong-url-gluster-1 Terminating pvc-55157405-2b06-11e9-bea2-fa163e625adc 1Gi RWO glusterfs-storage 2m upload-pvc Bound pvc-8a361be8-2881-11e9-bea2-fa163e625adc 1Gi RWO glusterfs-storage 3d upload-test Bound pvc-739007b6-2884-11e9-bea2-fa163e625adc 1Gi RWO glusterfs-storage 3d $ oc logs importer-pvc-with-wrong-url-gluster-1-q4wn8 I0207 18:45:57.424118 1 importer.go:45] Starting importer I0207 18:45:57.424319 1 importer.go:58] begin import process I0207 18:45:57.424329 1 importer.go:82] begin import process I0207 18:45:57.424339 1 dataStream.go:293] copying "https://distro.ibiblio.org/tinycorelinux/10.x/x86/release/TinyCore-current-wrong.iso" to "/data/disk.img"... E0207 18:45:57.515523 1 importer.go:85] Get https://distro.ibiblio.org/tinycorelinux/10.x/x86/release/TinyCore-current-wrong.iso: x509: certificate signed by unknown authority HTTP request errored kubevirt.io/containerized-data-importer/pkg/importer.(*DataStream).http /go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:255 kubevirt.io/containerized-data-importer/pkg/importer.(*DataStream).dataStreamSelector /go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:204 kubevirt.io/containerized-data-importer/pkg/importer.(*DataStream).constructReaders /go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:393 kubevirt.io/containerized-data-importer/pkg/importer.newDataStream /go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:163 kubevirt.io/containerized-data-importer/pkg/importer.NewDataStream /go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:125 kubevirt.io/containerized-data-importer/pkg/importer.CopyData /go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:299 main.main /go/src/kubevirt.io/containerized-data-importer/cmd/cdi-importer/importer.go:83 runtime.main /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/proc.go:198 runtime.goexit /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361 could not get data reader unable to construct readers kubevirt.io/containerized-data-importer/pkg/importer.newDataStream /go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:166 kubevirt.io/containerized-data-importer/pkg/importer.NewDataStream /go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:125 kubevirt.io/containerized-data-importer/pkg/importer.CopyData /go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:299 main.main /go/src/kubevirt.io/containerized-data-importer/cmd/cdi-importer/importer.go:83 runtime.main /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/proc.go:198 runtime.goexit /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361 unable to create data stream kubevirt.io/containerized-data-importer/pkg/importer.CopyData /go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:301 main.main /go/src/kubevirt.io/containerized-data-importer/cmd/cdi-importer/importer.go:83 runtime.main /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/proc.go:198 runtime.goexit /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361