Bug 1673683 - CDI Importer logic prevents the deletion of PVCs
Summary: CDI Importer logic prevents the deletion of PVCs
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Storage
Version: 1.4
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 2.0
Assignee: John Griffith
QA Contact: Natalie Gavrielov
URL:
Whiteboard:
: 1669116 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-07 18:56 UTC by Natalie Gavrielov
Modified: 2019-07-29 07:24 UTC (History)
9 users (show)

Fixed In Version: 1.5
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-07-29 07:24:09 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github kubevirt containerized-data-importer pull 650 None None None 2019-03-27 20:14:48 UTC

Description Natalie Gavrielov 2019-02-07 18:56:54 UTC
Description of problem:
When creating a PVC with a wrong URL (endpoint), the PVC cannot be deleted because the importer pod keeps retrying.

Version-Release number of selected component:
virt-cdi-importer:v1.4.0
virt-cdi-cloner:v1.4.0
virt-cdi-uploadserver:v1.4.0
virt-cdi-controller:v1.4.0-6

How reproducible: 100%

Steps to Reproduce:
1. Crate a PVC with an invalid http endpoint. 
2. Wait until importer pod is in retrying.
3. Delete the PVC.

Actual results:
1. Delete operation does not finish (CLI is stuck waiting for a response).
2. PVC is stuck in Terminating.
3. Importer pod continues retrying.

Expected results:
1. For the importer pod to stop retrying, and terminate.
2. For the PVC to delete.

Additional info:
Notes: 
1. Simple (yet not very elegant) workaround is to delete the pod (after trying to delete the PVC).
2. This is not a regression bug.

The yaml used:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    cdi.kubevirt.io/storage.contentType: kubevirt
    cdi.kubevirt.io/storage.import.endpoint: "https://distro.ibiblio.org/tinycorelinux/10.x/x86/release/TinyCore-current-wrong.iso"
    cdi.kubevirt.io/storage.import.secretName: ""
    cdi.kubevirt.io/storage.import.source: http
  name: pvc-with-wrong-url-gluster-1
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  selector: ~
  storageClassName: glusterfs-storage

Output:
$ oc create -f test-pvc.yaml
persistentvolumeclaim/pvc-with-wrong-url-gluster-1 created

$ oc get pods -w
NAME                                          READY     STATUS    RESTARTS   AGE
cdi-upload-upload-pvc                         1/1       Running   0          3d
cdi-upload-upload-test                        1/1       Running   0          3d
docker-registry-1-zrbss                       1/1       Running   0          11d
importer-pvc-with-wrong-url-gluster-1-q4wn8   0/1       Pending   0          7s
local-volume-provisioner-7xrj9                1/1       Running   0          9d
local-volume-provisioner-p6f2k                1/1       Running   0          9d
local-volume-provisioner-znfhs                1/1       Running   0          9d
registry-console-1-gkscv                      1/1       Running   0          11d
router-1-hsp6f                                1/1       Running   0          11d
virt-launcher-vmi-fedora-47b5v                1/1       Running   0          36m
importer-pvc-with-wrong-url-gluster-1-q4wn8   0/1       Pending   0         11s
importer-pvc-with-wrong-url-gluster-1-q4wn8   0/1       ContainerCreating   0         11s
importer-pvc-with-wrong-url-gluster-1-q4wn8   0/1       ContainerCreating   0         19s
importer-pvc-with-wrong-url-gluster-1-q4wn8   1/1       Running   0         19s
importer-pvc-with-wrong-url-gluster-1-q4wn8   0/1       Error     0         20s
importer-pvc-with-wrong-url-gluster-1-q4wn8   0/1       Error     1         21s
importer-pvc-with-wrong-url-gluster-1-q4wn8   0/1       CrashLoopBackOff   1         22s
importer-pvc-with-wrong-url-gluster-1-q4wn8   1/1       Running   2         36s
importer-pvc-with-wrong-url-gluster-1-q4wn8   0/1       Error     2         37s
importer-pvc-with-wrong-url-gluster-1-q4wn8   0/1       CrashLoopBackOff   2         50s
importer-pvc-with-wrong-url-gluster-1-q4wn8   1/1       Running   3         1m
importer-pvc-with-wrong-url-gluster-1-q4wn8   0/1       Error     3         1m

$ oc get pvc
NAME                                     STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
dv-with-secret-httpd-fedora-15-gluster   Bound     pvc-82922563-2afe-11e9-bea2-fa163e625adc   15Gi       RWO            glusterfs-storage   57m
local-pvc-2                              Bound     local-pv-2                                 1Gi        RWO            manual              2d
pvc-on-hdd-for-vmi-rhel                  Bound     local-pv-c204d9ce                          6521Mi     RWO            hdd                 1d
pvc-with-wrong-url-gluster-1             Bound     pvc-55157405-2b06-11e9-bea2-fa163e625adc   1Gi        RWO            glusterfs-storage   1m
upload-pvc                               Bound     pvc-8a361be8-2881-11e9-bea2-fa163e625adc   1Gi        RWO            glusterfs-storage   3d
upload-test                              Bound     pvc-739007b6-2884-11e9-bea2-fa163e625adc   1Gi        RWO            glusterfs-storage   3d


$ oc delete pvc pvc-with-wrong-url-gluster-1
persistentvolumeclaim "pvc-with-wrong-url-gluster-1" deleted
^C

$ oc get pods
NAME                                          READY     STATUS             RESTARTS   AGE
cdi-upload-upload-pvc                         1/1       Running            0          3d
cdi-upload-upload-test                        1/1       Running            0          3d
docker-registry-1-zrbss                       1/1       Running            0          11d
importer-pvc-with-wrong-url-gluster-1-q4wn8   0/1       CrashLoopBackOff   4          2m
local-volume-provisioner-7xrj9                1/1       Running            0          9d
local-volume-provisioner-p6f2k                1/1       Running            0          9d
local-volume-provisioner-znfhs                1/1       Running            0          9d
registry-console-1-gkscv                      1/1       Running            0          11d
router-1-hsp6f                                1/1       Running            0          11d
virt-launcher-vmi-fedora-47b5v                1/1       Running            0          38m

$ oc get pvc
NAME                                     STATUS        VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
dv-with-secret-httpd-fedora-15-gluster   Bound         pvc-82922563-2afe-11e9-bea2-fa163e625adc   15Gi       RWO            glusterfs-storage   58m
local-pvc-2                              Bound         local-pv-2                                 1Gi        RWO            manual              2d
pvc-on-hdd-for-vmi-rhel                  Bound         local-pv-c204d9ce                          6521Mi     RWO            hdd                 1d
pvc-with-wrong-url-gluster-1             Terminating   pvc-55157405-2b06-11e9-bea2-fa163e625adc   1Gi        RWO            glusterfs-storage   2m
upload-pvc                               Bound         pvc-8a361be8-2881-11e9-bea2-fa163e625adc   1Gi        RWO            glusterfs-storage   3d
upload-test                              Bound         pvc-739007b6-2884-11e9-bea2-fa163e625adc   1Gi        RWO            glusterfs-storage   3d

$ oc logs importer-pvc-with-wrong-url-gluster-1-q4wn8
I0207 18:45:57.424118       1 importer.go:45] Starting importer
I0207 18:45:57.424319       1 importer.go:58] begin import process
I0207 18:45:57.424329       1 importer.go:82] begin import process
I0207 18:45:57.424339       1 dataStream.go:293] copying "https://distro.ibiblio.org/tinycorelinux/10.x/x86/release/TinyCore-current-wrong.iso" to "/data/disk.img"...
E0207 18:45:57.515523       1 importer.go:85] Get https://distro.ibiblio.org/tinycorelinux/10.x/x86/release/TinyCore-current-wrong.iso: x509: certificate signed by unknown authority
HTTP request errored
kubevirt.io/containerized-data-importer/pkg/importer.(*DataStream).http
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:255
kubevirt.io/containerized-data-importer/pkg/importer.(*DataStream).dataStreamSelector
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:204
kubevirt.io/containerized-data-importer/pkg/importer.(*DataStream).constructReaders
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:393
kubevirt.io/containerized-data-importer/pkg/importer.newDataStream
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:163
kubevirt.io/containerized-data-importer/pkg/importer.NewDataStream
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:125
kubevirt.io/containerized-data-importer/pkg/importer.CopyData
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:299
main.main
	/go/src/kubevirt.io/containerized-data-importer/cmd/cdi-importer/importer.go:83
runtime.main
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/proc.go:198
runtime.goexit
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361
could not get data reader
unable to construct readers
kubevirt.io/containerized-data-importer/pkg/importer.newDataStream
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:166
kubevirt.io/containerized-data-importer/pkg/importer.NewDataStream
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:125
kubevirt.io/containerized-data-importer/pkg/importer.CopyData
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:299
main.main
	/go/src/kubevirt.io/containerized-data-importer/cmd/cdi-importer/importer.go:83
runtime.main
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/proc.go:198
runtime.goexit
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361
unable to create data stream
kubevirt.io/containerized-data-importer/pkg/importer.CopyData
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:301
main.main
	/go/src/kubevirt.io/containerized-data-importer/cmd/cdi-importer/importer.go:83
runtime.main
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/proc.go:198
runtime.goexit
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361

Comment 1 Adam Litke 2019-02-11 15:15:20 UTC
We should fix this by having the CDI controller delete any importer/cloner/uploader pods when a PVC is deleted.  As a workaround, the pod can be manually deleted.

Comment 2 Adam Litke 2019-02-11 15:21:14 UTC
*** Bug 1669116 has been marked as a duplicate of this bug. ***

Comment 3 Federico Simoncelli 2019-02-15 15:15:58 UTC
Should we document this for 1.4? If so can you link here the relevant documentation BZ?

Comment 4 Adam Litke 2019-02-22 20:10:44 UTC
Created doc bug here: https://bugzilla.redhat.com/show_bug.cgi?id=1678412

Comment 6 John Griffith 2019-03-27 20:05:22 UTC
A change was added in 1.5 that detects the PVC delete and kills the pod retry.  It's possible that we're hitting another corner case (or introduced a new one) but it was verified with DataVolumes and Import PVCs here (https://github.com/kubevirt/containerized-data-importer/commit/30c0e1b8140103502710985ffe308ee54b1155fc).  

Still need the docs update for prior versions, but just noting that change merged with a functional test.  Just noticed a fix was targeted for 2.0; I think 2.0 target is to rework the retry strategy entirely.

Comment 7 Qixuan Wang 2019-05-06 11:00:29 UTC
Tested with CDI v2.0.0-5, the bug has been fixed.

[root@dell-r640-005 ~]# oc delete pvc pvc-invalid-url
persistentvolumeclaim "pvc-invalid-url" deleted
[root@dell-r640-005 ~]# oc get pvc
No resources found.

Didn't verify on 1.4


Note You need to log in before you can comment on or make changes to this bug.