Description of problem: Failed to upload a local image via virtctl image-upload tool. PVC is pending with message: persistentvolume-controller waiting for a volume to be created, either by external provisioner "kubevirt.io/hostpath-provisioner" or manually created by system administrator PVC and DV can't get kubevirt.io/provisionOnNode: <node> as an annotation. Hostpath-provisioner is a "node aware" provisioner, how can it take effect in this case? Failed to upload using YAML + token + curl, neither hostpath-provisioner nor other provisioners (local volume and glusterfs) in my 1.4 cluster. So I can't say upload failed only when using hostpath, or when using hostpath, upload works but virtctl upload doesn't. Version-Release number of selected component (if applicable): hostpath-provisioner: v1.4.1-4 CDI v1.4.1-1 How reproducible: 100% Steps to Reproduce: 1. Upload local disk image via virtctl 2. Check PVC Actual results: 1. [cloud-user@cnv-executor-qwang-14-master-b19d80-1 ~]$ virtctl image-upload --image-path=./cirros-0.4.0-x86_64-disk.img --pvc-size=1Gi --storage-class=kubevirt-hostpath-provisioner --uploadproxy-url=https://cdi-uploadproxy-route-cdi.cloudapps.example.com --pvc-name=upload-hp-1 --insecure PVC qwang89-1/upload-hp-1 created Waiting for PVC upload-hp-1 upload pod to be running... timed out waiting for the condition 2. [cloud-user@cnv-executor-qwang-14-master-b19d80-1 ~]$ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE hostpath-cloned-dv Bound pvc-12c0ee20-ba87-11e9-9c00-fa163e34ff55 39Gi RWO kubevirt-hostpath-provisioner 39m upload-g Bound pvc-461bb5c3-ba8c-11e9-9c00-fa163e34ff55 1Gi RWO glusterfs-storage 1m upload-hp-1 Pending kubevirt-hostpath-provisioner 7s [cloud-user@cnv-executor-qwang-14-master-b19d80-1 ~]$ oc describe pvc upload-hp-1 Name: upload-hp-1 Namespace: qwang89-1 StorageClass: kubevirt-hostpath-provisioner Status: Pending Volume: Labels: <none> Annotations: cdi.kubevirt.io/storage.pod.phase=Pending cdi.kubevirt.io/storage.upload.target= volume.beta.kubernetes.io/storage-provisioner=kubevirt.io/hostpath-provisioner Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: VolumeMode: Filesystem Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ExternalProvisioning 1s (x5 over 24s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "kubevirt.io/hostpath-provisioner" or manually created by system administrator Expected results: Upload successfully. Additional info:
Pending on the d/s test environment, so we can not verify this bug now.
The issue is still existing in virt-cdi-operator:v2.2.0-3 $ oc get pod -n openshift-cnv cdi-operator-677b7c4744-xl86c -o yaml | grep image image: registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-virt-cdi-operator:v2.2.0-3 $ oc describe pvc test1-rootdisk Name: test1-rootdisk Namespace: default StorageClass: kubevirt-hostpath-provisioner Status: Pending Volume: Labels: app=containerized-data-importer cdi-controller=test1-rootdisk Annotations: cdi.kubevirt.io/storage.contentType: kubevirt cdi.kubevirt.io/storage.import.endpoint: https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img cdi.kubevirt.io/storage.import.importPodName: importer-test1-rootdisk-zd5kh cdi.kubevirt.io/storage.import.source: http cdi.kubevirt.io/storage.pod.phase: Pending volume.beta.kubernetes.io/storage-provisioner: kubevirt.io/hostpath-provisioner Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: VolumeMode: Block Mounted By: importer-test1-rootdisk-zd5kh Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ExternalProvisioning 10s (x21 over 4m51s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "kubevirt.io/hostpath-provisioner" or manually created by system administrator
Make sure your environment is working correctly before testing stuff against it. According to the PVC, it is still in pending state, so there is a problem with the provisioner and that has nothing to do with CDI. Can you post the virtctl command you used, as well as your storage classes and the pods running in openshift-cnv? Also since it looks like you are trying to use the hostpath provisioner, can you give me the output of "oc get hostpathprovisioner"
1. virtctl image-upload works with --storage-class=hostpath-provisioner. $ virtctl image-upload --uploadproxy-url=https://cdi-uploadproxy-openshift-cnv.apps.working.oc4 --pvc-name=cirros-hpp --pvc-size=2Gi --image-path=./cirros-0.4.0-x86_64-disk.img --insecure --storage-class=hostpath-provisioner PVC default/cirros-hpp created Waiting for PVC cirros-hpp upload pod to be running... Pod now running Uploading data to https://cdi-uploadproxy-openshift-cnv.apps.working.oc4 12.13 MiB / 12.13 MiB [====================================================================================================] 100.00% 2s Uploading ./cirros-0.4.0-x86_64-disk.img completed successfully 2. virtctl image-upload works if the default sc is hostpath-provisioner. $ oc get sc NAME PROVISIONER AGE hostpath-provisioner (default) kubevirt.io/hostpath-provisioner 130m local-sc kubernetes.io/no-provisioner 141m rook-ceph-block rook-ceph.rbd.csi.ceph.com 143m [cnv-qe-jenkins@cnv-executor-guohua ~]$ virtctl image-upload --uploadproxy-url=https://cdi-uploadproxy-openshift-cnv.apps.working.oc4 --pvc-name=cirros-hpp1 --pvc-size=2Gi --image-path=./cirros-0.4.0-x86_64-disk.img --insecure PVC default/cirros-hpp1 created Waiting for PVC cirros-hpp1 upload pod to be running... Pod now running Uploading data to https://cdi-uploadproxy-openshift-cnv.apps.working.oc4 12.13 MiB / 12.13 MiB [====================================================================================================] 100.00% 2s Uploading ./cirros-0.4.0-x86_64-disk.img completed successfully 3. Create a VM from UI by using URL as the provision source: - if the storage class is hostpath-provisioner, the PVC is pending. - if the storage class is rook-ceph-block or local-sc, the PVC and PV can be created and the VM can get into running. $ oc describe pvc test-rootdisk Name: test-rootdisk Namespace: default StorageClass: hostpath-provisioner Status: Pending Volume: Labels: app=containerized-data-importer cdi-controller=test-rootdisk Annotations: cdi.kubevirt.io/storage.contentType: kubevirt cdi.kubevirt.io/storage.import.endpoint: https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img cdi.kubevirt.io/storage.import.importPodName: importer-test-rootdisk-csrlv cdi.kubevirt.io/storage.import.source: http cdi.kubevirt.io/storage.pod.phase: Pending volume.beta.kubernetes.io/storage-provisioner: kubevirt.io/hostpath-provisioner volume.kubernetes.io/selected-node: host-172-16-0-18 Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: VolumeMode: Block Mounted By: importer-test-rootdisk-csrlv Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal WaitForFirstConsumer 4m37s persistentvolume-controller waiting for first consumer to be created before binding Warning ProvisioningFailed 23s (x3 over 4m37s) kubevirt.io/hostpath-provisioner_hostpath-provisioner-khzgs_0a359049-96bf-4510-9890-038024938b67 kubevirt.io/hostpath-provisioner does not support block volume provisioning Normal ExternalProvisioning 3s (x22 over 4m37s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "kubevirt.io/hostpath-provisioner" or manually created by system administrator [cnv-qe-jenkins@cnv-executor-guohua ~]$ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE cirros Bound pvc-eb26f290-fef4-4aac-a95d-1d66556af826 2Gi RWO rook-ceph-block 23m cirros-hpp Bound pvc-c16fd713-d4f9-4347-a1f5-6267f8510230 39Gi RWO hostpath-provisioner 7m56s cirros-hpp1 Bound pvc-7e79f54b-6901-4c6d-9b07-9496fa487e7d 39Gi RWO hostpath-provisioner 5m16s test-rootdisk Pending hostpath-provisioner 4m40s [cnv-qe-jenkins@cnv-executor-guohua ~]$ oc describe pvc test-rootdisk Name: test-rootdisk Namespace: default StorageClass: hostpath-provisioner Status: Pending Volume: Labels: app=containerized-data-importer cdi-controller=test-rootdisk Annotations: cdi.kubevirt.io/storage.contentType: kubevirt cdi.kubevirt.io/storage.import.endpoint: https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img cdi.kubevirt.io/storage.import.importPodName: importer-test-rootdisk-csrlv cdi.kubevirt.io/storage.import.source: http cdi.kubevirt.io/storage.pod.phase: Pending volume.beta.kubernetes.io/storage-provisioner: kubevirt.io/hostpath-provisioner volume.kubernetes.io/selected-node: host-172-16-0-18 Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: VolumeMode: Block Mounted By: importer-test-rootdisk-csrlv Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal WaitForFirstConsumer 4m44s persistentvolume-controller waiting for first consumer to be created before binding Warning ProvisioningFailed 30s (x3 over 4m44s) kubevirt.io/hostpath-provisioner_hostpath-provisioner-khzgs_0a359049-96bf-4510-9890-038024938b67 kubevirt.io/hostpath-provisioner does not support block volume provisioning Normal ExternalProvisioning 10s (x22 over 4m44s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "kubevirt.io/hostpath-provisioner" or manually created by system administrator 4. provide informations as c#7 requested. $ oc get hostpathprovisioner NAME AGE hostpath-provisioner 141m $ oc get pod -n openshift-cnv NAME READY STATUS RESTARTS AGE bridge-marker-lkbpx 1/1 Running 0 143m bridge-marker-pbcpm 1/1 Running 0 143m bridge-marker-qsbxr 1/1 Running 0 143m bridge-marker-tb2wq 1/1 Running 0 143m bridge-marker-v2n7w 1/1 Running 0 143m cdi-apiserver-779b7c455b-pctbg 1/1 Running 0 143m cdi-deployment-59756855fc-dj5gv 1/1 Running 0 143m cdi-operator-677b7c4744-cwln4 1/1 Running 1 144m cdi-uploadproxy-6456f9b5cb-8b99q 1/1 Running 0 143m cluster-network-addons-operator-5f7c965f7f-rcpzx 1/1 Running 0 144m hco-operator-66dd5d8b7f-p9qgn 1/1 Running 0 144m hostpath-provisioner-2dwsh 1/1 Running 0 136m hostpath-provisioner-khzgs 1/1 Running 0 136m hostpath-provisioner-operator-6695797977-6xszd 1/1 Running 0 144m hostpath-provisioner-vk5kp 1/1 Running 0 136m kube-cni-linux-bridge-plugin-2dfbx 1/1 Running 0 143m kube-cni-linux-bridge-plugin-62jbs 1/1 Running 0 143m kube-cni-linux-bridge-plugin-67v7h 1/1 Running 0 143m kube-cni-linux-bridge-plugin-8w5s7 1/1 Running 0 143m kube-cni-linux-bridge-plugin-vvbkx 1/1 Running 0 143m kubemacpool-mac-controller-manager-74b986d899-7dspc 1/1 Running 0 143m kubemacpool-mac-controller-manager-74b986d899-9bm5k 1/1 Running 0 143m kubevirt-node-labeller-fj829 1/1 Running 0 141m kubevirt-node-labeller-q2nkj 1/1 Running 0 141m kubevirt-node-labeller-wgb5n 1/1 Running 0 141m kubevirt-ssp-operator-55d54c556b-h4twg 1/1 Running 0 144m nmstate-handler-9j6pd 1/1 Running 0 143m nmstate-handler-g9xfk 1/1 Running 0 143m nmstate-handler-grf2j 1/1 Running 0 143m nmstate-handler-rzvf2 1/1 Running 0 143m nmstate-handler-sd7fv 1/1 Running 0 143m node-maintenance-operator-869dbc7ff9-xws9p 1/1 Running 0 144m ovs-cni-amd64-78rzp 2/2 Running 0 142m ovs-cni-amd64-99b8s 2/2 Running 0 142m ovs-cni-amd64-ddg2b 2/2 Running 0 142m ovs-cni-amd64-k7tg9 2/2 Running 0 142m ovs-cni-amd64-vshzr 2/2 Running 0 142m virt-api-556d5c95bc-vlmbv 1/1 Running 0 142m virt-api-556d5c95bc-xkj7b 1/1 Running 0 142m virt-controller-f64974c59-7mfr4 1/1 Running 0 142m virt-controller-f64974c59-r8z98 1/1 Running 0 142m virt-handler-jjk6s 1/1 Running 0 142m virt-handler-m2plk 1/1 Running 0 142m virt-handler-xq4pm 1/1 Running 0 142m virt-operator-845679f989-fqnkv 1/1 Running 1 144m virt-operator-845679f989-x9z9j 1/1 Running 0 144m virt-template-validator-55f48976dc-6h7lx 1/1 Running 0 141m virt-template-validator-55f48976dc-stvc5 1/1 Running 0 141m
It appears the UI is trying to create block mode disks, which are not possible for a hostpath provisioner. And that is why it remains pending. [cnv-qe-jenkins@cnv-executor-guohua ~]$ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE cirros Bound pvc-eb26f290-fef4-4aac-a95d-1d66556af826 2Gi RWO rook-ceph-block 23m cirros-hpp Bound pvc-c16fd713-d4f9-4347-a1f5-6267f8510230 39Gi RWO hostpath-provisioner 7m56s cirros-hpp1 Bound pvc-7e79f54b-6901-4c6d-9b07-9496fa487e7d 39Gi RWO hostpath-provisioner 5m16s test-rootdisk Pending hostpath-provisioner 4m40s [cnv-qe-jenkins@cnv-executor-guohua ~]$ oc describe pvc test-rootdisk Name: test-rootdisk Namespace: default StorageClass: hostpath-provisioner Status: Pending Volume: Labels: app=containerized-data-importer cdi-controller=test-rootdisk Annotations: cdi.kubevirt.io/storage.contentType: kubevirt cdi.kubevirt.io/storage.import.endpoint: https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img cdi.kubevirt.io/storage.import.importPodName: importer-test-rootdisk-csrlv cdi.kubevirt.io/storage.import.source: http cdi.kubevirt.io/storage.pod.phase: Pending volume.beta.kubernetes.io/storage-provisioner: kubevirt.io/hostpath-provisioner volume.kubernetes.io/selected-node: host-172-16-0-18 Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: VolumeMode: Block <------------- Mounted By: importer-test-rootdisk-csrlv Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal WaitForFirstConsumer 4m44s persistentvolume-controller waiting for first consumer to be created before binding Warning ProvisioningFailed 30s (x3 over 4m44s) kubevirt.io/hostpath-provisioner_hostpath-provisioner-khzgs_0a359049-96bf-4510-9890-038024938b67 kubevirt.io/hostpath-provisioner does not support block volume provisioning <--------- Normal ExternalProvisioning 10s (x22 over 4m44s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "kubevirt.io/hostpath-provisioner" or manually created by system administrator
I creates a bug https://bugzilla.redhat.com/show_bug.cgi?id=1782650 for UI, feel free to close this as virtctl upload image is fine with hpp.
We need to retest this (without using the UI this time)
Verified with Code: ----------------------------------------------- container-native-virtualization-virt-cdi-uploadserver:v2.2.0-3 OCP 4.3 CNV 2.2 Verified with the following scenario: ----------------------------------------------- virtctl image-upload --image-path=./cirros-0.4.0-x86_64-disk.img --pvc-size=100Mi --storage-class=hostpath-provisioner --uploadproxy-url=https://cdi-uploadproxy-openshift-cnv.apps.working.oc4 --pvc-name=upload-hp --insecure PVC default/upload-hp created Waiting for PVC upload-hp upload pod to be running... Pod now running Uploading data to https://cdi-uploadproxy-openshift-cnv.apps.working.oc4 12.13 MiB / 12.13 MiB [===============================================================================================================================================================================] 100.00% 2s Uploading ./cirros-0.4.0-x86_64-disk.img completed successfully
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:0307