Bug 1739456 - [hostpath-provisioner] Can't upload image using virtctl tool
Summary: [hostpath-provisioner] Can't upload image using virtctl tool
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Storage
Version: 1.4.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 2.2.0
Assignee: Alexander Wels
QA Contact: Qixuan Wang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-09 11:11 UTC by Qixuan Wang
Modified: 2020-01-30 16:27 UTC (History)
6 users (show)

Fixed In Version: hco-bundle-registry-container-v2.2.0-21 virt-cdi-operator-container-v2.2.0-2
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-01-30 16:27:13 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github kubevirt hostpath-provisioner pull 15 0 'None' closed Support Binding Mode WaitForFirstConsumer 2020-12-23 14:53:27 UTC
Red Hat Product Errata RHEA-2020:0307 0 None None None 2020-01-30 16:27:23 UTC

Description Qixuan Wang 2019-08-09 11:11:26 UTC
Description of problem:
Failed to upload a local image via virtctl image-upload tool. PVC is pending with message:
persistentvolume-controller  waiting for a volume to be created, either by external provisioner "kubevirt.io/hostpath-provisioner" or manually created by system administrator
PVC and DV can't get kubevirt.io/provisionOnNode: <node> as an annotation.

Hostpath-provisioner is a "node aware" provisioner, how can it take effect in this case?

Failed to upload using YAML + token + curl, neither hostpath-provisioner nor other provisioners (local volume and glusterfs) in my 1.4 cluster. So I can't say upload failed only when using hostpath, or when using hostpath, upload works but virtctl upload doesn't.


Version-Release number of selected component (if applicable):
hostpath-provisioner: v1.4.1-4
CDI v1.4.1-1


How reproducible:
100%


Steps to Reproduce:
1. Upload local disk image via virtctl
2. Check PVC


Actual results:
1. [cloud-user@cnv-executor-qwang-14-master-b19d80-1 ~]$ virtctl image-upload --image-path=./cirros-0.4.0-x86_64-disk.img --pvc-size=1Gi --storage-class=kubevirt-hostpath-provisioner --uploadproxy-url=https://cdi-uploadproxy-route-cdi.cloudapps.example.com --pvc-name=upload-hp-1 --insecure
PVC qwang89-1/upload-hp-1 created
Waiting for PVC upload-hp-1 upload pod to be running...
timed out waiting for the condition


2. [cloud-user@cnv-executor-qwang-14-master-b19d80-1 ~]$ oc get pvc
NAME                 STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                    AGE
hostpath-cloned-dv   Bound     pvc-12c0ee20-ba87-11e9-9c00-fa163e34ff55   39Gi       RWO            kubevirt-hostpath-provisioner   39m
upload-g             Bound     pvc-461bb5c3-ba8c-11e9-9c00-fa163e34ff55   1Gi        RWO            glusterfs-storage               1m
upload-hp-1          Pending                                                                        kubevirt-hostpath-provisioner   7s
[cloud-user@cnv-executor-qwang-14-master-b19d80-1 ~]$ oc describe pvc upload-hp-1
Name:          upload-hp-1
Namespace:     qwang89-1
StorageClass:  kubevirt-hostpath-provisioner
Status:        Pending
Volume:
Labels:        <none>
Annotations:   cdi.kubevirt.io/storage.pod.phase=Pending
               cdi.kubevirt.io/storage.upload.target=
               volume.beta.kubernetes.io/storage-provisioner=kubevirt.io/hostpath-provisioner
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Events:
  Type    Reason                Age               From                         Message
  ----    ------                ----              ----                         -------
  Normal  ExternalProvisioning  1s (x5 over 24s)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "kubevirt.io/hostpath-provisioner" or manually created by system administrator


Expected results:
Upload successfully.


Additional info:

Comment 5 Ying Cui 2019-11-25 06:00:23 UTC
Pending on the d/s test environment, so we can not verify this bug now.

Comment 6 Guohua Ouyang 2019-12-10 04:15:21 UTC
The issue is still existing in virt-cdi-operator:v2.2.0-3

$ oc get pod -n openshift-cnv cdi-operator-677b7c4744-xl86c -o yaml | grep image
    image: registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-virt-cdi-operator:v2.2.0-3

$ oc describe  pvc test1-rootdisk 
Name:          test1-rootdisk
Namespace:     default
StorageClass:  kubevirt-hostpath-provisioner
Status:        Pending
Volume:        
Labels:        app=containerized-data-importer
               cdi-controller=test1-rootdisk
Annotations:   cdi.kubevirt.io/storage.contentType: kubevirt
               cdi.kubevirt.io/storage.import.endpoint: https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
               cdi.kubevirt.io/storage.import.importPodName: importer-test1-rootdisk-zd5kh
               cdi.kubevirt.io/storage.import.source: http
               cdi.kubevirt.io/storage.pod.phase: Pending
               volume.beta.kubernetes.io/storage-provisioner: kubevirt.io/hostpath-provisioner
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Block
Mounted By:    importer-test1-rootdisk-zd5kh
Events:
  Type    Reason                Age                   From                         Message
  ----    ------                ----                  ----                         -------
  Normal  ExternalProvisioning  10s (x21 over 4m51s)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "kubevirt.io/hostpath-provisioner" or manually created by system administrator

Comment 7 Alexander Wels 2019-12-10 12:50:00 UTC
Make sure your environment is working correctly before testing stuff against it. According to the PVC, it is still in pending state, so there is a problem with the provisioner and that has nothing to do with CDI. Can you post the virtctl command you used, as well as your storage classes and the pods running in openshift-cnv? Also since it looks like you are trying to use the hostpath provisioner, can you give me the output of "oc get hostpathprovisioner"

Comment 8 Guohua Ouyang 2019-12-11 05:24:53 UTC
1. virtctl image-upload works with --storage-class=hostpath-provisioner.

$ virtctl image-upload --uploadproxy-url=https://cdi-uploadproxy-openshift-cnv.apps.working.oc4 --pvc-name=cirros-hpp --pvc-size=2Gi --image-path=./cirros-0.4.0-x86_64-disk.img  --insecure --storage-class=hostpath-provisioner
PVC default/cirros-hpp created
Waiting for PVC cirros-hpp upload pod to be running...
Pod now running
Uploading data to https://cdi-uploadproxy-openshift-cnv.apps.working.oc4

 12.13 MiB / 12.13 MiB [====================================================================================================] 100.00% 2s

Uploading ./cirros-0.4.0-x86_64-disk.img completed successfully

2. virtctl image-upload works if the default sc is hostpath-provisioner.

$ oc get sc
NAME                             PROVISIONER                        AGE
hostpath-provisioner (default)   kubevirt.io/hostpath-provisioner   130m
local-sc                         kubernetes.io/no-provisioner       141m
rook-ceph-block                  rook-ceph.rbd.csi.ceph.com         143m
[cnv-qe-jenkins@cnv-executor-guohua ~]$ virtctl image-upload --uploadproxy-url=https://cdi-uploadproxy-openshift-cnv.apps.working.oc4 --pvc-name=cirros-hpp1 --pvc-size=2Gi --image-path=./cirros-0.4.0-x86_64-disk.img  --insecure
PVC default/cirros-hpp1 created
Waiting for PVC cirros-hpp1 upload pod to be running...
Pod now running
Uploading data to https://cdi-uploadproxy-openshift-cnv.apps.working.oc4

 12.13 MiB / 12.13 MiB [====================================================================================================] 100.00% 2s

Uploading ./cirros-0.4.0-x86_64-disk.img completed successfully


3. Create a VM from UI by using URL as the provision source:
- if the storage class is hostpath-provisioner, the PVC is pending.
- if the storage class is rook-ceph-block or local-sc, the PVC and PV can be created and the VM can get into running.

$ oc describe  pvc test-rootdisk
Name:          test-rootdisk
Namespace:     default
StorageClass:  hostpath-provisioner
Status:        Pending
Volume:        
Labels:        app=containerized-data-importer
               cdi-controller=test-rootdisk
Annotations:   cdi.kubevirt.io/storage.contentType: kubevirt
               cdi.kubevirt.io/storage.import.endpoint: https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
               cdi.kubevirt.io/storage.import.importPodName: importer-test-rootdisk-csrlv
               cdi.kubevirt.io/storage.import.source: http
               cdi.kubevirt.io/storage.pod.phase: Pending
               volume.beta.kubernetes.io/storage-provisioner: kubevirt.io/hostpath-provisioner
               volume.kubernetes.io/selected-node: host-172-16-0-18
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Block
Mounted By:    importer-test-rootdisk-csrlv
Events:
  Type     Reason                Age                  From                                                                                              Message
  ----     ------                ----                 ----                                                                                              -------
  Normal   WaitForFirstConsumer  4m37s                persistentvolume-controller                                                                       waiting for first consumer to be created before binding
  Warning  ProvisioningFailed    23s (x3 over 4m37s)  kubevirt.io/hostpath-provisioner_hostpath-provisioner-khzgs_0a359049-96bf-4510-9890-038024938b67  kubevirt.io/hostpath-provisioner does not support block volume provisioning
  Normal   ExternalProvisioning  3s (x22 over 4m37s)  persistentvolume-controller                                                                       waiting for a volume to be created, either by external provisioner "kubevirt.io/hostpath-provisioner" or manually created by system administrator
[cnv-qe-jenkins@cnv-executor-guohua ~]$ oc get pvc
NAME            STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS           AGE
cirros          Bound     pvc-eb26f290-fef4-4aac-a95d-1d66556af826   2Gi        RWO            rook-ceph-block        23m
cirros-hpp      Bound     pvc-c16fd713-d4f9-4347-a1f5-6267f8510230   39Gi       RWO            hostpath-provisioner   7m56s
cirros-hpp1     Bound     pvc-7e79f54b-6901-4c6d-9b07-9496fa487e7d   39Gi       RWO            hostpath-provisioner   5m16s
test-rootdisk   Pending                                                                        hostpath-provisioner   4m40s
[cnv-qe-jenkins@cnv-executor-guohua ~]$ oc describe  pvc test-rootdisk
Name:          test-rootdisk
Namespace:     default
StorageClass:  hostpath-provisioner
Status:        Pending
Volume:        
Labels:        app=containerized-data-importer
               cdi-controller=test-rootdisk
Annotations:   cdi.kubevirt.io/storage.contentType: kubevirt
               cdi.kubevirt.io/storage.import.endpoint: https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
               cdi.kubevirt.io/storage.import.importPodName: importer-test-rootdisk-csrlv
               cdi.kubevirt.io/storage.import.source: http
               cdi.kubevirt.io/storage.pod.phase: Pending
               volume.beta.kubernetes.io/storage-provisioner: kubevirt.io/hostpath-provisioner
               volume.kubernetes.io/selected-node: host-172-16-0-18
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Block
Mounted By:    importer-test-rootdisk-csrlv
Events:
  Type     Reason                Age                   From                                                                                              Message
  ----     ------                ----                  ----                                                                                              -------
  Normal   WaitForFirstConsumer  4m44s                 persistentvolume-controller                                                                       waiting for first consumer to be created before binding
  Warning  ProvisioningFailed    30s (x3 over 4m44s)   kubevirt.io/hostpath-provisioner_hostpath-provisioner-khzgs_0a359049-96bf-4510-9890-038024938b67  kubevirt.io/hostpath-provisioner does not support block volume provisioning
  Normal   ExternalProvisioning  10s (x22 over 4m44s)  persistentvolume-controller                                                                       waiting for a volume to be created, either by external provisioner "kubevirt.io/hostpath-provisioner" or manually created by system administrator

4. provide informations as c#7 requested.

$ oc get hostpathprovisioner
NAME                   AGE
hostpath-provisioner   141m

$ oc get pod -n openshift-cnv
NAME                                                  READY   STATUS    RESTARTS   AGE
bridge-marker-lkbpx                                   1/1     Running   0          143m
bridge-marker-pbcpm                                   1/1     Running   0          143m
bridge-marker-qsbxr                                   1/1     Running   0          143m
bridge-marker-tb2wq                                   1/1     Running   0          143m
bridge-marker-v2n7w                                   1/1     Running   0          143m
cdi-apiserver-779b7c455b-pctbg                        1/1     Running   0          143m
cdi-deployment-59756855fc-dj5gv                       1/1     Running   0          143m
cdi-operator-677b7c4744-cwln4                         1/1     Running   1          144m
cdi-uploadproxy-6456f9b5cb-8b99q                      1/1     Running   0          143m
cluster-network-addons-operator-5f7c965f7f-rcpzx      1/1     Running   0          144m
hco-operator-66dd5d8b7f-p9qgn                         1/1     Running   0          144m
hostpath-provisioner-2dwsh                            1/1     Running   0          136m
hostpath-provisioner-khzgs                            1/1     Running   0          136m
hostpath-provisioner-operator-6695797977-6xszd        1/1     Running   0          144m
hostpath-provisioner-vk5kp                            1/1     Running   0          136m
kube-cni-linux-bridge-plugin-2dfbx                    1/1     Running   0          143m
kube-cni-linux-bridge-plugin-62jbs                    1/1     Running   0          143m
kube-cni-linux-bridge-plugin-67v7h                    1/1     Running   0          143m
kube-cni-linux-bridge-plugin-8w5s7                    1/1     Running   0          143m
kube-cni-linux-bridge-plugin-vvbkx                    1/1     Running   0          143m
kubemacpool-mac-controller-manager-74b986d899-7dspc   1/1     Running   0          143m
kubemacpool-mac-controller-manager-74b986d899-9bm5k   1/1     Running   0          143m
kubevirt-node-labeller-fj829                          1/1     Running   0          141m
kubevirt-node-labeller-q2nkj                          1/1     Running   0          141m
kubevirt-node-labeller-wgb5n                          1/1     Running   0          141m
kubevirt-ssp-operator-55d54c556b-h4twg                1/1     Running   0          144m
nmstate-handler-9j6pd                                 1/1     Running   0          143m
nmstate-handler-g9xfk                                 1/1     Running   0          143m
nmstate-handler-grf2j                                 1/1     Running   0          143m
nmstate-handler-rzvf2                                 1/1     Running   0          143m
nmstate-handler-sd7fv                                 1/1     Running   0          143m
node-maintenance-operator-869dbc7ff9-xws9p            1/1     Running   0          144m
ovs-cni-amd64-78rzp                                   2/2     Running   0          142m
ovs-cni-amd64-99b8s                                   2/2     Running   0          142m
ovs-cni-amd64-ddg2b                                   2/2     Running   0          142m
ovs-cni-amd64-k7tg9                                   2/2     Running   0          142m
ovs-cni-amd64-vshzr                                   2/2     Running   0          142m
virt-api-556d5c95bc-vlmbv                             1/1     Running   0          142m
virt-api-556d5c95bc-xkj7b                             1/1     Running   0          142m
virt-controller-f64974c59-7mfr4                       1/1     Running   0          142m
virt-controller-f64974c59-r8z98                       1/1     Running   0          142m
virt-handler-jjk6s                                    1/1     Running   0          142m
virt-handler-m2plk                                    1/1     Running   0          142m
virt-handler-xq4pm                                    1/1     Running   0          142m
virt-operator-845679f989-fqnkv                        1/1     Running   1          144m
virt-operator-845679f989-x9z9j                        1/1     Running   0          144m
virt-template-validator-55f48976dc-6h7lx              1/1     Running   0          141m
virt-template-validator-55f48976dc-stvc5              1/1     Running   0          141m

Comment 9 Alexander Wels 2019-12-11 12:20:32 UTC
It appears the UI is trying to create block mode disks, which are not possible for a hostpath provisioner. And that is why it remains pending.

[cnv-qe-jenkins@cnv-executor-guohua ~]$ oc get pvc
NAME            STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS           AGE
cirros          Bound     pvc-eb26f290-fef4-4aac-a95d-1d66556af826   2Gi        RWO            rook-ceph-block        23m
cirros-hpp      Bound     pvc-c16fd713-d4f9-4347-a1f5-6267f8510230   39Gi       RWO            hostpath-provisioner   7m56s
cirros-hpp1     Bound     pvc-7e79f54b-6901-4c6d-9b07-9496fa487e7d   39Gi       RWO            hostpath-provisioner   5m16s
test-rootdisk   Pending                                                                        hostpath-provisioner   4m40s
[cnv-qe-jenkins@cnv-executor-guohua ~]$ oc describe  pvc test-rootdisk
Name:          test-rootdisk
Namespace:     default
StorageClass:  hostpath-provisioner
Status:        Pending
Volume:        
Labels:        app=containerized-data-importer
               cdi-controller=test-rootdisk
Annotations:   cdi.kubevirt.io/storage.contentType: kubevirt
               cdi.kubevirt.io/storage.import.endpoint: https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
               cdi.kubevirt.io/storage.import.importPodName: importer-test-rootdisk-csrlv
               cdi.kubevirt.io/storage.import.source: http
               cdi.kubevirt.io/storage.pod.phase: Pending
               volume.beta.kubernetes.io/storage-provisioner: kubevirt.io/hostpath-provisioner
               volume.kubernetes.io/selected-node: host-172-16-0-18
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Block <-------------
Mounted By:    importer-test-rootdisk-csrlv
Events:
  Type     Reason                Age                   From                                                                                              Message
  ----     ------                ----                  ----                                                                                              -------
  Normal   WaitForFirstConsumer  4m44s                 persistentvolume-controller                                                                       waiting for first consumer to be created before binding
  Warning  ProvisioningFailed    30s (x3 over 4m44s)   kubevirt.io/hostpath-provisioner_hostpath-provisioner-khzgs_0a359049-96bf-4510-9890-038024938b67  kubevirt.io/hostpath-provisioner does not support block volume provisioning <---------
  Normal   ExternalProvisioning  10s (x22 over 4m44s)  persistentvolume-controller                                                                       waiting for a volume to be created, either by external provisioner "kubevirt.io/hostpath-provisioner" or manually created by system administrator

Comment 10 Guohua Ouyang 2019-12-12 02:45:46 UTC
I creates a bug https://bugzilla.redhat.com/show_bug.cgi?id=1782650 for UI, feel free to close this as virtctl upload image is fine with hpp.

Comment 11 Natalie Gavrielov 2019-12-18 13:24:10 UTC
We need to retest this (without using the UI this time)

Comment 12 Kevin Alon Goldblatt 2019-12-23 12:00:39 UTC
Verified with Code:
-----------------------------------------------
container-native-virtualization-virt-cdi-uploadserver:v2.2.0-3
OCP 4.3 CNV 2.2



Verified with the following scenario:
-----------------------------------------------
virtctl image-upload --image-path=./cirros-0.4.0-x86_64-disk.img --pvc-size=100Mi --storage-class=hostpath-provisioner --uploadproxy-url=https://cdi-uploadproxy-openshift-cnv.apps.working.oc4 --pvc-name=upload-hp --insecure
PVC default/upload-hp created
Waiting for PVC upload-hp upload pod to be running...
Pod now running
Uploading data to https://cdi-uploadproxy-openshift-cnv.apps.working.oc4

 12.13 MiB / 12.13 MiB [===============================================================================================================================================================================] 100.00% 2s

Uploading ./cirros-0.4.0-x86_64-disk.img completed successfully

Comment 14 errata-xmlrpc 2020-01-30 16:27:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:0307


Note You need to log in before you can comment on or make changes to this bug.