Bug 1739456
| Summary: | [hostpath-provisioner] Can't upload image using virtctl tool | ||
|---|---|---|---|
| Product: | Container Native Virtualization (CNV) | Reporter: | Qixuan Wang <qixuan.wang> |
| Component: | Storage | Assignee: | Alexander Wels <awels> |
| Status: | CLOSED ERRATA | QA Contact: | Qixuan Wang <qixuan.wang> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 1.4.1 | CC: | alitke, cnv-qe-bugs, gouyang, kgoldbla, ngavrilo, ycui |
| Target Milestone: | --- | ||
| Target Release: | 2.2.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | hco-bundle-registry-container-v2.2.0-21 virt-cdi-operator-container-v2.2.0-2 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-01-30 16:27:13 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Qixuan Wang
2019-08-09 11:11:26 UTC
Pending on the d/s test environment, so we can not verify this bug now. The issue is still existing in virt-cdi-operator:v2.2.0-3
$ oc get pod -n openshift-cnv cdi-operator-677b7c4744-xl86c -o yaml | grep image
image: registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-virt-cdi-operator:v2.2.0-3
$ oc describe pvc test1-rootdisk
Name: test1-rootdisk
Namespace: default
StorageClass: kubevirt-hostpath-provisioner
Status: Pending
Volume:
Labels: app=containerized-data-importer
cdi-controller=test1-rootdisk
Annotations: cdi.kubevirt.io/storage.contentType: kubevirt
cdi.kubevirt.io/storage.import.endpoint: https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
cdi.kubevirt.io/storage.import.importPodName: importer-test1-rootdisk-zd5kh
cdi.kubevirt.io/storage.import.source: http
cdi.kubevirt.io/storage.pod.phase: Pending
volume.beta.kubernetes.io/storage-provisioner: kubevirt.io/hostpath-provisioner
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Block
Mounted By: importer-test1-rootdisk-zd5kh
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 10s (x21 over 4m51s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "kubevirt.io/hostpath-provisioner" or manually created by system administrator
Make sure your environment is working correctly before testing stuff against it. According to the PVC, it is still in pending state, so there is a problem with the provisioner and that has nothing to do with CDI. Can you post the virtctl command you used, as well as your storage classes and the pods running in openshift-cnv? Also since it looks like you are trying to use the hostpath provisioner, can you give me the output of "oc get hostpathprovisioner" 1. virtctl image-upload works with --storage-class=hostpath-provisioner. $ virtctl image-upload --uploadproxy-url=https://cdi-uploadproxy-openshift-cnv.apps.working.oc4 --pvc-name=cirros-hpp --pvc-size=2Gi --image-path=./cirros-0.4.0-x86_64-disk.img --insecure --storage-class=hostpath-provisioner PVC default/cirros-hpp created Waiting for PVC cirros-hpp upload pod to be running... Pod now running Uploading data to https://cdi-uploadproxy-openshift-cnv.apps.working.oc4 12.13 MiB / 12.13 MiB [====================================================================================================] 100.00% 2s Uploading ./cirros-0.4.0-x86_64-disk.img completed successfully 2. virtctl image-upload works if the default sc is hostpath-provisioner. $ oc get sc NAME PROVISIONER AGE hostpath-provisioner (default) kubevirt.io/hostpath-provisioner 130m local-sc kubernetes.io/no-provisioner 141m rook-ceph-block rook-ceph.rbd.csi.ceph.com 143m [cnv-qe-jenkins@cnv-executor-guohua ~]$ virtctl image-upload --uploadproxy-url=https://cdi-uploadproxy-openshift-cnv.apps.working.oc4 --pvc-name=cirros-hpp1 --pvc-size=2Gi --image-path=./cirros-0.4.0-x86_64-disk.img --insecure PVC default/cirros-hpp1 created Waiting for PVC cirros-hpp1 upload pod to be running... Pod now running Uploading data to https://cdi-uploadproxy-openshift-cnv.apps.working.oc4 12.13 MiB / 12.13 MiB [====================================================================================================] 100.00% 2s Uploading ./cirros-0.4.0-x86_64-disk.img completed successfully 3. Create a VM from UI by using URL as the provision source: - if the storage class is hostpath-provisioner, the PVC is pending. - if the storage class is rook-ceph-block or local-sc, the PVC and PV can be created and the VM can get into running. $ oc describe pvc test-rootdisk Name: test-rootdisk Namespace: default StorageClass: hostpath-provisioner Status: Pending Volume: Labels: app=containerized-data-importer cdi-controller=test-rootdisk Annotations: cdi.kubevirt.io/storage.contentType: kubevirt cdi.kubevirt.io/storage.import.endpoint: https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img cdi.kubevirt.io/storage.import.importPodName: importer-test-rootdisk-csrlv cdi.kubevirt.io/storage.import.source: http cdi.kubevirt.io/storage.pod.phase: Pending volume.beta.kubernetes.io/storage-provisioner: kubevirt.io/hostpath-provisioner volume.kubernetes.io/selected-node: host-172-16-0-18 Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: VolumeMode: Block Mounted By: importer-test-rootdisk-csrlv Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal WaitForFirstConsumer 4m37s persistentvolume-controller waiting for first consumer to be created before binding Warning ProvisioningFailed 23s (x3 over 4m37s) kubevirt.io/hostpath-provisioner_hostpath-provisioner-khzgs_0a359049-96bf-4510-9890-038024938b67 kubevirt.io/hostpath-provisioner does not support block volume provisioning Normal ExternalProvisioning 3s (x22 over 4m37s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "kubevirt.io/hostpath-provisioner" or manually created by system administrator [cnv-qe-jenkins@cnv-executor-guohua ~]$ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE cirros Bound pvc-eb26f290-fef4-4aac-a95d-1d66556af826 2Gi RWO rook-ceph-block 23m cirros-hpp Bound pvc-c16fd713-d4f9-4347-a1f5-6267f8510230 39Gi RWO hostpath-provisioner 7m56s cirros-hpp1 Bound pvc-7e79f54b-6901-4c6d-9b07-9496fa487e7d 39Gi RWO hostpath-provisioner 5m16s test-rootdisk Pending hostpath-provisioner 4m40s [cnv-qe-jenkins@cnv-executor-guohua ~]$ oc describe pvc test-rootdisk Name: test-rootdisk Namespace: default StorageClass: hostpath-provisioner Status: Pending Volume: Labels: app=containerized-data-importer cdi-controller=test-rootdisk Annotations: cdi.kubevirt.io/storage.contentType: kubevirt cdi.kubevirt.io/storage.import.endpoint: https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img cdi.kubevirt.io/storage.import.importPodName: importer-test-rootdisk-csrlv cdi.kubevirt.io/storage.import.source: http cdi.kubevirt.io/storage.pod.phase: Pending volume.beta.kubernetes.io/storage-provisioner: kubevirt.io/hostpath-provisioner volume.kubernetes.io/selected-node: host-172-16-0-18 Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: VolumeMode: Block Mounted By: importer-test-rootdisk-csrlv Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal WaitForFirstConsumer 4m44s persistentvolume-controller waiting for first consumer to be created before binding Warning ProvisioningFailed 30s (x3 over 4m44s) kubevirt.io/hostpath-provisioner_hostpath-provisioner-khzgs_0a359049-96bf-4510-9890-038024938b67 kubevirt.io/hostpath-provisioner does not support block volume provisioning Normal ExternalProvisioning 10s (x22 over 4m44s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "kubevirt.io/hostpath-provisioner" or manually created by system administrator 4. provide informations as c#7 requested. $ oc get hostpathprovisioner NAME AGE hostpath-provisioner 141m $ oc get pod -n openshift-cnv NAME READY STATUS RESTARTS AGE bridge-marker-lkbpx 1/1 Running 0 143m bridge-marker-pbcpm 1/1 Running 0 143m bridge-marker-qsbxr 1/1 Running 0 143m bridge-marker-tb2wq 1/1 Running 0 143m bridge-marker-v2n7w 1/1 Running 0 143m cdi-apiserver-779b7c455b-pctbg 1/1 Running 0 143m cdi-deployment-59756855fc-dj5gv 1/1 Running 0 143m cdi-operator-677b7c4744-cwln4 1/1 Running 1 144m cdi-uploadproxy-6456f9b5cb-8b99q 1/1 Running 0 143m cluster-network-addons-operator-5f7c965f7f-rcpzx 1/1 Running 0 144m hco-operator-66dd5d8b7f-p9qgn 1/1 Running 0 144m hostpath-provisioner-2dwsh 1/1 Running 0 136m hostpath-provisioner-khzgs 1/1 Running 0 136m hostpath-provisioner-operator-6695797977-6xszd 1/1 Running 0 144m hostpath-provisioner-vk5kp 1/1 Running 0 136m kube-cni-linux-bridge-plugin-2dfbx 1/1 Running 0 143m kube-cni-linux-bridge-plugin-62jbs 1/1 Running 0 143m kube-cni-linux-bridge-plugin-67v7h 1/1 Running 0 143m kube-cni-linux-bridge-plugin-8w5s7 1/1 Running 0 143m kube-cni-linux-bridge-plugin-vvbkx 1/1 Running 0 143m kubemacpool-mac-controller-manager-74b986d899-7dspc 1/1 Running 0 143m kubemacpool-mac-controller-manager-74b986d899-9bm5k 1/1 Running 0 143m kubevirt-node-labeller-fj829 1/1 Running 0 141m kubevirt-node-labeller-q2nkj 1/1 Running 0 141m kubevirt-node-labeller-wgb5n 1/1 Running 0 141m kubevirt-ssp-operator-55d54c556b-h4twg 1/1 Running 0 144m nmstate-handler-9j6pd 1/1 Running 0 143m nmstate-handler-g9xfk 1/1 Running 0 143m nmstate-handler-grf2j 1/1 Running 0 143m nmstate-handler-rzvf2 1/1 Running 0 143m nmstate-handler-sd7fv 1/1 Running 0 143m node-maintenance-operator-869dbc7ff9-xws9p 1/1 Running 0 144m ovs-cni-amd64-78rzp 2/2 Running 0 142m ovs-cni-amd64-99b8s 2/2 Running 0 142m ovs-cni-amd64-ddg2b 2/2 Running 0 142m ovs-cni-amd64-k7tg9 2/2 Running 0 142m ovs-cni-amd64-vshzr 2/2 Running 0 142m virt-api-556d5c95bc-vlmbv 1/1 Running 0 142m virt-api-556d5c95bc-xkj7b 1/1 Running 0 142m virt-controller-f64974c59-7mfr4 1/1 Running 0 142m virt-controller-f64974c59-r8z98 1/1 Running 0 142m virt-handler-jjk6s 1/1 Running 0 142m virt-handler-m2plk 1/1 Running 0 142m virt-handler-xq4pm 1/1 Running 0 142m virt-operator-845679f989-fqnkv 1/1 Running 1 144m virt-operator-845679f989-x9z9j 1/1 Running 0 144m virt-template-validator-55f48976dc-6h7lx 1/1 Running 0 141m virt-template-validator-55f48976dc-stvc5 1/1 Running 0 141m It appears the UI is trying to create block mode disks, which are not possible for a hostpath provisioner. And that is why it remains pending.
[cnv-qe-jenkins@cnv-executor-guohua ~]$ oc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cirros Bound pvc-eb26f290-fef4-4aac-a95d-1d66556af826 2Gi RWO rook-ceph-block 23m
cirros-hpp Bound pvc-c16fd713-d4f9-4347-a1f5-6267f8510230 39Gi RWO hostpath-provisioner 7m56s
cirros-hpp1 Bound pvc-7e79f54b-6901-4c6d-9b07-9496fa487e7d 39Gi RWO hostpath-provisioner 5m16s
test-rootdisk Pending hostpath-provisioner 4m40s
[cnv-qe-jenkins@cnv-executor-guohua ~]$ oc describe pvc test-rootdisk
Name: test-rootdisk
Namespace: default
StorageClass: hostpath-provisioner
Status: Pending
Volume:
Labels: app=containerized-data-importer
cdi-controller=test-rootdisk
Annotations: cdi.kubevirt.io/storage.contentType: kubevirt
cdi.kubevirt.io/storage.import.endpoint: https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
cdi.kubevirt.io/storage.import.importPodName: importer-test-rootdisk-csrlv
cdi.kubevirt.io/storage.import.source: http
cdi.kubevirt.io/storage.pod.phase: Pending
volume.beta.kubernetes.io/storage-provisioner: kubevirt.io/hostpath-provisioner
volume.kubernetes.io/selected-node: host-172-16-0-18
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Block <-------------
Mounted By: importer-test-rootdisk-csrlv
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 4m44s persistentvolume-controller waiting for first consumer to be created before binding
Warning ProvisioningFailed 30s (x3 over 4m44s) kubevirt.io/hostpath-provisioner_hostpath-provisioner-khzgs_0a359049-96bf-4510-9890-038024938b67 kubevirt.io/hostpath-provisioner does not support block volume provisioning <---------
Normal ExternalProvisioning 10s (x22 over 4m44s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "kubevirt.io/hostpath-provisioner" or manually created by system administrator
I creates a bug https://bugzilla.redhat.com/show_bug.cgi?id=1782650 for UI, feel free to close this as virtctl upload image is fine with hpp. We need to retest this (without using the UI this time) Verified with Code: ----------------------------------------------- container-native-virtualization-virt-cdi-uploadserver:v2.2.0-3 OCP 4.3 CNV 2.2 Verified with the following scenario: ----------------------------------------------- virtctl image-upload --image-path=./cirros-0.4.0-x86_64-disk.img --pvc-size=100Mi --storage-class=hostpath-provisioner --uploadproxy-url=https://cdi-uploadproxy-openshift-cnv.apps.working.oc4 --pvc-name=upload-hp --insecure PVC default/upload-hp created Waiting for PVC upload-hp upload pod to be running... Pod now running Uploading data to https://cdi-uploadproxy-openshift-cnv.apps.working.oc4 12.13 MiB / 12.13 MiB [===============================================================================================================================================================================] 100.00% 2s Uploading ./cirros-0.4.0-x86_64-disk.img completed successfully Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:0307 |