Description of problem: Seems that even though scratchSpaceStorageClass is defined, scratch space required operations look for a default storage class to use. Version-Release number of selected component: CNV 2.3 How reproducible: Not sure yet. Steps to Reproduce: 1. Have one hostpath-provisioner storage class in the environment 2. The storage class is not defined as default (no default storage class) 3. scratchSpaceStorageClass is defined to be hostpath-provisioner 4. upload a data volume: virtctl image-upload dv dv-name --size=1Gi --image-path=cirros-0.4.0-x86_64-disk.qcow2 --insecure Actual results: DataVolume default/dv-name created Waiting for PVC dv-name upload pod to be ready... timed out waiting for the condition Expected results: Upload to succeed Additional info: After upload failed, I set a default storage class: $ oc patch storageclass hostpath-provisioner -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}' storageclass.storage.k8s.io/hostpath-provisioner patched oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE hostpath-provisioner (default) kubevirt.io/hostpath-provisioner Delete WaitForFirstConsumer false 2d18h and tried to upload again: $ virtctl image-upload dv dv-name --size=1Gi --image-path=cirros-0.4.0-x86_64-disk.qcow2 --insecure DataVolume default/dv-name created Waiting for PVC dv-name upload pod to be ready... Pod now ready Uploading data to https://cdi-uploadproxy-openshift-cnv.apps.bm-rhel.oc4 12.13 MiB / 12.13 MiB [===============================================================================================================================================] 100.00% 0s Uploading data completed successfully, waiting for processing to complete, you can hit ctrl-c without interrupting the progress Processing completed successfully Uploading cirros-0.4.0-x86_64-disk.qcow2 completed successfully $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE dv-name Bound pvc-ac2cc14e-2999-4546-8f78-1f7788c1c2f3 916Gi RWO hostpath-provisioner 13s $ oc get dv NAME PHASE PROGRESS AGE dv-name Succeeded 1m Output for the failed test: ------------------------------------------------------------------------- $ oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE hostpath-provisioner kubevirt.io/hostpath-provisioner Delete WaitForFirstConsumer false 2d17h ------------------------------------------------------------------------- $ oc get pods NAME READY STATUS RESTARTS AGE cdi-upload-dv-name 0/1 Pending 0 14s ------------------------------------------------------------------------- $ oc get pod cdi-upload-dv-name -oyaml apiVersion: v1 kind: Pod metadata: annotations: cdi.kubevirt.io/storage.createdByUploadController: "yes" creationTimestamp: "2020-04-27T07:46:04Z" labels: app: containerized-data-importer cdi.kubevirt.io: cdi-upload-server service: cdi-upload-dv-name name: cdi-upload-dv-name namespace: default ownerReferences: - apiVersion: v1 blockOwnerDeletion: true controller: true kind: PersistentVolumeClaim name: dv-name uid: f8e09ecc-7f20-439f-bafe-d5caf7e85a1e resourceVersion: "2723602" selfLink: /api/v1/namespaces/default/pods/cdi-upload-dv-name uid: d3479bd3-4f0e-4132-8a52-440138ef67d8 spec: containers: - args: - -v=1 env: - name: TLS_KEY value: | -----BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEA4g2yDfDqdf8K44NLDDMjqqMGEfxUxuCgpi8tCYQn8XpqDo3P Z751ELYggPIEucjSO5FlKt828LcJPr2jD8cDHPySHf5qMYhzrzsajN4t/lYV6/tb 1Eed5VajW3OZubc3hlSqMSD3VtpopOruSdHn4Vg1WUz3gf31u05TZ55YD8+FP0JG I8yNbggFRJSSNDJ5ssbk4FM5tCkUOs7M9LeR01GTi4KMOx7TVYqwr4FS8mGTDZN1 dbJzW7jZuNjLs1+MzsNRZAj33hxr7XAHCpGctdSiqwopS74bDEkqDwi9h1cSmskY 2xt9zUAP+RHrrUvDv4dy7V9rOG5AE+pCzkVMxwIDAQABAoIBAHRTceYkrLLN7Q+J F8yOYHNF8zC8NfAl1eXO7xEvldJ5GSymIy4S33EbF/zjDjWAt1b6CiKWvVIADtzf f5L16Y84FgUjUAhvXIC3HIGTL/3q1Nsb/XRu6j0DaYr9IQVl2EfjN7WSblWejS0Y zEjmoLngx4tNxSx6wMKDmv3+iPiQ7M06HM1KSwstA5xQjtmcK+2crJPVq44rk8fS wVGvLv0qFn8v+jwMW/7Fd+mNhAdvP9IofFvfk4d/wFV1Wm8T79F+0U9s57lPqeJr /QPa0q+0unO0vyaN2ig3knLVKQ+NyiL2aY+Hg4yrVKJaRA5f7ldTfiQyRl7ClVAV NVxEyxkCgYEA4qhop+lAuPdx8sFDhL1kV73S40aSBnOG7v3C+8Mev3HirsKLE/12 MU6/sLiA74IOwR+NmrwMRHdQI8pZrllJMhDrK+kZaje9bOcfIuXs1XI/7kWV2pLR 4O53MdstijUQoLEZWfMfHD/q6T68H24168ZYcXRzSLJltAPoz1hV2F0CgYEA/1FC GQWq/Ne/hQ5FzUCNtJMvwni5puhAqih2oklrsmm+mYkGlGziXFWW8crrH0YQOVQj whzx69OtP9UN9ZSUS1oLC1P6ZFD4t1Ygb4ytSokQYWkJIkx1KtnQ8oOT8FO71tOx F8mTluBJL5TGzoLfEBzwBXHYOElBD2TErPcL13MCgYEAwQ/v7S6KmcLzxXIHX/au lOo0yrCey232PnXacvZ1p16LVo6RRpq75S4Nq6rM8LXYeXm19Vo+HMeAdKv0jcSx yzqMBqALrOUDrdKEPAabIAisGfodtcsfiLe2wbnO5Y1jt2yCipLDKh3rq3draF5J Aaa2SRWG6SJD7PzzfkHhBWUCgYBgBubuVFRoPpkY6TJVrWbeuNEoupFQ9l+D4C4m gajQ9LasNn5UCtPkWIiqLmDaZVaqyTudwZc31Gh/7pkOebCA2r/2pcIyQsYY5nrS mb1/oCeaC5BU5x7WXBAbdmOI62Vluj8MLFOnVnoQPEu91jkhbVDVK2lgWiTvLj5O s6gvgwKBgQCFBXqvWD23Nrgd5W7a9uYDX8zXIJmphpnqZNmg/OefseNSSglD0hMA QS56B49bWEDEC+GdQ7ZLWYj4S4bgqihuqgNEB1VH15Z0GK3tBS08eXEhSWYzd3I1 VUNDrRYeGZK4BCjK0r0a1K0zasceliH5O+pFAXfgQHepANG+hKd+zA== -----END RSA PRIVATE KEY----- - name: TLS_CERT value: | -----BEGIN CERTIFICATE----- MIIDrDCCApSgAwIBAgIIN3h3XCb+vRUwDQYJKoZIhvcNAQELBQAwOzE5MDcGA1UE Awwwb3BlbnNoaWZ0LWNudl9jZGktdXBsb2Fkc2VydmVyLXNpZ25lckAxNTg3NzM3 Mjg3MB4XDTIwMDQyNzA3NDYwM1oXDTIxMDQyNzA3NDYwNFowHTEbMBkGA1UEAxMS Y2RpLXVwbG9hZC1kdi1uYW1lMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC AQEA4g2yDfDqdf8K44NLDDMjqqMGEfxUxuCgpi8tCYQn8XpqDo3PZ751ELYggPIE ucjSO5FlKt828LcJPr2jD8cDHPySHf5qMYhzrzsajN4t/lYV6/tb1Eed5VajW3OZ ubc3hlSqMSD3VtpopOruSdHn4Vg1WUz3gf31u05TZ55YD8+FP0JGI8yNbggFRJSS NDJ5ssbk4FM5tCkUOs7M9LeR01GTi4KMOx7TVYqwr4FS8mGTDZN1dbJzW7jZuNjL s1+MzsNRZAj33hxr7XAHCpGctdSiqwopS74bDEkqDwi9h1cSmskY2xt9zUAP+RHr rUvDv4dy7V9rOG5AE+pCzkVMxwIDAQABo4HRMIHOMA4GA1UdDwEB/wQEAwIFoDAT BgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBSVExQh m5O83tkSXOUeOg58kEGMJjAfBgNVHSMEGDAWgBRHbM5Bq62iqBRorkyPYUDMQx1R 6zBZBgNVHREEUjBQghJjZGktdXBsb2FkLWR2LW5hbWWCGmNkaS11cGxvYWQtZHYt bmFtZS5kZWZhdWx0gh5jZGktdXBsb2FkLWR2LW5hbWUuZGVmYXVsdC5zdmMwDQYJ KoZIhvcNAQELBQADggEBAOo1AUvlGIoOUS9YDralSYrZfFMJi579Vm3n7XgyMibH NPa82ykpaTyztAbb9a4rkcz3yoebmFHVIsi9EAxO/j8RlGs8RSDn11SHQN/Ozjt3 kS5e+YkePHbEUjiaYqhb04KY8rjsOy6dQvVJ1dk4p4SYiQEr6Lf+1tXqgFxHZezv NNc7W24Pb6qJBbVoRbT8tQQY5GX7MRiRAhzF3/9mv0vrI2WVu3vEMud8i2yCfNxe XwRNwz7hUGgbLV0GBQlCsF994RAV+3EvfPipW9LQoP6bb7A9Xc09ejcCmi18MBHZ hDLNd7zD+OIBciNZNcXmsimQIuP9k8ho9JuCpKT9Pes= -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIDVDCCAjygAwIBAgIBATANBgkqhkiG9w0BAQsFADA7MTkwNwYDVQQDDDBvcGVu c2hpZnQtY252X2NkaS11cGxvYWRzZXJ2ZXItc2lnbmVyQDE1ODc3MzcyODcwHhcN MjAwNDI0MTQwODA2WhcNMzAwNDIyMTQwODA3WjA7MTkwNwYDVQQDDDBvcGVuc2hp ZnQtY252X2NkaS11cGxvYWRzZXJ2ZXItc2lnbmVyQDE1ODc3MzcyODcwggEiMA0G CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDtIynMFcz2Nv0QXRIxnMEB+MA1owji iuyhr0cjp2HzcmkpFMLjxnS4v75h9UZ78w4vHO+gECB/p5k6GIcxV81HeMlDRI00 F4ELs5+hEmCbqJgb0PsHCCwvY+t7allDHJApCfnPS8FGVONJ9Zi22FG3bEMMbU+U 8TCHUVSQrKyg05HlfXUjDEt3I73mJx4ytmnvl1HxyHpW2iClZKD36VLUHf96DZrh jgUHdgCVJa5Qc/anS1+dsyYzdRR+O4uD3uz79NbCRgw9RXigbQhOAwksz0BRR0OG JJvvbI9SPW634hZAoes22i8f2DdCtdlqfOuOyhM+xICOn3ccPdgeTaVbAgMBAAGj YzBhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRH bM5Bq62iqBRorkyPYUDMQx1R6zAfBgNVHSMEGDAWgBRHbM5Bq62iqBRorkyPYUDM Qx1R6zANBgkqhkiG9w0BAQsFAAOCAQEAfuryljz1+9t67SsrN94Np7CIDdcVfSAi cekTM68t8JixqduRZdpj2T4zK8396pIUbAJ2GRal6U/z/EyF+dnLzAKPnGey+7mc rMqeQxJyrud4z2P+HYDAlcyXjQtGjcWPtxKW1+4vPNBw8OzRfXlE9GOhaZ1V80EX q7cqdSgSzZtpXAFxZK5T2NHOUAZ+U4kROpsPHTx3+kzKMWfLEUr8SYsBEvHwqOkq JjwXSNmJ+R9JeQhO3Q2apEUDR/NvmuD5NFRsnTG1HOGND6htfsiYKLpfNcM4GtnU na9ybWQgws692X0Cz1NIpW/sZahQD6yUNseXFhJYJufBjagkPMRYxw== -----END CERTIFICATE----- - name: CLIENT_CERT value: | -----BEGIN CERTIFICATE----- MIIDYjCCAkqgAwIBAgIBATANBgkqhkiG9w0BAQsFADBCMUAwPgYDVQQDDDdvcGVu c2hpZnQtY252X2NkaS11cGxvYWRzZXJ2ZXItY2xpZW50LXNpZ25lckAxNTg3NzM3 Mjg3MB4XDTIwMDQyNDE0MDgwNloXDTMwMDQyMjE0MDgwN1owQjFAMD4GA1UEAww3 b3BlbnNoaWZ0LWNudl9jZGktdXBsb2Fkc2VydmVyLWNsaWVudC1zaWduZXJAMTU4 NzczNzI4NzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANhK/iX1dU7o BTeJBbRywHhKllA4X5t7m4Xqxp9gSXd6znNG8wtL0OnCNalEFQ30QWxABGIItr12 rKFZTMhMI9gy/u03KHS9ovLfj/vkT1MGBQC8yQ+noG0JJqjuLyPO1zLAYRYYvmb5 Aau8W9tSO18yJ5s3j4R/j+qt7MPj9CD1upPz/LDOQ0Zcs3foLIqG5mqG6Sm24+m7 h6ZXEim13f3WgI2y8BOEAzUsgawCOrJLXeG5Ck3HetfmI55rOOXtAYM6vo27nVhQ FM8hQsy3Fo+mYtiFE9fpmvljdllNlhnfjfjsjUr9dmPGIfd9q59KYuqbRdvzXeA3 AUZWYxUXo7UCAwEAAaNjMGEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMB Af8wHQYDVR0OBBYEFI0pg+cpCjGSEPHfS6Q7SlmckY5mMB8GA1UdIwQYMBaAFI0p g+cpCjGSEPHfS6Q7SlmckY5mMA0GCSqGSIb3DQEBCwUAA4IBAQCsNCHYnsCP2oDY TCqsWnqcQLCM1UALVYo6xVAxpVMzVdG+3op6bzGY+mgFnrVfwK9O5o4sxvUE8X53 yB2hxMM2wBuPVpQns6OazK2lQ5ABJUqr72+uKVxl7I7WZdbYXx5Eb4AOLtcGfhDx C/9BxuIVbPUFmEikdR1xEKM4rv8PsVQ7RBKHesUQD9qP4DJsZFom8lgH6EbbusA2 L5ANl4zcJ2ZQYdlrb2XnRUwG0nSYxb3N0x1cyK2MQ0FAGz995pDRHue8V/FAj9Hb PqhxO6W2sjDCB8xTfg8yJAhhxRDTUBxDVF69FFrZWeuLmA/qHRWfXjBTGKVjhH/f uTDGX/6I -----END CERTIFICATE----- - name: UPLOAD_IMAGE_SIZE value: 1Gi - name: CLIENT_NAME value: client.upload-server.cdi.kubevirt.io image: registry.stage.redhat.io/container-native-virtualization/virt-cdi-uploadserver@sha256:7a6611d2f738c04bcc93f2062c074e8a80a0107be780df46bfe8cb73fc4c2d52 imagePullPolicy: IfNotPresent name: cdi-upload-server readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 8080 scheme: HTTP initialDelaySeconds: 2 periodSeconds: 5 successThreshold: 1 timeoutSeconds: 1 resources: limits: cpu: "0" memory: "0" requests: cpu: "0" memory: "0" terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /data name: cdi-data-vol - mountPath: /scratch name: cdi-scratch-vol - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: default-token-ctpkk readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true imagePullSecrets: - name: default-dockercfg-sfg85 priority: 0 restartPolicy: OnFailure schedulerName: default-scheduler securityContext: runAsUser: 0 serviceAccount: default apiVersion: cdi.kubevirt.io/v1alpha1 serviceAccountName: default terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - name: cdi-data-vol persistentVolumeClaim: claimName: dv-name - name: cdi-scratch-vol persistentVolumeClaim: claimName: dv-name-scratch - name: default-token-ctpkk secret: apiVersion: cdi.kubevirt.io/v1alpha1 defaultMode: 420 secretName: default-token-ctpkk status: conditions: - lastProbeTime: null lastTransitionTime: "2020-04-27T07:46:04Z" message: 'error while running "VolumeBinding" filter plugin for pod "cdi-upload-dv-name": pod has unbound immediate PersistentVolumeClaims' reason: Unschedulable status: "False" type: PodScheduled phase: Pending qosClass: BestEffort ------------------------------------------------------------------------- $ oc get dv dv-name -oyaml apiVersion: cdi.kubevirt.io/v1alpha1 kind: DataVolume metadata: creationTimestamp: "2020-04-27T07:46:04Z" generation: 2 name: dv-name namespace: default resourceVersion: "2723569" selfLink: /apis/cdi.kubevirt.io/v1alpha1/namespaces/default/datavolumes/dv-name uid: 8ced0050-0d83-41bb-bed5-b7ae2625eef5 spec: pvc: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi source: upload: {} status: phase: Pending ------------------------------------------------------------------------- $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE dv-name Pending 11s dv-name-scratch Pending hostpath-provisioner 10s $ oc get cdiconfig config -oyaml apiVersion: cdi.kubevirt.io/v1alpha1 kind: CDIConfig metadata: creationTimestamp: "2020-04-24T14:09:03Z" generation: 13 labels: app: containerized-data-importer cdi.kubevirt.io: "" name: config ownerReferences: - apiVersion: cdi.kubevirt.io/v1alpha1 blockOwnerDeletion: true controller: true kind: CDI name: cdi-kubevirt-hyperconverged uid: ae8cfeba-350e-4141-a1d8-22fa7f7ab3ab resourceVersion: "2383797" selfLink: /apis/cdi.kubevirt.io/v1alpha1/cdiconfigs/config uid: 5113a12a-25a5-4859-9f6f-42240b0e45e2 spec: scratchSpaceStorageClass: hostpath-provisioner status: defaultPodResourceRequirements: limits: cpu: "0" memory: "0" requests: cpu: "0" memory: "0" scratchSpaceStorageClass: hostpath-provisioner uploadProxyURL: cdi-uploadproxy-openshift-cnv.apps.bm-rhel.oc4 ------------------------------------------------------------------------- $ oc describe pvc dv-name Name: dv-name Namespace: default StorageClass: Status: Pending Volume: Labels: app=containerized-data-importer cdi-controller=dv-name Annotations: cdi.kubevirt.io/storage.pod.phase: Pending cdi.kubevirt.io/storage.pod.ready: false cdi.kubevirt.io/storage.upload.target: Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: VolumeMode: Filesystem Mounted By: cdi-upload-dv-name Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal FailedBinding 14s (x7 over 80s) persistentvolume-controller no persistent volumes available for this claim and no storage class is set ------------------------------------------------------------------------- $ oc describe pvc dv-name-scratch Name: dv-name-scratch Namespace: default StorageClass: hostpath-provisioner Status: Pending Volume: Labels: app=containerized-data-importer cdi-controller=cdi-upload-dv-name cdi.kubevirt.io/storage.import.importPvcName=dv-name Annotations: <none> Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: VolumeMode: Filesystem Mounted By: cdi-upload-dv-name Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal WaitForFirstConsumer 2s (x8 over 98s) persistentvolume-controller waiting for first consumer to be created before binding ------------------------------------------------------------------------- $ oc get nodes NAME STATUS ROLES AGE VERSION cnv-qe-01.cnvqe.lab.eng.rdu2.redhat.com Ready master 2d22h v1.17.1 cnv-qe-02.cnvqe.lab.eng.rdu2.redhat.com Ready master 2d22h v1.17.1 cnv-qe-03.cnvqe.lab.eng.rdu2.redhat.com Ready master 2d22h v1.17.1 cnv-qe-04.cnvqe.lab.eng.rdu2.redhat.com Ready worker 2d19h v1.17.1 cnv-qe-05.cnvqe.lab.eng.rdu2.redhat.com Ready worker 2d20h v1.17.1 cnv-qe-06.cnvqe.lab.eng.rdu2.redhat.com Ready worker 2d20h v1.17.1 Note: I'm having difficulty determining if it's a regression issue because we had similar scenarios, but not a scenario where we have only one storage class and it's not defined as default.
I do not consider this as a blocker, but I do think we should add it as a known issue in 2.3 docs I have added suggeted doc @Adam, @Natalie please review
I think this issue is not hpp specific. I assume that it will probably happen regardless of any storage type used. The main issue here is that scratchSpaceStorageClass configuration is being ignored and not having default storage class defined reveals it.
Agreed this looks like an issue with CDI and would therefore be storage agnostic. I agree that it should be documented for 2.3.
Alexander, please take a look.
So this is not a bug, here is what happened: 1. You ran: virtctl image-upload dv dv-name --size=1Gi --image-path=cirros-0.4.0-x86_64-disk.qcow2 --insecure 2. You have no default storage class. 3. The PVCs are created as expected, and in the yaml of the PVCs you can see that the scratch space PVC is using the hostpath-provisioner storage class, but the target PVC has no storage class, and no storage class is defined. 4. The POD cannot run, because the target PVC cannot be bound. The scratch space PVC is not bound because of WaitForFirstConsumer waiting for a consumer (which would normally be the upload pod, but it cannot run because the target PVC cannot be bound due to no storage class). In short it is doing exactly what it is supposed to do. If you specify the storage class in the virtcl image-upload command to be hostpath-provisioner, then this will all work. Please close after verifying that adding the storage class to the target PVC in the upload command works as expected.
Alexander is right, I was probably too tired when opening this one :), sorry for the hassle.