Description of problem: Trying to upload with virtctl outputs Unexpected return value 500 (upload fails) Version-Release number of selected component: CNV 2.0 ( hco-2.0.0-15) How reproducible: 100% Steps to Reproduce: Run: # virtctl image-upload --uploadproxy-url=https://cdi-uploadproxy-kubevirt-hyperconverged.apps.working.oc4 --pvc-name=upload-test --image-path=cirros-0.4.0-x86_64-disk.img --insecure --pvc-size=5Gi --storage-class local-sc Actual results: The following output: PVC local-storage/upload-test created Waiting for PVC upload-test upload pod to be running... Pod now running Uploading data to https://cdi-uploadproxy-kubevirt-hyperconverged.apps.working.oc4 12.13 MiB / 12.13 MiB [==========================================================================================================================================================================================================================================] 100.00% 0s Unexpected return value 500 Expected results: For the operation to finish successfully Additional info: Output from cdi-uploadproxy pod: # oc logs -f cdi-uploadproxy-547d56666f-fltrr -n kubevirt-hyperconverged I0506 15:08:57.853550 1 uploadproxy.go:54] Note: increase the -v level in the api deployment for more detailed logging, eg. -v=2 or -v=3 W0506 15:08:57.853956 1 client_config.go:548] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0507 08:16:31.524421 1 uploadproxy.go:170] Received valid token: pvc: upload-test, namespace: local-storage I0507 08:53:11.206323 1 uploadproxy.go:170] Received valid token: pvc: upload-test, namespace: local-storage Output from cdi-upload: # oc logs -f cdi-upload-upload-test I0507 08:53:02.809438 1 uploadserver.go:70] Upload destination: /data/disk.img I0507 08:53:02.809691 1 uploadserver.go:72] Running server on 0.0.0.0:8443 2019/05/07 08:53:11 http: TLS handshake error from 10.130.0.31:60596: EOF I0507 08:53:11.229911 1 data-processor.go:237] Calculating available size I0507 08:53:11.230713 1 data-processor.go:245] Checking out file system volume size. I0507 08:53:11.230884 1 data-processor.go:249] Request image size not empty. I0507 08:53:11.230897 1 data-processor.go:254] Target size 5Gi. I0507 08:53:11.231070 1 data-processor.go:167] New phase: TransferScratch I0507 08:53:11.231886 1 util.go:140] Writing data... I0507 08:53:11.352232 1 data-processor.go:167] New phase: Process I0507 08:53:11.352259 1 data-processor.go:167] New phase: Convert I0507 08:53:11.352264 1 data-processor.go:173] Validating image E0507 08:53:11.360599 1 qemu.go:156] Invalid JSON: E0507 08:53:11.363570 1 data-processor.go:164] unexpected end of JSON input Invalid json for image /scratch/tmpimage kubevirt.io/containerized-data-importer/pkg/image.(*qemuOperations).Info /go/src/kubevirt.io/containerized-data-importer/pkg/image/qemu.go:157 kubevirt.io/containerized-data-importer/pkg/image.(*qemuOperations).Validate /go/src/kubevirt.io/containerized-data-importer/pkg/image/qemu.go:172 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).validate /go/src/kubevirt.io/containerized-data-importer/pkg/importer/data-processor.go:174 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).convert /go/src/kubevirt.io/containerized-data-importer/pkg/importer/data-processor.go:183 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData /go/src/kubevirt.io/containerized-data-importer/pkg/importer/data-processor.go:157 kubevirt.io/containerized-data-importer/pkg/uploadserver.newUploadStreamProcessor /go/src/kubevirt.io/containerized-data-importer/pkg/uploadserver/uploadserver.go:260 kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).uploadHandler /go/src/kubevirt.io/containerized-data-importer/pkg/uploadserver/uploadserver.go:237 kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).uploadHandler-fm /go/src/kubevirt.io/containerized-data-importer/pkg/uploadserver/uploadserver.go:94 net/http.HandlerFunc.ServeHTTP /usr/lib/golang/src/net/http/server.go:1964 net/http.(*ServeMux).ServeHTTP /usr/lib/golang/src/net/http/server.go:2361 kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).ServeHTTP /go/src/kubevirt.io/containerized-data-importer/pkg/uploadserver/uploadserver.go:201 net/http.serverHandler.ServeHTTP /usr/lib/golang/src/net/http/server.go:2741 net/http.(*conn).serve /usr/lib/golang/src/net/http/server.go:1847 runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s:1333 Image validation failed kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).validate /go/src/kubevirt.io/containerized-data-importer/pkg/importer/data-processor.go:176 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).convert /go/src/kubevirt.io/containerized-data-importer/pkg/importer/data-processor.go:183 kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData /go/src/kubevirt.io/containerized-data-importer/pkg/importer/data-processor.go:157 kubevirt.io/containerized-data-importer/pkg/uploadserver.newUploadStreamProcessor /go/src/kubevirt.io/containerized-data-importer/pkg/uploadserver/uploadserver.go:260 kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).uploadHandler /go/src/kubevirt.io/containerized-data-importer/pkg/uploadserver/uploadserver.go:237 kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).uploadHandler-fm /go/src/kubevirt.io/containerized-data-importer/pkg/uploadserver/uploadserver.go:94 net/http.HandlerFunc.ServeHTTP /usr/lib/golang/src/net/http/server.go:1964 net/http.(*ServeMux).ServeHTTP /usr/lib/golang/src/net/http/server.go:2361 kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).ServeHTTP /go/src/kubevirt.io/containerized-data-importer/pkg/uploadserver/uploadserver.go:201 net/http.serverHandler.ServeHTTP /usr/lib/golang/src/net/http/server.go:2741 net/http.(*conn).serve /usr/lib/golang/src/net/http/server.go:1847 runtime.goexit /usr/lib/golang/src/runtime/asm_amd64.s:1333 I0507 08:53:11.363774 1 util.go:37] deleting file: /scratch/tmpimage E0507 08:53:11.365398 1 uploadserver.go:243] Saving stream failed: Image validation failed: Invalid json for image /scratch/tmpimage: unexpected end of JSON input
So it looks like everything worked up to the point where it was verifying the contents of the image. It failed there. I have seen that a few times on an import, and a retry solved it, obviously that doesn't work for upload.
Well, the "How reproducible" field needs to be updated from 100% to "Not sure" (it's not consistent as I assumed) thus, changing severity from "urgent" to "high".
Irit, is this the duplicate of bug 1683163? could you take a look?
It's a different bug. Michael is this the PR with the fix https://github.com/kubevirt/containerized-data-importer/pull/807 ?
No, that PR should not be related to this at all.
*** This bug has been marked as a duplicate of bug 1725730 ***