Bug 1707418

Summary: CDI Upload with virtctl results in Unexpected return value 500
Product: Container Native Virtualization (CNV) Reporter: Natalie Gavrielov <ngavrilo>
Component: StorageAssignee: Michael Henriksen <mhenriks>
Status: CLOSED DUPLICATE QA Contact: Natalie Gavrielov <ngavrilo>
Severity: high Docs Contact:
Priority: medium    
Version: 2.0CC: alitke, awels, cnv-qe-bugs, igoihman, mhenriks, sgordon, ycui
Target Milestone: ---   
Target Release: 2.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-21 14:25:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1683163, 1705080    

Description Natalie Gavrielov 2019-05-07 13:31:21 UTC 
Description of problem:
Trying to upload with virtctl outputs Unexpected return value 500 (upload fails)

Version-Release number of selected component:
CNV 2.0 ( hco-2.0.0-15)

How reproducible:
100%

Steps to Reproduce:
Run:
# virtctl image-upload --uploadproxy-url=https://cdi-uploadproxy-kubevirt-hyperconverged.apps.working.oc4 --pvc-name=upload-test --image-path=cirros-0.4.0-x86_64-disk.img --insecure --pvc-size=5Gi --storage-class local-sc


Actual results:
The following output:
PVC local-storage/upload-test created
Waiting for PVC upload-test upload pod to be running...
Pod now running
Uploading data to https://cdi-uploadproxy-kubevirt-hyperconverged.apps.working.oc4

 12.13 MiB / 12.13 MiB [==========================================================================================================================================================================================================================================] 100.00% 0s

Unexpected return value 500

Expected results:
For the operation to finish successfully

Additional info:
Output from cdi-uploadproxy pod:
# oc logs -f cdi-uploadproxy-547d56666f-fltrr -n kubevirt-hyperconverged
I0506 15:08:57.853550       1 uploadproxy.go:54] Note: increase the -v level in the api deployment for more detailed logging, eg. -v=2 or -v=3
W0506 15:08:57.853956       1 client_config.go:548] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0507 08:16:31.524421       1 uploadproxy.go:170] Received valid token: pvc: upload-test, namespace: local-storage
I0507 08:53:11.206323       1 uploadproxy.go:170] Received valid token: pvc: upload-test, namespace: local-storage


Output from cdi-upload:
# oc logs -f cdi-upload-upload-test
I0507 08:53:02.809438       1 uploadserver.go:70] Upload destination: /data/disk.img
I0507 08:53:02.809691       1 uploadserver.go:72] Running server on 0.0.0.0:8443
2019/05/07 08:53:11 http: TLS handshake error from 10.130.0.31:60596: EOF
I0507 08:53:11.229911       1 data-processor.go:237] Calculating available size
I0507 08:53:11.230713       1 data-processor.go:245] Checking out file system volume size.
I0507 08:53:11.230884       1 data-processor.go:249] Request image size not empty.
I0507 08:53:11.230897       1 data-processor.go:254] Target size 5Gi.
I0507 08:53:11.231070       1 data-processor.go:167] New phase: TransferScratch
I0507 08:53:11.231886       1 util.go:140] Writing data...
I0507 08:53:11.352232       1 data-processor.go:167] New phase: Process
I0507 08:53:11.352259       1 data-processor.go:167] New phase: Convert
I0507 08:53:11.352264       1 data-processor.go:173] Validating image
E0507 08:53:11.360599       1 qemu.go:156] Invalid JSON:

E0507 08:53:11.363570       1 data-processor.go:164] unexpected end of JSON input
Invalid json for image /scratch/tmpimage
kubevirt.io/containerized-data-importer/pkg/image.(*qemuOperations).Info
	/go/src/kubevirt.io/containerized-data-importer/pkg/image/qemu.go:157
kubevirt.io/containerized-data-importer/pkg/image.(*qemuOperations).Validate
	/go/src/kubevirt.io/containerized-data-importer/pkg/image/qemu.go:172
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).validate
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/data-processor.go:174
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).convert
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/data-processor.go:183
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/data-processor.go:157
kubevirt.io/containerized-data-importer/pkg/uploadserver.newUploadStreamProcessor
	/go/src/kubevirt.io/containerized-data-importer/pkg/uploadserver/uploadserver.go:260
kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).uploadHandler
	/go/src/kubevirt.io/containerized-data-importer/pkg/uploadserver/uploadserver.go:237
kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).uploadHandler-fm
	/go/src/kubevirt.io/containerized-data-importer/pkg/uploadserver/uploadserver.go:94
net/http.HandlerFunc.ServeHTTP
	/usr/lib/golang/src/net/http/server.go:1964
net/http.(*ServeMux).ServeHTTP
	/usr/lib/golang/src/net/http/server.go:2361
kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).ServeHTTP
	/go/src/kubevirt.io/containerized-data-importer/pkg/uploadserver/uploadserver.go:201
net/http.serverHandler.ServeHTTP
	/usr/lib/golang/src/net/http/server.go:2741
net/http.(*conn).serve
	/usr/lib/golang/src/net/http/server.go:1847
runtime.goexit
	/usr/lib/golang/src/runtime/asm_amd64.s:1333
Image validation failed
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).validate
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/data-processor.go:176
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).convert
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/data-processor.go:183
kubevirt.io/containerized-data-importer/pkg/importer.(*DataProcessor).ProcessData
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/data-processor.go:157
kubevirt.io/containerized-data-importer/pkg/uploadserver.newUploadStreamProcessor
	/go/src/kubevirt.io/containerized-data-importer/pkg/uploadserver/uploadserver.go:260
kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).uploadHandler
	/go/src/kubevirt.io/containerized-data-importer/pkg/uploadserver/uploadserver.go:237
kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).uploadHandler-fm
	/go/src/kubevirt.io/containerized-data-importer/pkg/uploadserver/uploadserver.go:94
net/http.HandlerFunc.ServeHTTP
	/usr/lib/golang/src/net/http/server.go:1964
net/http.(*ServeMux).ServeHTTP
	/usr/lib/golang/src/net/http/server.go:2361
kubevirt.io/containerized-data-importer/pkg/uploadserver.(*uploadServerApp).ServeHTTP
	/go/src/kubevirt.io/containerized-data-importer/pkg/uploadserver/uploadserver.go:201
net/http.serverHandler.ServeHTTP
	/usr/lib/golang/src/net/http/server.go:2741
net/http.(*conn).serve
	/usr/lib/golang/src/net/http/server.go:1847
runtime.goexit
	/usr/lib/golang/src/runtime/asm_amd64.s:1333
I0507 08:53:11.363774       1 util.go:37] deleting file: /scratch/tmpimage
E0507 08:53:11.365398       1 uploadserver.go:243] Saving stream failed: Image validation failed: Invalid json for image /scratch/tmpimage: unexpected end of JSON input
Comment 2 Alexander Wels 2019-05-07 15:19:52 UTC 
So it looks like everything worked up to the point where it was verifying the contents of the image. It failed there. I have seen that a few times on an import, and a retry solved it, obviously that doesn't work for upload.
Comment 4 Natalie Gavrielov 2019-05-12 14:31:50 UTC 
Well, the "How reproducible" field needs to be updated from 100% to "Not sure" (it's not consistent as I assumed) thus, changing severity from "urgent" to "high".
Comment 5 Ying Cui 2019-05-28 12:59:30 UTC 
Irit, is this the duplicate of bug 1683163? could you take a look?
Comment 6 Irit Goihman 2019-05-28 13:05:20 UTC 
It's a different bug.
Michael is this the PR with the fix https://github.com/kubevirt/containerized-data-importer/pull/807 ?
Comment 7 Michael Henriksen 2019-05-28 13:29:23 UTC 
No, that PR should not be related to this at all.
Comment 11 Adam Litke 2019-08-21 14:25:30 UTC 

*** This bug has been marked as a duplicate of bug 1725730 ***