Description of problem: Followed the description on document https://github.com/kubevirt/containerized-data-importer/blob/master/doc/exposing-upload-proxy.md#openshift. Used router wildcard certificate and generated hostname with re-encrypt route, then uploaded an image, I got "x509: certificate signed by unknown authority". So I had to add --insecure(virtctl) or -k(curl) to make upload successful. Seems https://github.com/kubevirt/kubevirt/blob/master/pkg/virtctl/imageupload/imageupload.go#L201 doesn't handle certificate verification except for skip. Version-Release number of selected component (if applicable): virt-cdi-uploadproxy v1.4.0-10 virt-cdi-uploadserver v1.4.0-10 How reproducible: 100% Steps to Reproduce: 1. Get tls.crt from cdi-upload-proxy-ca-key secret # oc get secret -n cdi cdi-upload-proxy-ca-key -o=jsonpath="{.data['tls\.crt']}" | base64 -d > tls.crt 2. Create a route with reencrypt temination # oc create route reencrypt -n cdi --service=cdi-uploadproxy --dest-ca-cert=tls.crt 3. Check the route # oc get route -n cdi 4. Upload an image using virtctl upload-image tool # virtctl image-upload --uploadproxy-url=https://$(oc get route cdi-uploadproxy -n cdi -o=jsonpath='{.status.ingress[0].host}') --pvc-name=upload-cirros-pvc-https-re --pvc-size=1Gi --image-path=cirros-0.4.0-x86_64-disk.img 5. Create a DataVolume and request an UploadToken, then upload an image using curl # curl -v -H "Authorization: Bearer $TOKEN" --data-binary @cirros-0.4.0-x86_64-disk.img https://cdi-uploadproxy-cdi.cloudapps.example.com/v1alpha1/upload Actual results: 3. [root@cnv-executor-qwang-master1 ~]# oc get route -n cdi NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD cdi-uploadproxy cdi-uploadproxy-cdi.cloudapps.example.com cdi-uploadproxy <all> reencrypt None cdi-uploadproxy-route cdi-uploadproxy-route-cdi.cloudapps.example.com cdi-uploadproxy <all> passthrough None 4. [root@cnv-executor-qwang-master1 ~]# virtctl image-upload --uploadproxy-url=https://$(oc get route cdi-uploadproxy -n cdi -o=jsonpath='{.status.ingress[0].host}') --pvc-name=upload-cirros-pvc-https-re --pvc-size=1Gi --image-path=cirros-0.4.0-x86_64-disk.img Using existing PVC qwang2/upload-cirros-pvc-https-re Uploading data to https://cdi-uploadproxy-cdi.cloudapps.example.com 0 B / 12.13 MiB [-----------------------------------------------------------------------------------------------------------------] 0.00% 0s Post https://cdi-uploadproxy-cdi.cloudapps.example.com/v1alpha1/upload: x509: certificate signed by unknown authority 5. [root@cnv-executor-qwang-master1 ~]# curl -v -H "Authorization: Bearer $TOKEN" --data-binary @cirros-0.4.0-x86_64-disk.img https://cdi-uploadproxy-cdi.cloudapps.example.com/v1alpha1/upload * About to connect() to cdi-uploadproxy-cdi.cloudapps.example.com port 443 (#0) * Trying 172.16.0.18... * Connected to cdi-uploadproxy-cdi.cloudapps.example.com (172.16.0.18) port 443 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * Server certificate: * subject: CN=*.cloudapps.example.com * start date: Mar 06 08:04:10 2019 GMT * expire date: Mar 05 08:04:11 2021 GMT * common name: *.cloudapps.example.com * issuer: CN=openshift-signer@1551858842 * NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER) * Peer's certificate issuer has been marked as not trusted by the user. * Closing connection 0 curl: (60) Peer's certificate issuer has been marked as not trusted by the user. More details here: http://curl.haxx.se/docs/sslcerts.html ..... Expected results: 4/5. Images should be upload successfully. Additional info:
Michael, can you take a look?
Hi, it would appear that the system running virtctl/curl was not configured to trust the Openshift router's default wildcard certificate (see *.cloudapps.example.com above). Is that true? If so, you can extract the wildcard cert by running 'oc get secret -n default router-certs -o=jsonpath="{.data['tls\.crt']}" | base64 -d > router.crt'. Then add that cert to the system trust store (drop the file in /etc/pki/ca-trust/source/anchors/ and run 'sudo update-ca-trust'). Be aware that the hostname virtctl/curl is connecting to must match the "*.cloudapps.example.com" domain. Otherwise you will have to do some dns configuration to avoid hostname validation failure.
Michael, yes, you are right. It worked for me. Shall we add the configuration to the document? [root@cnv-executor-qwang-master1 ~]# oc get secret -n default router-certs -o=jsonpath="{.data['tls\.crt']}" | base64 -d > router.crt [root@cnv-executor-qwang-master1 ~]# cp router.crt /etc/pki/ca-trust/source/anchors/ [root@cnv-executor-qwang-master1 ~]# update-ca-trust [root@cnv-executor-qwang-master1 ~]# oc get secret -n cdi cdi-upload-proxy-ca-key -o=jsonpath="{.data['tls\.crt']}" | base64 -d > tls.crt [root@cnv-executor-qwang-master1 ~]# oc create route reencrypt -n cdi --service=cdi-uploadproxy --dest-ca-cert=tls.crt route.route.openshift.io/cdi-uploadproxy created [root@cnv-executor-qwang-master1 ~]# virtctl image-upload --uploadproxy-url=https://$(oc get route cdi-uploadproxy -n cdi -o=jsonpath='{.status.ingress[0].host}') --pvc-size=1Gi --image-path=cirros-0.4.0-x86_64-disk.img --pvc-name=upload-reencrypt-1 PVC cdi/upload-reencrypt-1 created Waiting for PVC upload-reencrypt-1 upload pod to be running... Pod now running Uploading data to https://cdi-uploadproxy-cdi.cloudapps.example.com 12.13 MiB / 12.13 MiB [===========================================================================================================] 100.00% 0s Uploading cirros-0.4.0-x86_64-disk.img completed successfully
The reencrypt route is now created by default by the CDI operator. So there should be no need to document how to create the route. Adding the router wildcard cert to the system trust store is something that varies based on the system. And I don't see that topic covered in OpenShift docs anywhere. But I can add a note stating that the router wildcard cert should be added to the system CA trust store.
closing since the reported issue can be solved with a proper configuration.