Bug 1738442
| Summary: | CDI import to hostpath PV (or via DV) sometimes failed due to permission denied while qemu converting | ||
|---|---|---|---|
| Product: | Container Native Virtualization (CNV) | Reporter: | Qixuan Wang <qixuan.wang> |
| Component: | Storage | Assignee: | Alexander Wels <awels> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Qixuan Wang <qixuan.wang> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 1.4.1 | CC: | alitke, cnv-qe-bugs, grajaiya, ycui |
| Target Milestone: | --- | ||
| Target Release: | 1.4.1 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | hostpath-provisioner-container-v1.4.1-4 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-10-15 11:31:56 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
@Qixan, next time you have one these failures, can you leave it running and contact me so I can investigate the state of the nodes? I am completely unable to reproduce locally so having a reproducer would be nice. It occurred several times when I started to use hostpath-provisioner v1.4.1-2. I can't reproduce it after updating to v1.4.1-4. Okay, Lets leave this open for a bit, if after more testing you cannot reproduce in the latest version, we can close then. (In reply to Qixuan Wang from comment #2) > It occurred several times when I started to use hostpath-provisioner > v1.4.1-2. I can't reproduce it after updating to v1.4.1-4. Since it has not been reproduced even once in the latest v1.4.1-4 we should close this bug as 'CLOSED NOTABUG'. Similar issue (Bug 1743248) has been reported in version 2.1 That similar issue is not related. That issue was due to CDI setting a user on import. That has since been fixed. so I think this is back to needs QE to mark as not an issue. In that case can you please either mark this as verified or close it? |
Description of problem: Given hostpath-provisioner is deployed in the cluster. CDI import a disk image with the PVC annotations sometimes failed: qemu-img: /data/disk.img: error while converting raw: Could not create file: Permission denied Version-Release number of selected component (if applicable): brew-pulp.../container-native-virtualization/hostpath-provisioner:v1.4.1-2 CDI brew-pulp.../v1.4.1-1 CNV brew-pulp.../v1.4.1-3 openshift v3.11.104 How reproducible: Not 100% Steps to Reproduce: 1. Use hostpath to provision a PVC/DV for CDI import $ cat hostpath-pvc-import.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: "hostpath-pvc-import" labels: app: containerized-data-importer annotations: kubevirt.io/provisionOnNode: cnv-executor-qwang-cnv14-node-a9e382-1.example.com cdi.kubevirt.io/storage.import.endpoint: "https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img" cdi.kubevirt.io/storage.import.secretName: "" # Optional. The name of the secret containing credentials for the data source spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: kubevirt-hostpath-provisioner $ cat hostpath-dv-import.yaml apiVersion: cdi.kubevirt.io/v1alpha1 kind: DataVolume metadata: name: hostpath-dv-import annotations: kubevirt.io/provisionOnNode: cnv-executor-qwang-cnv14-node-a9e382-1.example.com spec: source: http: url: "https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img" pvc: accessModes: - ReadWriteOnce resources: requests: storage: 100Mi storageClassName: kubevirt-hostpath-provisioner 2. Check PVC/DV status [cloud-user@cnv-executor-qwang-cnv14-master-a9e382-1 ~]$ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE hostpath-pvc-import Bound pvc-2d33d367-b8e5-11e9-a860-fa163e026b79 1Gi RWO kubevirt-hostpath-provisioner 14s hostpath-dv-import Bound pvc-d5fff92f-b8dd-11e9-a860-fa163e026b79 100Mi RWO kubevirt-hostpath-provisioner 1h 3. Verify import is finished Actual results: 3. [cloud-user@cnv-executor-qwang-cnv14-master-a9e382-1 ~]$ oc get pod -n qwang87-1 NAME READY STATUS RESTARTS AGE importer-hostpath-pvc-import-dcbgb 0/1 CrashLoopBackOff 15 53m importer-hostpath-dv-import-fqln4 0/1 CrashLoopBackOff 20 1h 3. [cloud-user@cnv-executor-qwang-cnv14-master-a9e382-1 ~]$ oc logs -f importer-hostpath-pvc-import-dcbgb I0807 07:32:55.676032 1 importer.go:45] Starting importer I0807 07:32:55.676317 1 importer.go:65] begin import process I0807 07:32:55.676332 1 importer.go:91] begin import process I0807 07:32:55.676342 1 dataStream.go:299] copying "https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img" to "/data/disk.img"... I0807 07:32:56.072841 1 prlimit.go:107] ExecWithLimits qemu-img, [info --output=json https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img] I0807 07:32:57.163874 1 prlimit.go:107] ExecWithLimits qemu-img, [convert -p -f qcow2 -O raw json: {"file.driver": "https", "file.url": "https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img", "file.timeout": 3600} /data/disk.img] I0807 07:32:57.180370 1 qemu.go:192] 0.00 E0807 07:32:58.233953 1 prlimit.go:159] qemu-img [convert -p -f qcow2 -O raw json: {"file.driver": "https", "file.url": "https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img", "file.timeout": 3600} /data/disk.img] failed output is: E0807 07:32:58.234858 1 prlimit.go:160] (0.00/100%)qemu-img: /data/disk.img: error while converting raw: Could not create file: Permission denied E0807 07:32:58.235157 1 importer.go:94] exit status 1 qemu-img execution failed kubevirt.io/containerized-data-importer/pkg/system.ExecWithLimits /go/src/kubevirt.io/containerized-data-importer/pkg/system/prlimit.go:161 kubevirt.io/containerized-data-importer/pkg/image.(*qemuOperations).ConvertQcow2ToRawStream /go/src/kubevirt.io/containerized-data-importer/pkg/image/qemu.go:111 kubevirt.io/containerized-data-importer/pkg/importer.(*DataStream).copy /go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:713 kubevirt.io/containerized-data-importer/pkg/importer.CopyData /go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:316 main.main /go/src/kubevirt.io/containerized-data-importer/cmd/cdi-importer/importer.go:92 runtime.main /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/proc.go:198 runtime.goexit /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361 could not stream/convert qcow2 image to raw kubevirt.io/containerized-data-importer/pkg/image.(*qemuOperations).ConvertQcow2ToRawStream /go/src/kubevirt.io/containerized-data-importer/pkg/image/qemu.go:114 kubevirt.io/containerized-data-importer/pkg/importer.(*DataStream).copy /go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:713 kubevirt.io/containerized-data-importer/pkg/importer.CopyData /go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:316 main.main /go/src/kubevirt.io/containerized-data-importer/cmd/cdi-importer/importer.go:92 runtime.main /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/proc.go:198 runtime.goexit /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361 Streaming qcow2 to raw conversion failed kubevirt.io/containerized-data-importer/pkg/importer.(*DataStream).copy /go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:715 kubevirt.io/containerized-data-importer/pkg/importer.CopyData /go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:316 main.main /go/src/kubevirt.io/containerized-data-importer/cmd/cdi-importer/importer.go:92 runtime.main /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/proc.go:198 runtime.goexit /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361 Expected results: Successful conversion. Additional info: