Bug 1738442 - CDI import to hostpath PV (or via DV) sometimes failed due to permission denied while qemu converting
Summary: CDI import to hostpath PV (or via DV) sometimes failed due to permission deni...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Storage
Version: 1.4.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 1.4.1
Assignee: Alexander Wels
QA Contact: Qixuan Wang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-07 08:01 UTC by Qixuan Wang
Modified: 2019-10-15 11:31 UTC (History)
4 users (show)

Fixed In Version: hostpath-provisioner-container-v1.4.1-4
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-15 11:31:56 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Qixuan Wang 2019-08-07 08:01:40 UTC
Description of problem:
Given hostpath-provisioner is deployed in the cluster. CDI import a disk image with the PVC annotations sometimes failed:
qemu-img: /data/disk.img: error while converting raw: Could not create file: Permission denied


Version-Release number of selected component (if applicable):
brew-pulp.../container-native-virtualization/hostpath-provisioner:v1.4.1-2
CDI brew-pulp.../v1.4.1-1
CNV brew-pulp.../v1.4.1-3
openshift v3.11.104


How reproducible:
Not 100%


Steps to Reproduce:
1. Use hostpath to provision a PVC/DV for CDI import
$ cat hostpath-pvc-import.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: "hostpath-pvc-import"
  labels:
    app: containerized-data-importer
  annotations:
    kubevirt.io/provisionOnNode: cnv-executor-qwang-cnv14-node-a9e382-1.example.com
    cdi.kubevirt.io/storage.import.endpoint: "https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img"
    cdi.kubevirt.io/storage.import.secretName: "" # Optional. The name of the secret containing credentials for the data source
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: kubevirt-hostpath-provisioner


$ cat hostpath-dv-import.yaml
apiVersion: cdi.kubevirt.io/v1alpha1
kind: DataVolume
metadata:
  name: hostpath-dv-import
  annotations:
    kubevirt.io/provisionOnNode: cnv-executor-qwang-cnv14-node-a9e382-1.example.com
spec:
  source:
      http:
         url: "https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img"
  pvc:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 100Mi
storageClassName: kubevirt-hostpath-provisioner


2. Check PVC/DV status
[cloud-user@cnv-executor-qwang-cnv14-master-a9e382-1 ~]$ oc get pvc
NAME                  STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                    AGE
hostpath-pvc-import   Bound     pvc-2d33d367-b8e5-11e9-a860-fa163e026b79   1Gi        RWO            kubevirt-hostpath-provisioner   14s
hostpath-dv-import   Bound     pvc-d5fff92f-b8dd-11e9-a860-fa163e026b79   100Mi      RWO            kubevirt-hostpath-provisioner   1h


3. Verify import is finished


Actual results:
3. [cloud-user@cnv-executor-qwang-cnv14-master-a9e382-1 ~]$ oc get pod -n qwang87-1
NAME                                READY     STATUS             RESTARTS   AGE
importer-hostpath-pvc-import-dcbgb           0/1       CrashLoopBackOff   15         53m
importer-hostpath-dv-import-fqln4   0/1       CrashLoopBackOff   20         1h

3. [cloud-user@cnv-executor-qwang-cnv14-master-a9e382-1 ~]$ oc logs -f importer-hostpath-pvc-import-dcbgb
I0807 07:32:55.676032       1 importer.go:45] Starting importer
I0807 07:32:55.676317       1 importer.go:65] begin import process
I0807 07:32:55.676332       1 importer.go:91] begin import process
I0807 07:32:55.676342       1 dataStream.go:299] copying "https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img" to "/data/disk.img"...
I0807 07:32:56.072841       1 prlimit.go:107] ExecWithLimits qemu-img, [info --output=json https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img]
I0807 07:32:57.163874       1 prlimit.go:107] ExecWithLimits qemu-img, [convert -p -f qcow2 -O raw json: {"file.driver": "https", "file.url": "https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img", "file.timeout": 3600} /data/disk.img]
I0807 07:32:57.180370       1 qemu.go:192] 0.00
E0807 07:32:58.233953       1 prlimit.go:159] qemu-img [convert -p -f qcow2 -O raw json: {"file.driver": "https", "file.url": "https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img", "file.timeout": 3600} /data/disk.img] failed output is:
E0807 07:32:58.234858       1 prlimit.go:160]     (0.00/100%)qemu-img: /data/disk.img: error while converting raw: Could not create file: Permission denied
E0807 07:32:58.235157       1 importer.go:94] exit status 1
qemu-img execution failed
kubevirt.io/containerized-data-importer/pkg/system.ExecWithLimits
	/go/src/kubevirt.io/containerized-data-importer/pkg/system/prlimit.go:161
kubevirt.io/containerized-data-importer/pkg/image.(*qemuOperations).ConvertQcow2ToRawStream
	/go/src/kubevirt.io/containerized-data-importer/pkg/image/qemu.go:111
kubevirt.io/containerized-data-importer/pkg/importer.(*DataStream).copy
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:713
kubevirt.io/containerized-data-importer/pkg/importer.CopyData
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:316
main.main
	/go/src/kubevirt.io/containerized-data-importer/cmd/cdi-importer/importer.go:92
runtime.main
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/proc.go:198
runtime.goexit
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361
could not stream/convert qcow2 image to raw
kubevirt.io/containerized-data-importer/pkg/image.(*qemuOperations).ConvertQcow2ToRawStream
	/go/src/kubevirt.io/containerized-data-importer/pkg/image/qemu.go:114
kubevirt.io/containerized-data-importer/pkg/importer.(*DataStream).copy
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:713
kubevirt.io/containerized-data-importer/pkg/importer.CopyData
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:316
main.main
	/go/src/kubevirt.io/containerized-data-importer/cmd/cdi-importer/importer.go:92
runtime.main
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/proc.go:198
runtime.goexit
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361
Streaming qcow2 to raw conversion failed
kubevirt.io/containerized-data-importer/pkg/importer.(*DataStream).copy
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:715
kubevirt.io/containerized-data-importer/pkg/importer.CopyData
	/go/src/kubevirt.io/containerized-data-importer/pkg/importer/dataStream.go:316
main.main
	/go/src/kubevirt.io/containerized-data-importer/cmd/cdi-importer/importer.go:92
runtime.main
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/proc.go:198
runtime.goexit
	/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361


Expected results:
Successful conversion.


Additional info:

Comment 1 Alexander Wels 2019-08-14 14:05:32 UTC
@Qixan,

next time you have one these failures, can you leave it running and contact me so I can investigate the state of the nodes? I am completely unable to reproduce locally so having a reproducer would be nice.

Comment 2 Qixuan Wang 2019-08-15 05:11:53 UTC
It occurred several times when I started to use hostpath-provisioner v1.4.1-2. I can't reproduce it after updating to v1.4.1-4.

Comment 3 Alexander Wels 2019-08-15 11:25:35 UTC
Okay,

Lets leave this open for a bit, if after more testing you cannot reproduce in the latest version, we can close then.

Comment 4 Gowrishankar Rajaiyan 2019-08-18 04:38:27 UTC
(In reply to Qixuan Wang from comment #2)
> It occurred several times when I started to use hostpath-provisioner
> v1.4.1-2. I can't reproduce it after updating to v1.4.1-4.

Since it has not been reproduced even once in the latest v1.4.1-4 we should close this bug as 'CLOSED NOTABUG'.

Comment 5 Gowrishankar Rajaiyan 2019-08-19 13:04:50 UTC
Similar issue (Bug 1743248) has been reported in version 2.1

Comment 6 Alexander Wels 2019-08-21 14:33:21 UTC
That similar issue is not related. That issue was due to CDI setting a user on import. That has since been fixed. so I think this is back to needs QE to mark as not an issue.

Comment 7 Adam Litke 2019-09-03 19:15:14 UTC
In that case can you please either mark this as verified or close it?


Note You need to log in before you can comment on or make changes to this bug.