Bug 1658615
Summary: | Creating a DataVolume using a PVC as source results in non-sparsed image on the destination PVC | ||
---|---|---|---|
Product: | Container Native Virtualization (CNV) | Reporter: | Sergi Jimenez Romero <sjr> |
Component: | Storage | Assignee: | Adam Litke <alitke> |
Status: | CLOSED CURRENTRELEASE | QA Contact: | shiyang.wang <shiywang> |
Severity: | medium | Docs Contact: | |
Priority: | unspecified | ||
Version: | 1.3 | CC: | cnv-qe-bugs, ncredi, qixuan.wang, sjr, sreichar |
Target Milestone: | --- | ||
Target Release: | 1.4 | ||
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | v1.4.0-6 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2019-03-05 16:51:01 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Sergi Jimenez Romero
2018-12-12 14:18:28 UTC
I've run some tests locally after seeing the cloner script uses `tar cv` to send the file a named pipe and I think that's the problem here, please see the example below: * Test #1 - tar cv: ⏚ [sjr:~/Downloads] % qemu-img info Fedora-Cloud-Base-29-1.2.x86_64.raw image: Fedora-Cloud-Base-29-1.2.x86_64.raw file format: raw virtual size: 4.0G (4294967296 bytes) disk size: 778M ⏚ [sjr:~/Downloads] % mkfifo /tmp/testfifo ⏚ [sjr:~/Downloads] % tar cv Fedora-Cloud-Base-29-1.2.x86_64.raw > /tmp/testfifo Fedora-Cloud-Base-29-1.2.x86_64.raw * Results #1 - tar cv: ⏚ [sjr:~/Downloads/tmp] 4s 2 % cat /tmp/testfifo| tar xv Fedora-Cloud-Base-29-1.2.x86_64.raw ⏚ [sjr:~/Downloads/tmp] 10s % qemu-img info Fedora-Cloud-Base-29-1.2.x86_64.raw image: Fedora-Cloud-Base-29-1.2.x86_64.raw file format: raw virtual size: 4.0G (4294967296 bytes) disk size: 4.0G * Test #2 - tar cv --sparse: ⏚ [sjr:~/Downloads] 14s % tar cv --sparse Fedora-Cloud-Base-29-1.2.x86_64.raw > /tmp/testfifo Fedora-Cloud-Base-29-1.2.x86_64.raw * Results #2 - tar cv --sparse: ⏚ [sjr:~/Downloads/tmp2] % cat /tmp/testfifo| tar xv Fedora-Cloud-Base-29-1.2.x86_64.raw ⏚ [sjr:~/Downloads/tmp2] 11s % qemu-img info Fedora-Cloud-Base-29-1.2.x86_64.raw image: Fedora-Cloud-Base-29-1.2.x86_64.raw file format: raw virtual size: 4.0G (4294967296 bytes) disk size: 778M Looking at the GNU tar manual [0], it looks quite safe to add this parameter, I'd be willing to send the PR myself if it looks like the right solution, thoughts? [0] https://www.gnu.org/software/tar/manual/html_node/sparse.html *** Bug 1658614 has been marked as a duplicate of this bug. *** Yes. The --sparse option would be a good idea. I'd love it if you could submit the PR. Just posted the PR upstream: https://github.com/kubevirt/containerized-data-importer/pull/575 The PR https://github.com/kubevirt/containerized-data-importer/pull/575 has been merged. Thanks Sergi! Tested with openshift penshift v3.11.59, virt-cdi-importer:v1.4.0-6 1) I used the image you provided https://download.fedoraproject.org/pub/fedora/linux/releases/29/Cloud/x86_64/images/Fedora-Cloud-Base-29-1.2.x86_64.qcow2 file format: raw virtual size: 4.0G (4294967296 bytes) disk size: 707M and set PVC and DV storage to 1Gi. The importer finished without throwing storage limitation messages such as "no space", and it didn't resize the image to available space. But finally, no VM created. See: http://pastebin.test.redhat.com/706431 2) I also tried image http://fileshare.englab.nay.redhat.com/pub/libra/mnt/qa/scratch/OpenShift_QE/CleanImage/Fedora-Live-Workstation-x86_64-21-5.qcow2: file format: raw virtual size: 20G (21474836480 bytes) disk size: 889M The importer was in retry-loop, and I can't capture any storage limitation messages. Then I expand DataVolume to 21Gi(my cluster‘s upper limit). Re-created the DV generated PVC and I can see "no space" in the PVC event and then bound successfully, and the import progress went further. Unfortunately, it still in retry-loop and I never saw its finish. My questions: 1) Now the image is a sparse image, so during creation, it took up only as much actual disk space as the data contained (707M), 1Gi PVC is sufficient. In this sense, the bug has been fixed. 2) Perhaps it's an environmental issue, like https://bugzilla.redhat.com/show_bug.cgi?id=1662099 Make sense? Qixuan, the bug, at least when I reported it, it was affecting the cloner (image/process), but not the importer and the PR applies only to the cloner as well. Failures on the importer should not be related, I think. The importer, IIRC, was doing its job well when importing an sparse image, but the problem was on the cloning process as it was using tar without the --sparse option. Does that help? Thanks, Sergi, make sense. Tested with virt-cdi-importer:v1.4.0-8 I used test image https://dl.fedoraproject.org/pub/fedora/linux/releases/27/CloudImages/x86_64/images/Fedora-Cloud-Base-27-1.6.x86_64.qcow2 https://raw.githubusercontent.com/qwang1/cnv-test-file/master/golden-pvc.yaml https://raw.githubusercontent.com/qwang1/cnv-test-file/master/example-clone-dv.yaml [root@cnv-executor-qwang-1-master1 ~]# qemu-img info Fedora-Cloud-Base-27-1.6.x86_64.qcow2 image: Fedora-Cloud-Base-27-1.6.x86_64.qcow2 file format: qcow2 virtual size: 4.0G (4294967296 bytes) disk size: 222M cluster_size: 65536 Format specific information: compat: 0.10 CDI importer behavior changed recently. It checks if target size < virtual size, reject import. Here the qcow2 image was 222M (ActualSize), VM read 4.0Gi (VirtualSize), correct? @awels request.storage should >= 4.0G. Otherwise, I would get an error message: "Virtual image size 4294967296 is larger than available size 1018417152, shrink not yet supported." So I set requests.storage to 5Gi on golden-pvc.yaml. After importing completed, I used example-clone-dv.yaml whose requests.storage was 1Gi. If the image was sparsed, 1Gi would be sufficient. [root@cnv-executor-qwang-1-master1 ~]# oc get dv NAME AGE example-clone-dv 3s [root@cnv-executor-qwang-1-master1 ~]# oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE example-clone-dv Bound pvc-1f1532f1-30f6-11e9-8e8c-fa163eecabce 1Gi RWO glusterfs-storage 15s golden-pvc Bound pvc-8b0af6ed-30ef-11e9-8e8c-fa163eecabce 5Gi RWO glusterfs-storage 47m [root@cnv-executor-qwang-1-master1 ~]# oc get all NAME READY STATUS RESTARTS AGE pod/clone-source-pod-s96bw 1/1 Running 0 10s pod/clone-target-pod-rflmq 1/1 Running 0 10s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/glusterfs-dynamic-1f1532f1-30f6-11e9-8e8c-fa163eecabce ClusterIP 172.30.93.216 <none> 1/TCP 10s service/glusterfs-dynamic-8b0af6ed-30ef-11e9-8e8c-fa163eecabce ClusterIP 172.30.104.122 <none> 1/TCP 47m [root@cnv-executor-qwang-1-master1 ~]# oc logs -f pod/clone-source-pod-7g77q cloner: Starting clone source cloner: creating fifo pipe cloner: creating tarball of the image and redirecting it to /tmp/clone/socket/65c1e0fd-30f8-11e9-8e8c-fa163eecabce/pipe /tmp/clone/image / 5270437888 ./ ./disk.img / cloner: finished writing image to /tmp/clone/socket/65c1e0fd-30f8-11e9-8e8c-fa163eecabce/pipe [root@cnv-executor-qwang-1-master1 ~]# oc logs -f pod/clone-target-pod-tl7sf cloner: Starting clone target cloner: check if the fifo pipe was created by the cloning source pod cloner: 0: fifo pipe has not been created by the source pod. Waiting 3 seconds before checking again... cloner: check if the fifo pipe was created by the cloning source pod /tmp/clone/image / cloner: extract the image from /tmp/clone/socket/65c1e0fd-30f8-11e9-8e8c-fa163eecabce/pipe into /tmp/clone/image directory I0215 08:05:26.340401 8 clonertarget.go:53] Starting cloner target I0215 08:05:27.287010 8 clonertarget.go:102] Reading total size I0215 08:05:27.287139 8 clonertarget.go:144] total size: 5270437888 I0215 08:05:27.288033 8 util.go:97] begin untar... I0215 08:05:28.288366 8 clonertarget.go:127] 0.00 I0215 08:05:29.288524 8 clonertarget.go:127] 0.00 I0215 08:05:30.288685 8 clonertarget.go:127] 0.00 I0215 08:05:31.288801 8 clonertarget.go:127] 0.00 I0215 08:05:32.288994 8 clonertarget.go:127] 0.00 I0215 08:05:33.289126 8 clonertarget.go:127] 0.00 I0215 08:05:34.289256 8 clonertarget.go:127] 0.00 I0215 08:05:35.289385 8 clonertarget.go:127] 0.00 I0215 08:05:36.289516 8 clonertarget.go:127] 0.00 I0215 08:05:37.289750 8 clonertarget.go:127] 0.00 I0215 08:05:38.289916 8 clonertarget.go:127] 0.00 I0215 08:05:39.290044 8 clonertarget.go:127] 0.00 I0215 08:05:40.290213 8 clonertarget.go:127] 0.00 I0215 08:05:41.290428 8 clonertarget.go:127] 0.00 I0215 08:05:42.290567 8 clonertarget.go:127] 0.00 I0215 08:05:43.290787 8 clonertarget.go:127] 0.00 I0215 08:05:44.290949 8 clonertarget.go:127] 0.00 I0215 08:05:45.291809 8 clonertarget.go:127] 0.00 I0215 08:05:46.291948 8 clonertarget.go:127] 0.00 I0215 08:05:47.292076 8 clonertarget.go:127] 0.00 I0215 08:05:48.292229 8 clonertarget.go:127] 0.00 I0215 08:05:49.292358 8 clonertarget.go:127] 0.00 I0215 08:05:50.292484 8 clonertarget.go:127] 0.00 I0215 08:05:51.292628 8 clonertarget.go:127] 0.10 I0215 08:05:52.292795 8 clonertarget.go:127] 0.23 I0215 08:05:53.292930 8 clonertarget.go:127] 0.40 I0215 08:05:54.293059 8 clonertarget.go:127] 0.57 I0215 08:05:55.293186 8 clonertarget.go:127] 0.74 I0215 08:05:56.293332 8 clonertarget.go:127] 0.92 I0215 08:05:57.293458 8 clonertarget.go:127] 1.12 I0215 08:05:58.293610 8 clonertarget.go:127] 1.28 I0215 08:05:59.293786 8 clonertarget.go:127] 1.45 I0215 08:06:00.293934 8 clonertarget.go:127] 1.61 I0215 08:06:01.294053 8 clonertarget.go:127] 1.77 I0215 08:06:02.294189 8 clonertarget.go:127] 1.92 I0215 08:06:03.294343 8 clonertarget.go:127] 2.13 I0215 08:06:04.294495 8 clonertarget.go:127] 2.33 I0215 08:06:05.294619 8 clonertarget.go:127] 2.52 I0215 08:06:06.294803 8 clonertarget.go:127] 2.71 I0215 08:06:07.294936 8 clonertarget.go:127] 2.86 I0215 08:06:08.295104 8 clonertarget.go:127] 3.00 I0215 08:06:09.295264 8 clonertarget.go:127] 3.12 I0215 08:06:10.295421 8 clonertarget.go:127] 3.25 I0215 08:06:11.295574 8 clonertarget.go:127] 3.40 I0215 08:06:12.295701 8 clonertarget.go:127] 3.60 I0215 08:06:13.295844 8 clonertarget.go:127] 3.72 I0215 08:06:14.295970 8 clonertarget.go:127] 3.88 I0215 08:06:15.296087 8 clonertarget.go:127] 4.00 I0215 08:06:16.296229 8 clonertarget.go:127] 4.19 I0215 08:06:17.296390 8 clonertarget.go:127] 4.34 I0215 08:06:18.296562 8 clonertarget.go:127] 4.34 I0215 08:06:19.296753 8 clonertarget.go:127] 4.52 I0215 08:06:20.296888 8 clonertarget.go:127] 4.73 I0215 08:06:21.297016 8 clonertarget.go:127] 4.91 I0215 08:06:22.297140 8 clonertarget.go:127] 5.09 I0215 08:06:23.297319 8 clonertarget.go:127] 5.24 I0215 08:06:24.297464 8 clonertarget.go:127] 5.26 I0215 08:06:25.297609 8 clonertarget.go:127] 5.45 I0215 08:06:26.297755 8 clonertarget.go:127] 5.63 I0215 08:06:27.297881 8 clonertarget.go:127] 5.81 I0215 08:06:28.298059 8 clonertarget.go:127] 5.98 I0215 08:06:29.298227 8 clonertarget.go:127] 6.18 I0215 08:06:30.298361 8 clonertarget.go:127] 6.37 I0215 08:06:31.298494 8 clonertarget.go:127] 6.55 I0215 08:06:32.298623 8 clonertarget.go:127] 6.73 I0215 08:06:33.298760 8 clonertarget.go:127] 6.90 I0215 08:06:34.298907 8 clonertarget.go:127] 7.08 I0215 08:06:35.299053 8 clonertarget.go:127] 7.26 I0215 08:06:36.299179 8 clonertarget.go:127] 7.44 I0215 08:06:37.299345 8 clonertarget.go:127] 7.61 I0215 08:06:38.299489 8 clonertarget.go:127] 7.80 I0215 08:06:39.299967 8 clonertarget.go:127] 7.99 I0215 08:06:40.300124 8 clonertarget.go:127] 8.19 I0215 08:06:41.300232 8 clonertarget.go:127] 8.40 I0215 08:06:42.300401 8 clonertarget.go:127] 8.61 I0215 08:06:43.300600 8 clonertarget.go:127] 8.82 I0215 08:06:44.300834 8 clonertarget.go:127] 9.03 I0215 08:06:45.300981 8 clonertarget.go:127] 9.22 I0215 08:06:46.301105 8 clonertarget.go:127] 9.39 I0215 08:06:47.301823 8 clonertarget.go:127] 9.56 I0215 08:06:48.301949 8 clonertarget.go:127] 9.73 I0215 08:06:49.302086 8 clonertarget.go:127] 9.92 I0215 08:06:50.302248 8 clonertarget.go:127] 10.09 I0215 08:06:51.302403 8 clonertarget.go:127] 10.24 I0215 08:06:52.302536 8 clonertarget.go:127] 10.24 I0215 08:06:53.302725 8 clonertarget.go:127] 10.24 I0215 08:06:54.302851 8 clonertarget.go:127] 10.24 I0215 08:06:54.384040 8 clonertarget.go:98] clone complete / cloner: finished cloning image from /tmp/clone/socket/65c1e0fd-30f8-11e9-8e8c-fa163eecabce/pipe to /tmp/clone/image If the clone image was a non-sparse image, 1Gi couldn't satisfy it, so I think this bug has been fixed. Meanwhile, some questions came up: 1) The image was imported to PVC, where can I check it? Searched "disk.img" on all of my nodes and got nothing. 2) The above clone progress was truly complete? It was wired that jumped from 10% to complete immediately. 3) Where was the cloned image? There was not a directory called /tmp/clone/image. |