Bug 1791788
Summary: | blockdev: libvirt doesn't preserve the 'offset' and 'size' properties of JSON specified backing files faithfully | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux Advanced Virtualization | Reporter: | Richard W.M. Jones <rjones> | ||||||||||
Component: | libvirt | Assignee: | Peter Krempa <pkrempa> | ||||||||||
Status: | CLOSED ERRATA | QA Contact: | Han Han <hhan> | ||||||||||
Severity: | unspecified | Docs Contact: | |||||||||||
Priority: | unspecified | ||||||||||||
Version: | 8.2 | CC: | dyuan, jdenemar, jsuchane, libvirt-maint, lmen, pkrempa, tburke, xuzhang | ||||||||||
Target Milestone: | rc | ||||||||||||
Target Release: | --- | ||||||||||||
Hardware: | Unspecified | ||||||||||||
OS: | Unspecified | ||||||||||||
Whiteboard: | |||||||||||||
Fixed In Version: | libvirt-6.0.0-6.el8 | Doc Type: | If docs needed, set a value | ||||||||||
Doc Text: | Story Points: | --- | |||||||||||
Clone Of: | Environment: | ||||||||||||
Last Closed: | 2020-05-05 09:55:54 UTC | Type: | Bug | ||||||||||
Regression: | --- | Mount Type: | --- | ||||||||||
Documentation: | --- | CRM: | |||||||||||
Verified Versions: | Category: | --- | |||||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
Embargoed: | |||||||||||||
Attachments: |
|
Description
Richard W.M. Jones
2020-01-16 13:32:54 UTC
Test which fails is: https://github.com/libguestfs/virt-v2v/blob/master/tests/test-v2v-i-ova.sh Created attachment 1652781 [details]
test-v2v-i-ova.sh.log
Very verbose test output.
Roughly what's happening here is that /home/rjones/d/virt-v2v/tests/test-v2v-i-ova.d/test-ova.ova is an uncompressed tar file. We want to efficiently read the VMDK disk out of this tar file without unpacking the tar file so we construct the following json: URL: json:{ "file": { "driver": "raw", "offset": 10752, "size": 4063232, "file": { "driver": "file", "filename": "/home/rjones/d/virt-v2v/tests/test-v2v-i-ova.d/test-ova.ova" } } } This is set as a backing file of /home/rjones/d/virt-v2v/tmp/v2vovl2c4505.qcow2 We then use libvirt to open /home/rjones/d/virt-v2v/tmp/v2vovl2c4505.qcow2 and that is what appears to be failing. Oops, libvirt can't preserve the offset and size faithfully. Patches posted upstream: https://www.redhat.com/archives/libvir-list/2020-February/msg00216.html The following commits were pushed upstream to resolve this issue: e8a819e87f virStorageSourceParseBackingJSONRaw: Parse 'offset' and 'size' attributes 293e7750c9 tests: qemu: Add test data for the new <slice> element 0e644e6e47 qemu: Add support for slices of type 'storage' 9b804ef5ef tests: qemublock: Add cases for creating image overlays on top of disks with <slice> 73ca201467 qemu: block: Properly format storage slice into backing store strings f36d751fa6 qemu: domain: Store nodenames of slice in status XML bbf5d05cfd conf: Implement support for <slices> of disk source 44f0f76890 docs: Document the new <slices> sub-element of disk's <source> 8c43037688 qemu: block: forbid creation of storage sources with <slice> a6eeda986e qemuDomainValidateStorageSource: Reject unsupported slices c481881283 qemuBlockStorageSourceGetFormatRawProps: format 'offset' and 'size' for slice 6efa046165 util: virstoragefile: Add data structure for storing storage source slices 554ae62637 tests: virstorage: Add test data for json specified raw image with offset/size 4e93c47576 docs: formatdomain: Close <source> on one of disk examples 9fb7ccb3cf qemu: domain: Refactor formatting of node names into status XML 0d0d60ddc5 tests: virstorage: Add test cases for "json:" pseudo-URI without 'file' wrapper fd70f1b4d3 virStorageSourceParseBackingJSON: Prevent arbitrary nesting with format drivers f8e097570e virStorageSourceParseBackingJSON: Allow 'json:' pseudo URIs without 'file' wrapper 7e13ff8dc0 virStorageSourceJSONDriverParser: annotate 'format' drivers aadb34be34 virStorageSourceParseBackingJSON: Move deflattening of json: URIs out of recursion 4a6bc568cd virStorageSourceParseBackingJSON: Pass around original backing file string Test on libvirt-6.0.0-9.module+el8.2.0+5957+7ae8988e.x86_64 qemu-kvm-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64 Steps: 1. Boot VM with <slices> settings 2. Hot-plug and hot-unplug 3. Compare the values of allocation|capacity|physical from domstats and domblkinfo 4. Insert or eject cdrom media with <slices> 5. Create snapshots with <slices> with --reuse-external 6. Blockcommit from active layer or inactive layer 7. Blockcopy with --reuse-external 8. Set blkiotune to disk with <slices>. And get the values set. 9. Use qcow2 format to redo the steps above Step3 domstats&domblkinfo fails: a. Create /tmp/disk-raw as 300M raw image, full allocation b. Start VM with disk like following <disk type="file" device="disk"> <driver name="qemu" type="raw" copy_on_read="on" cache="none" discard="ignore" detect_zeroes="on" io="native"/> <source file="/tmp/disk-raw"> <slices> <slice type="storage" offset="104857600" size="104857600"/> </slices> </source> <backingStore/> <target dev="sdb" bus="scsi"/> <alias name="ua-slices"/> <iotune> <total_bytes_sec>10000000</total_bytes_sec> <group_name>slice</group_name> </iotune> </disk> c. Check domstats and domblkinfo results # virsh domstats pc --block|grep -Ei 'block.1.(allocation|capacity|physical)' block.1.allocation=0 block.1.capacity=104857600 block.1.physical=314576896 # virsh domblkinfo pc sdb|grep -Ei '(allocation|capacity|physical)' Capacity: 104857600 Allocation: 314576896 Physical: 314572800 The values of Allocation are not the same. See the log disk-raw.xml-stats-blkinfo.log Step6 blockcommit fails: a. Prepare a snapshot xml with <slices>, prepare a 300M SNAP_FILE <domainsnapshot> <disks> <disk name="vda" snapshot="no"> </disk> <disk name="sdb" snapshot="external"> <driver type="qcow2"/> <source file="SNAP_FILE"> <slices> <slice type="storage" offset="104857600" size="104857600"/> </slices> </source> </disk> <disk name="sdc" snapshot="no"> </disk> </disks> </domainsnapshot> b. Create 3 snapshots: # virsh snapshot-create pc snap1.xml --no-metadata --disk-only --reuse-external # virsh snapshot-create pc snap2.xml --no-metadata --disk-only --reuse-external # virsh snapshot-create pc snap3.xml --no-metadata --disk-only --reuse-external Then the xml will be like: <disk type="file" device="disk"> <driver name="qemu" type="qcow2" cache="none" io="native" copy_on_read="on" discard="ignore" detect_zeroes="on"/> <source file="/tmp/snap3" index="8"> <slices> <slice type="storage" offset="104857600" size="104857600"/> </slices> </source> <backingStore type="file" index="7"> <format type="qcow2"/> <source file="/tmp/snap2"> <slices> <slice type="storage" offset="104857600" size="104857600"/> </slices> </source> <backingStore type="file" index="6"> <format type="qcow2"/> <source file="/tmp/snap1"> <slices> <slice type="storage" offset="104857600" size="104857600"/> </slices> </source> <backingStore type="file" index="3"> <format type="raw"/> <source file="/tmp/disk-raw"> <slices> <slice type="storage" offset="104857600" size="104857600"/> </slices> </source> <backingStore/> </backingStore> </backingStore> </backingStore> <target dev="sdb" bus="scsi"/> <iotune> <total_bytes_sec>10000000</total_bytes_sec> <group_name>slice</group_name> </iotune> <alias name="ua-slices"/> <address type="drive" controller="0" bus="0" target="0" unit="1"/> </disk> c. blockcommit from inactive layer and active layer # virsh blockcommit pc sdb --top /tmp/snap1 --shallow --wait --verbose Block commit: [100 %] # virsh blockcommit pc sdb --active --wait --verbose --pivot error: internal error: child reported (status=125): Requested operation is not valid: Setting different SELinux label on /tmp/disk-raw which is already in use See the log disk-raw.xml-blockcommit.log Step7 blockcopy fails on qcow2 image: a. Prepare the src and dest image at the size of 300M like following: |=== raw ===|=== qcow2 ===|=== raw ===| |<-- 100M ->|<-- 100M --->|<-- 100M ->| b. Prepare VM with disk xml: <disk type="file" device="disk"> <driver name="qemu" type="qcow2" copy_on_read="off" cache="none" discard="ignore" detect_zeroes="on" io="native"/> <source file="/tmp/disk-qcow2"> <slices> <slice type="storage" offset="104857600" size="104857600"/> </slices> </source> <backingStore/> <target dev="sdb" bus="scsi"/> <alias name="ua-slices"/> <iotune> <total_bytes_sec>10000000</total_bytes_sec> <group_name>slice</group_name> </iotune> </disk> Dest disk xml: <disk type="file" device="disk"> <driver name="qemu" type="qcow2" copy_on_read="off" cache="none" discard="ignore" detect_zeroes="on" io="native"/> <source file="/tmp/copy"> <slices> <slice type="storage" offset="104857600" size="104857600"/> </slices> </source> <backingStore/> <target dev="sdb" bus="scsi"/> <alias name="ua-slices"/> <iotune> <total_bytes_sec>10000000</total_bytes_sec> <group_name>slice</group_name> </iotune> </disk> c. Do blockcopy with --reuse-external # virsh blockcopy pc sdb --xml dest.xml --wait --verbose --pivot --reuse-external --transient-job Block Copy: [ 96 %]error: Copy failed See the log disk-qcow2.xml-blockcopy.log Created attachment 1669163 [details]
The logs and scripts of last comment
Peter, please check the 3 failures in comment11 (In reply to Han Han from comment #11) > Test on libvirt-6.0.0-9.module+el8.2.0+5957+7ae8988e.x86_64 > qemu-kvm-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64 > > Steps: > 1. Boot VM with <slices> settings > 2. Hot-plug and hot-unplug > 3. Compare the values of allocation|capacity|physical from domstats and > domblkinfo > 4. Insert or eject cdrom media with <slices> > 5. Create snapshots with <slices> with --reuse-external > 6. Blockcommit from active layer or inactive layer > 7. Blockcopy with --reuse-external > 8. Set blkiotune to disk with <slices>. And get the values set. > 9. Use qcow2 format to redo the steps above > > Step3 domstats&domblkinfo fails: > a. Create /tmp/disk-raw as 300M raw image, full allocation > b. Start VM with disk like following > <disk type="file" device="disk"> > <driver name="qemu" type="raw" copy_on_read="on" cache="none" > discard="ignore" detect_zeroes="on" io="native"/> > <source file="/tmp/disk-raw"> > <slices> > <slice type="storage" offset="104857600" size="104857600"/> > </slices> > </source> > <backingStore/> > <target dev="sdb" bus="scsi"/> > <alias name="ua-slices"/> > <iotune> > <total_bytes_sec>10000000</total_bytes_sec> > <group_name>slice</group_name> > </iotune> > </disk> > > c. Check domstats and domblkinfo results > # virsh domstats pc --block|grep -Ei 'block.1.(allocation|capacity|physical)' > block.1.allocation=0 > block.1.capacity=104857600 > block.1.physical=314576896 > # virsh domblkinfo pc sdb|grep -Ei '(allocation|capacity|physical)' > Capacity: 104857600 > Allocation: 314576896 > Physical: 314572800 > > The values of Allocation are not the same. > See the log disk-raw.xml-stats-blkinfo.log Did you write to the disk from the guest? The same happens to raw image without a slice and is based on the way we collect the stats for raw images: qemu-img create -f raw /path/to/image 20M Start VM. # virsh domstats upstream | grep allocation block.0.allocation=0 block.1.allocation=0 # virsh domblkinfo upstream vda Capacity: 20971520 Allocation: 4096 Physical: 20971520 Write 10MiB to the image from the guest: # virsh domstats upstream | grep allocation block.0.allocation=0 block.1.allocation=10485760 # virsh domblkinfo upstream vda Capacity: 20971520 Allocation: 10485760 Physical: 20971520 Note that 'block.0' is a cdrom so it's never written to. > Step6 blockcommit fails: [...] > c. blockcommit from inactive layer and active layer > # virsh blockcommit pc sdb --top /tmp/snap1 --shallow --wait --verbose > Block commit: [100 %] > > # virsh blockcommit pc sdb --active --wait --verbose --pivot > error: internal error: child reported (status=125): Requested operation is > not valid: Setting different SELinux label on /tmp/disk-raw which is already > in use > https://bugzilla.redhat.com/show_bug.cgi?id=1803551 > > Step7 blockcopy fails on qcow2 image: > a. Prepare the src and dest image at the size of 300M like following: > |=== raw ===|=== qcow2 ===|=== raw ===| > |<-- 100M ->|<-- 100M --->|<-- 100M ->| > > b. Prepare VM with disk xml: > <disk type="file" device="disk"> > <driver name="qemu" type="qcow2" copy_on_read="off" cache="none" > discard="ignore" detect_zeroes="on" io="native"/> > > <source file="/tmp/disk-qcow2"> > <slices> > <slice type="storage" offset="104857600" size="104857600"/> > </slices> > </source> > <backingStore/> > <target dev="sdb" bus="scsi"/> > <alias name="ua-slices"/> > <iotune> > <total_bytes_sec>10000000</total_bytes_sec> > <group_name>slice</group_name> > </iotune> > </disk> > Dest disk xml: > <disk type="file" device="disk"> > <driver name="qemu" type="qcow2" copy_on_read="off" cache="none" > discard="ignore" detect_zeroes="on" io="native"/> > > <source file="/tmp/copy"> > <slices> > <slice type="storage" offset="104857600" size="104857600"/> > </slices> > </source> > <backingStore/> > <target dev="sdb" bus="scsi"/> > <alias name="ua-slices"/> > <iotune> > <total_bytes_sec>10000000</total_bytes_sec> > <group_name>slice</group_name> > </iotune> > </disk> > > c. Do blockcopy with --reuse-external > # virsh blockcopy pc sdb --xml dest.xml --wait --verbose --pivot > --reuse-external --transient-job > Block Copy: [ 96 %]error: > > Copy failed This failed with: 2020-03-11 07:00:00.434+0000: 159506: debug : qemuMonitorJSONIOProcessLine:220 : Line [{"return": [{"current-progress": 103809024, "status": "concluded", "total-progress": 104857600, "type": "mirror", "id": "copy-sdb-libvirt-7-format", "error": "No space left on device"}], "id": "libvirt-420"}] The slice you've created is probably too small to hold the image + metadata overhead of qcow2. Try creating the copy without the slice, see how big the qcow2 is and then compare it with the size of the slice or re-try with a bigger destination image. (In reply to Peter Krempa from comment #14) > (In reply to Han Han from comment #11) > > Test on libvirt-6.0.0-9.module+el8.2.0+5957+7ae8988e.x86_64 > > qemu-kvm-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64 > > > > Steps: > > 1. Boot VM with <slices> settings > > 2. Hot-plug and hot-unplug > > 3. Compare the values of allocation|capacity|physical from domstats and > > domblkinfo > > 4. Insert or eject cdrom media with <slices> > > 5. Create snapshots with <slices> with --reuse-external > > 6. Blockcommit from active layer or inactive layer > > 7. Blockcopy with --reuse-external > > 8. Set blkiotune to disk with <slices>. And get the values set. > > 9. Use qcow2 format to redo the steps above > > > > Step3 domstats&domblkinfo fails: > > a. Create /tmp/disk-raw as 300M raw image, full allocation > > b. Start VM with disk like following > > <disk type="file" device="disk"> > > <driver name="qemu" type="raw" copy_on_read="on" cache="none" > > discard="ignore" detect_zeroes="on" io="native"/> > > <source file="/tmp/disk-raw"> > > <slices> > > <slice type="storage" offset="104857600" size="104857600"/> > > </slices> > > </source> > > <backingStore/> > > <target dev="sdb" bus="scsi"/> > > <alias name="ua-slices"/> > > <iotune> > > <total_bytes_sec>10000000</total_bytes_sec> > > <group_name>slice</group_name> > > </iotune> > > </disk> > > > > c. Check domstats and domblkinfo results > > # virsh domstats pc --block|grep -Ei 'block.1.(allocation|capacity|physical)' > > block.1.allocation=0 > > block.1.capacity=104857600 > > block.1.physical=314576896 > > # virsh domblkinfo pc sdb|grep -Ei '(allocation|capacity|physical)' > > Capacity: 104857600 > > Allocation: 314576896 > > Physical: 314572800 > > > > The values of Allocation are not the same. > > See the log disk-raw.xml-stats-blkinfo.log > > Did you write to the disk from the guest? No > > The same happens to raw image without a slice and is based on the way we > collect the stats for raw images: > > qemu-img create -f raw /path/to/image 20M > > Start VM. > > # virsh domstats upstream | grep allocation > block.0.allocation=0 > block.1.allocation=0 > > # virsh domblkinfo upstream vda > Capacity: 20971520 > Allocation: 4096 > Physical: 20971520 > > Write 10MiB to the image from the guest: > > # virsh domstats upstream | grep allocation > block.0.allocation=0 > block.1.allocation=10485760 > > # virsh domblkinfo upstream vda > Capacity: 20971520 > Allocation: 10485760 > Physical: 20971520 > > Note that 'block.0' is a cdrom so it's never written to. I got the same results here. After write 10M to the disk in vm, the allocation values from domstats and domblkinfo become same. > > > Step6 blockcommit fails: > > [...] > > > c. blockcommit from inactive layer and active layer > > # virsh blockcommit pc sdb --top /tmp/snap1 --shallow --wait --verbose > > Block commit: [100 %] > > > > # virsh blockcommit pc sdb --active --wait --verbose --pivot > > error: internal error: child reported (status=125): Requested operation is > > not valid: Setting different SELinux label on /tmp/disk-raw which is already > > in use > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1803551 It passes on libvirt-6.0.0-10.module+el8.2.0+5984+dce93708.x86_64 > > > > > Step7 blockcopy fails on qcow2 image: > > a. Prepare the src and dest image at the size of 300M like following: > > |=== raw ===|=== qcow2 ===|=== raw ===| > > |<-- 100M ->|<-- 100M --->|<-- 100M ->| > > > > b. Prepare VM with disk xml: > > <disk type="file" device="disk"> > > <driver name="qemu" type="qcow2" copy_on_read="off" cache="none" > > discard="ignore" detect_zeroes="on" io="native"/> > > > > <source file="/tmp/disk-qcow2"> > > <slices> > > <slice type="storage" offset="104857600" size="104857600"/> > > </slices> > > </source> > > <backingStore/> > > <target dev="sdb" bus="scsi"/> > > <alias name="ua-slices"/> > > <iotune> > > <total_bytes_sec>10000000</total_bytes_sec> > > <group_name>slice</group_name> > > </iotune> > > </disk> > > Dest disk xml: > > <disk type="file" device="disk"> > > <driver name="qemu" type="qcow2" copy_on_read="off" cache="none" > > discard="ignore" detect_zeroes="on" io="native"/> > > > > <source file="/tmp/copy"> > > <slices> > > <slice type="storage" offset="104857600" size="104857600"/> > > </slices> > > </source> > > <backingStore/> > > <target dev="sdb" bus="scsi"/> > > <alias name="ua-slices"/> > > <iotune> > > <total_bytes_sec>10000000</total_bytes_sec> > > <group_name>slice</group_name> > > </iotune> > > </disk> > > > > c. Do blockcopy with --reuse-external > > # virsh blockcopy pc sdb --xml dest.xml --wait --verbose --pivot > > --reuse-external --transient-job > > Block Copy: [ 96 %]error: > > > > Copy failed > > This failed with: > > 2020-03-11 07:00:00.434+0000: 159506: debug : > qemuMonitorJSONIOProcessLine:220 : Line [{"return": [{"current-progress": > 103809024, "status": "concluded", "total-progress": 104857600, "type": > "mirror", "id": "copy-sdb-libvirt-7-format", "error": "No space left on > device"}], "id": "libvirt-420"}] > > The slice you've created is probably too small to hold the image + metadata > overhead of qcow2. Try creating the copy without the slice, see how big the > qcow2 is and then compare it with the size of the slice or re-try with a > bigger destination image. I will try that with bigger qcow2 image. Created attachment 1669226 [details] The logs and scripts for comment11 The size for qcow2 format in comment11 is not right. It should contain the size of metadata and data. For a full allocated 100M virtual size qcow2 file, the size should be: ➜ ~ qemu-img create /tmp/b -f qcow2 100M -o preallocation=full Formatting '/tmp/b', fmt=qcow2 size=104857600 cluster_size=65536 preallocation=full lazy_refcounts=off refcount_bits=16 ➜ ~ du -b /tmp/b 105185280 /tmp/b Blockcopy on qcow2 passes after updating the size Rich Jones, Could you please help to check if libvirt-6.0.0-10.module+el8.2.0+5984+dce93708.x86_64 works on the case of comment0 ? Some build dependencies are missing on my machine so I cannot run the `make check` for v2v Created attachment 1669590 [details]
test-v2v-i-ova.sh-with-libvirt-6.0.0-10.txt
Yes it works with libvirt-6.0.0-10.module+el8.2.0+5984+dce93708.x86_64
The log from the test is attached.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2017 |