Description of problem: A virt-v2v test fails with: Original error from libvirt: internal error: process exited while connecting to monitor: 2020-01-16T13:27:42.500175Z qemu-system-x86_64: -blockdev {"node-name":"libvirt-4-format","read-only":true,"discard":"unmap","cache":{"dir ect":false,"no-flush":true},"driver":"vmdk","file":"libvirt-4-storage","backing" :null}: invalid VMDK image descriptor [code=1 int1=-1] Version-Release number of selected component (if applicable): libvirt from git (@ac67964e73) qemu-4.2.0-2.fc32.x86_64 How reproducible: 100% Steps to Reproduce: 1. Check out and compile virt-v2v from source. 2. Check out and compile libvirt from source. 3. In the libvirt source tree: ./run src/libvirtd 4. In the virt-v2v source tree: PATH=~/d/libvirt/build/tools:$PATH ~/d/libvirt/build/run make check
Test which fails is: https://github.com/libguestfs/virt-v2v/blob/master/tests/test-v2v-i-ova.sh
Created attachment 1652781 [details] test-v2v-i-ova.sh.log Very verbose test output.
Roughly what's happening here is that /home/rjones/d/virt-v2v/tests/test-v2v-i-ova.d/test-ova.ova is an uncompressed tar file. We want to efficiently read the VMDK disk out of this tar file without unpacking the tar file so we construct the following json: URL: json:{ "file": { "driver": "raw", "offset": 10752, "size": 4063232, "file": { "driver": "file", "filename": "/home/rjones/d/virt-v2v/tests/test-v2v-i-ova.d/test-ova.ova" } } } This is set as a backing file of /home/rjones/d/virt-v2v/tmp/v2vovl2c4505.qcow2 We then use libvirt to open /home/rjones/d/virt-v2v/tmp/v2vovl2c4505.qcow2 and that is what appears to be failing.
Oops, libvirt can't preserve the offset and size faithfully.
Patches posted upstream: https://www.redhat.com/archives/libvir-list/2020-February/msg00216.html
Version 3: https://www.redhat.com/archives/libvir-list/2020-February/msg00505.html
The following commits were pushed upstream to resolve this issue: e8a819e87f virStorageSourceParseBackingJSONRaw: Parse 'offset' and 'size' attributes 293e7750c9 tests: qemu: Add test data for the new <slice> element 0e644e6e47 qemu: Add support for slices of type 'storage' 9b804ef5ef tests: qemublock: Add cases for creating image overlays on top of disks with <slice> 73ca201467 qemu: block: Properly format storage slice into backing store strings f36d751fa6 qemu: domain: Store nodenames of slice in status XML bbf5d05cfd conf: Implement support for <slices> of disk source 44f0f76890 docs: Document the new <slices> sub-element of disk's <source> 8c43037688 qemu: block: forbid creation of storage sources with <slice> a6eeda986e qemuDomainValidateStorageSource: Reject unsupported slices c481881283 qemuBlockStorageSourceGetFormatRawProps: format 'offset' and 'size' for slice 6efa046165 util: virstoragefile: Add data structure for storing storage source slices 554ae62637 tests: virstorage: Add test data for json specified raw image with offset/size 4e93c47576 docs: formatdomain: Close <source> on one of disk examples 9fb7ccb3cf qemu: domain: Refactor formatting of node names into status XML 0d0d60ddc5 tests: virstorage: Add test cases for "json:" pseudo-URI without 'file' wrapper fd70f1b4d3 virStorageSourceParseBackingJSON: Prevent arbitrary nesting with format drivers f8e097570e virStorageSourceParseBackingJSON: Allow 'json:' pseudo URIs without 'file' wrapper 7e13ff8dc0 virStorageSourceJSONDriverParser: annotate 'format' drivers aadb34be34 virStorageSourceParseBackingJSON: Move deflattening of json: URIs out of recursion 4a6bc568cd virStorageSourceParseBackingJSON: Pass around original backing file string
Test on libvirt-6.0.0-9.module+el8.2.0+5957+7ae8988e.x86_64 qemu-kvm-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64 Steps: 1. Boot VM with <slices> settings 2. Hot-plug and hot-unplug 3. Compare the values of allocation|capacity|physical from domstats and domblkinfo 4. Insert or eject cdrom media with <slices> 5. Create snapshots with <slices> with --reuse-external 6. Blockcommit from active layer or inactive layer 7. Blockcopy with --reuse-external 8. Set blkiotune to disk with <slices>. And get the values set. 9. Use qcow2 format to redo the steps above Step3 domstats&domblkinfo fails: a. Create /tmp/disk-raw as 300M raw image, full allocation b. Start VM with disk like following <disk type="file" device="disk"> <driver name="qemu" type="raw" copy_on_read="on" cache="none" discard="ignore" detect_zeroes="on" io="native"/> <source file="/tmp/disk-raw"> <slices> <slice type="storage" offset="104857600" size="104857600"/> </slices> </source> <backingStore/> <target dev="sdb" bus="scsi"/> <alias name="ua-slices"/> <iotune> <total_bytes_sec>10000000</total_bytes_sec> <group_name>slice</group_name> </iotune> </disk> c. Check domstats and domblkinfo results # virsh domstats pc --block|grep -Ei 'block.1.(allocation|capacity|physical)' block.1.allocation=0 block.1.capacity=104857600 block.1.physical=314576896 # virsh domblkinfo pc sdb|grep -Ei '(allocation|capacity|physical)' Capacity: 104857600 Allocation: 314576896 Physical: 314572800 The values of Allocation are not the same. See the log disk-raw.xml-stats-blkinfo.log Step6 blockcommit fails: a. Prepare a snapshot xml with <slices>, prepare a 300M SNAP_FILE <domainsnapshot> <disks> <disk name="vda" snapshot="no"> </disk> <disk name="sdb" snapshot="external"> <driver type="qcow2"/> <source file="SNAP_FILE"> <slices> <slice type="storage" offset="104857600" size="104857600"/> </slices> </source> </disk> <disk name="sdc" snapshot="no"> </disk> </disks> </domainsnapshot> b. Create 3 snapshots: # virsh snapshot-create pc snap1.xml --no-metadata --disk-only --reuse-external # virsh snapshot-create pc snap2.xml --no-metadata --disk-only --reuse-external # virsh snapshot-create pc snap3.xml --no-metadata --disk-only --reuse-external Then the xml will be like: <disk type="file" device="disk"> <driver name="qemu" type="qcow2" cache="none" io="native" copy_on_read="on" discard="ignore" detect_zeroes="on"/> <source file="/tmp/snap3" index="8"> <slices> <slice type="storage" offset="104857600" size="104857600"/> </slices> </source> <backingStore type="file" index="7"> <format type="qcow2"/> <source file="/tmp/snap2"> <slices> <slice type="storage" offset="104857600" size="104857600"/> </slices> </source> <backingStore type="file" index="6"> <format type="qcow2"/> <source file="/tmp/snap1"> <slices> <slice type="storage" offset="104857600" size="104857600"/> </slices> </source> <backingStore type="file" index="3"> <format type="raw"/> <source file="/tmp/disk-raw"> <slices> <slice type="storage" offset="104857600" size="104857600"/> </slices> </source> <backingStore/> </backingStore> </backingStore> </backingStore> <target dev="sdb" bus="scsi"/> <iotune> <total_bytes_sec>10000000</total_bytes_sec> <group_name>slice</group_name> </iotune> <alias name="ua-slices"/> <address type="drive" controller="0" bus="0" target="0" unit="1"/> </disk> c. blockcommit from inactive layer and active layer # virsh blockcommit pc sdb --top /tmp/snap1 --shallow --wait --verbose Block commit: [100 %] # virsh blockcommit pc sdb --active --wait --verbose --pivot error: internal error: child reported (status=125): Requested operation is not valid: Setting different SELinux label on /tmp/disk-raw which is already in use See the log disk-raw.xml-blockcommit.log Step7 blockcopy fails on qcow2 image: a. Prepare the src and dest image at the size of 300M like following: |=== raw ===|=== qcow2 ===|=== raw ===| |<-- 100M ->|<-- 100M --->|<-- 100M ->| b. Prepare VM with disk xml: <disk type="file" device="disk"> <driver name="qemu" type="qcow2" copy_on_read="off" cache="none" discard="ignore" detect_zeroes="on" io="native"/> <source file="/tmp/disk-qcow2"> <slices> <slice type="storage" offset="104857600" size="104857600"/> </slices> </source> <backingStore/> <target dev="sdb" bus="scsi"/> <alias name="ua-slices"/> <iotune> <total_bytes_sec>10000000</total_bytes_sec> <group_name>slice</group_name> </iotune> </disk> Dest disk xml: <disk type="file" device="disk"> <driver name="qemu" type="qcow2" copy_on_read="off" cache="none" discard="ignore" detect_zeroes="on" io="native"/> <source file="/tmp/copy"> <slices> <slice type="storage" offset="104857600" size="104857600"/> </slices> </source> <backingStore/> <target dev="sdb" bus="scsi"/> <alias name="ua-slices"/> <iotune> <total_bytes_sec>10000000</total_bytes_sec> <group_name>slice</group_name> </iotune> </disk> c. Do blockcopy with --reuse-external # virsh blockcopy pc sdb --xml dest.xml --wait --verbose --pivot --reuse-external --transient-job Block Copy: [ 96 %]error: Copy failed See the log disk-qcow2.xml-blockcopy.log
Created attachment 1669163 [details] The logs and scripts of last comment
Peter, please check the 3 failures in comment11
(In reply to Han Han from comment #11) > Test on libvirt-6.0.0-9.module+el8.2.0+5957+7ae8988e.x86_64 > qemu-kvm-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64 > > Steps: > 1. Boot VM with <slices> settings > 2. Hot-plug and hot-unplug > 3. Compare the values of allocation|capacity|physical from domstats and > domblkinfo > 4. Insert or eject cdrom media with <slices> > 5. Create snapshots with <slices> with --reuse-external > 6. Blockcommit from active layer or inactive layer > 7. Blockcopy with --reuse-external > 8. Set blkiotune to disk with <slices>. And get the values set. > 9. Use qcow2 format to redo the steps above > > Step3 domstats&domblkinfo fails: > a. Create /tmp/disk-raw as 300M raw image, full allocation > b. Start VM with disk like following > <disk type="file" device="disk"> > <driver name="qemu" type="raw" copy_on_read="on" cache="none" > discard="ignore" detect_zeroes="on" io="native"/> > <source file="/tmp/disk-raw"> > <slices> > <slice type="storage" offset="104857600" size="104857600"/> > </slices> > </source> > <backingStore/> > <target dev="sdb" bus="scsi"/> > <alias name="ua-slices"/> > <iotune> > <total_bytes_sec>10000000</total_bytes_sec> > <group_name>slice</group_name> > </iotune> > </disk> > > c. Check domstats and domblkinfo results > # virsh domstats pc --block|grep -Ei 'block.1.(allocation|capacity|physical)' > block.1.allocation=0 > block.1.capacity=104857600 > block.1.physical=314576896 > # virsh domblkinfo pc sdb|grep -Ei '(allocation|capacity|physical)' > Capacity: 104857600 > Allocation: 314576896 > Physical: 314572800 > > The values of Allocation are not the same. > See the log disk-raw.xml-stats-blkinfo.log Did you write to the disk from the guest? The same happens to raw image without a slice and is based on the way we collect the stats for raw images: qemu-img create -f raw /path/to/image 20M Start VM. # virsh domstats upstream | grep allocation block.0.allocation=0 block.1.allocation=0 # virsh domblkinfo upstream vda Capacity: 20971520 Allocation: 4096 Physical: 20971520 Write 10MiB to the image from the guest: # virsh domstats upstream | grep allocation block.0.allocation=0 block.1.allocation=10485760 # virsh domblkinfo upstream vda Capacity: 20971520 Allocation: 10485760 Physical: 20971520 Note that 'block.0' is a cdrom so it's never written to. > Step6 blockcommit fails: [...] > c. blockcommit from inactive layer and active layer > # virsh blockcommit pc sdb --top /tmp/snap1 --shallow --wait --verbose > Block commit: [100 %] > > # virsh blockcommit pc sdb --active --wait --verbose --pivot > error: internal error: child reported (status=125): Requested operation is > not valid: Setting different SELinux label on /tmp/disk-raw which is already > in use > https://bugzilla.redhat.com/show_bug.cgi?id=1803551 > > Step7 blockcopy fails on qcow2 image: > a. Prepare the src and dest image at the size of 300M like following: > |=== raw ===|=== qcow2 ===|=== raw ===| > |<-- 100M ->|<-- 100M --->|<-- 100M ->| > > b. Prepare VM with disk xml: > <disk type="file" device="disk"> > <driver name="qemu" type="qcow2" copy_on_read="off" cache="none" > discard="ignore" detect_zeroes="on" io="native"/> > > <source file="/tmp/disk-qcow2"> > <slices> > <slice type="storage" offset="104857600" size="104857600"/> > </slices> > </source> > <backingStore/> > <target dev="sdb" bus="scsi"/> > <alias name="ua-slices"/> > <iotune> > <total_bytes_sec>10000000</total_bytes_sec> > <group_name>slice</group_name> > </iotune> > </disk> > Dest disk xml: > <disk type="file" device="disk"> > <driver name="qemu" type="qcow2" copy_on_read="off" cache="none" > discard="ignore" detect_zeroes="on" io="native"/> > > <source file="/tmp/copy"> > <slices> > <slice type="storage" offset="104857600" size="104857600"/> > </slices> > </source> > <backingStore/> > <target dev="sdb" bus="scsi"/> > <alias name="ua-slices"/> > <iotune> > <total_bytes_sec>10000000</total_bytes_sec> > <group_name>slice</group_name> > </iotune> > </disk> > > c. Do blockcopy with --reuse-external > # virsh blockcopy pc sdb --xml dest.xml --wait --verbose --pivot > --reuse-external --transient-job > Block Copy: [ 96 %]error: > > Copy failed This failed with: 2020-03-11 07:00:00.434+0000: 159506: debug : qemuMonitorJSONIOProcessLine:220 : Line [{"return": [{"current-progress": 103809024, "status": "concluded", "total-progress": 104857600, "type": "mirror", "id": "copy-sdb-libvirt-7-format", "error": "No space left on device"}], "id": "libvirt-420"}] The slice you've created is probably too small to hold the image + metadata overhead of qcow2. Try creating the copy without the slice, see how big the qcow2 is and then compare it with the size of the slice or re-try with a bigger destination image.
(In reply to Peter Krempa from comment #14) > (In reply to Han Han from comment #11) > > Test on libvirt-6.0.0-9.module+el8.2.0+5957+7ae8988e.x86_64 > > qemu-kvm-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64 > > > > Steps: > > 1. Boot VM with <slices> settings > > 2. Hot-plug and hot-unplug > > 3. Compare the values of allocation|capacity|physical from domstats and > > domblkinfo > > 4. Insert or eject cdrom media with <slices> > > 5. Create snapshots with <slices> with --reuse-external > > 6. Blockcommit from active layer or inactive layer > > 7. Blockcopy with --reuse-external > > 8. Set blkiotune to disk with <slices>. And get the values set. > > 9. Use qcow2 format to redo the steps above > > > > Step3 domstats&domblkinfo fails: > > a. Create /tmp/disk-raw as 300M raw image, full allocation > > b. Start VM with disk like following > > <disk type="file" device="disk"> > > <driver name="qemu" type="raw" copy_on_read="on" cache="none" > > discard="ignore" detect_zeroes="on" io="native"/> > > <source file="/tmp/disk-raw"> > > <slices> > > <slice type="storage" offset="104857600" size="104857600"/> > > </slices> > > </source> > > <backingStore/> > > <target dev="sdb" bus="scsi"/> > > <alias name="ua-slices"/> > > <iotune> > > <total_bytes_sec>10000000</total_bytes_sec> > > <group_name>slice</group_name> > > </iotune> > > </disk> > > > > c. Check domstats and domblkinfo results > > # virsh domstats pc --block|grep -Ei 'block.1.(allocation|capacity|physical)' > > block.1.allocation=0 > > block.1.capacity=104857600 > > block.1.physical=314576896 > > # virsh domblkinfo pc sdb|grep -Ei '(allocation|capacity|physical)' > > Capacity: 104857600 > > Allocation: 314576896 > > Physical: 314572800 > > > > The values of Allocation are not the same. > > See the log disk-raw.xml-stats-blkinfo.log > > Did you write to the disk from the guest? No > > The same happens to raw image without a slice and is based on the way we > collect the stats for raw images: > > qemu-img create -f raw /path/to/image 20M > > Start VM. > > # virsh domstats upstream | grep allocation > block.0.allocation=0 > block.1.allocation=0 > > # virsh domblkinfo upstream vda > Capacity: 20971520 > Allocation: 4096 > Physical: 20971520 > > Write 10MiB to the image from the guest: > > # virsh domstats upstream | grep allocation > block.0.allocation=0 > block.1.allocation=10485760 > > # virsh domblkinfo upstream vda > Capacity: 20971520 > Allocation: 10485760 > Physical: 20971520 > > Note that 'block.0' is a cdrom so it's never written to. I got the same results here. After write 10M to the disk in vm, the allocation values from domstats and domblkinfo become same. > > > Step6 blockcommit fails: > > [...] > > > c. blockcommit from inactive layer and active layer > > # virsh blockcommit pc sdb --top /tmp/snap1 --shallow --wait --verbose > > Block commit: [100 %] > > > > # virsh blockcommit pc sdb --active --wait --verbose --pivot > > error: internal error: child reported (status=125): Requested operation is > > not valid: Setting different SELinux label on /tmp/disk-raw which is already > > in use > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1803551 It passes on libvirt-6.0.0-10.module+el8.2.0+5984+dce93708.x86_64 > > > > > Step7 blockcopy fails on qcow2 image: > > a. Prepare the src and dest image at the size of 300M like following: > > |=== raw ===|=== qcow2 ===|=== raw ===| > > |<-- 100M ->|<-- 100M --->|<-- 100M ->| > > > > b. Prepare VM with disk xml: > > <disk type="file" device="disk"> > > <driver name="qemu" type="qcow2" copy_on_read="off" cache="none" > > discard="ignore" detect_zeroes="on" io="native"/> > > > > <source file="/tmp/disk-qcow2"> > > <slices> > > <slice type="storage" offset="104857600" size="104857600"/> > > </slices> > > </source> > > <backingStore/> > > <target dev="sdb" bus="scsi"/> > > <alias name="ua-slices"/> > > <iotune> > > <total_bytes_sec>10000000</total_bytes_sec> > > <group_name>slice</group_name> > > </iotune> > > </disk> > > Dest disk xml: > > <disk type="file" device="disk"> > > <driver name="qemu" type="qcow2" copy_on_read="off" cache="none" > > discard="ignore" detect_zeroes="on" io="native"/> > > > > <source file="/tmp/copy"> > > <slices> > > <slice type="storage" offset="104857600" size="104857600"/> > > </slices> > > </source> > > <backingStore/> > > <target dev="sdb" bus="scsi"/> > > <alias name="ua-slices"/> > > <iotune> > > <total_bytes_sec>10000000</total_bytes_sec> > > <group_name>slice</group_name> > > </iotune> > > </disk> > > > > c. Do blockcopy with --reuse-external > > # virsh blockcopy pc sdb --xml dest.xml --wait --verbose --pivot > > --reuse-external --transient-job > > Block Copy: [ 96 %]error: > > > > Copy failed > > This failed with: > > 2020-03-11 07:00:00.434+0000: 159506: debug : > qemuMonitorJSONIOProcessLine:220 : Line [{"return": [{"current-progress": > 103809024, "status": "concluded", "total-progress": 104857600, "type": > "mirror", "id": "copy-sdb-libvirt-7-format", "error": "No space left on > device"}], "id": "libvirt-420"}] > > The slice you've created is probably too small to hold the image + metadata > overhead of qcow2. Try creating the copy without the slice, see how big the > qcow2 is and then compare it with the size of the slice or re-try with a > bigger destination image. I will try that with bigger qcow2 image.
Created attachment 1669226 [details] The logs and scripts for comment11 The size for qcow2 format in comment11 is not right. It should contain the size of metadata and data. For a full allocated 100M virtual size qcow2 file, the size should be: ➜ ~ qemu-img create /tmp/b -f qcow2 100M -o preallocation=full Formatting '/tmp/b', fmt=qcow2 size=104857600 cluster_size=65536 preallocation=full lazy_refcounts=off refcount_bits=16 ➜ ~ du -b /tmp/b 105185280 /tmp/b Blockcopy on qcow2 passes after updating the size
Rich Jones, Could you please help to check if libvirt-6.0.0-10.module+el8.2.0+5984+dce93708.x86_64 works on the case of comment0 ? Some build dependencies are missing on my machine so I cannot run the `make check` for v2v
Created attachment 1669590 [details] test-v2v-i-ova.sh-with-libvirt-6.0.0-10.txt Yes it works with libvirt-6.0.0-10.module+el8.2.0+5984+dce93708.x86_64 The log from the test is attached.
According to the results from comment11 to comment18, mark it verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2017