Bug 1791788

Test which fails is:

https://github.com/libguestfs/virt-v2v/blob/master/tests/test-v2v-i-ova.sh

Created attachment 1652781 [details]
test-v2v-i-ova.sh.log

Very verbose test output.

Roughly what's happening here is that /home/rjones/d/virt-v2v/tests/test-v2v-i-ova.d/test-ova.ova
is an uncompressed tar file.  We want to efficiently read the VMDK disk out of this
tar file without unpacking the tar file so we construct the following json: URL:

        json:{ "file": { "driver": "raw", "offset": 10752, "size": 4063232, "file": { "driver": "file", "filename": "/home/rjones/d/virt-v2v/tests/test-v2v-i-ova.d/test-ova.ova" } } }

This is set as a backing file of /home/rjones/d/virt-v2v/tmp/v2vovl2c4505.qcow2

We then use libvirt to open /home/rjones/d/virt-v2v/tmp/v2vovl2c4505.qcow2
and that is what appears to be failing.

Oops, libvirt can't preserve the offset and size faithfully.

Patches posted upstream:

https://www.redhat.com/archives/libvir-list/2020-February/msg00216.html

Version 3:

https://www.redhat.com/archives/libvir-list/2020-February/msg00505.html

The following commits were pushed upstream to resolve this issue:

e8a819e87f virStorageSourceParseBackingJSONRaw: Parse 'offset' and 'size' attributes
293e7750c9 tests: qemu: Add test data for the new <slice> element
0e644e6e47 qemu: Add support for slices of type 'storage'
9b804ef5ef tests: qemublock: Add cases for creating image overlays on top of disks with <slice>
73ca201467 qemu: block: Properly format storage slice into backing store strings
f36d751fa6 qemu: domain: Store nodenames of slice in status XML
bbf5d05cfd conf: Implement support for <slices> of disk source
44f0f76890 docs: Document the new <slices> sub-element of disk's <source>
8c43037688 qemu: block: forbid creation of storage sources with <slice>
a6eeda986e qemuDomainValidateStorageSource: Reject unsupported slices
c481881283 qemuBlockStorageSourceGetFormatRawProps: format 'offset' and 'size' for slice
6efa046165 util: virstoragefile: Add data structure for storing storage source slices
554ae62637 tests: virstorage: Add test data for json specified raw image with offset/size
4e93c47576 docs: formatdomain: Close <source> on one of disk examples
9fb7ccb3cf qemu: domain: Refactor formatting of node names into status XML
0d0d60ddc5 tests: virstorage: Add test cases for "json:" pseudo-URI without 'file' wrapper
fd70f1b4d3 virStorageSourceParseBackingJSON: Prevent arbitrary nesting with format drivers
f8e097570e virStorageSourceParseBackingJSON: Allow 'json:' pseudo URIs without 'file' wrapper
7e13ff8dc0 virStorageSourceJSONDriverParser: annotate 'format' drivers
aadb34be34 virStorageSourceParseBackingJSON: Move deflattening of json: URIs out of recursion
4a6bc568cd virStorageSourceParseBackingJSON: Pass around original backing file string

Test on libvirt-6.0.0-9.module+el8.2.0+5957+7ae8988e.x86_64 qemu-kvm-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64

Steps:
1. Boot VM with <slices> settings
2. Hot-plug and hot-unplug
3. Compare the values of allocation|capacity|physical  from domstats and domblkinfo
4. Insert or eject cdrom media with <slices>
5. Create snapshots with <slices> with --reuse-external
6. Blockcommit from active layer or inactive layer
7. Blockcopy with --reuse-external
8. Set blkiotune to disk with <slices>. And get the values set.
9. Use qcow2 format to redo the steps above

Step3 domstats&domblkinfo fails:
a. Create /tmp/disk-raw as 300M raw image, full allocation
b. Start VM with disk like following
<disk type="file" device="disk">
      <driver name="qemu" type="raw" copy_on_read="on" cache="none" discard="ignore" detect_zeroes="on" io="native"/>
      <source file="/tmp/disk-raw">
        <slices>
          <slice type="storage" offset="104857600" size="104857600"/>
        </slices>
      </source>
      <backingStore/>
      <target dev="sdb" bus="scsi"/>
      <alias name="ua-slices"/>
      <iotune>
               <total_bytes_sec>10000000</total_bytes_sec>
              <group_name>slice</group_name>
           </iotune>
    </disk>

c. Check domstats and domblkinfo results
# virsh domstats pc --block|grep -Ei 'block.1.(allocation|capacity|physical)'
  block.1.allocation=0
  block.1.capacity=104857600
  block.1.physical=314576896
# virsh domblkinfo pc sdb|grep -Ei '(allocation|capacity|physical)'
Capacity:       104857600
Allocation:     314576896
Physical:       314572800

The values of Allocation are not the same.
See the log disk-raw.xml-stats-blkinfo.log


Step6 blockcommit fails:
a. Prepare a snapshot xml with <slices>, prepare a 300M SNAP_FILE
<domainsnapshot>
  <disks>
    <disk name="vda" snapshot="no">
    </disk>
    <disk name="sdb" snapshot="external">
      <driver type="qcow2"/>
      <source file="SNAP_FILE">
        <slices>
          <slice type="storage" offset="104857600" size="104857600"/>
        </slices>
      </source>
    </disk>
    <disk name="sdc" snapshot="no">
    </disk>
  </disks>
</domainsnapshot>

b. Create 3 snapshots:
# virsh snapshot-create pc snap1.xml --no-metadata --disk-only --reuse-external
# virsh snapshot-create pc snap2.xml --no-metadata --disk-only --reuse-external
# virsh snapshot-create pc snap3.xml --no-metadata --disk-only --reuse-external
Then the xml will be like:
<disk type="file" device="disk">                                                                                                                                                                        
      <driver name="qemu" type="qcow2" cache="none" io="native" copy_on_read="on" discard="ignore" detect_zeroes="on"/>                                                                                            
      <source file="/tmp/snap3" index="8">
        <slices>
          <slice type="storage" offset="104857600" size="104857600"/>
        </slices>
      </source>
      <backingStore type="file" index="7">
        <format type="qcow2"/>
        <source file="/tmp/snap2">
          <slices>
            <slice type="storage" offset="104857600" size="104857600"/>
          </slices>
        </source>
        <backingStore type="file" index="6">
          <format type="qcow2"/>
          <source file="/tmp/snap1">
            <slices>
              <slice type="storage" offset="104857600" size="104857600"/>
            </slices>
          </source>
          <backingStore type="file" index="3">
            <format type="raw"/>
            <source file="/tmp/disk-raw">
              <slices>
                <slice type="storage" offset="104857600" size="104857600"/>
              </slices>
            </source>
            <backingStore/>
          </backingStore>
        </backingStore>
      </backingStore>
      <target dev="sdb" bus="scsi"/>
      <iotune>
        <total_bytes_sec>10000000</total_bytes_sec>
        <group_name>slice</group_name>
      </iotune>
      <alias name="ua-slices"/>
      <address type="drive" controller="0" bus="0" target="0" unit="1"/>
    </disk>

c. blockcommit from inactive layer and active layer
# virsh blockcommit pc sdb --top /tmp/snap1 --shallow --wait --verbose
Block commit: [100 %]

# virsh blockcommit pc sdb --active --wait --verbose --pivot
error: internal error: child reported (status=125): Requested operation is not valid: Setting different SELinux label on /tmp/disk-raw which is already in use

See the log disk-raw.xml-blockcommit.log


Step7 blockcopy fails on qcow2 image:
a. Prepare the src and dest image at the size of 300M like following:
|=== raw ===|=== qcow2 ===|=== raw ===|
|<-- 100M ->|<-- 100M --->|<-- 100M ->|

b. Prepare VM with disk xml:
<disk type="file" device="disk">
      <driver name="qemu" type="qcow2" copy_on_read="off" cache="none" discard="ignore" detect_zeroes="on" io="native"/>                                                                                          
      <source file="/tmp/disk-qcow2">
        <slices>
          <slice type="storage" offset="104857600" size="104857600"/>
        </slices>
      </source>
      <backingStore/>
      <target dev="sdb" bus="scsi"/>
      <alias name="ua-slices"/>
      <iotune>
               <total_bytes_sec>10000000</total_bytes_sec>
              <group_name>slice</group_name>
           </iotune>
    </disk>
Dest disk xml:
<disk type="file" device="disk">
      <driver name="qemu" type="qcow2" copy_on_read="off" cache="none" discard="ignore" detect_zeroes="on" io="native"/>                                                                                          
      <source file="/tmp/copy">
        <slices>
          <slice type="storage" offset="104857600" size="104857600"/>
        </slices>
      </source>
      <backingStore/>
      <target dev="sdb" bus="scsi"/>
      <alias name="ua-slices"/>
      <iotune>
               <total_bytes_sec>10000000</total_bytes_sec>
              <group_name>slice</group_name>
           </iotune>
    </disk>

c. Do blockcopy with --reuse-external
# virsh blockcopy pc sdb --xml dest.xml --wait --verbose --pivot --reuse-external --transient-job
Block Copy: [ 96 %]error:                                                                                                                                                                                          
Copy failed  

See the log disk-qcow2.xml-blockcopy.log

Created attachment 1669163 [details]
The logs and scripts of last comment

Peter, please check the 3 failures in comment11

(In reply to Han Han from comment #11)
> Test on libvirt-6.0.0-9.module+el8.2.0+5957+7ae8988e.x86_64
> qemu-kvm-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64
> 
> Steps:
> 1. Boot VM with <slices> settings
> 2. Hot-plug and hot-unplug
> 3. Compare the values of allocation|capacity|physical  from domstats and
> domblkinfo
> 4. Insert or eject cdrom media with <slices>
> 5. Create snapshots with <slices> with --reuse-external
> 6. Blockcommit from active layer or inactive layer
> 7. Blockcopy with --reuse-external
> 8. Set blkiotune to disk with <slices>. And get the values set.
> 9. Use qcow2 format to redo the steps above
> 
> Step3 domstats&domblkinfo fails:
> a. Create /tmp/disk-raw as 300M raw image, full allocation
> b. Start VM with disk like following
> <disk type="file" device="disk">
>       <driver name="qemu" type="raw" copy_on_read="on" cache="none"
> discard="ignore" detect_zeroes="on" io="native"/>
>       <source file="/tmp/disk-raw">
>         <slices>
>           <slice type="storage" offset="104857600" size="104857600"/>
>         </slices>
>       </source>
>       <backingStore/>
>       <target dev="sdb" bus="scsi"/>
>       <alias name="ua-slices"/>
>       <iotune>
>                <total_bytes_sec>10000000</total_bytes_sec>
>               <group_name>slice</group_name>
>            </iotune>
>     </disk>
> 
> c. Check domstats and domblkinfo results
> # virsh domstats pc --block|grep -Ei 'block.1.(allocation|capacity|physical)'
>   block.1.allocation=0
>   block.1.capacity=104857600
>   block.1.physical=314576896
> # virsh domblkinfo pc sdb|grep -Ei '(allocation|capacity|physical)'
> Capacity:       104857600
> Allocation:     314576896
> Physical:       314572800
> 
> The values of Allocation are not the same.
> See the log disk-raw.xml-stats-blkinfo.log

Did you write to the disk from the guest?

The same happens to raw image without a slice and is based on the way we collect the stats for raw images:

qemu-img create -f raw /path/to/image 20M

Start VM.

# virsh domstats upstream | grep allocation
  block.0.allocation=0
  block.1.allocation=0

# virsh domblkinfo upstream vda
Capacity:       20971520
Allocation:     4096
Physical:       20971520

Write 10MiB to the image from the guest:

# virsh domstats upstream | grep allocation
  block.0.allocation=0
  block.1.allocation=10485760

# virsh domblkinfo upstream vda
Capacity:       20971520
Allocation:     10485760
Physical:       20971520

Note that 'block.0' is a cdrom so it's never written to.

> Step6 blockcommit fails:

[...]

> c. blockcommit from inactive layer and active layer
> # virsh blockcommit pc sdb --top /tmp/snap1 --shallow --wait --verbose
> Block commit: [100 %]
> 
> # virsh blockcommit pc sdb --active --wait --verbose --pivot
> error: internal error: child reported (status=125): Requested operation is
> not valid: Setting different SELinux label on /tmp/disk-raw which is already
> in use
> 

https://bugzilla.redhat.com/show_bug.cgi?id=1803551

> 
> Step7 blockcopy fails on qcow2 image:
> a. Prepare the src and dest image at the size of 300M like following:
> |=== raw ===|=== qcow2 ===|=== raw ===|
> |<-- 100M ->|<-- 100M --->|<-- 100M ->|
> 
> b. Prepare VM with disk xml:
> <disk type="file" device="disk">
>       <driver name="qemu" type="qcow2" copy_on_read="off" cache="none"
> discard="ignore" detect_zeroes="on" io="native"/>                           
> 
>       <source file="/tmp/disk-qcow2">
>         <slices>
>           <slice type="storage" offset="104857600" size="104857600"/>
>         </slices>
>       </source>
>       <backingStore/>
>       <target dev="sdb" bus="scsi"/>
>       <alias name="ua-slices"/>
>       <iotune>
>                <total_bytes_sec>10000000</total_bytes_sec>
>               <group_name>slice</group_name>
>            </iotune>
>     </disk>
> Dest disk xml:
> <disk type="file" device="disk">
>       <driver name="qemu" type="qcow2" copy_on_read="off" cache="none"
> discard="ignore" detect_zeroes="on" io="native"/>                           
> 
>       <source file="/tmp/copy">
>         <slices>
>           <slice type="storage" offset="104857600" size="104857600"/>
>         </slices>
>       </source>
>       <backingStore/>
>       <target dev="sdb" bus="scsi"/>
>       <alias name="ua-slices"/>
>       <iotune>
>                <total_bytes_sec>10000000</total_bytes_sec>
>               <group_name>slice</group_name>
>            </iotune>
>     </disk>
> 
> c. Do blockcopy with --reuse-external
> # virsh blockcopy pc sdb --xml dest.xml --wait --verbose --pivot
> --reuse-external --transient-job
> Block Copy: [ 96 %]error:                                                   
> 
> Copy failed 

This failed with:

2020-03-11 07:00:00.434+0000: 159506: debug : qemuMonitorJSONIOProcessLine:220 : Line [{"return": [{"current-progress": 103809024, "status": "concluded", "total-progress": 104857600, "type": "mirror", "id": "copy-sdb-libvirt-7-format", "error": "No space left on device"}], "id": "libvirt-420"}]

The slice you've created is probably too small to hold the image + metadata overhead of qcow2. Try creating the copy without the slice, see how big the qcow2 is and then compare it with the size of the slice or re-try with a bigger destination image.

(In reply to Peter Krempa from comment #14)
> (In reply to Han Han from comment #11)
> > Test on libvirt-6.0.0-9.module+el8.2.0+5957+7ae8988e.x86_64
> > qemu-kvm-4.2.0-13.module+el8.2.0+5898+fb4bceae.x86_64
> > 
> > Steps:
> > 1. Boot VM with <slices> settings
> > 2. Hot-plug and hot-unplug
> > 3. Compare the values of allocation|capacity|physical  from domstats and
> > domblkinfo
> > 4. Insert or eject cdrom media with <slices>
> > 5. Create snapshots with <slices> with --reuse-external
> > 6. Blockcommit from active layer or inactive layer
> > 7. Blockcopy with --reuse-external
> > 8. Set blkiotune to disk with <slices>. And get the values set.
> > 9. Use qcow2 format to redo the steps above
> > 
> > Step3 domstats&domblkinfo fails:
> > a. Create /tmp/disk-raw as 300M raw image, full allocation
> > b. Start VM with disk like following
> > <disk type="file" device="disk">
> >       <driver name="qemu" type="raw" copy_on_read="on" cache="none"
> > discard="ignore" detect_zeroes="on" io="native"/>
> >       <source file="/tmp/disk-raw">
> >         <slices>
> >           <slice type="storage" offset="104857600" size="104857600"/>
> >         </slices>
> >       </source>
> >       <backingStore/>
> >       <target dev="sdb" bus="scsi"/>
> >       <alias name="ua-slices"/>
> >       <iotune>
> >                <total_bytes_sec>10000000</total_bytes_sec>
> >               <group_name>slice</group_name>
> >            </iotune>
> >     </disk>
> > 
> > c. Check domstats and domblkinfo results
> > # virsh domstats pc --block|grep -Ei 'block.1.(allocation|capacity|physical)'
> >   block.1.allocation=0
> >   block.1.capacity=104857600
> >   block.1.physical=314576896
> > # virsh domblkinfo pc sdb|grep -Ei '(allocation|capacity|physical)'
> > Capacity:       104857600
> > Allocation:     314576896
> > Physical:       314572800
> > 
> > The values of Allocation are not the same.
> > See the log disk-raw.xml-stats-blkinfo.log
> 
> Did you write to the disk from the guest?
No
> 
> The same happens to raw image without a slice and is based on the way we
> collect the stats for raw images:
> 
> qemu-img create -f raw /path/to/image 20M
> 
> Start VM.
> 
> # virsh domstats upstream | grep allocation
>   block.0.allocation=0
>   block.1.allocation=0
> 
> # virsh domblkinfo upstream vda
> Capacity:       20971520
> Allocation:     4096
> Physical:       20971520
> 
> Write 10MiB to the image from the guest:
> 
> # virsh domstats upstream | grep allocation
>   block.0.allocation=0
>   block.1.allocation=10485760
> 
> # virsh domblkinfo upstream vda
> Capacity:       20971520
> Allocation:     10485760
> Physical:       20971520
> 
> Note that 'block.0' is a cdrom so it's never written to.

I got the same results here. After write 10M to the disk in vm, the allocation values from domstats and domblkinfo become same.
> 
> > Step6 blockcommit fails:
> 
> [...]
> 
> > c. blockcommit from inactive layer and active layer
> > # virsh blockcommit pc sdb --top /tmp/snap1 --shallow --wait --verbose
> > Block commit: [100 %]
> > 
> > # virsh blockcommit pc sdb --active --wait --verbose --pivot
> > error: internal error: child reported (status=125): Requested operation is
> > not valid: Setting different SELinux label on /tmp/disk-raw which is already
> > in use
> > 
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1803551
It passes on libvirt-6.0.0-10.module+el8.2.0+5984+dce93708.x86_64
> 
> > 
> > Step7 blockcopy fails on qcow2 image:
> > a. Prepare the src and dest image at the size of 300M like following:
> > |=== raw ===|=== qcow2 ===|=== raw ===|
> > |<-- 100M ->|<-- 100M --->|<-- 100M ->|
> > 
> > b. Prepare VM with disk xml:
> > <disk type="file" device="disk">
> >       <driver name="qemu" type="qcow2" copy_on_read="off" cache="none"
> > discard="ignore" detect_zeroes="on" io="native"/>                           
> > 
> >       <source file="/tmp/disk-qcow2">
> >         <slices>
> >           <slice type="storage" offset="104857600" size="104857600"/>
> >         </slices>
> >       </source>
> >       <backingStore/>
> >       <target dev="sdb" bus="scsi"/>
> >       <alias name="ua-slices"/>
> >       <iotune>
> >                <total_bytes_sec>10000000</total_bytes_sec>
> >               <group_name>slice</group_name>
> >            </iotune>
> >     </disk>
> > Dest disk xml:
> > <disk type="file" device="disk">
> >       <driver name="qemu" type="qcow2" copy_on_read="off" cache="none"
> > discard="ignore" detect_zeroes="on" io="native"/>                           
> > 
> >       <source file="/tmp/copy">
> >         <slices>
> >           <slice type="storage" offset="104857600" size="104857600"/>
> >         </slices>
> >       </source>
> >       <backingStore/>
> >       <target dev="sdb" bus="scsi"/>
> >       <alias name="ua-slices"/>
> >       <iotune>
> >                <total_bytes_sec>10000000</total_bytes_sec>
> >               <group_name>slice</group_name>
> >            </iotune>
> >     </disk>
> > 
> > c. Do blockcopy with --reuse-external
> > # virsh blockcopy pc sdb --xml dest.xml --wait --verbose --pivot
> > --reuse-external --transient-job
> > Block Copy: [ 96 %]error:                                                   
> > 
> > Copy failed 
> 
> This failed with:
> 
> 2020-03-11 07:00:00.434+0000: 159506: debug :
> qemuMonitorJSONIOProcessLine:220 : Line [{"return": [{"current-progress":
> 103809024, "status": "concluded", "total-progress": 104857600, "type":
> "mirror", "id": "copy-sdb-libvirt-7-format", "error": "No space left on
> device"}], "id": "libvirt-420"}]
> 
> The slice you've created is probably too small to hold the image + metadata
> overhead of qcow2. Try creating the copy without the slice, see how big the
> qcow2 is and then compare it with the size of the slice or re-try with a
> bigger destination image.

I will try that with bigger qcow2 image.

Created attachment 1669226 [details]
The logs and scripts for comment11

The size for qcow2 format in comment11 is not right. It should contain the size of metadata and data.
For a full allocated 100M virtual size qcow2 file, the size should be:
➜  ~ qemu-img create /tmp/b -f qcow2 100M -o preallocation=full                                                                                 
Formatting '/tmp/b', fmt=qcow2 size=104857600 cluster_size=65536 preallocation=full lazy_refcounts=off refcount_bits=16
➜  ~ du -b /tmp/b
105185280       /tmp/b

Blockcopy on qcow2 passes after updating the size

Rich Jones,
Could you please help to check if libvirt-6.0.0-10.module+el8.2.0+5984+dce93708.x86_64 works on the case of comment0 ?
Some build dependencies are missing on my machine so I cannot run the `make check` for v2v

Created attachment 1669590 [details]
test-v2v-i-ova.sh-with-libvirt-6.0.0-10.txt

Yes it works with libvirt-6.0.0-10.module+el8.2.0+5984+dce93708.x86_64
The log from the test is attached.

According to the results from comment11 to comment18, mark it verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2017