Bug 2047660

This was dropped when I modularised virt-v2v, and I intended to add it
back but forgot.  Should be an easy change.

https://github.com/libguestfs/virt-v2v/blob/68af35d48ca845133ede948d36ee351d171e3de8/TODO#L14

This isn't quite as easy as I thought it would be.  I had to ask a question
about how the compress filter works in qemu:

https://lists.nongnu.org/archive/html/qemu-devel/2022-01/msg06108.html

This looked like it was going to be an easy fix but it revealed
problems in nbdcopy which will take more effort to fix.  This is
RHEL 9.1 material.

See this thread for background:
https://listman.redhat.com/archives/libguestfs/2022-January/thread.html#00213

(In reply to Richard W.M. Jones from comment #3)
> This looked like it was going to be an easy fix but it revealed
> problems in nbdcopy which will take more effort to fix.  This is
> RHEL 9.1 material.
> 
> See this thread for background:
> https://listman.redhat.com/archives/libguestfs/2022-January/thread.html#00213

Hi Rich, the link doesn't work; can you please summarize, or provide a new link? Thanks!

I *think* it's this thread:

https://listman.redhat.com/archives/libguestfs/2022-January/027901.html

We can create a compressed qcow2 server and write to it using:

$ qemu-nbd --image-opts driver=compress,file.driver=qcow2,file.file.driver=file,file.file.filename=new.qcow2

qemu-nbd (correctly) advertises and requires you to write in whole cluster-size
(IIRC?) blocks because of the way that cluster-level qcow2 compression works.
nbdcopy however ignores this size requirement, and fixing that is hard work.

Putting this on the backlog as it's non-trivial to fix.

Patch posted:
https://listman.redhat.com/archives/libguestfs/2022-July/029390.html

Note this will require an updated libnbd, hence make this bug depend on
bug 2059288.

Upstream in:

https://github.com/libguestfs/virt-v2v/commit/71c4301909cb307def02ebcd0e89beee4138e7f2
https://github.com/libguestfs/virt-v2v/commit/53690a0c602a4286fdb9408fdf6a01cc352697ec
https://github.com/libguestfs/virt-v2v/commit/7505750972b49e1a448c519a27998bd5f20be60a

Verified with the versions:
qemu-guest-agent-7.0.0-8.el9.x86_64
libnbd-1.12.5-1.el9.x86_64
virt-v2v-2.0.7-1.el9.x86_64

Steps:
1. Convert a VM via virt-v2v with -oo compressed option.
# virsh list --all |grep rhel9.1
 -    esx7.0-rhel9.1-x86_64                      shut off

#  virt-v2v -i libvirt -ic qemu:///system esx7.0-rhel9.1-x86_64   -o libvirt -os default -oo compressed -on  esx7.0-rhel9.1-x86_64-compress -of qcow2
[   0.0] Setting up the source: -ic qemu:///system -i libvirt esx7.0-rhel9.1-x86_64
[   1.2] Opening the source
[   5.7] Inspecting the source
[  12.8] Checking for sufficient free disk space in the guest
[  12.8] Converting Red Hat Enterprise Linux 9.1 Beta (Plow) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 119.4] Mapping filesystem data to avoid copying unused and blank areas
[ 120.7] Closing the overlay
[ 121.0] Assigning disks to buses
[ 121.0] Checking if the guest needs BIOS or UEFI to boot
[ 121.0] Setting up the destination: -o libvirt -os default
[ 122.2] Copying disk 1/1
█ 100% [****************************************]
[ 195.0] Creating output metadata
[ 195.0] Finishing off

2. Check compressed VM and start VM to check the checkpoints.

# virsh list --all |grep rhel9.1
 7    esx7.0-rhel9.1-x86_64-compress             running
 -    esx7.0-rhel9.1-x86_64                      shut off



rjones, I also tried on the VM with nvme-disk that was converted from ESXi. But it failed to start due to CPU incompatible.

# virsh list --all |grep nvme
 -    esx6.7-rhel8.6-nvme-disk                   shut off

# virt-v2v -i libvirt -ic qemu:///system esx6.7-rhel8.6-nvme-disk  -o libvirt -os default -oo compressed -on  esx6.7-rhel8.6-nvme-disk-compress -of qcow2
[   0.2] Setting up the source: -ic qemu:///system -i libvirt esx6.7-rhel8.6-nvme-disk
[   1.3] Opening the source
[   5.8] Inspecting the source
[  15.2] Checking for sufficient free disk space in the guest
[  15.2] Converting Red Hat Enterprise Linux 8.6 Beta (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 122.1] Mapping filesystem data to avoid copying unused and blank areas
[ 123.7] Closing the overlay
[ 124.0] Assigning disks to buses
[ 124.0] Checking if the guest needs BIOS or UEFI to boot
[ 124.0] Setting up the destination: -o libvirt -os default
[ 125.2] Copying disk 1/1
█ 100% [****************************************]
[ 221.4] Creating output metadata
[ 221.4] Finishing off

# virsh list --all |grep nvme
 -    esx6.7-rhel8.6-nvme-disk                   shut off
 -    esx6.7-rhel8.6-nvme-disk-compress          shut off

# virsh start esx6.7-rhel8.6-nvme-disk-compress
error: Failed to start domain 'esx6.7-rhel8.6-nvme-disk-compress'
error: the CPU is incompatible with host CPU: Host CPU does not provide required features: svm


The diff of dumpxml between original and compressed one:

  <cpu mode='custom' match='exact' check='none'>
    <model fallback='forbid'>qemu64</model>
  </cpu>
......

  <cpu mode='custom' match='minimum' check='partial'>
    <model fallback='allow'>qemu64</model>
  </cpu>


Do I need to file a new bug against this issue?

Hi Vera,

(In reply to Vera from comment #10)
> rjones, I also tried on the VM with nvme-disk that was converted from ESXi.
> But it failed to start due to CPU incompatible.
> 
> # virsh list --all |grep nvme
>  -    esx6.7-rhel8.6-nvme-disk                   shut off
> 
> # virt-v2v -i libvirt -ic qemu:///system esx6.7-rhel8.6-nvme-disk  -o
> libvirt -os default -oo compressed -on  esx6.7-rhel8.6-nvme-disk-compress
> -of qcow2
> [   0.2] Setting up the source: -ic qemu:///system -i libvirt
> esx6.7-rhel8.6-nvme-disk
> [   1.3] Opening the source
> [   5.8] Inspecting the source
> [  15.2] Checking for sufficient free disk space in the guest
> [  15.2] Converting Red Hat Enterprise Linux 8.6 Beta (Ootpa) to run on KVM
> virt-v2v: This guest has virtio drivers installed.
> [ 122.1] Mapping filesystem data to avoid copying unused and blank areas
> [ 123.7] Closing the overlay
> [ 124.0] Assigning disks to buses
> [ 124.0] Checking if the guest needs BIOS or UEFI to boot
> [ 124.0] Setting up the destination: -o libvirt -os default
> [ 125.2] Copying disk 1/1
> █ 100% [****************************************]
> [ 221.4] Creating output metadata
> [ 221.4] Finishing off
> 
> # virsh list --all |grep nvme
>  -    esx6.7-rhel8.6-nvme-disk                   shut off
>  -    esx6.7-rhel8.6-nvme-disk-compress          shut off
> 
> # virsh start esx6.7-rhel8.6-nvme-disk-compress
> error: Failed to start domain 'esx6.7-rhel8.6-nvme-disk-compress'
> error: the CPU is incompatible with host CPU: Host CPU does not provide
> required features: svm
> 
> 
> The diff of dumpxml between original and compressed one:
> 
>   <cpu mode='custom' match='exact' check='none'>
>     <model fallback='forbid'>qemu64</model>
>   </cpu>
> ......
> 
>   <cpu mode='custom' match='minimum' check='partial'>
>     <model fallback='allow'>qemu64</model>
>   </cpu>
> 
> 
> Do I need to file a new bug against this issue?

The CPU model question in virt-v2v is complicated. Recently we've had a related bug: BZ 2076013. Maybe we should extend the logic we added there.

The idea is that the "qemu64" CPU model is basically never right. For bug 2076013, we made a very surgical change -- we override it for RHEL9 guests only. Perhaps we should make the coverage more extensive (in this case, the guest is a RHEL8 OS). The comparison is tricky because here the problem is not *guest-OS* dependent.

I think that filing a new BZ would be useful. We should investigate there what the original <cpu> and the converted <cpu> elements mean precisely.

In case you file a new bug, please also include the output of "virsh capabilities" there, executed on the host in question.

Anyway I defer to Rich on whether we need a new bug for this. I don't think that we absolutely need to investigate this, as I'd never expect libvirt-to-libvirt conversion to be used in production.

(In reply to Laszlo Ersek from comment #11)
> Hi Vera,
> 
> (In reply to Vera from comment #10)
> 
> The CPU model question in virt-v2v is complicated. Recently we've had a
> related bug: BZ 2076013. Maybe we should extend the logic we added there.
> 
> The idea is that the "qemu64" CPU model is basically never right. For bug
> 2076013, we made a very surgical change -- we override it for RHEL9 guests
> only. Perhaps we should make the coverage more extensive (in this case, the
> guest is a RHEL8 OS). The comparison is tricky because here the problem is
> not *guest-OS* dependent.
> 
> I think that filing a new BZ would be useful. We should investigate there
> what the original <cpu> and the converted <cpu> elements mean precisely.
> 
> In case you file a new bug, please also include the output of "virsh
> capabilities" there, executed on the host in question.
> 
> Anyway I defer to Rich on whether we need a new bug for this. I don't think
> that we absolutely need to investigate this, as I'd never expect
> libvirt-to-libvirt conversion to be used in production.

Thanks Laszlo. I just file a new bug Bug 2107503 against this issue.

And this bug will be marked as Verified if you can investigate on that new bug.

Verified with the versions:
libguestfs-1.48.4-1.el9.x86_64
qemu-kvm-7.0.0-9.el9.x86_64
virt-v2v-2.0.7-4.el9.x86_64
libvirt-8.5.0-4.el9.x86_64

steps:
1. Convert ESX VM via virt-v2v with -oo compressed option to libvirt.

# virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1  -it vddk -io vddk-libdir=/home/vddk7.0.3 -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA  -ip /v2v-ops/esxpw esx6.7-rhel9.0-x86_64 -oo compressed -of qcow2
[   0.0] Setting up the source: -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk esx6.7-rhel9.0-x86_64
[   1.9] Opening the source
[   7.2] Inspecting the source
[  13.4] Checking for sufficient free disk space in the guest
[  13.4] Converting Red Hat Enterprise Linux 9.0 Beta (Plow) to run on KVM
virt-v2v: The QEMU Guest Agent will be installed for this guest at first 
boot.
virt-v2v: This guest has virtio drivers installed.
[ 123.2] Mapping filesystem data to avoid copying unused and blank areas
[ 124.2] Closing the overlay
[ 124.4] Assigning disks to buses
[ 124.4] Checking if the guest needs BIOS or UEFI to boot
[ 124.4] Setting up the destination: -o libvirt
[ 126.1] Copying disk 1/1
█ 100% [****************************************]
[ 311.2] Creating output metadata
[ 311.2] Finishing off

2. Check compressed VM and start VM to check the checkpoints.

# virsh start esx6.7-rhel9.0-x86_64
Domain 'esx6.7-rhel9.0-x86_64' started

# virsh list |grep rhel9.0
 18   esx6.7-rhel9.0-x86_64   running

Moving to Verified.

It might be an idea to check that the output is really compressed.
You can do this using:

qemu-img check output.qcow2

(The output file should be in libvirt's default pool, probably
in /var/lib/libvirt/images/ )

For a compressed file you will see something like:

$ qemu-img check fedora-35.qcow2 
No errors were found on the image.
17518/98304 = 17.82% allocated, 78.74% fragmented, 73.75% compressed clusters
                                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^

For an uncompressed file you will see something like:

$ qemu-img check fedora-35.qcow2 
No errors were found on the image.
17518/98304 = 17.82% allocated, 0.00% fragmented, 0.00% compressed clusters
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^

We expect that the number of compressed clusters will be larger than 0%
(exactly how big depends on the source guest).

Note that as guests as being used, the number of compressed clusters
is expected to decline.  This is because qemu cannot create new compressed
clusters for running guests.

As rjones suggested, adding steps to check the compressed option really works.

Step3: check the number of compressed clusters.
# qemu-img check esx6.7-rhel9.0-x86_64-sda
No errors were found on the image.
73366/262144 = 27.99% allocated, 82.51% fragmented, 75.60% compressed clusters
Image end offset: 2799435776


Also tried on the one without -oo compressed:
# qemu-img check rhel9-luks-sda 
No errors were found on the image.
35888/163840 = 21.90% allocated, 16.02% fragmented, 0.00% compressed clusters
Image end offset: 2353594368

Looks good, thanks for testing this.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: virt-v2v security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7968