RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2107503 - RHEL 8.6 VM with "qemu64" CPU model can't start because "the CPU is incompatible with host CPU: Host CPU does not provide required features: svm"
Summary: RHEL 8.6 VM with "qemu64" CPU model can't start because "the CPU is incompati...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: virt-v2v
Version: 9.1
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Laszlo Ersek
QA Contact: Vera
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-07-15 09:37 UTC by Vera
Modified: 2022-11-15 10:24 UTC (History)
9 users (show)

Fixed In Version: virt-v2v-2.0.7-3.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-11-15 09:56:15 UTC
Type: Feature Request
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
the output of "virsh capabilities" on the conversion host (9.83 KB, text/plain)
2022-07-15 09:37 UTC, Vera
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 2047660 0 unspecified CLOSED Add '--compressed' support in modular v2v 2023-06-02 07:24:17 UTC
Red Hat Issue Tracker RHELPLAN-127839 0 None None None 2022-07-15 09:48:48 UTC
Red Hat Issue Tracker RHELPLAN-127840 0 None None None 2022-07-15 09:48:50 UTC
Red Hat Product Errata RHSA-2022:7968 0 None None None 2022-11-15 09:56:24 UTC

Description Vera 2022-07-15 09:37:01 UTC
Created attachment 1897349 [details]
the output of "virsh capabilities" on the conversion host

Description of problem:
VM with "qemu64" CPU model can't start after "-oo compressed" conversion via virt-v2v

Version-Release number of selected component (if applicable):
qemu-guest-agent-7.0.0-8.el9.x86_64
libnbd-1.12.5-1.el9.x86_64
virt-v2v-2.0.7-1.el9.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Prepare a VM with "qemu64" CPU model 
# virsh list --all |grep nvme
 -    esx6.7-rhel8.6-nvme-disk                   shut off
# virsh dumpxml esx6.7-rhel8.6-nvme-disk
...
  <cpu mode='custom' match='exact' check='none'>
    <model fallback='forbid'>qemu64</model>
  </cpu>
...

2. Convert with -oo compressed option.
# virt-v2v -i libvirt -ic qemu:///system esx6.7-rhel8.6-nvme-disk  -o libvirt -os default -oo compressed -on  esx6.7-rhel8.6-nvme-disk-compress -of qcow2
[   0.2] Setting up the source: -ic qemu:///system -i libvirt esx6.7-rhel8.6-nvme-disk
[   1.3] Opening the source
[   5.8] Inspecting the source
[  15.2] Checking for sufficient free disk space in the guest
[  15.2] Converting Red Hat Enterprise Linux 8.6 Beta (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 122.1] Mapping filesystem data to avoid copying unused and blank areas
[ 123.7] Closing the overlay
[ 124.0] Assigning disks to buses
[ 124.0] Checking if the guest needs BIOS or UEFI to boot
[ 124.0] Setting up the destination: -o libvirt -os default
[ 125.2] Copying disk 1/1
█ 100% [****************************************]
[ 221.4] Creating output metadata
[ 221.4] Finishing off

3.Start VM and check the checkpoints.

# virsh list --all |grep nvme
 -    esx6.7-rhel8.6-nvme-disk                   shut off
 -    esx6.7-rhel8.6-nvme-disk-compress          shut off

# virsh start esx6.7-rhel8.6-nvme-disk-compress
error: Failed to start domain 'esx6.7-rhel8.6-nvme-disk-compress'
error: the CPU is incompatible with host CPU: Host CPU does not provide required features: svm

# virsh dumpxml esx6.7-rhel8.6-nvme-disk-compress
...
  <cpu mode='custom' match='minimum' check='partial'>
    <model fallback='allow'>qemu64</model>
  </cpu>
...

Actual results:
VM can't start into OS successfully

Expected results:
VM can start into OS successfully

Additional info:
Please check attachment on  the output of "virsh capabilities"

Comment 1 Richard W.M. Jones 2022-07-15 09:53:58 UTC
Changing the summary because I don't think this has anything to do with
-oo compressed.  It's similar to bug 2076013.

Vera, could you attach the full debug output from the virt-v2v conversion please?

Comment 4 Laszlo Ersek 2022-07-15 14:04:14 UTC
I guess we need to figure out, based on the libvirt documentation, what the difference is, between

  <cpu mode='custom' match='exact' check='none'>
    <model fallback='forbid'>qemu64</model>
  </cpu>

and

  <cpu match='minimum'>
    <model fallback='allow'>qemu64</model>
  </cpu>

The qemu64 model is supposed to work on all hosts, per <https://gitlab.com/qemu-project/qemu/-/blob/master/docs/system/cpu-models-x86.rst.inc>.

from https://libvirt.org/formatdomain.html#cpu-model-and-topology:

- mode='custom' is the default, so there is no difference in that regard

- match='minimum' is laxer than match='exact', so the conversion output is more lenient than the original

- model fallback='allow' is laxer than fallback='forbid', so again the conversion output is more lenient

- the culprit seems to be that we don't output check='none'. In the original domain def, check='none' means:

> Libvirt does no checking and it is up to the hypervisor to refuse to start the domain if it cannot provide the requested CPU. With QEMU this means no
> checking is done at all since the default behavior of QEMU is to emit warnings, but start the domain anyway.

However when we omit it in the conversion output, check='none' is *not* the default:

> Once the domain starts, libvirt will automatically change the check attribute to the best supported value to ensure the virtual CPU does not change when
> the domain is migrated to another host

and indeed we can see that libvirt fills it in as "partial"

> Libvirt will check the guest CPU specification before starting a domain [...]

I think we need to add the check='none' attribute to the "cpu" element in the libvirt output, as we don't care about migratability here.

Comment 5 Laszlo Ersek 2022-07-15 14:11:02 UTC
wild guess (untested):

diff --git a/output/create_libvirt_xml.ml b/output/create_libvirt_xml.ml
index 531a4f75bf3e..0343d3194268 100644
--- a/output/create_libvirt_xml.ml
+++ b/output/create_libvirt_xml.ml
@@ -192,6 +192,7 @@ let create_libvirt_xml ?pool source inspect
            List.push_back cpu_attrs ("mode", "host-passthrough");
      | Some model ->
          List.push_back cpu_attrs ("match", "minimum");
+         List.push_back cpu_attrs ("check", "none");
          (match source.s_cpu_vendor with
           | None -> ()
           | Some vendor ->

Comment 7 Laszlo Ersek 2022-07-20 11:09:55 UTC
Turns out I can reproduce, and therefore test, this, myself.

[v2v PATCH] output/create_libvirt_xml: generate @check='none' CPU attribute
Message-Id: <20220720110913.14058-1-lersek>
https://listman.redhat.com/archives/libguestfs/2022-July/029504.html

Comment 8 Laszlo Ersek 2022-07-22 07:36:51 UTC
[v2v PATCH v2] output/create_libvirt_xml: relax VCPU feature checking for "qemu64"
Message-Id: <20220722073627.6511-1-lersek>
https://listman.redhat.com/archives/libguestfs/2022-July/029519.html

Comment 9 Laszlo Ersek 2022-07-22 13:00:02 UTC
(In reply to Laszlo Ersek from comment #8)
> [v2v PATCH v2] output/create_libvirt_xml: relax VCPU feature checking for "qemu64"
> Message-Id: <20220722073627.6511-1-lersek>
> https://listman.redhat.com/archives/libguestfs/2022-July/029519.html

Upstream commit e5297c3180fd.

Comment 16 Vera 2022-08-01 13:16:58 UTC
Verified with the versions:
qemu-img-7.0.0-9.el9.x86_64
nbdkit-1.30.8-1.el9.x86_64
libvirt-8.5.0-3.el9.x86_64
libnbd-1.12.6-1.el9.x86_64
virt-v2v-2.0.7-4.el9.x86_64


Steps:
1. Prepare a VM with "qemu64" CPU model 
# virsh list --all |grep nvme
 -    esx6.7-rhel8.6-nvme-disk                   shut off
# virsh dumpxml esx6.7-rhel8.6-nvme-disk
...
  <cpu mode='custom' match='exact' check='none'>
    <model fallback='forbid'>qemu64</model>
  </cpu>
...

2. Convert with -oo compressed option.
# virt-v2v -i libvirt -ic qemu:///system esx6.7-rhel8.6-nvme-disk  -o libvirt -os default -oo compressed -on  esx6.7-rhel8.6-nvme-disk-compress -of qcow2
[   0.0] Setting up the source: -ic qemu:///system -i libvirt esx6.7-rhel8.6-nvme-disk
[   1.1] Opening the source
[   5.7] Inspecting the source
[  14.6] Checking for sufficient free disk space in the guest
[  14.6] Converting Red Hat Enterprise Linux 8.6 Beta (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 119.3] Mapping filesystem data to avoid copying unused and blank areas
[ 121.5] Closing the overlay
[ 121.8] Assigning disks to buses
[ 121.8] Checking if the guest needs BIOS or UEFI to boot
[ 121.8] Setting up the destination: -o libvirt -os default
[ 123.0] Copying disk 1/1
█ 100% [****************************************]
[ 259.5] Creating output metadata
[ 259.5] Finishing off

3.Start VM and check the checkpoints.

# virsh list --all |grep nvme
 -    esx6.7-rhel8.6-nvme-disk            shut off
 -    esx6.7-rhel8.6-nvme-disk-compress   shut off

# virsh start esx6.7-rhel8.6-nvme-disk-compress
Domain 'esx6.7-rhel8.6-nvme-disk-compress' started

# virsh list --all |grep nvme
 4    esx6.7-rhel8.6-nvme-disk-compress   running
 -    esx6.7-rhel8.6-nvme-disk            shut off


# virsh dumpxml esx6.7-rhel8.6-nvme-disk-compress
...
 <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Cascadelake-Server</model>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='umip'/>
    <feature policy='require' name='pku'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='xsaves'/>
    <feature policy='require' name='ibpb'/>
    <feature policy='require' name='ibrs'/>
    <feature policy='require' name='amd-stibp'/>
    <feature policy='require' name='amd-ssbd'/>
    <feature policy='require' name='rdctl-no'/>
    <feature policy='require' name='ibrs-all'/>
    <feature policy='require' name='skip-l1dfl-vmentry'/>
    <feature policy='require' name='mds-no'/>
    <feature policy='require' name='pschange-mc-no'/>
    <feature policy='require' name='tsx-ctrl'/>
    <feature policy='disable' name='hle'/>
    <feature policy='disable' name='rtm'/>
    <feature policy='disable' name='mpx'/>
  </cpu>
...

It is working with this version.

Lazsole, I have a question on this. From the patch, When the source domain specifies a CPU model, is it correct @check='full'?

Comment 17 Laszlo Ersek 2022-08-01 13:54:40 UTC
Hi Vera,

something doesn't add up here. In your comment 16, the source domain seems to have an explicit "qemu64" CPU model, but the converted domain's CPU model is "Cascadelake-Server".

virt-v2v should never do this. Can you please repeat the test and:

- while the converted domain is running, attach the output of "virsh dumpxml --inactive" (not just "virsh dumpxml"),

- provide the full conversion log?

From my experimentation, what happens is that libvirt modifies the <cpu> element, but only for the *running* instance of the domain. For example, if I have a local domain with

  <cpu mode='custom' match='exact' check='none'>
    <model fallback='forbid'>qemu64</model>
  </cpu>

and I start it, I get the following command outputs (with the domain running):

- virsh dumpxml --inactive:

  <cpu mode='custom' match='exact' check='none'>
    <model fallback='forbid'>qemu64</model>
  </cpu>

- virsh dumpxml:

  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>qemu64</model>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='lahf_lm'/>
    <feature policy='disable' name='svm'/>
  </cpu>

Note that @check changes from "none" to "full", from the config version to the live version, of the domain XML. Additionally, some CPU flags get explicitly listed, such as the disablement of "svm". However, the model name itself does *not* change, from qemu64 to anything else.

So this is the justification for my two requests above:

- please pass "--inactive" to "virsh dump" (we care about the config version, not the live version, of the domain XML),
- please upload the full conversion log (that's how we best see what virt-v2v creates from what).


Thank you.

Comment 20 Vera 2022-08-02 03:05:02 UTC
Hi Laszlo,

Please check the 2 attachments on the log as requested.

Thanks.

Comment 21 Laszlo Ersek 2022-08-03 10:30:22 UTC
Thank you.

I think everything is fine. The conversion output, and the "virsh dump --inactive" command, record the following output domain XML fragment (respectively):

  <cpu match='minimum' check='none'>
    <model fallback='allow'>qemu64</model>
  </cpu>

  <cpu mode='custom' match='minimum' check='none'>
    <model fallback='allow'>qemu64</model>
  </cpu>

This is what the patch intends to do.

Given that the guest actually boots and works, I think the only lesson here is that for a *running* (live) domain, the "live" domain XML may contain a <cpu> element that significantly differs from the "config" (= inactive) variant of the same element. Apparently libvirtd translates the "config" fragment

  <cpu mode='custom' match='minimum' check='none'>
    <model fallback='allow'>qemu64</model>
  </cpu>

to a very different "live" fragment, but -- importantly -- with the correct, desired effect.

So I think it's fine to set this BZ to VERIFIED.

Comment 22 Vera 2022-08-04 01:54:02 UTC
Laszlo, Thanks for confirmation.

Marking the bug to VERIFIED.

Comment 29 errata-xmlrpc 2022-11-15 09:56:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: virt-v2v security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7968


Note You need to log in before you can comment on or make changes to this bug.