RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1834714 - libvirt refuses to start a domain with qemu64 CPU which is reported as usable in domain capabilities
Summary: libvirt refuses to start a domain with qemu64 CPU which is reported as usable...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: libvirt
Version: 9.0
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: rc
: ---
Assignee: Tim Wiederhake
QA Contact: yalzhang@redhat.com
URL:
Whiteboard:
Depends On: 1727122
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-12 09:36 UTC by Karel Šimon
Modified: 2022-09-26 11:40 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-09-26 11:40:41 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Karel Šimon 2020-05-12 09:36:34 UTC 
Description of problem:
kubevirt flakyfinder reports failing test related to node-labeller (see here [1]. See line with `[rfe_id:273][crit:high][vendor:cnv-qe][level:component]VMIlifecycle [rfe_id:273][crit:high][vendor:cnv-qe][level:component]Creating a VirtualMachineInstance with node feature discovery [test_id:1639]the vmi with cpu.model matching a nfd label on a node should be scheduled `)

This test [2] tests if vmi with supported cpu model runs on the node. Supported cpus are acquired from virt-launcher from this command[3]. Output of virsh domcapabilities is then parsed by node-labeller and it labels nodes with acquired supported cpu models. Then these labels are used for scheduling vms with a defined cpu model on a node which supports it.

I tested on my local pc command virsh domcapabilities which generates this output[4] (KVM device was available). Then I created a VM with `qemu64` cpu model (you can see in that output, that it is supported), but it fails with command:
Unexpected Warning event received: testvmibthqvfpqq2ssx78ndd2ds8fqwqdvgbg5rgk4p645btrhqcsj,63bb7287-86bf-408a-abae-ac28ae158cbb: server error. command SyncVMI failed: "LibvirtError(Code=91, Domain=31, Message='the CPU is incompatible with host CPU: Host CPU does not provide required features: svm'
I noticed this same error occurs for Opteron_G2 (maybe there are more cpu models. I encountered this only with qemu64 and Opteron_G2).

Jiri Denemark adviced to use workaround: adding check='none' attribute to the
domain's <cpu> element

[1]https://storage.googleapis.com/kubevirt-prow/reports/flakefinder/kubevirt/kubevirt/flakefinder-2020-05-07-168h.html
[2]https://github.com/kubevirt/kubevirt/blob/master/tests/vmi_lifecycle_test.go#L909
[3]https://github.com/kubevirt/kubevirt/blob/master/cmd/virt-launcher/node-labeller/node-labeller.sh#L16
[4]http://pastebin.test.redhat.com/863576

Version-Release number of selected component (if applicable):
virsh --version
5.1.0


How reproducible:
100%

Steps to Reproduce:
1.
2.
3.

Actual results:
Unexpected Warning event received: testvmibthqvfpqq2ssx78ndd2ds8fqwqdvgbg5rgk4p645btrhqcsj,63bb7287-86bf-408a-abae-ac28ae158cbb: server error. command SyncVMI failed: "LibvirtError(Code=91, Domain=31, Message='the CPU is incompatible with host CPU: Host CPU does not provide required features: svm'

Expected results:
kubevirt vmi runs without error

Additional info:
Comment 1 Jiri Denemark 2020-05-13 09:22:52 UTC 
The usable='yes' attribute in the output of domcapabilities says QEMU is able
to start a domain with the corresponding CPU model without disabling any of
the features included in the model. Unfortunately, in case of qemu64 this
depends on the machine type. While qemu64 requires svm with pc-i440fx-2.1 or
older machine types, newer machine types change the model to not require svm.
With pc-i440fx-2.5 or newer qemu64 does not even require sse4a.

Unfortunately, libvirt doesn't know about all this and still checks whether
the host supports svm and thus refuses to start a domain with qemu64 CPU on an
Intel host. I think the solution would be connected to the support for
versioned CPU models (see bug 1727122). In the meantime, you could workaround
this by adding check='none' attribute to the domain's <cpu> element. Doing so
is pretty safe when used with models that are reported as usable (I wouldn't
suggest using it with other models, though).
Comment 3 John Ferlan 2021-09-08 13:31:19 UTC 
Bulk update: Move RHEL-AV bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release.
Comment 4 John Ferlan 2021-09-09 15:31:53 UTC 
Return to RHEL8, will add RHV as Dependent Product.

If a resolution / testing is required in RHEL9, a clone will need to be created.
Comment 5 yalzhang@redhat.com 2021-09-27 09:56:47 UTC 
reproduce on 
# rpm -q libvirt qemu-kvm
libvirt-7.6.0-3.module+el8.5.0+12510+80564ecf.x86_64
qemu-kvm-6.0.0-30.module+el8.5.0+12586+476da3e1.x86_64

# virsh dumpxml rhel
...
<os>
    <type arch='x86_64' machine='pc-q35-rhel8.5.0'>hvm</type>
    <boot dev='hd'/>
  </os>
...
 <cpu mode='custom' match='exact' check='partial'>
    <model fallback='forbid'>qemu64</model>
  </cpu>
...
# virsh start rhel 
error: Failed to start domain 'rhel'
error: the CPU is incompatible with host CPU: Host CPU does not provide required features: svm
Comment 7 RHEL Program Management 2021-11-12 07:27:08 UTC 
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.
Comment 9 yalzhang@redhat.com 2022-03-10 06:56:17 UTC 
Hi, I have several questions about it, could you please check below steps? Thank you!

Currently, guest will crash with default cpu model qemu64. Does it need any fix? like:
1) Update the default cpu model from "qemu64" to "host-model";
2) When we start a vm with cpu as qemu64, libvirt report an error like "qemu64" is not supported; 

# rpm -q libvirt qemu-kvm
libvirt-8.0.0-6.el9.x86_64
qemu-kvm-6.2.0-11.el9.x86_64

1. Ignore the cpu model setting in vm xml;
2. After save the xml, it will use the default one:
# virsh dumpxml rhel | grep /cpu -B2
  <cpu mode='custom' match='exact' check='none'>
    <model fallback='forbid'>qemu64</model>
  </cpu>
3. Start the vm, the guest will crash:
[    1.774228] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00
Fatal glibc error: CPU does not support x86-64-v2
[    1.775720] CPU: 6 PID: 1 Comm: init Not tainted 5.14.0-58.el9.x86_64 #1
[    1.776883] Hardware name: Red Hat KVM/RHEL, BIOS 1.15.0-1.el9 04/01/2014
[    1.778063] Call Trace:
[    1.778601]  dump_stack_lvl+0x34/0x44
[    1.779312]  panic+0x102/0x2d4
[    1.779932]  do_exit.cold+0x87/0x9f
[    1.780618]  do_group_exit+0x33/0xa0
[    1.781314]  __x64_sys_exit_group+0x14/0x20
[    1.782104]  do_syscall_64+0x3b/0x90
[    1.782803]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[    1.783722] RIP: 0033:0x7f7764f35191
[    1.784419] Code: c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa be e7 00 00 00 ba 3c 00 00 00 eb 0d 89 d0 0f 05 48 3d 00 f0 ff ff 77 1c f4 89 f0 0f 05 <48> 3d 00 f0 ff ff 76 e7 f7 d8 89 05 3f 00 01 00 eb dd 0f 1f 44 00
[    1.787518] RSP: 002b:00007ffcd05b7898 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
[    1.788878] RAX: ffffffffffffffda RBX: 00007f7764f2fe30 RCX: 00007f7764f35191
[    1.790102] RDX: 000000000000003c RSI: 00000000000000e7 RDI: 000000000000007f
[    1.791326] RBP: 00005603942ce040 R08: 00007ffcd05b7409 R09: 0000000000000000
[    1.792547] R10: 00000000ffffffff R11: 0000000000000246 R12: 000000000000000d
[    1.793769] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000001
[    1.795089] Kernel Offset: 0x22a00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[    1.796925] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00 ]--
Comment 10 Jaroslav Suchanek 2022-03-17 14:03:16 UTC 
(In reply to yalzhang from comment #9)
> Hi, I have several questions about it, could you please check below steps?
> Thank you!
> 
> Currently, guest will crash with default cpu model qemu64. Does it need any
> fix? like:
> 1) Update the default cpu model from "qemu64" to "host-model";
> 2) When we start a vm with cpu as qemu64, libvirt report an error like
> "qemu64" is not supported; 
> 
> # rpm -q libvirt qemu-kvm
> libvirt-8.0.0-6.el9.x86_64
> qemu-kvm-6.2.0-11.el9.x86_64
> 
> 1. Ignore the cpu model setting in vm xml;
> 2. After save the xml, it will use the default one:
> # virsh dumpxml rhel | grep /cpu -B2
>   <cpu mode='custom' match='exact' check='none'>
>     <model fallback='forbid'>qemu64</model>
>   </cpu>
> 3. Start the vm, the guest will crash:
> [    1.774228] Kernel panic - not syncing: Attempted to kill init!
> exitcode=0x00007f00
> Fatal glibc error: CPU does not support x86-64-v2

Yalan Zhang, would you file a separate kernel bug for this? What is the version of guest OS?
Comment 11 yalzhang@redhat.com 2022-03-18 09:58:47 UTC 
> 
> Yalan Zhang, would you file a separate kernel bug for this? What is the
> version of guest OS?

Sorry, I found there is an existing bug to cover the issue in comment 9: 
Bug 1925384 - RHEL9 KVM guest kernel panic during system boot up with default cpu mode

Current bug is a different one with comment 9.
Comment 12 Riccardo Pittau 2022-03-30 13:10:38 UTC 
As far as I can see the bug mentioned by Yalan in comment 11 is different
We're also experiencing the same issue and having check='none' in the cpu entry in the domain does not help
An example:
<domain type='qemu'>
  <name>node-0</name>
  <uuid>26e6aba4-bb4e-43ff-82bf-e89229a1aacb</uuid>
  <memory unit='KiB'>3145728</memory>
  <currentMemory unit='KiB'>3145728</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-4.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/var/lib/libvirt/images/OVMF_CODE.fd</loader>
    <nvram template='/var/lib/libvirt/images/OVMF_VARS.fd'>/var/lib/libvirt/images/OVMF_VARS.fd-node-0</nvram>
    <bootmenu enable='no'/>
    <bios useserial='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact' check='none'>
    <model fallback='forbid'>qemu64</model>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='unsafe'/>
      <source file='/opt/libvirt/images/node-0.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </disk>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='piix3-uhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='pci' index='1' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:28:45:cf'/>
      <source dev='tap-node-0i1' mode='vepa'/>
      <model type='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </interface>
    <interface type='direct'>
      <mac address='52:54:00:b6:47:99'/>
      <source dev='tap-node-0i2' mode='vepa'/>
      <model type='virtio'/>
      <boot order='2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x02' function='0x0'/>
    </interface>
    <serial type='file'>
      <source path='/opt/stack/logs/ironic-bm-logs/node-0_console.log'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <serial type='pty'>
      <target type='isa-serial' port='1'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='file'>
      <source path='/opt/stack/logs/ironic-bm-logs/node-0_console.log'/>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>


[    8.624131] Run /init as init process
Fatal glibc error: CPU does not support x86-64-v2
[    8.637563] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00
[    8.638317] CPU: 0 PID: 1 Comm: init Not tainted 5.14.0-75.el9.x86_64 #1
[    8.638412] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 0.0.0 02/06/2015
[    8.638412] Call Trace:
[    8.638412]  dump_stack_lvl+0x34/0x44
[    8.638412]  panic+0x102/0x2d4
[    8.638412]  do_exit.cold+0x87/0x9f
[    8.638412]  do_group_exit+0x33/0xa0
[    8.638412]  __x64_sys_exit_group+0x14/0x20
[    8.638412]  do_syscall_64+0x3b/0x90
[    8.638412]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[    8.638412] RIP: 0033:0x7fb846640191
[    8.638412] Code: c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa be e7 00 00 00 ba 3c 00 00 00 eb 0d 89 d0 0f 05 48 3d 00 f0 ff ff 77 1c f4 89 f0 0f 05 <48> 3d 00 f0 ff ff 76 e7 f7 d8 89 05 3f 00 01 00 eb dd 0f 1f 44 00
[    8.638412] RSP: 002b:00007fff7aa772f8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
[    8.638412] RAX: ffffffffffffffda RBX: 00007fb84663ae30 RCX: 00007fb846640191
[    8.638412] RDX: 000000000000003c RSI: 00000000000000e7 RDI: 000000000000007f
[    8.638412] RBP: 000055f1cc394040 R08: 00007fff7aa76e69 R09: 0000000000000000
[    8.638412] R10: 00000000ffffffff R11: 0000000000000246 R12: 000000000000000d
[    8.638412] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000001
[    8.638412] Kernel Offset: 0x16800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[    8.638412] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00 ]---
Comment 13 Tim Wiederhake 2022-03-30 16:10:12 UTC 
The kernel panic helpfully points out: "Fatal glibc error: CPU does not support x86-64-v2".

"x86-64-v2" requires the following cpu features: lm, cmov, cx8, fpu, fxsr, mmx, syscall, sse2, cx16, lahf, popcnt, sse4_1, sse4_2, and ssse3.

The qemu64 model does not include sse4 nor ssse3, libvirt's version of the qemu64 model additionally lacks support for lahf.

You can either resort to running guests built for x86-64-v1 (which is supported by qemu64), or adjust the cpu model in you domain xml; either by adding the required features to the qemu64 cpu or by choosing a different model.

Hope that helps!
Note You need to log in before you can comment on or make changes to this bug.