Bug 1609818

Summary: Error starting domain: the CPU is incompatible with host CPU: Host CPU does not provide required features: spec-ctrl
Product: Red Hat Enterprise Linux 7 Reporter: Martin Krajnak <mkrajnak>
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.6CC: berrange, hartsjc, jdenemar, jkoten, mkrajnak, tburke, tpelka
Target Milestone: rcKeywords: FutureFeature
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-30 15:12:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Error msg dialog none

Description Martin Krajnak 2018-07-30 14:28:57 UTC
Created attachment 1471563 [details]
Error msg dialog

Description of problem:
After recent update of kernel I am not able to run any VM

Version-Release number of selected component (if applicable):
kernel-3.10.0-927.el7.x86_64
kernel-3.10.0-925.el7.x86_64

libvirt-daemon-driver-storage-scsi-4.5.0-4.el7.x86_64
libvirt-daemon-driver-network-4.5.0-4.el7.x86_64
libvirt-gobject-1.0.0-1.el7.x86_64
libvirt-daemon-driver-nwfilter-4.5.0-4.el7.x86_64
libvirt-daemon-driver-nodedev-4.5.0-4.el7.x86_64
libvirt-glib-1.0.0-1.el7.x86_64
libvirt-daemon-driver-storage-core-4.5.0-4.el7.x86_64
libvirt-daemon-driver-storage-iscsi-4.5.0-4.el7.x86_64
libvirt-daemon-config-network-4.5.0-4.el7.x86_64
libvirt-daemon-driver-storage-gluster-4.5.0-4.el7.x86_64
libvirt-daemon-driver-interface-4.5.0-4.el7.x86_64
libvirt-python-4.5.0-1.el7.x86_64
libvirt-libs-4.5.0-4.el7.x86_64
libvirt-daemon-driver-qemu-4.5.0-4.el7.x86_64
libvirt-daemon-driver-storage-4.5.0-4.el7.x86_64
libvirt-daemon-4.5.0-4.el7.x86_64
libvirt-daemon-driver-storage-rbd-4.5.0-4.el7.x86_64
libvirt-daemon-kvm-4.5.0-4.el7.x86_64
libvirt-daemon-driver-secret-4.5.0-4.el7.x86_64
libvirt-daemon-driver-storage-mpath-4.5.0-4.el7.x86_64
libvirt-gconfig-1.0.0-1.el7.x86_64
libvirt-daemon-driver-storage-logical-4.5.0-4.el7.x86_64
libvirt-daemon-driver-storage-disk-4.5.0-4.el7.x86_64

How reproducible:
always

Steps to Reproduce:
1.Open virt manager
2.Try to run VM

Actual results:
VM fails to run with the message:

Error starting domain: the CPU is incompatible with host CPU: Host CPU does not provide required features: spec-ctrl

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 89, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 125, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 82, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1508, in startup
    self._backend.create()
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1080, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: the CPU is incompatible with host CPU: Host CPU does not provide required features: spec-ctrl


Expected results:
vm should start

Additional info:
probably similar to Fedora 28 bug 1584219

Also I am able to run vms when I boot the kernel-3.10.0-924.el7.x86_64

Comment 4 Martin Krajnak 2018-07-30 14:33:01 UTC
Also the HW is lenovo thinkpad t470s, cpu info:

 lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    2
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 142
Model name:            Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz
Stepping:              9
CPU MHz:               1461.328
CPU max MHz:           3900.0000
CPU min MHz:           400.0000
BogoMIPS:              5808.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              4096K
NUMA node0 CPU(s):     0-3
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp

Comment 6 Daniel Berrangé 2018-07-30 14:40:17 UTC
"spec-ctrl" is one fo the features added for the Spectre flaw at start of this year. It required a new microcode to be installed to become available. It sounds like virt-manager has somehow picked a CPU that requires spec-ctrl, but your machine lacks the microcode.

It would be useful to have the libvirt XML config for the guest in question to see what CPU it has requested. eg "virsh dumpxml $GUESTNAME", and also the output of "virsh capabilities"  and 'virsh domcapabilities kvm'

Comment 7 Martin Krajnak 2018-07-30 14:51:19 UTC
[root@t470s mkrajnak]# virsh dumpxml rhel7.6
<domain type='kvm'>
  <name>rhel7.6</name>
  <uuid>4816fc4f-b9d0-45b0-b1c0-b293f72f45e2</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='allow'>Skylake-Client-IBRS</model>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/mkrajnak/storage/rhel7.5.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:42:a2:d5'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
      <image compression='off'/>
    </graphics>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>

[root@t470s mkrajnak]# virsh capabilities
<capabilities>

  <host>
    <uuid>e0fed64c-3131-11b2-a85c-92ce1ce5772f</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Skylake-Client</model>
      <vendor>Intel</vendor>
      <microcode version='112'/>
      <topology sockets='1' cores='2' threads='2'/>
      <feature name='ds'/>
      <feature name='acpi'/>
      <feature name='ss'/>
      <feature name='ht'/>
      <feature name='tm'/>
      <feature name='pbe'/>
      <feature name='dtes64'/>
      <feature name='monitor'/>
      <feature name='ds_cpl'/>
      <feature name='vmx'/>
      <feature name='smx'/>
      <feature name='est'/>
      <feature name='tm2'/>
      <feature name='xtpr'/>
      <feature name='pdcm'/>
      <feature name='osxsave'/>
      <feature name='tsc_adjust'/>
      <feature name='clflushopt'/>
      <feature name='xsaves'/>
      <feature name='pdpe1gb'/>
      <feature name='invtsc'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='2048'/>
      <pages unit='KiB' size='1048576'/>
    </cpu>
    <power_management>
      <suspend_mem/>
      <suspend_disk/>
      <suspend_hybrid/>
    </power_management>
    <iommu support='no'/>
    <migration_features>
      <live/>
      <uri_transports>
        <uri_transport>tcp</uri_transport>
        <uri_transport>rdma</uri_transport>
      </uri_transports>
    </migration_features>
    <topology>
      <cells num='1'>
        <cell id='0'>
          <memory unit='KiB'>16415972</memory>
          <pages unit='KiB' size='4'>4103993</pages>
          <pages unit='KiB' size='2048'>0</pages>
          <pages unit='KiB' size='1048576'>0</pages>
          <distances>
            <sibling id='0' value='10'/>
          </distances>
          <cpus num='4'>
            <cpu id='0' socket_id='0' core_id='0' siblings='0,2'/>
            <cpu id='1' socket_id='0' core_id='1' siblings='1,3'/>
            <cpu id='2' socket_id='0' core_id='0' siblings='0,2'/>
            <cpu id='3' socket_id='0' core_id='1' siblings='1,3'/>
          </cpus>
        </cell>
      </cells>
    </topology>
    <cache>
      <bank id='0' level='3' type='both' size='4' unit='MiB' cpus='0-3'/>
    </cache>
    <secmodel>
      <model>selinux</model>
      <doi>0</doi>
      <baselabel type='kvm'>system_u:system_r:svirt_t:s0</baselabel>
      <baselabel type='qemu'>system_u:system_r:svirt_tcg_t:s0</baselabel>
    </secmodel>
    <secmodel>
      <model>dac</model>
      <doi>0</doi>
      <baselabel type='kvm'>+107:+107</baselabel>
      <baselabel type='qemu'>+107:+107</baselabel>
    </secmodel>
  </host>

  <guest>
    <os_type>hvm</os_type>
    <arch name='i686'>
      <wordsize>32</wordsize>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <machine maxCpus='240'>pc-i440fx-rhel7.0.0</machine>
      <machine canonical='pc-i440fx-rhel7.0.0' maxCpus='240'>pc</machine>
      <machine maxCpus='240'>rhel6.0.0</machine>
      <machine maxCpus='240'>rhel6.1.0</machine>
      <machine maxCpus='240'>rhel6.2.0</machine>
      <machine maxCpus='240'>rhel6.3.0</machine>
      <machine maxCpus='240'>rhel6.4.0</machine>
      <machine maxCpus='240'>rhel6.5.0</machine>
      <machine maxCpus='240'>rhel6.6.0</machine>
      <domain type='qemu'/>
      <domain type='kvm'>
        <emulator>/usr/libexec/qemu-kvm</emulator>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <disksnapshot default='off' toggle='no'/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
      <pae/>
      <nonpae/>
    </features>
  </guest>

  <guest>
    <os_type>hvm</os_type>
    <arch name='x86_64'>
      <wordsize>64</wordsize>
      <emulator>/usr/libexec/qemu-kvm</emulator>
      <machine maxCpus='240'>pc-i440fx-rhel7.0.0</machine>
      <machine canonical='pc-i440fx-rhel7.0.0' maxCpus='240'>pc</machine>
      <machine maxCpus='240'>rhel6.0.0</machine>
      <machine maxCpus='240'>rhel6.1.0</machine>
      <machine maxCpus='240'>rhel6.2.0</machine>
      <machine maxCpus='240'>rhel6.3.0</machine>
      <machine maxCpus='240'>rhel6.4.0</machine>
      <machine maxCpus='240'>rhel6.5.0</machine>
      <machine maxCpus='240'>rhel6.6.0</machine>
      <domain type='qemu'/>
      <domain type='kvm'>
        <emulator>/usr/libexec/qemu-kvm</emulator>
      </domain>
    </arch>
    <features>
      <cpuselection/>
      <deviceboot/>
      <disksnapshot default='off' toggle='no'/>
      <acpi default='on' toggle='yes'/>
      <apic default='on' toggle='no'/>
    </features>
  </guest>

</capabilities>

Thanks, I'll try to update the microcode ASAP

Comment 8 Daniel Berrangé 2018-07-30 15:03:30 UTC
Ok your guest has been given the -IBRS variant which requires spec-ctrl

    <model fallback='allow'>Skylake-Client-IBRS</model>

but the host does not support this:

    <model>Skylake-Client</model>

So either don't request this -IBRS CPU for your guest, or you'll need the new microcode.

Comment 9 Martin Krajnak 2018-07-30 15:12:43 UTC
Ok, I noticed just now that I have that CPU param, not sure where it came from I was just confused since it worked on different kernel.

So if anyone hits this issue all you have to do is:

1.Open virt-manager
2.Go to parameters of VM
3.Go to cpu section
4.Check "Copy host CPU configuration" and click Apply

that fixed the problem for me

thanks a lot Daniel

Comment 10 James Hartsock 2018-09-17 13:52:44 UTC
The virt-install way would be to add '--cpu host', or at least it worked for me.