Bug 1532401 - VIrt install fails with CPU feature spec-ctrl not found
Summary: VIrt install fails with CPU feature spec-ctrl not found
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.5
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: Libvirt Maintainers
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-01-08 22:33 UTC by Christian Trautman
Modified: 2018-05-25 08:38 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-01-17 11:34:34 UTC


Attachments (Terms of Use)
sosreport (8.37 MB, application/x-xz)
2018-01-08 22:33 UTC, Christian Trautman
no flags Details

Description Christian Trautman 2018-01-08 22:33:15 UTC
Created attachment 1378745 [details]
sosreport

Description of problem:
Running virt install after installing QEMU-kvm-rhev fails with "CPU feature spec-ctrl not found"


Version-Release number of selected component (if applicable):
kernel 3.10.0-826.el7.x86_64

#qemu-kvm-rhev

qemu-img-rhev-2.6.0-28.el7_3.9.x86_64
libvirt-daemon-driver-qemu-3.9.0-7.el7.x86_64
ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
qemu-kvm-common-rhev-2.6.0-28.el7_3.9.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.9.x86_64


How reproducible:
Always

Steps to Reproduce:
1. Provision system with latest compose for 7.5 RHEL-7.5-20180107.n.0

2. Add openstack repo file
cat <<EOT >> /etc/yum.repos.d/osp8-rhel.repo
[osp8-rhel7]
name=osp8-rhel7
baseurl=http://download.lab.bos.redhat.com/rel-eng/OpenStack/8.0-RHEL-7/latest/RH7-RHOS-8.0/x86_64/os/
enabled=1
gpgcheck=0
skip_if_unavailable=1
EOT

3. Install qemu-kvm-rhev

4. Install virt-install
yum install -y virt-install libvirt
systemctl start libvirtd

5. Set some variables 
LOCATION="http://download-node-02.eng.bos.redhat.com/released/RHEL-7/7.3/Server/x86_64/os/"
CPUS=3
vm=master
bridge=virbr0
master_image=master.qcow2
image_path=/var/lib/libvirt/images/
dist=rhel73
location=$LOCATION

6. create kickstart file
cat << KS_CFG > $dist-vm.ks
# System authorization information
auth --enableshadow --passalgo=sha512

# Use network installation
url --url=$location

# Use text mode install
text
# Run the Setup Agent on first boot
firstboot --enable
ignoredisk --only-use=vda
# Keyboard layouts
keyboard --vckeymap=us --xlayouts='us'
# System language
lang en_US.UTF-8

# Network information
network  --bootproto=dhcp --device=eth0 --ipv6=auto --activate
# Root password
rootpw  redhat
# Do not configure the X Window System
skipx
# System timezone
timezone US/Eastern --isUtc --ntpservers=10.16.31.254,clock.util.phx2.redhat.com,clock02.util.phx2.redhat.com
# System bootloader configuration
bootloader --location=mbr --timeout=5 --append="crashkernel=auto rhgb quiet console=ttyS0,115200"
# Partition clearing information
autopart --type=plain
clearpart --all --initlabel --drives=vda
zerombr

%packages
@base
@core
@network-tools
%end
shutdown

7. Run virt-install
virt-install --name=$vm\
	 --virt-type=kvm\
	 --disk path=$image_path/$master_image,format=qcow2,,size=3,bus=virtio\
	 --vcpus=$CPUS\
	 --ram=4096\
	 --network bridge=$bridge\
	 --graphics none\
	 --extra-args="$extra"\
	 --initrd-inject=$dist-vm.ks\
	 --location=$location\
	 --noreboot\
         --serial pty\
         --serial file,path=/tmp/$vm.console
Actual results:
Fails with 
2018-01-08T22:20:57.854198Z qemu-kvm: CPU feature spec-ctrl not found
2018-01-08 22:20:57.863+0000: shutting down, reason=failed

Expected results:
Virt-install should work

Additional info:
This has been working as this is part of a setup script we use often in our CI. I only noticed this issue when trying to run our scripts today. It appears the compose from January 2nd works OK. So something between then and the latest compose has broken this. 

SOSreport attached

Comment 3 Christian Trautman 2018-01-15 18:04:08 UTC
I believe the current nightly build has corrected this error. My setup script on the current 7.5 nightly in Beaker did not fail with this error. Will fully verify later.

Comment 4 Eduardo Habkost 2018-01-16 14:41:16 UTC
According to sosreport, the guest didn't have spec_ctrl available.  I don't know if this is a libvirt or virt-install bug, but moving it to libvirt so it can be investigated.

(What's the default CPU model used by virt-install?)

Comment 5 Eduardo Habkost 2018-01-16 14:42:38 UTC
For reference, libvirt logs on sosreport shows that it is (incorrectly) using "-cpu Broadwell,+spec-ctrl,+rtm,+hle".

Comment 6 Eduardo Habkost 2018-01-16 14:43:41 UTC
sos_commands/virsh/virsh_-r_capabilities:

<capabilities>
  <host>
    <uuid>4c4c4544-0030-3110-8037-b8c04f484b32</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Broadwell-IBRS</model>
      <vendor>Intel</vendor>
      <microcode version='184549413'/>
      <topology sockets='1' cores='12' threads='2'/>
      <feature name='vme'/>
      <feature name='ds'/>
      <feature name='acpi'/>
      <feature name='ss'/>
      <feature name='ht'/>
      <feature name='tm'/>
      <feature name='pbe'/>
      <feature name='dtes64'/>
      <feature name='monitor'/>
      <feature name='ds_cpl'/>
      <feature name='vmx'/>
      <feature name='smx'/>
      <feature name='est'/>
      <feature name='tm2'/>
      <feature name='xtpr'/>
      <feature name='pdcm'/>
      <feature name='dca'/>
      <feature name='osxsave'/>
      <feature name='f16c'/>
      <feature name='rdrand'/>
      <feature name='arat'/>
      <feature name='tsc_adjust'/>
      <feature name='cmt'/>
      <feature name='stibp'/>
      <feature name='xsaveopt'/>
      <feature name='mbm_total'/>
      <feature name='mbm_local'/>
      <feature name='pdpe1gb'/>
      <feature name='abm'/>
      <feature name='invtsc'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='2048'/>
      <pages unit='KiB' size='1048576'/>
    </cpu>
[...]

Comment 7 Eduardo Habkost 2018-01-16 14:44:54 UTC
sosreport-netqe25.knqe.lab.eng.bos.redhat.com-20180108172208/proc/cpuinfo:

processor       : 47
vendor_id       : GenuineIntel
cpu family      : 6
model           : 79
model name      : Intel(R) Xeon(R) CPU E5-2687W v4 @ 3.00GHz
stepping        : 1
microcode       : 0xb000025
cpu MHz         : 2866.149
cache size      : 30720 KB
physical id     : 1
siblings        : 24
core id         : 13
cpu cores       : 12
apicid          : 59
initial apicid  : 59
fpu             : yes
fpu_exception   : yes
cpuid level     : 20
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts
bogomips        : 6004.38
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

Comment 8 Jiri Denemark 2018-01-17 11:34:34 UTC
This would suggest libvirt detected the new feature in the host CPU via CPUID, but the kernel is not new enough to recognize it. Moreover, qemu-kvm-rhev is pretty old and doesn't support the new feature either:

* Fri Mar 24 2017 Miroslav Rezanina <mrezanin@redhat.com> - rhev-2.6.0-28.el7_3.9

I think virt-install by default takes the CPU model from libvirt's capabilities XML, which is Broadwell-IBRS. Since qemu-kvm-rhev does not support the new CPU model, libvirt uses Broadwell,+spec-ctrl,+rtm,+hle instead.

In other words, this was just a temporary issue with the compose when libvirt is updated while others are not updated yet. The problem can be fixed by detecting CPU capabilities via QEMU and teaching virt-install to use the CPU from domain capabilities. Libvirt does so with QEMU-2.10.

Comment 9 Dan Yasny 2018-04-18 20:52:02 UTC
Just ran into this problem:

# virt-install --name node-2    --disk path=/var/lib/libvirt/images/node-2-disk1.qcow2,device=disk,bus=virtio,format=qcow2,cache=unsafe    --boot hd   --network network:management --virt-type kvm --cpu host-model --ram 8196 --vcpus 4 --os-variant rhel7 --import --noautoconsole --autostart --vnc --rng /dev/urandom

Starting install...
ERROR    internal error: process exited while connecting to monitor: 2018-04-18T20:47:24.690482Z qemu-kvm: CPU feature spec-ctrl not found
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect qemu:///system start node-2
otherwise, please restart your installation.


Versions:
qemu-kvm-rhev-2.6.0-28.el7_3.12.x86_64
qemu-kvm-common-rhev-2.6.0-28.el7_3.12.x86_64
ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
qemu-img-rhev-2.6.0-28.el7_3.12.x86_64
libvirt-daemon-driver-qemu-3.9.0-14.el7_5.2.x86_64
libvirt-3.9.0-14.el7_5.2.x86_64
libvirt-libs-3.9.0-14.el7_5.2.x86_64
libvirt-daemon-kvm-3.9.0-14.el7_5.2.x86_64
kernel-3.10.0-862.el7.x86_64
kernel-tools-libs-3.10.0-862.el7.x86_64
kernel-headers-3.10.0-862.el7.x86_64
kernel-tools-3.10.0-862.el7.x86_64


# virsh capabilities
<capabilities>

  <host>
    <uuid>4c4c4544-0054-3210-8034-b4c04f39354a</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Westmere-IBRS</model>
      <vendor>Intel</vendor>
      <microcode version='30'/>
      <topology sockets='1' cores='6' threads='2'/>
      <feature name='vme'/>
      <feature name='ds'/>
      <feature name='acpi'/>
      <feature name='ss'/>
      <feature name='ht'/>
      <feature name='tm'/>
      <feature name='pbe'/>
      <feature name='pclmuldq'/>
      <feature name='dtes64'/>
      <feature name='monitor'/>
      <feature name='ds_cpl'/>
      <feature name='vmx'/>
      <feature name='smx'/>
      <feature name='est'/>
      <feature name='tm2'/>
      <feature name='xtpr'/>
      <feature name='pdcm'/>
      <feature name='pcid'/>
      <feature name='dca'/>
      <feature name='arat'/>
      <feature name='stibp'/>
      <feature name='pdpe1gb'/>
      <feature name='rdtscp'/>
      <feature name='invtsc'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='2048'/>
      <pages unit='KiB' size='1048576'/>
    </cpu>
...


This is with a RHEL7.5 host installed and updated from RHN. 3 months after this BZ has been closed.

Should I reopen the BZ?

Comment 10 Jiri Denemark 2018-04-19 07:40:17 UTC
No, just update qemu-kvm-rhev, 2.6.0-28.el7_3.12.x86_64 is pretty old (released on Fri Jun 23 2017) and definitely not from 7.5.


Note You need to log in before you can comment on or make changes to this bug.