Bug 1658406

Summary: mode="host-model" VMs include broken "arch-facilities" flag name [libvirt]
Product: Red Hat Enterprise Linux 7 Reporter: Eduardo Habkost <ehabkost>
Component: libvirtAssignee: Jiri Denemark <jdenemar>
Status: CLOSED ERRATA QA Contact: jiyan <jiyan>
Severity: urgent Docs Contact:
Priority: high    
Version: 7.6CC: berrange, dzheng, ehabkost, fjin, gsun, haizhao, jdenemar, jiyan, kchamart, lhuang, mtessun, yalzhang
Target Milestone: rcKeywords: Regression, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-4.5.0-11.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1658407 1664793 (view as bug list) Environment:
Last Closed: 2019-08-06 13:14:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1664793    

Description Eduardo Habkost 2018-12-12 00:53:44 UTC 
Description of problem:

When using mode="host-model" on a 7.6 host, libvirt may inadvertently enable the "arch-facilities" feature in the domain XML.

"arch-facilities" was a feature that was never included in the 7.5 kernel, but was added to the CPU feature table in qemu-kvm.  This was not a problem in RHEL-7.5 because KVM never reported the feature on GET_SUPPORTED_CPUID.

The upstream kernel added support to ARCH_CAPABILITIES later.  QEMU called the feature "arch-capabilities", but live-migration support for the feature isn't available yet (neither upstream or in the RHEL qemu-kvm package).

Then the 7.6 kernel was updated, and ARCH_CAPABILITIES was added to GET_SUPPORTED_CPUID.  Now qemu-kvm reports arch-facilities as a supported feature, mode="host-model" will enable arch-facilities in the domain XML, and we have a VM that can't be safely live-migrated.


Version-Release number of selected component (if applicable):

libvirt-4.5.0-10.el7
qemu-kvm-1.5.3-160.el7
kernel-3.10.0-957.el7


Steps to Reproduce:
Create VM with mode="host-model" on a host having arch_capabilities on /proc/cpuinfo

Actual results:
"arch-facilities" is in the domain XML after the VM is started.

Expected results:
"arch-facilities" should never be enabled in a VM.


Additional info:
It will be difficult to fix this without breaking existing VM configurations. But fixing it as soon as possible in a 7.6.z update will make sure less customers will have new VMs created with the broken arch-facilities CPU feature.

Problem was detected by code analysis while reviewing the patch for bug 1633150.  I didn't reproduce it yet.


Proposed solution:
On the libvirt side: make mode="host-model" never enable arch-facilities.


Additional actions that are (or were) being considered:

On the QEMU side: removing the "arch-facilities" feature unconditionally.  This would prevent existing VMs using the "arch-facilities" feature from running.

On the QEMU side: keep "arch-facilities" working, but always report it as unavailable.  I'm not sure we want that.  This could break libvirt feature checks in the same way as removing the feature unconditionally.

On the QEMU side: make "+arch-facilities" a no-op.  This would keep existing VMs working, but may confuse a running guest.  I'm not sure this would benefit users.

On the QEMU side: make "+arch-facilities" a migration blocker.  Probably a good idea, to make sure customers update their configurations before trying to live migrate.
Comment 4 jiyan 2018-12-22 07:07:26 UTC 
Several days ago, I reproduced this bug in RHEL-7.5 occasionally in the following comment.
But tried for many times, I can not reproduce it any more in neither RHEL-7.5 or RHEL7.6, will continue to do more research.
https://bugzilla.redhat.com/show_bug.cgi?id=1655032#c10

Prepare a RHEL-7.5 host.

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.5 (Maipo)

# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    2
Core(s) per socket:    8
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 158
Model name:            Genuine Intel(R) CPU 0000 @ 2.60GHz
Stepping:              12
CPU MHz:               4385.278
CPU max MHz:           4400.0000
CPU min MHz:           800.0000
BogoMIPS:              5184.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              16384K
NUMA node0 CPU(s):     0-15
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 ibpb ibrs stibp dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp spec_ctrl intel_stibp arch_capabilities

Obtain the output of 'domcapabilities' in RHEL-7.5.
# virsh domcapbilities > 75.xml

# cat 75.xml |grep "arch-facilities"
      <feature policy='require' name='arch-facilities'/>

In another RHEL-8 host.
Copy 75.xml to RHEL-8 host.
Obtain the output of 'domcapabilities' in RHEL-8 host.
# virsh domcapabilities > 8.xml
Merge 75.xml and 8.xml. compute cpu baseline in RHEL-8 host
# cat 75.xml > all.xml
# cat 8.xml >> all.xml

# virsh hypervisor-cpu-baseline all.xml 
error: internal error: Unknown CPU feature arch-facilities
Comment 5 jiyan 2018-12-24 10:46:51 UTC 
Reproduced in the following components:
Version:
# rpm -qa libvirt qemu-kvm kernel
libvirt-4.5.0-10.el7_6.3.x86_64
qemu-kvm-1.5.3-160.el7.x86_64
kernel-3.10.0-957.el7.x86_64

Steps:
1. Prepare a RHEL7.6 physical host with 'arch_capabilities' cpu flag
# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.6 (Maipo)

# lscpu
Flags:      ... arch_capabilities

2. Prepare a shutdown VM with 'host-model' CPU configuration, Start VM , check the dumpxml and qemu cmd line
# virsh domstate test1
shut off

# virsh dumpxml test1 |grep "<cpu" -A2
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>

# virsh start test1
Domain test1 started

# virsh dumpxml test1 |grep "<cpu" -A30
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Skylake-Client-IBRS</model>
    <vendor>Intel</vendor>
    <feature policy='disable' name='ds'/>
    <feature policy='disable' name='acpi'/>
    <feature policy='require' name='ss'/>
    <feature policy='disable' name='ht'/>
    <feature policy='disable' name='tm'/>
    <feature policy='disable' name='pbe'/>
    <feature policy='disable' name='dtes64'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='disable' name='ds_cpl'/>
    <feature policy='disable' name='vmx'/>
    <feature policy='disable' name='smx'/>
    <feature policy='disable' name='est'/>
    <feature policy='disable' name='tm2'/>
    <feature policy='disable' name='xtpr'/>
    <feature policy='disable' name='pdcm'/>
    <feature policy='disable' name='osxsave'/>
    <feature policy='disable' name='tsc_adjust'/>
    <feature policy='require' name='clflushopt'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-facilities'/>
    <feature policy='require' name='ssbd'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='disable' name='arat'/>
    <feature policy='disable' name='rdtscp'/>
  </cpu>

# ps -ef |grep test1 |grep "arch-facilities"
qemu     28838     1  1 05:42 ?        00:00:00 /usr/libexec/qemu-kvm -name test1 -S -machine rhel6.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu Skylake-Client-IBRS,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+osxsave,+tsc_adjust,+clflushopt,+stibp,** +arch-facilities **,+ssbd,+pdpe1gb
Comment 6 jiyan 2018-12-24 10:54:00 UTC 
As for the doubt in comment 4. I think maybe it is caused by 'qemu-kvm-rhev'. This issue can not be reproduced in 'qemu-kvm-rhev' (Refer to the following steps).

Version:
# rpm -qa libvirt qemu-kvm-rhev kernel
libvirt-4.5.0-10.el7_6.3.x86_64
qemu-kvm-rhev-2.12.0-19.el7_6.2.x86_64
kernel-3.10.0-957.el7.x86_64

Steps:
1. Prepare a RHEL7.6 physical host with 'arch_capabilities' cpu flag
# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.6 (Maipo)

# lscpu
Flags:      ... arch_capabilities

2. Prepare a shutdown VM with 'host-model' CPU configuration, Start VM , check the dumpxml and qemu cmd line
# virsh domstate test1
shut off

# virsh dumpxml test1 |grep "<cpu" -A3
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>
# virsh start test1
Domain test1 started

# virsh dumpxml test1 |grep "<cpu" -A14
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Skylake-Client-IBRS</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='clflushopt'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='ssbd'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='disable' name='arat'/>
    <feature policy='disable' name='rdtscp'/>
  </cpu>

# ps -ef |grep test1 
qemu     29313     1  3 05:52 ?        00:00:00 /usr/libexec/qemu-kvm -name guest=test1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-9-test1/master-key.aes -machine rhel6.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu Skylake-Client-IBRS,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,stibp=on,ssbd=on,pdpe1gb=on
Comment 7 Eduardo Habkost 2018-12-26 17:28:53 UTC 
(In reply to jiyan from comment #6)
> As for the doubt in comment 4. I think maybe it is caused by
> 'qemu-kvm-rhev'. This issue can not be reproduced in 'qemu-kvm-rhev' (Refer
> to the following steps).

Yes.  arch-facilities isn't available when using RHEL-7.5 kernel + RHEL-7.5 qemu-kvm-rhev (because RHEL-7.5 kernel lacks the feature), nor when using RHEL-7.6 kernel + RHEL-7.6 qemu-kvm-rhev (because RHEL-7.6 qemu-kvm-rhev lacks the feature).

arch-facilities is only reported when using RHEL-7.[56] qemu-kvm + RHEL-7.6 kernel.
Comment 12 jiyan 2019-04-25 02:07:32 UTC 
Version:
libvirt-4.5.0-14.el7.x86_64
qemu-kvm-rhev-2.12.0-27.el7.x86_64
kernel-3.10.0-1040.el7.x86_64

Steps:
1. Check the related cpu flags in physical host
# lscpu |grep arch
Flags:                 ... arch_capabilities

2. Prepare a shutdown VM with the following configuration
# virsh domstate avocado-vt-vm1
shut off

# virsh dumpxml avocado-vt-vm1 |grep "<cpu" -A3
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>

3. Start VM, check dumpxml file and qemu cmd line 
# virsh start avocado-vt-vm1
Domain avocado-vt-vm1 started

# virsh dumpxml avocado-vt-vm1 |grep "<cpu" -A30
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Skylake-Client-IBRS</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='clflushopt'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='ssbd'/>
    <feature policy='require' name='pdpe1gb'/>
  </cpu>

# ps -ef |grep avocado-vt-vm1
.. -cpu Skylake-Client-IBRS,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,stibp=on,ssbd=on,pdpe1gb=on ...

In step-2 and step-3, there is no 'arch-facilities' related info in dumpxml and qemu cmd line, which is expected.
Move this bug to be verified.
Comment 14 errata-xmlrpc 2019-08-06 13:14:35 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2294