RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1658406 - mode="host-model" VMs include broken "arch-facilities" flag name [libvirt]
Summary: mode="host-model" VMs include broken "arch-facilities" flag name [libvirt]
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.6
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: rc
: ---
Assignee: Jiri Denemark
QA Contact: jiyan
URL:
Whiteboard:
Depends On:
Blocks: 1664793
TreeView+ depends on / blocked
 
Reported: 2018-12-12 00:53 UTC by Eduardo Habkost
Modified: 2019-08-06 13:14 UTC (History)
12 users (show)

Fixed In Version: libvirt-4.5.0-11.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1658407 1664793 (view as bug list)
Environment:
Last Closed: 2019-08-06 13:14:35 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:2294 0 None None None 2019-08-06 13:14:52 UTC

Description Eduardo Habkost 2018-12-12 00:53:44 UTC 
Description of problem:

When using mode="host-model" on a 7.6 host, libvirt may inadvertently enable the "arch-facilities" feature in the domain XML.

"arch-facilities" was a feature that was never included in the 7.5 kernel, but was added to the CPU feature table in qemu-kvm.  This was not a problem in RHEL-7.5 because KVM never reported the feature on GET_SUPPORTED_CPUID.

The upstream kernel added support to ARCH_CAPABILITIES later.  QEMU called the feature "arch-capabilities", but live-migration support for the feature isn't available yet (neither upstream or in the RHEL qemu-kvm package).

Then the 7.6 kernel was updated, and ARCH_CAPABILITIES was added to GET_SUPPORTED_CPUID.  Now qemu-kvm reports arch-facilities as a supported feature, mode="host-model" will enable arch-facilities in the domain XML, and we have a VM that can't be safely live-migrated.


Version-Release number of selected component (if applicable):

libvirt-4.5.0-10.el7
qemu-kvm-1.5.3-160.el7
kernel-3.10.0-957.el7


Steps to Reproduce:
Create VM with mode="host-model" on a host having arch_capabilities on /proc/cpuinfo

Actual results:
"arch-facilities" is in the domain XML after the VM is started.

Expected results:
"arch-facilities" should never be enabled in a VM.


Additional info:
It will be difficult to fix this without breaking existing VM configurations. But fixing it as soon as possible in a 7.6.z update will make sure less customers will have new VMs created with the broken arch-facilities CPU feature.

Problem was detected by code analysis while reviewing the patch for bug 1633150.  I didn't reproduce it yet.


Proposed solution:
On the libvirt side: make mode="host-model" never enable arch-facilities.


Additional actions that are (or were) being considered:

On the QEMU side: removing the "arch-facilities" feature unconditionally.  This would prevent existing VMs using the "arch-facilities" feature from running.

On the QEMU side: keep "arch-facilities" working, but always report it as unavailable.  I'm not sure we want that.  This could break libvirt feature checks in the same way as removing the feature unconditionally.

On the QEMU side: make "+arch-facilities" a no-op.  This would keep existing VMs working, but may confuse a running guest.  I'm not sure this would benefit users.

On the QEMU side: make "+arch-facilities" a migration blocker.  Probably a good idea, to make sure customers update their configurations before trying to live migrate.
Comment 4 jiyan 2018-12-22 07:07:26 UTC 
Several days ago, I reproduced this bug in RHEL-7.5 occasionally in the following comment.
But tried for many times, I can not reproduce it any more in neither RHEL-7.5 or RHEL7.6, will continue to do more research.
https://bugzilla.redhat.com/show_bug.cgi?id=1655032#c10

Prepare a RHEL-7.5 host.

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.5 (Maipo)

# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    2
Core(s) per socket:    8
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 158
Model name:            Genuine Intel(R) CPU 0000 @ 2.60GHz
Stepping:              12
CPU MHz:               4385.278
CPU max MHz:           4400.0000
CPU min MHz:           800.0000
BogoMIPS:              5184.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              16384K
NUMA node0 CPU(s):     0-15
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 ibpb ibrs stibp dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp spec_ctrl intel_stibp arch_capabilities

Obtain the output of 'domcapabilities' in RHEL-7.5.
# virsh domcapbilities > 75.xml

# cat 75.xml |grep "arch-facilities"
      <feature policy='require' name='arch-facilities'/>

In another RHEL-8 host.
Copy 75.xml to RHEL-8 host.
Obtain the output of 'domcapabilities' in RHEL-8 host.
# virsh domcapabilities > 8.xml
Merge 75.xml and 8.xml. compute cpu baseline in RHEL-8 host
# cat 75.xml > all.xml
# cat 8.xml >> all.xml

# virsh hypervisor-cpu-baseline all.xml 
error: internal error: Unknown CPU feature arch-facilities
Comment 5 jiyan 2018-12-24 10:46:51 UTC 
Reproduced in the following components:
Version:
# rpm -qa libvirt qemu-kvm kernel
libvirt-4.5.0-10.el7_6.3.x86_64
qemu-kvm-1.5.3-160.el7.x86_64
kernel-3.10.0-957.el7.x86_64

Steps:
1. Prepare a RHEL7.6 physical host with 'arch_capabilities' cpu flag
# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.6 (Maipo)

# lscpu
Flags:      ... arch_capabilities

2. Prepare a shutdown VM with 'host-model' CPU configuration, Start VM , check the dumpxml and qemu cmd line
# virsh domstate test1
shut off

# virsh dumpxml test1 |grep "<cpu" -A2
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>

# virsh start test1
Domain test1 started

# virsh dumpxml test1 |grep "<cpu" -A30
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Skylake-Client-IBRS</model>
    <vendor>Intel</vendor>
    <feature policy='disable' name='ds'/>
    <feature policy='disable' name='acpi'/>
    <feature policy='require' name='ss'/>
    <feature policy='disable' name='ht'/>
    <feature policy='disable' name='tm'/>
    <feature policy='disable' name='pbe'/>
    <feature policy='disable' name='dtes64'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='disable' name='ds_cpl'/>
    <feature policy='disable' name='vmx'/>
    <feature policy='disable' name='smx'/>
    <feature policy='disable' name='est'/>
    <feature policy='disable' name='tm2'/>
    <feature policy='disable' name='xtpr'/>
    <feature policy='disable' name='pdcm'/>
    <feature policy='disable' name='osxsave'/>
    <feature policy='disable' name='tsc_adjust'/>
    <feature policy='require' name='clflushopt'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-facilities'/>
    <feature policy='require' name='ssbd'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='disable' name='arat'/>
    <feature policy='disable' name='rdtscp'/>
  </cpu>

# ps -ef |grep test1 |grep "arch-facilities"
qemu     28838     1  1 05:42 ?        00:00:00 /usr/libexec/qemu-kvm -name test1 -S -machine rhel6.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu Skylake-Client-IBRS,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+osxsave,+tsc_adjust,+clflushopt,+stibp,** +arch-facilities **,+ssbd,+pdpe1gb
Comment 6 jiyan 2018-12-24 10:54:00 UTC 
As for the doubt in comment 4. I think maybe it is caused by 'qemu-kvm-rhev'. This issue can not be reproduced in 'qemu-kvm-rhev' (Refer to the following steps).

Version:
# rpm -qa libvirt qemu-kvm-rhev kernel
libvirt-4.5.0-10.el7_6.3.x86_64
qemu-kvm-rhev-2.12.0-19.el7_6.2.x86_64
kernel-3.10.0-957.el7.x86_64

Steps:
1. Prepare a RHEL7.6 physical host with 'arch_capabilities' cpu flag
# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.6 (Maipo)

# lscpu
Flags:      ... arch_capabilities

2. Prepare a shutdown VM with 'host-model' CPU configuration, Start VM , check the dumpxml and qemu cmd line
# virsh domstate test1
shut off

# virsh dumpxml test1 |grep "<cpu" -A3
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>
# virsh start test1
Domain test1 started

# virsh dumpxml test1 |grep "<cpu" -A14
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Skylake-Client-IBRS</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='clflushopt'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='ssbd'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='disable' name='arat'/>
    <feature policy='disable' name='rdtscp'/>
  </cpu>

# ps -ef |grep test1 
qemu     29313     1  3 05:52 ?        00:00:00 /usr/libexec/qemu-kvm -name guest=test1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-9-test1/master-key.aes -machine rhel6.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu Skylake-Client-IBRS,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,stibp=on,ssbd=on,pdpe1gb=on
Comment 7 Eduardo Habkost 2018-12-26 17:28:53 UTC 
(In reply to jiyan from comment #6)
> As for the doubt in comment 4. I think maybe it is caused by
> 'qemu-kvm-rhev'. This issue can not be reproduced in 'qemu-kvm-rhev' (Refer
> to the following steps).

Yes.  arch-facilities isn't available when using RHEL-7.5 kernel + RHEL-7.5 qemu-kvm-rhev (because RHEL-7.5 kernel lacks the feature), nor when using RHEL-7.6 kernel + RHEL-7.6 qemu-kvm-rhev (because RHEL-7.6 qemu-kvm-rhev lacks the feature).

arch-facilities is only reported when using RHEL-7.[56] qemu-kvm + RHEL-7.6 kernel.
Comment 12 jiyan 2019-04-25 02:07:32 UTC 
Version:
libvirt-4.5.0-14.el7.x86_64
qemu-kvm-rhev-2.12.0-27.el7.x86_64
kernel-3.10.0-1040.el7.x86_64

Steps:
1. Check the related cpu flags in physical host
# lscpu |grep arch
Flags:                 ... arch_capabilities

2. Prepare a shutdown VM with the following configuration
# virsh domstate avocado-vt-vm1
shut off

# virsh dumpxml avocado-vt-vm1 |grep "<cpu" -A3
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>

3. Start VM, check dumpxml file and qemu cmd line 
# virsh start avocado-vt-vm1
Domain avocado-vt-vm1 started

# virsh dumpxml avocado-vt-vm1 |grep "<cpu" -A30
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Skylake-Client-IBRS</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='clflushopt'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='ssbd'/>
    <feature policy='require' name='pdpe1gb'/>
  </cpu>

# ps -ef |grep avocado-vt-vm1
.. -cpu Skylake-Client-IBRS,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,stibp=on,ssbd=on,pdpe1gb=on ...

In step-2 and step-3, there is no 'arch-facilities' related info in dumpxml and qemu cmd line, which is expected.
Move this bug to be verified.
Comment 14 errata-xmlrpc 2019-08-06 13:14:35 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2294
Note You need to log in before you can comment on or make changes to this bug.