Bug 1697627

Summary:

CPU mode=host-model needs to include MSR features

Product:

Red Hat Enterprise Linux 8

Reporter:

Eduardo Habkost <ehabkost>

Component:

libvirt

Assignee:

Jiri Denemark <jdenemar>

Status:

CLOSED ERRATA

QA Contact:

jiyan <jiyan>

Severity:

medium

Docs Contact:

Priority:

high

Version:

8.0

CC:

ailan, jdenemar, jiyan, kchamart, knoel, rbalakri, yalzhang, yuhuang

Target Milestone:

Target Release:

8.0

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

libvirt-4.5.0-29.el8

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2019-11-05 20:48:28 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

1709970, 1747185

Bug Blocks:

1710589

Attachments:

Description	Flags
domcap_and_cap_info_on_qemu-kvm-2.12.0-81	none
domcap_and_cap_info_on_qemu-kvm-v4.1.0-rc1-50-g23da9e297b	none

Description Eduardo Habkost 2019-04-08 20:56:23 UTC

When using CPU mode=host-model, the bug reported at bug 1687578 must not be present and the vulnerability report on the guest should match the host.  This means features in MSR_IA32_ARCH_CAPABILITIES need to be included in the CPU configuration and passed to the guest.

Comment 4 Jiri Denemark 2019-06-20 12:35:37 UTC

This is now implemented upstream in a several series ending with

commit 2674d00ed484091faf2b6e6b1efe58ee9a72b96b
Refs: v5.4.0-300-g2674d00ed4
Author:     Jiri Denemark <jdenemar>
AuthorDate: Wed Jun 19 22:22:09 2019 +0200
Commit:     Jiri Denemark <jdenemar>
CommitDate: Thu Jun 20 14:02:36 2019 +0200

    qemu: Drop MSR features from host-model with old QEMU

    With QEMU versions which lack "unavailable-features" we use CPUID based
    detection of features which were enabled or disabled once QEMU starts.
    Thus using MSR features with host-model would result in all of them
    being marked as disabled in the active domain definition even though
    QEMU did not actually disable them.

    Let's make sure we add MSR features to host-model only when
    "unavailable-features" property is supported by QEMU.

    Signed-off-by: Jiri Denemark <jdenemar>
    Reviewed-by: Ján Tomko <jtomko>

Comment 7 jiyan 2019-07-16 09:14:48 UTC

Version:
libvirt-4.5.0-30.module+el8.1.0+3574+3a63752b.x86_64
qemu-kvm-2.12.0-81.module+el8.1.0+3619+dfe1ae01.x86_64
kernel-4.18.0-107.el8.x86_64

ENV: 
Prepare a physical host with msr cpu flag
# lscpu |grep msr
Model name:          AMD EPYC 7601 32-Core Processor
Flags:               fpu vme de pse tsc msr

Steps:
S1: Check the diff between cpu_map.xml; capabilities and domcapabilities
1. Compare the diff of capabilities of libvirt-4.5.0-24 and libvirt-4.5.0-30
# diff capabilities_4.5.0-24.xml capabilities_4.5.0-30.xml 
No output

2. Compare the diff of domcapabilities of libvirt-4.5.0-24 and libvirt-4.5.0-30
# diff domcapabilities_4.5.0-24.xml domcapabilities_4.5.0-30.xml 
29a30
>       <feature policy='require' name='arch-capabilities'/>

3. Compare the diff of cpu_map.xml of libvirt-4.5.0-24 and libvirt-4.5.0-30
# diff cpu_map_4.5.0-24.xml cpu_map_4.5.0-30.xml 
339a340,342
>     <feature name='arch-capabilities'> <!-- arch_capabilities, arch-facilities -->
>       <cpuid eax_in='0x07' ecx_in='0x00' edx='0x20000000'/>
>     </feature>
481a485,504
>     <!-- IA32_ARCH_CAPABILITIES features -->
>     <feature name='rdctl-no'>
>       <msr index='0x10a' edx='0x00000000' eax='0x00000001'/>
>     </feature>
>     <feature name='ibrs-all'>
>       <msr index='0x10a' edx='0x00000000' eax='0x00000002'/>
>     </feature>
>     <feature name='rsba'>
>       <msr index='0x10a' edx='0x00000000' eax='0x00000004'/>
>     </feature>
>     <feature name='skip-l1dfl-vmentry'>
>       <msr index='0x10a' edx='0x00000000' eax='0x00000008'/>
>     </feature>
>     <feature name='ssb-no'>
>       <msr index='0x10a' edx='0x00000000' eax='0x00000010'/>
>     </feature>
>     <feature name='mds-no'>
>       <msr index='0x10a' edx='0x00000000' eax='0x00000020'/>
>     </feature>

4. Use the output of domcapabilities as the input of “hypervisor-cpu-baseline/compare”
# virsh hypervisor-cpu-baseline domcapabilities_4.5.0-30.xml 
<cpu mode='custom' match='exact'>
  <model fallback='forbid'>EPYC-IBPB</model>
  <vendor>AMD</vendor>
  <feature policy='require' name='x2apic'/>
  <feature policy='require' name='tsc-deadline'/>
  <feature policy='require' name='hypervisor'/>
  <feature policy='require' name='tsc_adjust'/>
  <feature policy='require' name='arch-capabilities'/>
  <feature policy='require' name='cmp_legacy'/>
  <feature policy='require' name='perfctr_core'/>
  <feature policy='require' name='invtsc'/>
  <feature policy='require' name='virt-ssbd'/>
  <feature policy='disable' name='monitor'/>
  <feature policy='disable' name='svm'/>
</cpu>

# virsh hypervisor-cpu-compare domcapabilities_4.5.0-30.xml 
CPU described in domcapabilities_4.5.0-30.xml is identical to the CPU provided by hypervisor on the host



S2. Start VM with host-model cpu conf
# virsh domstate test
shut off

# virsh dumpxml test --inactive |grep "<cpu" -A5
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>

# virsh start test
Domain test started

# virsh dumpxml test |grep "<cpu" -A17
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>EPYC-IBPB</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='tsc-deadline'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='require' name='perfctr_core'/>
    <feature policy='require' name='virt-ssbd'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='disable' name='svm'/>
    <feature policy='require' name='topoext'/>
  </cpu>

# ps -ef |grep test
qemu      98413      1 51 00:50 ?        
-cpu EPYC-IBPB,x2apic=on,tsc-deadline=on,hypervisor=on,tsc_adjust=on,arch-capabilities=on,cmp_legacy=on,perfctr_core=on,virt-ssbd=on,monitor=off,svm=off 

# virsh console test
Connected to domain test
Escape character is ^]

Red Hat Enterprise Linux 8.1 Beta (Ootpa)
Kernel 4.18.0-107.el8.x86_64 on an x86_64

localhost login: root
Password: 
[root@localhost ~]# lscpu |grep arch_capabilities
Flags:               fpu vme de pse tsc ... arch_capabilities

Comment 8 jiyan 2019-07-16 09:20:46 UTC

S3: Test upgrading libvirt-4.5.0-24 to libvirt-4.5.0-30
1. Check the diff between  cpu_map.xml; capabilities and domcapabilities
As S1 shows

2. Start VM under libvirt-4.5.0-24; upgrade libvirt and restart VM
# rpm -qa libvirt
libvirt-4.5.0-24.module+el8.1.0+3205+41ff0a42.x86_64

# virsh domstate test
shut off

# virsh dumpxml test --inactive |grep "<cpu" -A2
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>

# virsh start test
Domain test started

# virsh dumpxml test |grep "<cpu" -A15
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>EPYC-IBPB</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='tsc-deadline'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='require' name='perfctr_core'/>
    <feature policy='require' name='virt-ssbd'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='disable' name='svm'/>
    <feature policy='require' name='topoext'/>
  </cpu>

# yum update libvirt*
# rpm -qa libvirt 
libvirt-4.5.0-30.module+el8.1.0+3574+3a63752b.x86_64

# systemctl restart libvirtd

# virsh dumpxml test |grep "<cpu" -A15
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>EPYC-IBPB</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='tsc-deadline'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='require' name='perfctr_core'/>
    <feature policy='require' name='virt-ssbd'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='disable' name='svm'/>
    <feature policy='require' name='topoext'/>
  </cpu>

# virsh managedsave test
Domain test state saved by libvirt

# virsh start test
Domain test started

# virsh dumpxml test |grep "<cpu" -A15
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>EPYC-IBPB</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='tsc-deadline'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='require' name='perfctr_core'/>
    <feature policy='require' name='virt-ssbd'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='disable' name='svm'/>
    <feature policy='require' name='topoext'/>
  </cpu>

# virsh destroy test;virsh start test
Domain test destroyed

Domain test started

# virsh dumpxml test |grep "<cpu" -A15
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>EPYC-IBPB</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='tsc-deadline'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='require' name='perfctr_core'/>
    <feature policy='require' name='virt-ssbd'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='disable' name='svm'/>
    <feature policy='require' name='topoext'/>
  </cpu>

# ps -ef |grep test
qemu      98413      1 51 00:50 ?        
-cpu EPYC-IBPB,x2apic=on,tsc-deadline=on,hypervisor=on,tsc_adjust=on,arch-capabilities=on,cmp_legacy=on,perfctr_core=on,virt-ssbd=on,monitor=off,svm=off 

# virsh console test
Connected to domain test
Escape character is ^]

Red Hat Enterprise Linux 8.1 Beta (Ootpa)
Kernel 4.18.0-107.el8.x86_64 on an x86_64

localhost login: root
Password: 
[root@localhost ~]# lscpu |grep arch_capabilities
Flags:               fpu vme de pse tsc ... arch_capabilities




S4: Test libvirt-4.5.0-30 with lower qemu-kvm version
Version:
libvirt-4.5.0-30.module+el8.1.0+3574+3a63752b.x86_64
qemu-kvm-2.12.0-65.module+el8.1.0+2983+b2ae9c0a.x86_64
kernel-4.18.0-107.el8.x86_64

Steps:
# virsh domstate test
shut off

# virsh dumpxml test --inactive |grep "<cpu" -A2
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>

# virsh start test
Domain test started

# virsh dumpxml test |grep "<cpu" -A13
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>EPYC-IBPB</model>
    <vendor>AMD</vendor>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='tsc-deadline'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='require' name='perfctr_core'/>
    <feature policy='require' name='virt-ssbd'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='disable' name='svm'/>
    <feature policy='require' name='topoext'/>
  </cpu>

# ps -ef |grep test
qemu     103593      1 98 01:20 ?    
-cpu EPYC-IBPB,x2apic=on,tsc-deadline=on,hypervisor=on,tsc_adjust=on,cmp_legacy=on,perfctr_core=on,virt-ssbd=on,monitor=off,svm=off

# virsh console test
Connected to domain test
Escape character is ^]

Red Hat Enterprise Linux 8.1 Beta (Ootpa)
Kernel 4.18.0-107.el8.x86_64 on an x86_64

localhost login: root
Password: 
[root@localhost ~]# lscpu |grep arch_capabilities
No output

Comment 9 jiyan 2019-07-23 03:25:59 UTC

Please ignore the msg in comment 7 and comment 8; which was executed on the physical machine not actually supports MSR.

Comment 10 jiyan 2019-07-23 03:45:15 UTC

Hi jiri As we have discussed yesterday. 
I give the 2 results of testing MSR on different QEMU versions:
S1: "libvirt-4.5.0-30.module+el8.1.0+3574+3a63752b.x86_64" + "qemu-kvm-2.12.0-81.module+el8.1.0+3619+dfe1ae01.x86_64"
S2: "libvirt-4.5.0-30.module+el8.1.0+3574+3a63752b.x86_64" + "QEMU emulator version 4.0.91 (v4.1.0-rc1-50-g23da9e297b)"


S1: "libvirt-4.5.0-30.module+el8.1.0+3574+3a63752b.x86_64" + "qemu-kvm-2.12.0-81.module+el8.1.0+3619+dfe1ae01.x86_64"
Version:
kernel-4.18.0-119.el8.x86_64
libvirt-4.5.0-30.module+el8.1.0+3574+3a63752b.x86_64
qemu-kvm-2.12.0-81.module+el8.1.0+3619+dfe1ae01.x86_64

Steps：
# lscpu |grep arch_capabilities
Flags:               ... arch_capabilities

# virsh capabilities 
<capabilities>

  <host>
    <uuid>03fa4990-320e-4990-fa03-0e329049fa03</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Skylake-Client-IBRS</model>
      <vendor>Intel</vendor>
      <microcode version='184'/>
      <topology sockets='1' cores='8' threads='2'/>
      ...
      <feature name='arch-capabilities'/>
      <feature name='ssbd'/>
      <feature name='xsaves'/>
      <feature name='pdpe1gb'/>
      <feature name='invtsc'/>
      <feature name='rdctl-no'/>
      <feature name='ibrs-all'/>
      <feature name='skip-l1dfl-vmentry'/>
      <feature name='mds-no'/>
      ...
    </cpu>

# virsh domcapabilities 
<domainCapabilities>
  <path>/usr/libexec/qemu-kvm</path>
  ...
  <cpu>
    <mode name='host-passthrough' supported='yes'/>
    <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Skylake-Client-IBRS</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='ss'/>
      <feature policy='require' name='vmx'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='clflushopt'/>
      <feature policy='require' name='umip'/>
      <feature policy='require' name='md-clear'/>
      <feature policy='require' name='stibp'/>
      <feature policy='require' name='arch-capabilities'/>
      <feature policy='require' name='ssbd'/>
      <feature policy='require' name='xsaves'/>
      <feature policy='require' name='pdpe1gb'/>
      <feature policy='require' name='invtsc'/>
    </mode>

# virsh domstate vm
shut off

# virsh dumpxml vm --inactive |grep "<cpu" -A2
  <emulator>/usr/libexec/qemu-kvm</emulator>

  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>

# virsh start vm
Domain vm started

# virsh dumpxml vm|grep "<cpu" -A16
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Skylake-Client-IBRS</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='clflushopt'/>
    <feature policy='require' name='umip'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='ssbd'/>
    <feature policy='require' name='xsaves'/>
    <feature policy='require' name='pdpe1gb'/>
  </cpu>

# virsh console vm
Connected to domain vm
Escape character is ^]

Red Hat Enterprise Linux 8.1 Beta (Ootpa)
Kernel 4.18.0-107.el8.x86_64 on an x86_64

localhost login: root
Password: 
[root@localhost ~]# lscpu 
...
Flags:               fpu vme ... arch_capabilities

S2: "libvirt-4.5.0-30.module+el8.1.0+3574+3a63752b.x86_64" + "QEMU emulator version 4.0.91 (v4.1.0-rc1-50-g23da9e297b)"
Version:
kernel-4.18.0-119.el8.x86_64
libvirt-4.5.0-30.module+el8.1.0+3574+3a63752b.x86_64
QEMU version:
# /usr/local/bin/qemu-system-x86_64 -version
QEMU emulator version 4.0.91 (v4.1.0-rc1-50-g23da9e297b)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers

Steps：
# lscpu |grep arch_capabilities
Flags:               ... arch_capabilities

# virsh capabilities |more
<capabilities>

  <host>
    <uuid>03fa4990-320e-4990-fa03-0e329049fa03</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Skylake-Client-IBRS</model>
      <vendor>Intel</vendor>
      <microcode version='184'/>
      <topology sockets='1' cores='8' threads='2'/>
      <feature name='ds'/>
      <feature name='acpi'/>
      ...
      <feature name='arch-capabilities'/>
      <feature name='ssbd'/>
      <feature name='xsaves'/>
      <feature name='pdpe1gb'/>
      <feature name='invtsc'/>
      <feature name='rdctl-no'/>
      <feature name='ibrs-all'/>
      <feature name='skip-l1dfl-vmentry'/>
      <feature name='mds-no'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='2048'/>
      <pages unit='KiB' size='1048576'/>
    </cpu>

# virsh domcapabilities |more
<domainCapabilities>
  <path>/usr/local/bin/qemu-system-x86_64</path>
  ...
  <cpu>
    <mode name='host-passthrough' supported='yes'/>
    <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Skylake-Client-IBRS</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='ss'/>
      <feature policy='require' name='vmx'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='clflushopt'/>
      <feature policy='require' name='umip'/>
      <feature policy='require' name='md-clear'/>
      <feature policy='require' name='stibp'/>
      <feature policy='require' name='arch-capabilities'/>
      <feature policy='require' name='ssbd'/>
      <feature policy='require' name='xsaves'/>
      <feature policy='require' name='pdpe1gb'/>
      <feature policy='require' name='invtsc'/>
      <feature policy='require' name='rdctl-no'/>
      <feature policy='require' name='ibrs-all'/>
      <feature policy='require' name='skip-l1dfl-vmentry'/>
      <feature policy='require' name='mds-no'/>
    </mode>

# virsh domstate vm
shut off

# virsh dumpxml vm --inactive |grep "<cpu" -A2
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>

# virsh start vm
Domain vm started

# virsh dumpxml vm |grep "<cpu" -A20
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Skylake-Client-IBRS</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='clflushopt'/>
    <feature policy='require' name='umip'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='ssbd'/>
    <feature policy='require' name='xsaves'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='rdctl-no'/>
    <feature policy='require' name='ibrs-all'/>
    <feature policy='require' name='skip-l1dfl-vmentry'/>
    <feature policy='require' name='mds-no'/>
    <feature policy='disable' name='mpx'/>
  </cpu>

# virsh console vm
Connected to domain vm
Escape character is ^]

Red Hat Enterprise Linux 8.1 Beta (Ootpa)
Kernel 4.18.0-107.el8.x86_64 on an x86_64

localhost login: root
Password: 
Last login: Tue Jul 23 09:30:59 on ttyS0
[root@localhost ~]# lscpu
...
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves arat umip md_clear arch_capabilities

Comment 11 jiyan 2019-07-23 03:48:41 UTC

Created attachment 1592726 [details]
domcap_and_cap_info_on_qemu-kvm-2.12.0-81

Comment 12 jiyan 2019-07-23 03:49:11 UTC

Created attachment 1592727 [details]
domcap_and_cap_info_on_qemu-kvm-v4.1.0-rc1-50-g23da9e297b

Comment 13 jiyan 2019-07-23 03:53:33 UTC

Hi jiri 
As the test steps show in comment 10; and the attachments in comment 11 and comment 12 give the output of "virsh (dom)capabilities" on differnt QEMU version.
On both "qemu-kvm-2.12.0-81.module+el8.1.0+3619+dfe1ae01.x86_64" and "QEMU emulator version 4.0.91 (v4.1.0-rc1-50-g23da9e297b)"; MSR related features can be displayed through "virsh capabilities"
But only on "QEMU emulator version 4.0.91 (v4.1.0-rc1-50-g23da9e297b)"; MSR related features can be showed through "virsh domcapabilities" and in the dumpxml of VM which started with host-model cpu conf.

Comment 15 Jiri Denemark 2019-07-23 11:03:45 UTC

Interesting, this would suggest that qemu-kvm-2.12.0-81 does not properly
support the MSR features. Which is strange, since the corresponding QEMU bug
1709970 is VERIFIED with qemu-kvm-2.12.0-77. Can you confirm the features are
not available in the guest even when started with host-passthrough CPU?

Comment 16 Jiri Denemark 2019-07-24 07:57:29 UTC

It seems the needinfo flag was removed by accident. I'm adding it back to get
an answer to my question in comment 15.

Comment 17 jiyan 2019-07-24 09:37:26 UTC

Test it according to comment 15; there is also no related features in guest OS.

# rpm -qa libvirt qemu-kvm kernel
kernel-4.18.0-119.el8.x86_64
libvirt-4.5.0-30.module+el8.1.0+3574+3a63752b.x86_64
qemu-kvm-2.12.0-81.module+el8.1.0+3619+dfe1ae01.x86_64

# virsh domstate test
shut off

# virsh dumpxml test --inactive |grep "<cpu" 
  <cpu mode='host-passthrough' check='none'/>

# virsh start test
Domain test started

# virsh dumpxml test |grep "<cpu" 
  <cpu mode='host-passthrough' check='none'/>

# virsh console test
Connected to domain test
Escape character is ^]

Red Hat Enterprise Linux 8.1 Beta (Ootpa)
Kernel 4.18.0-107.el8.x86_64 on an x86_64

localhost login: root
Password: 
Last login: Wed Jul 24 10:34:14 on ttyS0
[root@localhost ~]# 
[root@localhost ~]# lscpu 
...
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves arat umip md_clear arch_capabilities


[root@localhost ~]# cat /proc/cpuinfo 
..
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves arat umip md_clear arch_capabilities
...

Comment 18 Jiri Denemark 2019-07-24 10:08:30 UTC

In that case it seems something is wrong with the 8.1 build of QEMU. Eduardo,
can you check why everything works with upstream QEMU, but there are no MSR
features enabled by qemu-kvm-2.12.0-81?

Comment 19 Eduardo Habkost 2019-08-22 16:14:05 UTC

(In reply to Jiri Denemark from comment #18)
> In that case it seems something is wrong with the 8.1 build of QEMU. Eduardo,
> can you check why everything works with upstream QEMU, but there are no MSR
> features enabled by qemu-kvm-2.12.0-81?

I'm investigating it.  The code looks OK, so I'm now looking for a machine I can borrow where I can debug the issue.

Comment 20 jiyan 2019-08-27 08:41:45 UTC

Hi Eduardo
is there any progress?

Comment 21 Eduardo Habkost 2019-08-29 19:53:33 UTC

I don't see what's wrong: the features are returned by query-cpu-model-expansion:

[virt-rhel8-qemu-kvm]# rpm -q qemu-kvm
qemu-kvm-2.12.0-85.module+el8.1.0+4010+d6842f29.x86_64
[virt-rhel8-qemu-kvm]# ps ax | grep qemu-kvm
19536 pts/2    Sl+    0:12 /usr/libexec/qemu-kvm -machine accel=kvm -qmp unix:/tmp/qmp,server
20862 pts/1    S+     0:00 grep --color=auto qemu-kvm
[virt-rhel8-qemu-kvm]# echo 'query-cpu-model-expansion type=full model={"name":"host"}' | python2 ./scripts/qmp/qmp-shell -p /tmp/qmp | egrep 'rdctl-no|ibrs-all|rsba|skip-l1dfl-vmentry|ssb-no|mds-no'
                "mds-no": true, 
                "rdctl-no": true, 
                "ssb-no": false, 
                "ibrs-all": true, 
                "skip-l1dfl-vmentry": true, 
                "rsba": false, 


Jiri, do you think the lack of the "unavailable-features" QOM property might explain the problem?

Comment 22 Eduardo Habkost 2019-08-29 20:01:19 UTC

I've submitted a scratch build including the "unavailable-features" QOM property, to help us check if that's the root cause:
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=23239910

Comment 23 Eduardo Habkost 2019-08-29 20:34:28 UTC

I can confirm that the "unavailable-features" QOM property is necessary:

# rpm -q qemu-kvm
qemu-kvm-2.12.0-85.module+el8.1.0+4010+d6842f29.x86_64                                                                                                                                                                                         
# virsh domcapabilities  | xmllint --xpath '/domainCapabilities/cpu' - | head -n20                                                                                                                           
<cpu>                                                                                                                                                                                                                                          
    <mode name="host-passthrough" supported="yes"/>                                                                                                                                                                                            
    <mode name="host-model" supported="yes">                                                                                                                                                                                                   
      <model fallback="forbid">Cascadelake-Server</model>                                                                                                                                                                                      
      <vendor>Intel</vendor>                                                                                                                                                                                                                   
      <feature policy="require" name="ss"/>                                                                                                                                                                                                    
      <feature policy="require" name="vmx"/>                                                                                                                                                                                                   
      <feature policy="require" name="hypervisor"/>                                                                                                                                                                                            
      <feature policy="require" name="tsc_adjust"/>                                                                                                                                                                                            
      <feature policy="require" name="umip"/>                                                                                                                                                                                                  
      <feature policy="require" name="pku"/>                                                                                                                                                                                                   
      <feature policy="require" name="md-clear"/>
      <feature policy="require" name="stibp"/>
      <feature policy="require" name="arch-capabilities"/>
      <feature policy="require" name="xsaves"/>
      <feature policy="require" name="invtsc"/>
    </mode>
    <mode name="custom" supported="yes">
      <model usable="yes">qemu64</model>
      <model usable="yes">qemu32</model>


# rpm -q qemu-kvm
qemu-kvm-2.12.0-85.el8.unav.features.v1.x86_64
# virsh domcapabilities  | xmllint --xpath '/domainCapabilities/cpu' - | head -n20
<cpu>
    <mode name="host-passthrough" supported="yes"/>
    <mode name="host-model" supported="yes">
      <model fallback="forbid">Cascadelake-Server</model>
      <vendor>Intel</vendor>
      <feature policy="require" name="ss"/>
      <feature policy="require" name="vmx"/>
      <feature policy="require" name="hypervisor"/>
      <feature policy="require" name="tsc_adjust"/>
      <feature policy="require" name="umip"/>
      <feature policy="require" name="pku"/>
      <feature policy="require" name="md-clear"/>
      <feature policy="require" name="stibp"/>
      <feature policy="require" name="arch-capabilities"/>
      <feature policy="require" name="xsaves"/>
      <feature policy="require" name="invtsc"/>
      <feature policy="require" name="rdctl-no"/>
      <feature policy="require" name="ibrs-all"/>
      <feature policy="require" name="skip-l1dfl-vmentry"/>
      <feature policy="require" name="mds-no"/>

Comment 24 Eduardo Habkost 2019-08-29 20:38:53 UTC

Bug validation with qemu-kvm-4.1.0 (from RHEL-AV) should suffice to confirm the libvirt code is working.

I will create a new qemu-kvm BZ for "filtered-features".

Comment 25 Eduardo Habkost 2019-08-29 20:43:44 UTC

(In reply to Eduardo Habkost from comment #24)
> I will create a new qemu-kvm BZ for "filtered-features".

Bug 1747185 created.

Comment 26 jiyan 2019-09-04 03:33:53 UTC

Since Bug 1747185 is on_qa now; trying to verify this bug on libvirt-4.5.0-33.scrmod+el8.1.0+4090+e8e6ad83.x86_64

Version:
libvirt-4.5.0-33.scrmod+el8.1.0+4090+e8e6ad83.x86_64
qemu-kvm-2.12.0-86.module+el8.1.0+4146+4ed2d185.x86_64
kernel-4.18.0-141.el8.x86_64

Steps:
1. Prepare a host with the following cpu flags:
# lscpu 
Flags:               ...avx512_vnni ... arch_capabilities

2. Check /usr/share/libvirt/cpu_map.xml; it contains the following info as expected
# cat /usr/share/libvirt/cpu_map.xml  |grep  "IA32_ARCH_CAPABILITIES features" -A18
    <!-- IA32_ARCH_CAPABILITIES features -->
    <feature name='rdctl-no'>
      <msr index='0x10a' edx='0x00000000' eax='0x00000001'/>
    </feature>
    <feature name='ibrs-all'>
      <msr index='0x10a' edx='0x00000000' eax='0x00000002'/>
    </feature>
    <feature name='rsba'>
      <msr index='0x10a' edx='0x00000000' eax='0x00000004'/>
    </feature>
    <feature name='skip-l1dfl-vmentry'>
      <msr index='0x10a' edx='0x00000000' eax='0x00000008'/>
    </feature>
    <feature name='ssb-no'>
      <msr index='0x10a' edx='0x00000000' eax='0x00000010'/>
    </feature>
    <feature name='mds-no'>
      <msr index='0x10a' edx='0x00000000' eax='0x00000020'/>
    </feature>

3. Check "virsh capabilities" and "virsh domcapabilities"; they also contain the cpu related info as expected
# virsh capabilities
...
      <model>Cascadelake-Server</model>
...
      <feature name='arch-capabilities'/>
      <feature name='xsaves'/>
      <feature name='mbm_total'/>
      <feature name='mbm_local'/>
      <feature name='invtsc'/>
      <feature name='rdctl-no'/>
      <feature name='ibrs-all'/>
      <feature name='skip-l1dfl-vmentry'/>
      <feature name='mds-no'/>
...

# virsh domcapabilities
    <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Cascadelake-Server</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='ss'/>
      <feature policy='require' name='vmx'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='umip'/>
      <feature policy='require' name='pku'/>
      <feature policy='require' name='md-clear'/>
      <feature policy='require' name='stibp'/>
      <feature policy='require' name='arch-capabilities'/>
      <feature policy='require' name='xsaves'/>
      <feature policy='require' name='invtsc'/>
      <feature policy='require' name='rdctl-no'/>
      <feature policy='require' name='ibrs-all'/>
      <feature policy='require' name='skip-l1dfl-vmentry'/>
      <feature policy='require' name='mds-no'/>
    </mode>

4. Start VM with host-model cpu conf
# virsh domstate test
shut off

# virsh dumpxml test --inactive |grep "<cpu" -A2
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>

# virsh start test
Domain test started

# virsh dumpxml test |grep "<cpu" -A20
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Cascadelake-Server</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='umip'/>
    <feature policy='require' name='pku'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='xsaves'/>
    <feature policy='require' name='rdctl-no'/>
    <feature policy='require' name='ibrs-all'/>
    <feature policy='require' name='skip-l1dfl-vmentry'/>
    <feature policy='require' name='mds-no'/>
    <feature policy='disable' name='mpx'/>
  </cpu>

# ps -ef |grep test
qemu      56015      1 87 23:29 ?        
-cpu Cascadelake-Server,ss=on,vmx=on,hypervisor=on,tsc-adjust=on,umip=on,pku=on,md-clear=on,stibp=on,arch-capabilities=on,xsaves=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on


# virsh console test
only arch-capabilities cpu flag can be seen in "lscpu"; msr related cpu flags can be seen through "virsh capabilities" after installing libvirt pkg in guest oS

All the test results are as expected; move this bug to be verified.

Comment 27 Jiri Denemark 2019-09-05 14:18:00 UTC

(In reply to Eduardo Habkost from comment #21)
> Jiri, do you think the lack of the "unavailable-features" QOM property might
> explain the problem?

Oh yes. When "unavailable-features" QOM property is not supported, libvirt
just filters out all MSR features from host-model because it wouldn't be able
to check whether they are really enabled when QEMU starts.

Comment 29 errata-xmlrpc 2019-11-05 20:48:28 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3345