Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1839926

Summary:	5 more -noTSX x86 CPU models are newly introduced
Product:	Red Hat Enterprise Linux Advanced Virtualization	Reporter:	jiyan <jiyan>
Component:	libvirt	Assignee:	Jiri Denemark <jdenemar>
Status:	CLOSED ERRATA	QA Contact:	yalzhang <yalzhang>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	8.3	CC:	dyuan, jdenemar, lhuang, lmen, virt-maint, xuzhang, yafu, yalzhang
Target Milestone:	rc	Flags:	pm-rhel: mirror+
Target Release:	8.3
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	libvirt-6.4.0-1.el8	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1840008 (view as bug list)		Environment:
Last Closed:	2020-11-17 17:48:38 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1840008

Description jiyan 2020-05-26 03:18:54 UTC

Description of problem:
5 more -noTSX x86 CPU models are newly introduced since libvirt-6.3.0-1.module+el8.3.0+6478+69f490bb.x86_64, including Skylake-Server-noTSX-IBRS, Skylake-Client-noTSX-IBRS, Icelake-Server-noTSX, Icelake-Client-noTSX, Cascadelake-Server-noTSX

Version-Release number of selected component (if applicable):
kernel-4.18.0-202.el8.x86_64
qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420.x86_64
libvirt-6.3.0-1.module+el8.3.0+6478+69f490bb.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Tested the following steps with libvirt-6.0.0-17.module+el8.3.0+6423+e4cb6418.x86_64.
# ls /usr/share/libvirt/cpu_map/ > 60017-libvirt
# virsh domcapabilities > 60017-libvirt-domcap

2. Tested the following steps with libvirt-6.3.0-1.module+el8.3.0+6478+69f490bb.x86_64
# ls /usr/share/libvirt/cpu_map/ > 6301-libvirt
# virsh domcapabilities > 6301-libvirt-domcap

3. Diff the files in step-1 and step-2
# diff 60017-libvirt 6301-libvirt 
15a16
> x86_Cascadelake-Server-noTSX.xml
29a31
> x86_Icelake-Client-noTSX.xml
30a33
> x86_Icelake-Server-noTSX.xml
54a58
> x86_Skylake-Client-noTSX-IBRS.xml
56a61
> x86_Skylake-Server-noTSX-IBRS.xml


# diff 60017-libvirt-domcap 6301-libvirt-domcap 
60a62
>       <model usable='yes'>Skylake-Server-noTSX-IBRS</model>
62a65
>       <model usable='yes'>Skylake-Client-noTSX-IBRS</model>
76a80
>       <model usable='no'>Icelake-Server-noTSX</model>
77a82
>       <model usable='no'>Icelake-Client-noTSX</model>
86a92
>       <model usable='no'>Cascadelake-Server-noTSX</model>

Actual results:
Showed in the steps above, Skylake-Server-noTSX-IBRS, Skylake-Client-noTSX-IBRS, Icelake-Server-noTSX, Icelake-Client-noTSX, Cascadelake-Server-noTSX, 5 -noTSX cpu models are newly introduced.

Expected results:


Additional info:
1. There is a commit info from upstream QEMU.

commit dd17a4eb
Author: Christian Ehrhardt <christian.ehrhardt>
Date: Tue Mar 10 11:48:06 2020 +0100

cpu_map: Add more -noTSX x86 CPU models

One of the mitigation methods for TAA[1] is to disable TSX
support on the host system. Linux added a mechanism to disable
TSX globally through the kernel command line, and many Linux
distributions now default to tsx=off. This makes existing CPU
models that have HLE and RTM enabled not usable anymore.

Add new versions of all CPU models that have the HLE and RTM
features enabled, that can be used when TSX is disabled in the
host system.

On systems disabling the features without those types defined
in cpu-maps users end up without modern CPU types in the list
of usable CPUs to use in the likes of virsh domcapabilities
or tools higher in the stack like virt-manager.

This adds:
-Cascadelake-Server-noTSX
-Icelake-Client-noTSX
-Icelake-Server-noTSX
-Skylake-Server-noTSX-IBRS
-Skylake-Client-noTSX-IBRS

Introduced in QEMU by commit v4.2.0-rc2-3-g9ab2237f19 (function)
and commit v4.2.0-rc2-4-g02fa60d101 (names)

2. I filed this bugs cause this is a new function introduced in libvirt, which needs some patch info and testing.

Comment 1 jiyan 2020-05-26 09:12:09 UTC

Also found the new changes about decode element in cpu_map.xml as follows on libvirt-6.3.0-1.module+el8.3.0+6478+69f490bb.x86_64.

# cat /usr/share/libvirt/cpu_map/* | grep decode
    <decode host='on' guest='on'/>
...
    <decode host='on' guest='off'/>
...

Which showed in the upstream patch: 

cpu_map: Add <decode> element to x86 CPU model definitions
commit 17cdefe5
Author: Jiri Denemark <jdenemar>
Date: Tue Mar 17 22:32:16 2020 +0100

cpu_map: Don't use new noTSX models for host-model CPUs

Host-model CPU definitions (and domain capabilities) will use the
original CPU models (without noTSX in their name) and explicitly disable
hle and rtm features. This way domains with host-model CPUs will be
migratable even to older versions of libvirt which do not support the
noTSX model variants.

The new models will be advertised in host capabilities and they may
be used explicitly with custom CPUs.

Signed-off-by: Jiri Denemark <jdenemar>
Reviewed-by: Christian Ehrhardt <christian.ehrhardt>
Tested-by: Christian Ehrhardt <christian.ehrhardt>

commit 7cd896ef
Author: Jiri Denemark <jdenemar>
Date: Tue Mar 17 22:27:49 2020 +0100

cpu_x86: Honor CPU models' <decode> element

Signed-off-by: Jiri Denemark <jdenemar>
Reviewed-by: Christian Ehrhardt <christian.ehrhardt>
Tested-by: Christian Ehrhardt <christian.ehrhardt>

Comment 4 yalzhang@redhat.com 2020-09-07 14:11:29 UTC

Hi Jiri, I have tested on a Cascadelake-Server physical host with hle but not rtm, the result is as below. And I have a question about step 3 result below:  why it shows   "<feature policy='require' name='amd-stibp'/>  <feature policy='require' name='amd-ssbd'/>" on such Intel host? Please help to confirm, Thank you!

1. Check the hardware info of the host:

# rpm -q libvirt-libs qemu-kvm kernel
libvirt-libs-6.6.0-4.module+el8.3.0+7883+3d717aa8.x86_64
qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64
Kernel-4.18.0-234.el8.x86_64

# lscpu 
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              40
On-line CPU(s) list: 0-39
Thread(s) per core:  2
Core(s) per socket:  10
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz
Stepping:            7
CPU MHz:             1000.832
BogoMIPS:            4400.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            14080K
NUMA node0 CPU(s):   0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38
NUMA node1 CPU(s):   1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke avx512_vnni md_clear flush_l1d arch_capabilities

# lscpu | grep rtm
# (no outputs)
# lscpu | grep hle
... tsc_adjust bmi1 hle avx2…

2. Check virsh capabilities recognized it as "Cascadelake-Server-noTSX";

# virsh capabilities 
<capabilities>

  <host>
    <uuid>4c4c4544-004a-5410-8035-b4c04f593233</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Cascadelake-Server-noTSX</model>   *****
      <vendor>Intel</vendor>
      <microcode version='83898113'/>
      <counter name='tsc' frequency='2194843000' scaling='yes'/>
      <topology sockets='1' dies='1' cores='10' threads='2'/>
      <feature name='ds'/>
      <feature name='acpi'/>
      <feature name='ss'/>
      <feature name='ht'/>
      <feature name='tm'/>
      <feature name='pbe'/>
      <feature name='dtes64'/>
      <feature name='monitor'/>
      <feature name='ds_cpl'/>
      <feature name='vmx'/>
      <feature name='smx'/>
      <feature name='est'/>
      <feature name='tm2'/>
      <feature name='xtpr'/>
      <feature name='pdcm'/>
      <feature name='dca'/>
      <feature name='osxsave'/>
      <feature name='tsc_adjust'/>
      <feature name='cmt'/>
      <feature name='intel-pt'/>
      <feature name='pku'/>
      <feature name='ospke'/>
      <feature name='md-clear'/>
      <feature name='stibp'/>
      <feature name='arch-capabilities'/>
      <feature name='xsaves'/>
      <feature name='mbm_total'/>
      <feature name='mbm_local'/>
      <feature name='invtsc'/>
      <feature name='rdctl-no'/>
      <feature name='ibrs-all'/>
      <feature name='skip-l1dfl-vmentry'/>
      <feature name='mds-no'/>
      <feature name='tsx-ctrl'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='2048'/>
      <pages unit='KiB' size='1048576'/>
    </cpu>


3. Check virsh domcapabilities shows it as "Cascadelake-Server" with hle and rtm disabled which is as expected; and model usable for "Cascadelake-Server-noTSX" is yes, model usable for "Cascadelake-Server" is no. The result is as expected.

# virsh domcapabilities
...
    <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Cascadelake-Server</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='ss'/>
      <feature policy='require' name='vmx'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='umip'/>
      <feature policy='require' name='pku'/>
      <feature policy='require' name='md-clear'/>
      <feature policy='require' name='stibp'/>
      <feature policy='require' name='arch-capabilities'/>
      <feature policy='require' name='xsaves'/>
      <feature policy='require' name='invtsc'/>
      <feature policy='require' name='ibpb'/>
 ***    <feature policy='require' name='amd-stibp'/>   ***
 ***   <feature policy='require' name='amd-ssbd'/>   ***
      <feature policy='require' name='rdctl-no'/>
      <feature policy='require' name='ibrs-all'/>
      <feature policy='require' name='skip-l1dfl-vmentry'/>
      <feature policy='require' name='mds-no'/>
      <feature policy='require' name='pschange-mc-no'/>
      <feature policy='disable' name='hle'/>  ***
      <feature policy='disable' name='rtm'/>  ***
    </mode>
…
  <mode name='custom' supported='yes'>
…...  
      <model usable='yes'>Conroe</model>
      <model usable='yes'>Cascadelake-Server-noTSX</model> ***
      <model usable='no'>Cascadelake-Server</model> ***
      <model usable='yes'>Broadwell-noTSX-IBRS</model>
      <model usable='yes'>Broadwell-noTSX</model>
      <model usable='no'>Broadwell-IBRS</model>
      <model usable='no'>Broadwell</model>
      <model usable='yes'>486</model>
    </mode>

Comment 5 yalzhang@redhat.com 2020-09-09 05:39:59 UTC

Test on Skylake-Client-IBRS host with hle and rtm flags, the result is as expected.
Start a vm with host-passthrough cpu model with hle and rtm disabled, and login the vm:
# lscpu | grep hle  
# (no outputs)
# lscpu | grep rtm 
# (no outputs)
# rpm -q libvirt-libs qemu-kvm
libvirt-libs-6.6.0-4.module+el8.3.0+7883+3d717aa8.x86_64
qemu-kvm-5.1.0-4.module+el8.3.0+7846+ae9b566f.x86_64

 # virsh capabilities 
<capabilities>

  <host>
    <uuid>f1da8f96-56d7-4d28-a3ef-703bfe97fa54</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Skylake-Client-noTSX-IBRS</model>
…
</cpu>

# virsh domcapabilities 
…
 <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Skylake-Client-IBRS</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='ss'/>
    ...
      <feature policy='disable' name='hle'/>
      <feature policy='disable' name='rtm'/>
      <feature policy='disable' name='mpx'/>
    </mode>
 <mode name='custom' supported='yes'>
 <model usable='no'>Skylake-Server-noTSX-IBRS</model>
      <model usable='no'>Skylake-Server-IBRS</model>
      <model usable='no'>Skylake-Server</model>
      <model usable='yes'>Skylake-Client-noTSX-IBRS</model>
      <model usable='no'>Skylake-Client-IBRS</model>
....

Comment 6 Jiri Denemark 2020-09-16 08:18:04 UTC

(In reply to yalzhang from comment #4)
> # virsh domcapabilities
> ...
>     <mode name='host-model' supported='yes'>
>       <model fallback='forbid'>Cascadelake-Server</model>
>       <vendor>Intel</vendor>
>       <feature policy='require' name='ss'/>
>       <feature policy='require' name='vmx'/>
>       <feature policy='require' name='hypervisor'/>
>       <feature policy='require' name='tsc_adjust'/>
>       <feature policy='require' name='umip'/>
>       <feature policy='require' name='pku'/>
>       <feature policy='require' name='md-clear'/>
>       <feature policy='require' name='stibp'/>
>       <feature policy='require' name='arch-capabilities'/>
>       <feature policy='require' name='xsaves'/>
>       <feature policy='require' name='invtsc'/>
>       <feature policy='require' name='ibpb'/>
>  ***    <feature policy='require' name='amd-stibp'/>   ***
>  ***   <feature policy='require' name='amd-ssbd'/>   ***
>       <feature policy='require' name='rdctl-no'/>
>       <feature policy='require' name='ibrs-all'/>
>       <feature policy='require' name='skip-l1dfl-vmentry'/>
>       <feature policy='require' name='mds-no'/>
>       <feature policy='require' name='pschange-mc-no'/>
>       <feature policy='disable' name='hle'/>  ***
>       <feature policy='disable' name='rtm'/>  ***
>     </mode>

Features that show as enabled in domain capabilities even though they are not
available in the host capabilities are enabled and/or emulated by QEMU/KVM
regardless of what the host supports.

Comment 7 yalzhang@redhat.com 2020-09-16 14:45:20 UTC

Test on Skylake-Server-IBRS host:
 
1. Start L1 guest with cpu setting as below:
# virsh dumpxml rhel --inactive | grep /cpu -B3
<cpu mode='host-passthrough' check='none' migratable='on'>
    <feature policy='disable' name='hle'/>
    <feature policy='disable' name='rtm'/>
  </cpu>

2.login L1 guest and install the latest virt module, then check the cpu related info:
# lscpu | grep hle
# (no outputs)
# lscpu | grep rtm
# (no outputs)
# virsh capabilities 
<capabilities>

  <host>
    <uuid>75b068b8-2482-43f9-88e8-17b0f33e9b83</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Skylake-Server-noTSX-IBRS</model>
...

# virsh domcapabilities 
<domainCapabilities>
  <path>/usr/libexec/qemu-kvm</path>
  <domain>kvm</domain>
...
 <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Skylake-Server-IBRS</model>
      <vendor>Intel</vendor>
...
     <feature policy='disable' name='hle'/>
      <feature policy='disable' name='rtm'/>
      <feature policy='disable' name='mpx'/>
    </mode>
  <mode name='custom' supported='yes'>
    <model usable='yes'>Skylake-Server-noTSX-IBRS</model>
      <model usable='no'>Skylake-Server-IBRS</model>
      <model usable='no'>Skylake-Server</model>
      <model usable='yes'>Skylake-Client-noTSX-IBRS</model>
      <model usable='no'>Skylake-Client-IBRS</model>
      <model usable='no'>Skylake-Client</model>
...

3. start a L2 guest with host-model, guest can start successfully with " <cpu mode='host-model' check='partial'/>";

4. start L2 guest with custom 'Skylake-Server-noTSX-IBRS':
# virsh dumpxml rh  | grep  /cpu -B3
  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='forbid'>Skylake-Server-noTSX-IBRS</model>
    <vendor>Intel</vendor>
  </cpu>

# virsh start rh 
Domain rh started

The result is as expected.

Comment 8 yalzhang@redhat.com 2020-09-29 02:39:13 UTC

Hi Jiri,

I have loan a icelake host, but the output of "virsh capabilities" show it is 'Westmere-IBRS' with several features. And 'virsh domcapabilities' show it is  Icelake-Server with several features disabled. I have checked the cpu flags, and found many flags like 'avx' does not exists. I have below questions, could you please help to have a check? Thank you!

1) How can libvirt recognize the cpu module in "virsh capabilities" and "virsh domcapabilities"? 
2) Some flags do not exits, is it disabled by kernel?
3) Do you think this is expected? 

# rpm -q libvirt-libs qemu-kvm
libvirt-libs-6.6.0-6.module+el8.3.0+8125+aefcf088.x86_64
qemu-kvm-5.1.0-10.module+el8.3.0+8254+568ca30d.x86_64

1.
# virsh capabilities 
<capabilities>

  <host>
    <uuid>a5a5a5a5-a5a5-0d14-080b-0708a5a5a5a5</uuid>
    <cpu>
      <arch>x86_64</arch>
 ******  <model>Westmere-IBRS</model>   ********
      <vendor>Intel</vendor>
      <microcode version='2600534023'/>
      <counter name='tsc' frequency='2200000000' scaling='yes'/>
      <topology sockets='1' dies='1' cores='24' threads='1'/>
      <feature name='vme'/>
      <feature name='ds'/>
      <feature name='acpi'/>
      <feature name='ss'/>
      <feature name='ht'/>
      <feature name='tm'/>
      <feature name='pbe'/>
      <feature name='pclmuldq'/>
      <feature name='dtes64'/>
      <feature name='monitor'/>
      <feature name='ds_cpl'/>
      <feature name='vmx'/>
      <feature name='smx'/>
      <feature name='est'/>
      <feature name='tm2'/>
      <feature name='xtpr'/>
      <feature name='pdcm'/>
      <feature name='x2apic'/>
      <feature name='movbe'/>
      <feature name='tsc-deadline'/>
      <feature name='xsave'/>
      <feature name='osxsave'/>
      <feature name='rdrand'/>
      <feature name='arat'/>
      <feature name='fsgsbase'/>
      <feature name='tsc_adjust'/>
      <feature name='smep'/>
      <feature name='erms'/>
      <feature name='cmt'/>
      <feature name='rdseed'/>
      <feature name='smap'/>
      <feature name='clflushopt'/>
      <feature name='clwb'/>
      <feature name='intel-pt'/>
      <feature name='sha-ni'/>
      <feature name='umip'/>
      <feature name='waitpkg'/>
      <feature name='gfni'/>
      <feature name='rdpid'/>
      <feature name='cldemote'/>
      <feature name='movdiri'/>
      <feature name='movdir64b'/>
      <feature name='md-clear'/>
      <feature name='stibp'/>
      <feature name='arch-capabilities'/>
      <feature name='ssbd'/>
      <feature name='xsaveopt'/>
      <feature name='xsavec'/>
      <feature name='xgetbv1'/>
      <feature name='xsaves'/>
      <feature name='mbm_total'/>
      <feature name='mbm_local'/>
      <feature name='rdtscp'/>
      <feature name='3dnowprefetch'/>
      <feature name='invtsc'/>
      <feature name='rdctl-no'/>
      <feature name='ibrs-all'/>
      <feature name='skip-l1dfl-vmentry'/>
      <feature name='mds-no'/>
      <feature name='pschange-mc-no'/>
      <feature name='taa-no'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='2048'/>
    </cpu>
...

2. 
# virsh domcapabilities 
<domainCapabilities>
  <path>/usr/libexec/qemu-kvm</path>
  <domain>kvm</domain>
  <machine>pc-i440fx-rhel7.6.0</machine>
  <arch>x86_64</arch>
  <vcpu max='240'/>
  <iothreads supported='yes'/>
  <os supported='yes'>
    <enum name='firmware'/>
    <loader supported='yes'>
      <value>/usr/share/OVMF/OVMF_CODE.secboot.fd</value>
      <enum name='type'>
        <value>rom</value>
        <value>pflash</value>
      </enum>
      <enum name='readonly'>
        <value>yes</value>
        <value>no</value>
      </enum>
      <enum name='secure'>
        <value>no</value>
      </enum>
    </loader>
  </os>
  <cpu>
    <mode name='host-passthrough' supported='yes'>
      <enum name='hostPassthroughMigratable'>
        <value>on</value>
        <value>off</value>
      </enum>
    </mode>
    <mode name='host-model' supported='yes'>
  ******    <model fallback='forbid'>Icelake-Server</model> ******
      <vendor>Intel</vendor>
      <feature policy='require' name='ss'/>
      <feature policy='require' name='vmx'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='sha-ni'/>
      <feature policy='require' name='waitpkg'/>
      <feature policy='require' name='rdpid'/>
      <feature policy='require' name='cldemote'/>
      <feature policy='require' name='movdiri'/>
      <feature policy='require' name='movdir64b'/>
      <feature policy='require' name='md-clear'/>
      <feature policy='require' name='stibp'/>
      <feature policy='require' name='arch-capabilities'/>
      <feature policy='require' name='xsaves'/>
      <feature policy='require' name='invtsc'/>
      <feature policy='require' name='ibpb'/>
      <feature policy='require' name='amd-stibp'/>
      <feature policy='require' name='amd-ssbd'/>
      <feature policy='require' name='rdctl-no'/>
      <feature policy='require' name='ibrs-all'/>
      <feature policy='require' name='skip-l1dfl-vmentry'/>
      <feature policy='require' name='mds-no'/>
      <feature policy='require' name='pschange-mc-no'/>
      <feature policy='disable' name='fma'/>
      <feature policy='disable' name='pcid'/>
      <feature policy='disable' name='avx'/>
      <feature policy='disable' name='f16c'/>
      <feature policy='disable' name='bmi1'/>
      <feature policy='disable' name='hle'/>
      <feature policy='disable' name='avx2'/>
      <feature policy='disable' name='bmi2'/>
      <feature policy='disable' name='invpcid'/>
      <feature policy='disable' name='rtm'/>
      <feature policy='disable' name='mpx'/>
      <feature policy='disable' name='avx512f'/>
      <feature policy='disable' name='avx512dq'/>
      <feature policy='disable' name='adx'/>
      <feature policy='disable' name='intel-pt'/>
      <feature policy='disable' name='avx512cd'/>
      <feature policy='disable' name='avx512bw'/>
      <feature policy='disable' name='avx512vl'/>
      <feature policy='disable' name='avx512vbmi'/>
      <feature policy='disable' name='pku'/>
      <feature policy='disable' name='avx512vbmi2'/>
      <feature policy='disable' name='vaes'/>
      <feature policy='disable' name='vpclmulqdq'/>
      <feature policy='disable' name='avx512vnni'/>
      <feature policy='disable' name='avx512bitalg'/>
      <feature policy='disable' name='avx512-vpopcntdq'/>
      <feature policy='disable' name='la57'/>
      <feature policy='disable' name='pdpe1gb'/>
      <feature policy='disable' name='abm'/>
      <feature policy='disable' name='wbnoinvd'/>
    </mode>
    <mode name='custom' supported='yes'>
      <model usable='yes'>qemu64</model>
      <model usable='yes'>qemu32</model>
      <model usable='no'>phenom</model>
      <model usable='yes'>pentium3</model>
      <model usable='yes'>pentium2</model>
      <model usable='yes'>pentium</model>
      <model usable='yes'>n270</model>
      <model usable='yes'>kvm64</model>
      <model usable='yes'>kvm32</model>
      <model usable='yes'>coreduo</model>
      <model usable='yes'>core2duo</model>
      <model usable='no'>athlon</model>
      <model usable='yes'>Westmere-IBRS</model>
      <model usable='yes'>Westmere</model>
      <model usable='no'>Skylake-Server-noTSX-IBRS</model> 
      <model usable='no'>Skylake-Server-IBRS</model>
      <model usable='no'>Skylake-Server</model>
      <model usable='no'>Skylake-Client-noTSX-IBRS</model>
      <model usable='no'>Skylake-Client-IBRS</model>
      <model usable='no'>Skylake-Client</model>
      <model usable='no'>SandyBridge-IBRS</model>
      <model usable='no'>SandyBridge</model>
      <model usable='yes'>Penryn</model>
      <model usable='no'>Opteron_G5</model>
      <model usable='no'>Opteron_G4</model>
      <model usable='no'>Opteron_G3</model>
      <model usable='yes'>Opteron_G2</model>
      <model usable='yes'>Opteron_G1</model>
      <model usable='yes'>Nehalem-IBRS</model>
      <model usable='yes'>Nehalem</model>
      <model usable='no'>IvyBridge-IBRS</model>
      <model usable='no'>IvyBridge</model>
 ****     <model usable='no'>Icelake-Server-noTSX</model>  *****  should be yes??
      <model usable='no'>Icelake-Server</model>
 ****     <model usable='no'>Icelake-Client-noTSX</model> ****  should be yes??
      <model usable='no'>Icelake-Client</model>
      <model usable='no'>Haswell-noTSX-IBRS</model>
      <model usable='no'>Haswell-noTSX</model>
      <model usable='no'>Haswell-IBRS</model>
      <model usable='no'>Haswell</model>
      <model usable='no'>EPYC-IBPB</model>
      <model usable='no'>EPYC</model>
      <model usable='no'>Dhyana</model>
      <model usable='no'>Cooperlake</model>
      <model usable='yes'>Conroe</model>
      <model usable='no'>Cascadelake-Server-noTSX</model>
      <model usable='no'>Cascadelake-Server</model>
      <model usable='no'>Broadwell-noTSX-IBRS</model>
      <model usable='no'>Broadwell-noTSX</model>
      <model usable='no'>Broadwell-IBRS</model>
      <model usable='no'>Broadwell</model>
      <model usable='yes'>486</model>
    </mode>
  </cpu>
  <devices>
    <disk supported='yes'>
      <enum name='diskDevice'>
        <value>disk</value>
        <value>cdrom</value>
        <value>floppy</value>
        <value>lun</value>
      </enum>
      <enum name='bus'>
        <value>ide</value>
        <value>fdc</value>
        <value>scsi</value>
        <value>virtio</value>
        <value>usb</value>
        <value>sata</value>
      </enum>
      <enum name='model'>
        <value>virtio</value>
        <value>virtio-transitional</value>
        <value>virtio-non-transitional</value>
      </enum>
    </disk>
    <graphics supported='yes'>
      <enum name='type'>
        <value>sdl</value>
        <value>vnc</value>
        <value>spice</value>
      </enum>
    </graphics>
    <video supported='yes'>
      <enum name='modelType'>
        <value>vga</value>
        <value>cirrus</value>
        <value>qxl</value>
        <value>virtio</value>
        <value>none</value>
        <value>bochs</value>
        <value>ramfb</value>
      </enum>
    </video>
    <hostdev supported='yes'>
      <enum name='mode'>
        <value>subsystem</value>
      </enum>
      <enum name='startupPolicy'>
        <value>default</value>
        <value>mandatory</value>
        <value>requisite</value>
        <value>optional</value>
      </enum>
      <enum name='subsysType'>
        <value>usb</value>
        <value>pci</value>
        <value>scsi</value>
      </enum>
      <enum name='capsType'/>
      <enum name='pciBackend'/>
    </hostdev>
    <rng supported='yes'>
      <enum name='model'>
        <value>virtio</value>
        <value>virtio-transitional</value>
        <value>virtio-non-transitional</value>
      </enum>
      <enum name='backendModel'>
        <value>random</value>
        <value>egd</value>
        <value>builtin</value>
      </enum>
    </rng>
  </devices>
  <features>
    <gic supported='no'/>
    <vmcoreinfo supported='yes'/>
    <genid supported='yes'/>
    <backingStoreInput supported='yes'/>
    <backup supported='no'/>
    <sev supported='no'/>
  </features>
</domainCapabilities>

3. Check host cpu:
# lscpu 
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              24
On-line CPU(s) list: 0-23
Thread(s) per core:  1
Core(s) per socket:  24
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               134
Model name:          Intel Atom(R) P5362 processor
Stepping:            5
CPU MHz:             1039.772
CPU max MHz:         2200.0000
CPU min MHz:         800.0000
BogoMIPS:            4400.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            4608K
L3 cache:            15360K
NUMA node0 CPU(s):   0-23
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust smep erms cqm rdt_a rdseed smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect dtherm arat pln pts umip waitpkg gfni rdpid cldemote movdiri movdir64b md_clear flush_l1d arch_capabilities

# lscpu | grep avx  -->no outputs

4. Start vm with cpu type is "host-model", start the vm, and check the live xml, the cpu type is Icelake-Server with hle and rtm disabled, which is expected.

# virsh dumpxml rhel
...
<cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Icelake-Server</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='sha-ni'/>
    <feature policy='require' name='waitpkg'/>
    <feature policy='require' name='rdpid'/>
    <feature policy='require' name='cldemote'/>
    <feature policy='require' name='movdiri'/>
    <feature policy='require' name='movdir64b'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='xsaves'/>
    <feature policy='require' name='ibpb'/>
    <feature policy='require' name='amd-stibp'/>
    <feature policy='require' name='amd-ssbd'/>
    <feature policy='require' name='rdctl-no'/>
    <feature policy='require' name='ibrs-all'/>
    <feature policy='require' name='skip-l1dfl-vmentry'/>
    <feature policy='require' name='mds-no'/>
    <feature policy='require' name='pschange-mc-no'/>
    <feature policy='disable' name='fma'/>
    <feature policy='disable' name='pcid'/>
    <feature policy='disable' name='avx'/>
    <feature policy='disable' name='f16c'/>
    <feature policy='disable' name='bmi1'/>
    <feature policy='disable' name='hle'/>
    <feature policy='disable' name='avx2'/>
    <feature policy='disable' name='bmi2'/>
    <feature policy='disable' name='invpcid'/>
    <feature policy='disable' name='rtm'/>
    <feature policy='disable' name='mpx'/>
    <feature policy='disable' name='avx512f'/>
    <feature policy='disable' name='avx512dq'/>
    <feature policy='disable' name='adx'/>
    <feature policy='disable' name='intel-pt'/>
    <feature policy='disable' name='avx512cd'/>
    <feature policy='disable' name='avx512bw'/>
    <feature policy='disable' name='avx512vl'/>
    <feature policy='disable' name='avx512vbmi'/>
    <feature policy='disable' name='pku'/>
    <feature policy='disable' name='avx512vbmi2'/>
    <feature policy='disable' name='vaes'/>
    <feature policy='disable' name='vpclmulqdq'/>
    <feature policy='disable' name='avx512vnni'/>
    <feature policy='disable' name='avx512bitalg'/>
    <feature policy='disable' name='avx512-vpopcntdq'/>
    <feature policy='disable' name='la57'/>
    <feature policy='disable' name='pdpe1gb'/>
    <feature policy='disable' name='abm'/>
    <feature policy='disable' name='wbnoinvd'/>
  </cpu>
...

Comment 9 Jiri Denemark 2020-10-07 08:36:44 UTC

(In reply to yalzhang from comment #8)
> 1) How can libvirt recognize the cpu module in "virsh capabilities" and
> "virsh domcapabilities"? 

Libvirt tries to find a model with matching signature (family, model, stepping
numbers) if possible and uses a "shortest additional features list" heuristics
if a model with matching signature is not present in our CPU map. But we
cannot disable features present in the selected model for virsh capabilities,
which means libvirt has to show a CPU model that is fully supported by the
host (no features have to be disabled) plus a list of additional features on
top of it.

> 2) Some flags do not exits, is it disabled by kernel?

Unlikely, the CPU just does not support them.

> 3) Do you think this is expected?

> 2. 
> # virsh domcapabilities 
> <domainCapabilities>
>   <cpu>
>     <mode name='host-passthrough' supported='yes'>
>       <enum name='hostPassthroughMigratable'>
>         <value>on</value>
>         <value>off</value>
>       </enum>
>     </mode>
>     <mode name='host-model' supported='yes'>
>   ******    <model fallback='forbid'>Icelake-Server</model> ******
>       <vendor>Intel</vendor>
...
>       <feature policy='disable' name='fma'/>
>       <feature policy='disable' name='pcid'/>
>       <feature policy='disable' name='avx'/>
>       <feature policy='disable' name='f16c'/>
>       <feature policy='disable' name='bmi1'/>
>       <feature policy='disable' name='hle'/>
>       <feature policy='disable' name='avx2'/>
>       <feature policy='disable' name='bmi2'/>
>       <feature policy='disable' name='invpcid'/>
>       <feature policy='disable' name='rtm'/>
>       <feature policy='disable' name='mpx'/>
>       <feature policy='disable' name='avx512f'/>
>       <feature policy='disable' name='avx512dq'/>
>       <feature policy='disable' name='adx'/>
>       <feature policy='disable' name='intel-pt'/>
>       <feature policy='disable' name='avx512cd'/>
>       <feature policy='disable' name='avx512bw'/>
>       <feature policy='disable' name='avx512vl'/>
>       <feature policy='disable' name='avx512vbmi'/>
>       <feature policy='disable' name='pku'/>
>       <feature policy='disable' name='avx512vbmi2'/>
>       <feature policy='disable' name='vaes'/>
>       <feature policy='disable' name='vpclmulqdq'/>
>       <feature policy='disable' name='avx512vnni'/>
>       <feature policy='disable' name='avx512bitalg'/>
>       <feature policy='disable' name='avx512-vpopcntdq'/>
>       <feature policy='disable' name='la57'/>
>       <feature policy='disable' name='pdpe1gb'/>
>       <feature policy='disable' name='abm'/>
>       <feature policy='disable' name='wbnoinvd'/>

Wow, that's a lot of disabled features. I think the CPU is not in fact
Icelake-Server.

>  ****     <model usable='no'>Icelake-Server-noTSX</model>  *****  should be
> yes??
>       <model usable='no'>Icelake-Server</model>
>  ****     <model usable='no'>Icelake-Client-noTSX</model> ****  should be
> yes??

No, usable='yes' would mean you can just create a domain with Icelake-Server
guest CPU and expect all its features to be enabled in the guest. Which is
definitely not going to happen on this host.

> 3. Check host cpu:
> # lscpu 
> Architecture:        x86_64
> CPU op-mode(s):      32-bit, 64-bit
> Byte Order:          Little Endian
> CPU(s):              24
> On-line CPU(s) list: 0-23
> Thread(s) per core:  1
> Core(s) per socket:  24
> Socket(s):           1
> NUMA node(s):        1
> Vendor ID:           GenuineIntel
> CPU family:          6
> Model:               134
> Model name:          Intel Atom(R) P5362 processor
> Stepping:            5

Oh, it's Atom. The CPU signature (family 6, model 134, stepping 5) actually
matches the one of Icelake-Server CPUs in our CPU map, but it doesn't really
look like Icelake-Server. I wonder what it is, ark.intel.com does not show
this CPU anywhere. The closest (by name) are Intel Atom P59* which are
SnowRidge CPUs.

Most likely the signatures for Icelake CPU models are incorrect or at least
incomplete in our CPU map. Mainly because only a few CPUs of this architecture
were actually released. The second issue is we're missing SnowRidge CPU model,
which would seem to be the right match for this host.

That said, the results are expected with the current state of libvirt's CPU
map.

Comment 12 yalzhang@redhat.com 2020-10-14 05:52:23 UTC

on ice-lake cpu which lack of "mpx" and "intel_pt"

1. on host:
# virsh capabilities 
<capabilities>

  <host>
    <uuid>88888888-8887-1814-080b-152ba5a5a5a5</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Broadwell-IBRS</model>
      <vendor>Intel</vendor>
      <microcode version='2332033632'/>
      <counter name='tsc' frequency='2700000000' scaling='yes'/>
      <topology sockets='1' dies='1' cores='24' threads='2'/>
      <feature name='vme'/>
      <feature name='ds'/>
      <feature name='acpi'/>
      <feature name='ss'/>
      <feature name='ht'/>
      <feature name='tm'/>
      <feature name='pbe'/>
      <feature name='dtes64'/>
      <feature name='monitor'/>
      <feature name='ds_cpl'/>
      <feature name='vmx'/>
      <feature name='smx'/>
      <feature name='est'/>
      <feature name='tm2'/>
      <feature name='xtpr'/>
      <feature name='pdcm'/>
      <feature name='dca'/>
      <feature name='osxsave'/>
      <feature name='f16c'/>
      <feature name='rdrand'/>
      <feature name='arat'/>
      <feature name='tsc_adjust'/>
      <feature name='cmt'/>
      <feature name='avx512f'/>
      <feature name='avx512dq'/>
      <feature name='avx512ifma'/>
      <feature name='clflushopt'/>
      <feature name='clwb'/>
      <feature name='intel-pt'/>
      <feature name='avx512cd'/>
      <feature name='sha-ni'/>
      <feature name='avx512bw'/>
      <feature name='avx512vl'/>
      <feature name='avx512vbmi'/>
      <feature name='umip'/>
      <feature name='pku'/>
      <feature name='ospke'/>
      <feature name='avx512vbmi2'/>
      <feature name='gfni'/>
      <feature name='vaes'/>
      <feature name='vpclmulqdq'/>
      <feature name='avx512vnni'/>
      <feature name='avx512bitalg'/>
      <feature name='avx512-vpopcntdq'/>
      <feature name='la57'/>
      <feature name='rdpid'/>
      <feature name='md-clear'/>
      <feature name='pconfig'/>
      <feature name='stibp'/>
      <feature name='arch-capabilities'/>
      <feature name='ssbd'/>
      <feature name='xsaveopt'/>
      <feature name='xsavec'/>
      <feature name='xgetbv1'/>
      <feature name='xsaves'/>
      <feature name='mbm_total'/>
      <feature name='mbm_local'/>
      <feature name='pdpe1gb'/>
      <feature name='abm'/>
      <feature name='invtsc'/>
      <feature name='wbnoinvd'/>
      <feature name='rdctl-no'/>
      <feature name='ibrs-all'/>
      <feature name='skip-l1dfl-vmentry'/>
      <feature name='mds-no'/>
      <feature name='pschange-mc-no'/>
      <pages unit='KiB' size='4'/>
      <pages unit='KiB' size='2048'/>
      <pages unit='KiB' size='1048576'/>
    </cpu>
...
# virsh domcapabilities
...
 <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Icelake-Server</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='ss'/>
      <feature policy='require' name='vmx'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='avx512ifma'/>
      <feature policy='require' name='sha-ni'/>
      <feature policy='require' name='rdpid'/>
      <feature policy='require' name='md-clear'/>
      <feature policy='require' name='stibp'/>
      <feature policy='require' name='arch-capabilities'/>
      <feature policy='require' name='xsaves'/>
      <feature policy='require' name='invtsc'/>
      <feature policy='require' name='ibpb'/>
      <feature policy='require' name='amd-stibp'/>
      <feature policy='require' name='amd-ssbd'/>
      <feature policy='require' name='rdctl-no'/>
      <feature policy='require' name='ibrs-all'/>
      <feature policy='require' name='skip-l1dfl-vmentry'/>
      <feature policy='require' name='mds-no'/>
      <feature policy='require' name='pschange-mc-no'/>
      <feature policy='disable' name='mpx'/>
      <feature policy='disable' name='intel-pt'/>
    </mode>
...
2. start a vm with cpu 
  <cpu mode='host-passthrough' check='partial' migratable='on'>
    <feature policy='disable' name='hle'/>
    <feature policy='disable' name='rtm'/>
  </cpu>

Login the guest, and check
# lscpu | grep hle
# lscpu | grep rtm
#  (no outputs)
# virsh domcapabilities
…
 <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Icelake-Server</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='ss'/>
      <feature policy='require' name='vmx'/>
…
    <feature policy='disable' name='hle'/>
      <feature policy='disable' name='rtm'/>
      <feature policy='disable' name='mpx'/>
      <feature policy='disable' name='intel-pt'/>
    </mode>
…
<mode name='custom' supported='yes'>
<model usable='yes'>Icelake-Server-noTSX</model>
      <model usable='no'>Icelake-Server</model>
      <model usable='yes'>Icelake-Client-noTSX</model>
      <model usable='no'>Icelake-Client</model>
…
 </mode>
  </cpu>

the result is as expected.

Comment 14 errata-xmlrpc 2020-11-17 17:48:38 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5137