Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1693433

Summary:	[RFE] - libvirt support for cascade lake cpu - Slow Train
Product:	Red Hat Enterprise Linux 8	Reporter:	Karen Noel <knoel>
Component:	libvirt	Assignee:	Jiri Denemark <jdenemar>
Status:	CLOSED ERRATA	QA Contact:	jiyan <jiyan>
Severity:	unspecified	Docs Contact:
Priority:	high
Version:	8.0	CC:	berrange, chhu, dyuan, hhuang, jdenemar, jiyan, kchamart, knoel, lhuang, mjankula, mtessun, pbonzini, phrdina, rbalakri, virt-bugs, wchadwic, weizhan, xuzhang, yalzhang
Target Milestone:	rc	Keywords:	FutureFeature, HardwareEnablement
Target Release:	8.0	Flags:	knoel: mirror+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	libvirt-4.5.0-29.el8	Doc Type:	Enhancement
Doc Text:		Story Points:	---
Clone Of:	1678381	Environment:
Last Closed:	2019-11-05 20:48:28 UTC	Type:	Feature Request
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1629906, 1677209, 1678381, 1745045, 1749522
Bug Blocks:	1674420, 1701002

Description Karen Noel 2019-03-27 19:49:29 UTC

+++ This bug was initially created as a clone of Bug #1678381 +++

QEMU 3.1.0 in added the Cascadelake-Server CPU, so we need matching support in libvirt

+++ This bug was initially created as a clone of Bug #1677209 +++

This bug was initially created as a copy of Bug #1674411


Description of problem:
add support for cascade lake cpu to rhel7

Feature Description:
       Introduce new cpu model in qemu-kvm for Cascade platform.

       instructions detailed info, please ref https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf

--- Additional comment from Karen Noel on 2019-03-27 19:46:15 UTC ---

Looks like Cascade Lake lines up with RHEL 8.1. Thanks.

Comment 1 Paolo Bonzini 2019-03-29 13:17:21 UTC

The main thing to add here is MSR-based CPU features, so it's not just adding a CPU model.

Comment 2 Jiri Denemark 2019-03-29 14:38:10 UTC

Exactly. I already have the patches for MSR features functional. I just need
to polish the series, update tests, and deal with some QEMU interaction issues.

Comment 7 jiyan 2019-08-02 15:35:57 UTC

Verified this bug on libvirt-4.5.0-31.module+el8.1.0+3808+3325c1a3.x86_64.

Version:
libvirt-4.5.0-31.module+el8.1.0+3808+3325c1a3.x86_64
qemu-kvm-2.12.0-82.module+el8.1.0+3738+0d8c0249.x86_64
kernel-4.18.0-126.el8.x86_64

S1: Check cascadelake related info and start VM with host-model
Steps:
1. Prepare a physical host with the following cpu info
# lscpu
Model name:          Intel(R) Xeon(R) Platinum 8260L CPU @ 2.40GHz
Flags:               fpu ... avx512_vnni ...

2. Check cpu_map.xml; domcapabilities; capabilities
# cat /usr/share/libvirt/cpu_map.xml |grep Cas -A100
    <model name='Cascadelake-Server'>
      <signature family='6' model='85'/> <!-- 050654 -->
      <vendor name='Intel'/>
      <feature name='3dnowprefetch'/>
      <feature name='abm'/>
...
      <feature name='avx512f'/>
      <feature name='avx512vl'/>
      <feature name='avx512vnni'/>
...

# virsh capabilities 
<capabilities>

  <host>
    <uuid>88888888-8887-0813-0412-1f25a5a5a5a5</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Cascadelake-Server</model>
      <vendor>Intel</vendor>
      <microcode version='67108892'/>
      <topology sockets='1' cores='24' threads='2'/>
      <feature name='ds'/>
      <feature name='acpi'/>
      <feature name='ss'/>
      ...
    </cpu>

# virsh domcapabilities 
  <cpu>
    <mode name='host-passthrough' supported='yes'/>
    <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Cascadelake-Server</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='ss'/>
      <feature policy='require' name='vmx'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='umip'/>
      <feature policy='require' name='pku'/>
      <feature policy='require' name='md-clear'/>
      <feature policy='require' name='stibp'/>
      <feature policy='require' name='arch-capabilities'/>
      <feature policy='require' name='xsaves'/>
      <feature policy='require' name='invtsc'/>
    </mode>

3. Prepare a shutdown VM with the following conf; Start VM and check qemu cmd line info and dumpxml
# virsh domstate vm1
shut off

# virsh dumpxml vm1 --inactive |grep "<cpu" -A2
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>

# virsh start vm1
Domain vm1 started

# virsh dumpxml vm1 |grep "<cpu" -A20
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Cascadelake-Server</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='umip'/>
    <feature policy='require' name='pku'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='xsaves'/>
    <feature policy='disable' name='mpx'/>
  </cpu>

# ps -ef |grep vm1
qemu      49456      1 33 11:24 ?        
...
-cpu Cascadelake-Server,ss=on,vmx=on,hypervisor=on,tsc_adjust=on,umip=on,pku=on,md-clear=on,stibp=on,arch-capabilities=on,xsaves=on

4. Login to VM and check avx512_vnni cpu flag
# virsh console vm1
Connected to domain vm1
Escape character is ^]

Red Hat Enterprise Linux 8.1 Beta (Ootpa)
Kernel 4.18.0-107.el8.x86_64 on an x86_64

localhost login: root
Password: 
Last login: Fri Aug  2 22:28:55 on ttyS0
[root@localhost ~]# lscpu
...
CPU family:          6
Model:               85
Model name:          Intel Xeon Processor (Cascadelake)
...
Flags:               fpu vme ... avx512_vnni ...

Comment 8 jiyan 2019-08-03 01:47:50 UTC

S2: Test other features with cpu and check cpu related cpu info
2.1 Create snapshot for VM and check cpu info in snapshot dumpxml 
# virsh snapshot-create-as vm1 s1 --disk-only --diskspec hda,file=/tmp/s1
Domain snapshot s1 created

# virsh snapshot-dumpxml vm1 s1 
...
    <cpu mode='custom' match='exact' check='partial'>
      <model fallback='forbid'>Cascadelake-Server</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='ss'/>
      <feature policy='require' name='vmx'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='umip'/>
      <feature policy='require' name='pku'/>
      <feature policy='require' name='md-clear'/>
      <feature policy='require' name='stibp'/>
      <feature policy='require' name='arch-capabilities'/>
      <feature policy='require' name='xsaves'/>
    </cpu>
...


2.2 Save VM and restore VM; check save-img-dumpxml file
# virsh save vm1 /tmp/save.1
Domain vm1 saved to /tmp/save.1

# virsh save-image-dumpxml /tmp/save.1
  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='forbid'>Cascadelake-Server</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='umip'/>
    <feature policy='require' name='pku'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='xsaves'/>
  </cpu>

# virsh restore /tmp/save.1 
Domain restored from /tmp/save.1


2.3 Managedsave VM and start VM again
# virsh managedsave vm1 
Domain vm1 state saved by libvirt

# virsh start vm1 
Domain vm1 started

# systemctl restart libvirtd

# ps -ef |grep vm1
qemu      57038      1  2 21:20 ?        
-cpu Cascadelake-Server,ss=on,vmx=on,hypervisor=on,tsc_adjust=on,umip=on,pku=on,md-clear=on,stibp=on,arch-capabilities=on,xsaves=on,mpx=off


2.4 Test vcpupin and vcpu hot-plug
# virsh domstate vm1
running

# virsh dumpxml vm1 |grep vcpu
  <vcpu placement='static' current='3'>8</vcpu>

# virsh vcpuinfo vm1
VCPU:           0
CPU:            58
State:          running
CPU time:       12.3s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           1
CPU:            167
State:          running
CPU time:       3.6s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           2
CPU:            191
State:          running
CPU time:       3.8s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

# virsh setvcpus vm1 --count 6

# virsh vcpupin vm1 3 176

# virsh vcpupin vm1 4 0-99,^76

# virsh vcpupin vm1 5 100-101

# virsh vcpuinfo vm1
VCPU:           0
CPU:            1
State:          running
CPU time:       24.6s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           1
CPU:            48
State:          running
CPU time:       6.1s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           2
CPU:            155
State:          running
CPU time:       6.4s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           3
CPU:            176
State:          running
CPU time:       2.2s
CPU Affinity:   --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------y---------------

VCPU:           4
CPU:            55
State:          running
CPU time:       2.2s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy-yyyyyyyyyyyyyyyyyyyyyyy--------------------------------------------------------------------------------------------

VCPU:           5
CPU:            100
State:          running
CPU time:       2.6s
CPU Affinity:   ----------------------------------------------------------------------------------------------------yy------------------------------------------------------------------------------------------

# virsh vcpupin vm1
VCPU: CPU Affinity
----------------------------------
   0: 0-191
   1: 0-191
   2: 0-191
   3: 176
   4: 0-75,77-99
   5: 100-101
   6: 0-191
   7: 0-191

# virsh console vm1
Connected to domain vm1
Escape character is ^]

Red Hat Enterprise Linux 8.1 Beta (Ootpa)
Kernel 4.18.0-107.el8.x86_64 on an x86_64

localhost login: root
Password: 
Last login: Fri Aug  2 23:27:11 on ttyS0
[root@localhost ~]# lscpu
...
CPU(s):              6
On-line CPU(s) list: 0-5
Model name:          Intel Xeon Processor (Cascadelake)


S3: Negative test cascadelake cpu model on physical host does not support this func
# virsh domstate vm1
shut off

# virsh dumpxml vm1 --inactive |grep "<cpu" -A3
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Cascadelake-Server</model>
    <vendor>Intel</vendor>
  </cpu>

# virsh start vm1
error: Failed to start domain vm1
error: operation failed: guest CPU doesn't match specification: extra features: hypervisor, missing features: hle,rtm,mpx,avx512f,avx512dq,rdseed,adx,smap,clflushopt,clwb,avx512cd,avx512bw,avx512vl,pku,avx512vnni,xsavec,xgetbv1,3dnowprefetch

All the test results are expected; move this bug to be verified.

Comment 9 Jaroslav Suchanek 2019-09-09 10:42:05 UTC

*** Bug 1749522 has been marked as a duplicate of this bug. ***

Comment 11 errata-xmlrpc 2019-11-05 20:48:28 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3345