Bug 1678381

Summary: [RFE] - libvirt support for cascade lake cpu - Fast Train
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: Daniel Berrangé <berrange>
Component: libvirtAssignee: Jiri Denemark <jdenemar>
Status: CLOSED ERRATA QA Contact: jiyan <jiyan>
Severity: unspecified Docs Contact:
Priority: high    
Version: 8.0CC: chhu, dyuan, jdenemar, jiyan, jsuchane, kchamart, knoel, lhuang, lijin, mjankula, rbalakri, virt-bugs, xuzhang, yalzhang
Target Milestone: rcKeywords: FutureFeature
Target Release: 8.1Flags: areis: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-5.3.0-1.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1677209
: 1693433 1745045 1749522 (view as bug list) Environment:
Last Closed: 2019-11-06 07:12:59 UTC Type: Feature Request
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1677209, 1710552    
Bug Blocks: 1674420, 1693433, 1745045, 1749522    

Description Daniel Berrangé 2019-02-18 16:33:37 UTC 
QEMU 3.1.0 in added the Cascadelake-Server CPU, so we need matching support in libvirt

+++ This bug was initially created as a clone of Bug #1677209 +++

This bug was initially created as a copy of Bug #1674411

I am copying this bug because: 



Description of problem:
add support for cascade lake cpu to rhel7

Feature Description:
       Introduce new cpu model in qemu-kvm for Cascade platform.

       instructions detailed info, please ref https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf

--- Additional comment from Jiri Denemark on 2019-02-14 09:49:46 GMT ---
Comment 3 Jiri Denemark 2019-06-20 07:44:14 UTC 
Added upstream by

commit 2878278c74cc450a7e28a3830ed0ceff3126f12f
Refs: v5.2.0-226-g2878278c74
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Mar 11 16:39:25 2019 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Apr 12 22:53:39 2019 +0200

    cpu_map: Add Cascadelake-Server CPU model

    Introduced in QEMU 3.1.0 by commit
    c7a88b52f62b30c04158eeb07f73e3f72221b6a8

    Signed-off-by: Jiri Denemark <jdenemar>
    Reviewed-by: Ján Tomko <jtomko>
Comment 5 jiyan 2019-08-03 03:14:26 UTC 
Verified this bug on libvirt-5.5.0-2.module+el8.1.0+3773+7dd501bf.x86_64

Version:
libvirt-5.5.0-2.module+el8.1.0+3773+7dd501bf.x86_64
qemu-kvm-4.0.0-6.module+el8.1.0+3736+a2aefea3.x86_64
kernel-4.18.0-126.el8.x86_64

Steps:
1. Prepare a physical host and check cascade lake related cpu info
# cat /usr/share/libvirt/cpu_map/x86_Cascadelake-Server.xml 
<cpus>
  <model name='Cascadelake-Server'>
    <signature family='6' model='85'/> <!-- 050654 -->
    <vendor name='Intel'/>
    <feature name='3dnowprefetch'/>
    <feature name='abm'/>
...
    <feature name='avx512vl'/>
    <feature name='avx512vnni'/>
...
    <feature name='xsaveopt'/>
  </model>
</cpus>

# virsh domcapabilities
...
  <cpu>
    <mode name='host-passthrough' supported='yes'/>
    <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Cascadelake-Server</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='ss'/>
      <feature policy='require' name='vmx'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='umip'/>
      <feature policy='require' name='pku'/>
      <feature policy='require' name='md-clear'/>
      <feature policy='require' name='stibp'/>
      <feature policy='require' name='arch-capabilities'/>
      <feature policy='require' name='xsaves'/>
      <feature policy='require' name='invtsc'/>
    </mode>
...

# virsh capabilities
<capabilities>
...
      <model>Cascadelake-Server</model>
      <vendor>Intel</vendor>
      <microcode version='67108892'/>
      <counter name='tsc' frequency='2394375000' scaling='yes'/>
      <topology sockets='1' cores='24' threads='2'/>
...
      <feature name='ss'/>
...
      <feature name='skip-l1dfl-vmentry'/>
      <feature name='mds-no'/>
...
      <pages unit='KiB' size='1048576'/>
    </cpu>
...

2. Test (hypervisor)-cpu-baseline/compare with the output of domcapabilities and capabilities
In local host:
# virsh domcapabilities > dom1.xml
# virsh capabilities > cap1.xml

In another host:
# virsh domcapabilities > dom2.xml
# virsh capabilities > cap2.xml
# # scp dom2.xml cap2.xml root@{{localhost}}:/root

In local host:
# cat cap1.xml cap2.xml >> capall.xml
# cat dom1.xml dom2.xml >> domall.xml
# virsh cpu-compare capall.xml 
# virsh cpu-baseline capall.xml 
# virsh hypervisor-cpu-compare capall.xml 
# virsh hypervisor-cpu-baseline capall.xml 
# virsh cpu-compare domall.xml 
# virsh cpu-baseline domall.xml 
# virsh hypervisor-cpu-compare domall.xml 
# virsh hypervisor-cpu-baseline domall.xml 
No abnormal err for this 8 cmds.

3. Prepare a shutdown Vm with host-model cpu conf and start VM
# virsh domstate vm1
shut off

# virsh dumpxml vm1 --inactive |grep "<cpu" -A2
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>

# virsh start vm1
Domain vm1 started

4. Check qemu cmd line; active dumpxml and login to VM
# virsh dumpxml vm1 |grep "<cpu" -A20
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Cascadelake-Server</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='umip'/>
    <feature policy='require' name='pku'/>
    <feature policy='require' name='md-clear'/>
    <feature policy='require' name='stibp'/>
    <feature policy='require' name='arch-capabilities'/>
    <feature policy='require' name='xsaves'/>
  </cpu>

# ps -ef |grep vm1
qemu      44538      1 76 22:55 ?        
-cpu Cascadelake-Server,ss=on,vmx=on,hypervisor=on,tsc_adjust=on,umip=on,pku=on,md-clear=on,stibp=on,arch-capabilities=on,xsaves=on

# virsh console vm1
Connected to domain vm1
Escape character is ^]

Red Hat Enterprise Linux 8.1 Beta (Ootpa)
Kernel 4.18.0-107.el8.x86_64 on an x86_64

localhost login: root
Password: 
[root@localhost ~]# lscpu
...
Model name:          Intel Xeon Processor (Cascadelake)
...
Flags:               fpu vme ... avx512_vnni ...

5. Create snapshot for VM and check snapshot dumpxml
# virsh snapshot-create-as vm1 s1 --disk-only --diskspec vda,file=/tmp/s1
# virsh snapshot-dumpxml vm1 s1
# systemctl restart libvirtd
# virsh snapshot-dumpxml vm1 s1

6. Save VM state and check save-image-dumpxml file 
[because of Bug 1559845 - [Fast] Nested KVM: support for migration of nested hypervisors - Fast Train]
# virsh save vm1 /tmp/vm1.save
error: Failed to save domain vm1 to /tmp/vm1.save
error: internal error: unable to execute QEMU command 'migrate': Nested VMX virtualization does not support live migration yet

7. Test vcpu affinity and hot-plugging vcpus
# virsh domstate vm1
running

# virsh dumpxml vm1 |grep vcpu
  <vcpu placement='static' current='4'>8</vcpu>

# virsh vcpuinfo vm1
VCPU:           0
CPU:            58
State:          running
CPU time:       9.7s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           1
CPU:            2
State:          running
CPU time:       2.7s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           2
CPU:            73
State:          running
CPU time:       3.6s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           3
CPU:            24
State:          running
CPU time:       2.8s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

# virsh vcpupin vm1 5 0-99

# virsh vcpupin vm1 6 100-191

# virsh vcpupin vm1 7 0-191,^5,^109

# virsh vcpupin vm1 7 0-191,^5,^109

# virsh vcpuinfo vm1
VCPU:           0
CPU:            58
State:          running
CPU time:       9.8s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           1
CPU:            2
State:          running
CPU time:       2.9s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           2
CPU:            73
State:          running
CPU time:       3.8s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           3
CPU:            24
State:          running
CPU time:       3.1s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           4
CPU:            159
State:          running
CPU time:       0.1s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

VCPU:           5
CPU:            63
State:          running
CPU time:       0.1s
CPU Affinity:   yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy--------------------------------------------------------------------------------------------

VCPU:           6
CPU:            100
State:          running
CPU time:       0.4s
CPU Affinity:   ----------------------------------------------------------------------------------------------------yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

# virsh console vm1
Connected to domain vm1
Escape character is ^]

Red Hat Enterprise Linux 8.1 Beta (Ootpa)
Kernel 4.18.0-107.el8.x86_64 on an x86_64

localhost login: root
Password: 
Last login: Sat Aug  3 10:56:49 on ttyS0
[root@localhost ~]# lscpu 
...
On-line CPU(s) list: 0-6
...
Model name:          Intel Xeon Processor (Cascadelake)

8.Test nagative scenarios: start Vm with Cascadelake cpu model on host does not support Cascadelake
# lscpu
...
Model:               60
Model name:          Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz

# virsh domstate vm1
shut off

# virsh dumpxml vm1 --inactive
  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='forbid'>Cascadelake-Server</model>
    <vendor>Intel</vendor>
  </cpu>

# virsh start vm1
error: Failed to start domain vm1
error: the CPU is incompatible with host CPU: Host CPU does not provide required features: hle, rtm, mpx, avx512f, avx512dq, rdseed, adx, smap, clflushopt, clwb, avx512cd, avx512bw, avx512vl, avx512vnni, xsavec, xgetbv1, 3dnowprefetch

All the results are expected; move this bug to be verified.
Comment 7 errata-xmlrpc 2019-11-06 07:12:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3723