Bug 1447612

Summary: Enhance libvirt to present virtual L3 cache info for vcpus
Product: Red Hat Enterprise Linux 7 Reporter: Jaroslav Reznik <jreznik>
Component: libvirtAssignee: Jiri Denemark <jdenemar>
Status: CLOSED ERRATA QA Contact: Luyao Huang <lhuang>
Severity: urgent Docs Contact:
Priority: high    
Version: 7.4CC: berrange, chayang, coli, dyuan, hhuang, jdenemar, jmelvin, juzhang, kchamart, knoel, michen, mtessun, pbonzini, rbalakri, rkrcmar, salmy, snagar, virt-bugs, virt-maint, xuzhang, yalzhang
Target Milestone: rcKeywords: FutureFeature, Upstream, ZStream
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-2.0.0-10.el7_3.9 Doc Type: Enhancement
Doc Text:
Feature: Level-3 CPU cache can be enabled for virtual CPUs using <cache level'3' mode='emulate'/> sub-element of the <cpu> element in domain XML. Reason: Reporting an L3 cache in a guest can increase performance in some workloads. Result: Performance in some workloads can significantly increase after an L3 CPU cache is enabled.
Story Points: ---
Clone Of: 1428952 Environment:
Last Closed: 2017-05-25 15:37:08 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1428952    
Bug Blocks:    

Description Jaroslav Reznik 2017-05-03 10:24:21 UTC
This bug has been copied from bug #1428952 and has been proposed
to be backported to 7.3 z-stream (EUS).

Comment 6 Luyao Huang 2017-05-08 10:59:13 UTC
Verify this bug with libvirt-2.0.0-10.el7_3.9.x86_64, qemu-kvm-rhev-2.6.0-28.el7_3.9.x86_64:

S1: l3 + host-model

1. prepare a guest which enable l3 cache:

  <cpu mode='host-model'>
    <model fallback='allow'/>
    <cache level='3' mode='emulate'/>
  </cpu>

2. start guest and recheck guest xml and qemu cmdline:

# virsh start r7
Domain r7 started

# virsh dumpxml r7 |grep -a2 'cache'
  <cpu mode='host-model'>
    <model fallback='allow'/>
    <cache level='3' mode='emulate'/>
  </cpu>
  <clock offset='utc'>

# ps aux|grep qemu
qemu      6106 49.2  1.1 2164540 372040 ?      Sl   06:06   0:26 ..... -cpu Opteron_G5,+vme,+ht,+monitor,+osxsave,+bmi1,+mmxext,+fxsr_opt,+cmp_legacy,+extapic,+cr8legacy,+osvw,+ibs,+skinit,+wdt,+lwp,+tce,+nodeid_msr,+topoext,+perfctr_core,+perfctr_nb,l3-cache=on

3. restart libvirtd and recheck guest:

# service libvirtd restart
Redirecting to /bin/systemctl restart libvirtd.service

# virsh dumpxml r7 |grep -a2 'cache'
  <cpu mode='host-model'>
    <model fallback='allow'/>
    <cache level='3' mode='emulate'/>
  </cpu>
  <clock offset='utc'>

4. login guest and check the cache info:

IN GUEST:

# lscpu |grep cache
L1d cache:             64K
L1i cache:             64K
L2 cache:              512K
L3 cache:              16384K


S2: disable + host-model

1. modify the guest xml like this:

  <cpu mode='host-model'>
    <model fallback='allow'/>
    <cache mode='disable'/>
  </cpu>

2. start guest and check xml and qemu cmdline:

# virsh start r7
Domain r7 started

# virsh dumpxml r7 |grep -a2 'cache'
  <cpu mode='host-model'>
    <model fallback='allow'/>
    <cache mode='disable'/>
  </cpu>
  <clock offset='utc'>

# ps aux|grep qemu
qemu      6437 13.4  1.1 2148144 373056 ?      Sl   06:13   0:27 ...... -cpu Opteron_G5,+vme,+ht,+monitor,+osxsave,+bmi1,+mmxext,+fxsr_opt,+cmp_legacy,+extapic,+cr8legacy,+osvw,+ibs,+skinit,+wdt,+lwp,+tce,+nodeid_msr,+topoext,+perfctr_core,+perfctr_nb


S3: host-passthrough + l3

1. modify the guest xml like this:
# virsh dumpxml r7 |grep -a1 'cache'
  <cpu mode='host-passthrough'>
    <cache level='3' mode='emulate'/>
  </cpu>

2. start guest and check xml and qemu cmdline:

# virsh start r7
Domain r7 started

# virsh dumpxml r7 |grep -a1 'cache'
  <cpu mode='host-passthrough'>
    <cache level='3' mode='emulate'/>
  </cpu>

# ps aux|grep qemu
qemu      6659  104  0.1 2112292 50536 ?       Sl   06:19   0:05 ..... -cpu host,l3-cache=on

3. restart libvirtd and recheck guest:

# virsh dumpxml r7 |grep -a1 'cache'
  <cpu mode='host-passthrough'>
    <cache level='3' mode='emulate'/>
  </cpu>

4. login guest and check the cache info:

IN GUEST:

# lscpu |grep cache
L1d cache:             64K
L1i cache:             64K
L2 cache:              512K
L3 cache:              16384K


S4: host-passthrough + passthrough

1. modify the guest xml like this:
  <cpu mode='host-passthrough'>
    <cache mode='passthrough'/>
  </cpu>


2. start guest and recheck guest xml & qemu cmdline:

# virsh start r7
Domain r7 started

# virsh dumpxml r7 |grep -a1 'cache'
  <cpu mode='host-passthrough'>
    <cache mode='passthrough'/>
  </cpu>

# ps aux|grep qemu
qemu      6968 15.4  1.1 2147116 379864 ?      Sl   06:25   0:26 ...... -cpu host,host-cache-info=on


3. restart libvirtd and recheck guest:

# service libvirtd restart
Redirecting to /bin/systemctl restart libvirtd.service

# virsh dumpxml r7 |grep -a1 'cache'
  <cpu mode='host-passthrough'>
    <cache mode='passthrough'/>
  </cpu>


4. login guest and check the cache info:

IN GUEST:

# lscpu |grep cache
L1d cache:             16K
L1i cache:             64K
L2 cache:              2048K
L3 cache:              12288K


S5: default cpu mode + l3

1. modify guest xml like this:

# virsh dumpxml r7 |grep -a1 'cache'
  <cpu>
    <cache level='3' mode='emulate'/>
  </cpu>

2. start guest and recheck guest xml and qemu cmdline:

# virsh start r7
Domain r7 started

# virsh dumpxml r7 |grep -a1 'cache'
  <cpu>
    <cache level='3' mode='emulate'/>
  </cpu>

# ps aux|grep qemu
qemu      7285  107  0.1 2114336 49244 ?       Sl   06:37   0:04 ...... -cpu qemu64,l3-cache=on

3. restart libvirtd and recheck:

# virsh dumpxml r7 |grep -a1 'cache'
  <cpu>
    <cache level='3' mode='emulate'/>
  </cpu>

4. login guest:

# lscpu |grep cache
L1d cache:             64K
L1i cache:             64K
L2 cache:              512K
L3 cache:              16384K


S6: some invalid config

1. passthrough + host-model

# virsh edit r7
error: unsupported configuration: CPU cache mode 'passthrough' can only be used with 'host-passthrough' CPUs
Failed. Try again? [y,n,i,f,?]: 

2. cache level 2

# virsh edit r7
error: unsupported configuration: CPU cache mode 'emulate' can only be used with level='3'
Failed. Try again? [y,n,i,f,?]: 

3. cache level 3 + disable

# virsh edit r7
error: unsupported configuration: unsupported CPU cache level for mode 'disable'
Failed. Try again? [y,n,i,f,?]: 

4. old qemu + l3 cache:

# virsh start r7
error: Failed to start domain r7
error: internal error: qemu unexpectedly closed the monitor: 2017-05-08T10:52:08.249316Z qemu-kvm: unrecognized feature l3-cache

5. old qemu + passthrough cache:

# virsh start r7
error: Failed to start domain r7
error: internal error: qemu unexpectedly closed the monitor: 2017-05-08T10:53:12.563395Z qemu-kvm: unrecognized feature host-cache-info

Comment 9 Luyao Huang 2017-05-09 01:46:41 UTC
Move this bug status to verify according to comment 6

Comment 11 errata-xmlrpc 2017-05-25 15:37:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1304