RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1428952 - Enhance libvirt to present virtual L3 cache info for vcpus
Summary: Enhance libvirt to present virtual L3 cache info for vcpus
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.4
Hardware: x86_64
OS: Unspecified
high
urgent
Target Milestone: rc
: 7.4
Assignee: Jiri Denemark
QA Contact: Luyao Huang
URL:
Whiteboard:
Depends On: 1428534
Blocks: 1434537 1447612
TreeView+ depends on / blocked
 
Reported: 2017-03-03 16:48 UTC by Paolo Bonzini
Modified: 2020-06-11 13:22 UTC (History)
21 users (show)

Fixed In Version: libvirt-3.2.0-4.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1428534
: 1447612 (view as bug list)
Environment:
Last Closed: 2017-08-01 17:24:15 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1846 0 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2017-08-01 18:02:50 UTC

Comment 2 Daniel Berrangé 2017-04-20 11:39:07 UTC 
In addition to the l3-cache property, we should consider this RFE to cover exposure of the 'host-cache-info' property too.
Comment 3 Jiri Denemark 2017-04-26 15:48:26 UTC 
Patches sent upstream for review: https://www.redhat.com/archives/libvir-list/2017-April/msg01190.html
Comment 4 Jiri Denemark 2017-04-27 20:58:14 UTC 
Implemented upstream by

commit e841a41169dd726af13f29cb57fbafe658b7011a
Refs: v3.2.0-283-ge841a4116
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Apr 24 15:38:41 2017 +0200
Commit:     Jiri Denemark <jdenemar>
CommitDate: Thu Apr 27 21:50:29 2017 +0200

    conf: Rename mode parameter in virCPUDefParseXML

    The type of this parameter is virCPUType so calling it 'mode' is pretty
    strange, 'type' is a much better name.

    Signed-off-by: Jiri Denemark <jdenemar>

commit a646a6016a7c62688988f7cfaa22e1340c2ce761
Refs: v3.2.0-284-ga646a6016
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Apr 24 15:40:07 2017 +0200
Commit:     Jiri Denemark <jdenemar>
CommitDate: Thu Apr 27 22:07:14 2017 +0200

    Add support for CPU cache specification

    This patch introduces

        <cache level='N' mode='emulate'/>
       	<cache mode='passthrough'/>
       	<cache mode='disable'/>

    sub element of /domain/cpu. Currently only a single <cache> element is
    allowed.

    Signed-off-by: Jiri Denemark <jdenemar>

commit df13c0b477ffda460eed259c3b8aab7255f11199
Refs: v3.2.0-285-gdf13c0b47
Author:     Jiri Denemark <jdenemar>
AuthorDate: Tue Apr 25 19:07:19 2017 +0200
Commit:     Jiri Denemark <jdenemar>
CommitDate: Thu Apr 27 22:41:10 2017 +0200

    qemu: Add support for guest CPU cache

    This patch maps /domain/cpu/cache element into -cpu parameters:

    - <cache mode='passthrough'/> is translated to host-cache-info=on
    - <cache level='3' mode='emulate'/> is transformed into l3-cache=on
    - <cache mode='disable'/> is turned in host-cache-info=off,l3-cache=off

    Any other <cache> element is forbidden.

    The tricky part is detecting whether QEMU supports the CPU properties.

    The 'host-cache-info' property is introduced in v2.4.0-1389-ge265e3e480,
    earlier QEMU releases enabled host-cache-info by default and had no way
    to disable it. If the property is present, it defaults to 'off' for any
    QEMU until at least 2.9.0.

    The 'l3-cache' property was introduced later by v2.7.0-200-g14c985cffa.
    Earlier versions worked as if l3-cache=off was passed. For any QEMU
    until at least 2.9.0 l3-cache is 'off' by default.

    QEMU 2.9.0 was the first release which supports probing both properties
    by running device-list-properties with typename=host-x86_64-cpu. Older
    QEMU releases did not support device-list-properties command for CPU
    devices. Thus we can't really rely on probing them and we can just use
    query-cpu-model-expansion QMP command as a witness.

    Because the cache property probing is only reliable for QEMU >= 2.9.0
    when both are already supported for quite a few releases, we let QEMU
    report an error if a specific cache mode is explicitly requested. The
    other mode (or both if a user requested CPU cache to be disabled) is
    explicitly turned off for QEMU >= 2.9.0 to avoid any surprises in case
    the QEMU defaults change. Any older QEMU already turns them off so not
    doing so explicitly does not make any harm.

    Signed-off-by: Jiri Denemark <jdenemar>
Comment 10 Luyao Huang 2017-05-09 05:34:36 UTC 
Test with libvirt-3.2.0-4.el7.x86_64, qemu-kvm-rhev-2.9.0-3.el7.x86_64:

S1: l3 + host-model

1. prepare a guest which enable l3 cache:

  <cpu mode='host-model'>
    <model fallback='allow'/>
    <cache level='3' mode='emulate'/>
  </cpu>

2. start guest and recheck guest xml and qemu cmdline:

# virsh start r7
Domain r7 started

# virsh dumpxml r7 |grep  'cache'
    <cache level='3' mode='emulate'/>

# ps aux|grep qemu
qemu     12079 38.1  1.0 2165764 357216 ?      Sl   22:53   0:25 ...... -cpu Opteron_G5,vme=on,x2apic=on,tsc-deadline=on,hypervisor=on,arat=on,tsc_adjust=on,bmi1=on,mmxext=on,fxsr_opt=on,cmp_legacy=on,cr8legacy=on,osvw=on,rdtscp=off,svm=off,l3-cache=on

3. restart libvirtd and recheck guest

4. login guest and check cache:

# lscpu |grep cache
L1d cache:             64K
L1i cache:             64K
L2 cache:              512K
L3 cache:              16384K


S2: disable + host-model

1. modify the guest xml like this:

  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
    <cache mode='disable'/>
  </cpu>

2.  start guest and recheck guest xml and qemu cmdline:

# virsh start r7
Domain r7 started

# virsh dumpxml r7 |grep -a4 cache
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Opteron_G5</model>
    <vendor>AMD</vendor>
    <cache mode='disable'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='tsc-deadline'/>
    <feature policy='require' name='hypervisor'/>

# ps aux|grep qemu
qemu     12517 23.2  1.1 2170900 365120 ?      Sl   23:14   0:26 ... -cpu Opteron_G5,vme=on,x2apic=on,tsc-deadline=on,hypervisor=on,arat=on,tsc_adjust=on,bmi1=on,mmxext=on,fxsr_opt=on,cmp_legacy=on,cr8legacy=on,osvw=on,rdtscp=off,svm=off,l3-cache=off

3. restart libvirtd and recheck guest

4. login guest and check cache:

# lscpu |grep cache
L1d cache:             64K
L1i cache:             64K
L2 cache:              512K


S3 host-passthrough + l3

1. modify the guest xml like this:

  <cpu mode='host-passthrough' check='partial'>
    <cache level='3' mode='emulate'/>
  </cpu>

2.  start guest and recheck guest xml and qemu cmdline:

# virsh start r7
Domain r7 started

# virsh dumpxml r7 |grep -a1 cache
  <cpu mode='host-passthrough' check='partial'>
    <cache level='3' mode='emulate'/>
  </cpu>

# ps aux|grep qemu
qemu     12834 44.4  1.1 2204684 360968 ?      Sl   23:24   0:25 ..... -cpu host,l3-cache=on,host-cache-info=off

3. restart libvirtd and recheck guest

4. login guest and check cache:

# lscpu |grep cache
L1d cache:             64K
L1i cache:             64K
L2 cache:              512K
L3 cache:              16384K

S4 host-passthrough + passthrough

1. modify the guest xml like this:
  <cpu mode='host-passthrough' check='partial'>
    <cache mode='passthrough'/>
  </cpu>

2. start guest and recheck guest xml & qemu cmdline:

# virsh start r7
Domain r7 started

# virsh dumpxml r7 |grep -a1 cache
  <cpu mode='host-passthrough' check='partial'>
    <cache mode='passthrough'/>
  </cpu>

# ps aux|grep qemu
qemu     13158 95.5  1.1 2146228 387888 ?      Sl   23:30   0:25 ..... -cpu host,host-cache-info=on,l3-cache=off

3. restart libvirtd and recheck guest

4. login guest and check cache info:

# lscpu |grep cache
L1d cache:             16K
L1i cache:             64K
L2 cache:              2048K
L3 cache:              12288K


S5: host-passthrough + disable

1. modify the guest xml like this:

  <cpu mode='host-passthrough' check='partial'>
    <cache mode='disable'/>
  </cpu>

2. start guest and recheck guest xml & qemu cmdline:

# virsh start r7
Domain r7 started

# virsh dumpxml r7 |grep -a1 cache
  <cpu mode='host-passthrough' check='partial'>
    <cache mode='disable'/>
  </cpu>

# ps aux|grep qemu
qemu     13479 84.7  1.1 2159544 366840 ?      Sl   23:37   0:26 ... -cpu host,host-cache-info=off,l3-cache=off

3. restart libvirtd and recheck guest

4. login guest and check cache info:

# lscpu |grep cache
L1d cache:             64K
L1i cache:             64K
L2 cache:              512K


S6 default cpu mode + l3

1. modify guest xml like this:

  <cpu>
    <cache level='3' mode='emulate'/>
  </cpu>

2. start guest and recheck guest xml & qemu cmdline:
# virsh dumpxml r7 |grep -a1 cache
  <cpu>
    <cache level='3' mode='emulate'/>
  </cpu>

# ps aux |grep qemu
qemu     13779 76.1  1.2 2156472 395268 ?      Sl   23:42   0:23 .... -cpu qemu64,l3-cache=on 

3. restart libvirtd and recheck guest

4. login guest and check cache info:

# lscpu |grep cache
L1d cache:             64K
L1i cache:             64K
L2 cache:              512K
L3 cache:              16384K


S7 default cpu mode + disable

1. modify guest xml like this:

  <cpu>
    <cache mode='disable'/>
  </cpu>

2. start guest and recheck guest xml & qemu cmdline:

# virsh dumpxml r7 |grep -a1 cache
  <cpu>
    <cache mode='disable'/>
  </cpu>

# ps aux |grep qemu
qemu     14107 92.8  0.1 2120612 44808 ?       Sl   23:48   0:04 ...... -cpu qemu64,l3-cache=off

3. restart libvirtd and recheck guest

4. login guest and check cache info:

# lscpu |grep cache
L1d cache:             64K
L1i cache:             64K
L2 cache:              512K


S8 invalid cases

1. passthrough + host-model

# virsh edit r7
error: unsupported configuration: CPU cache mode 'passthrough' can only be used with 'host-passthrough' CPUs
Failed. Try again? [y,n,i,f,?]: 

2. cache level 2

# virsh edit r7
error: unsupported configuration: CPU cache mode 'emulate' can only be used with level='3'
Failed. Try again? [y,n,i,f,?]: 

3. cache level 3 + disable

# virsh edit r7
error: unsupported configuration: unsupported CPU cache level for mode 'disable'
Failed. Try again? [y,n,i,f,?]:
Comment 11 Luyao Huang 2017-05-09 05:59:44 UTC 
Test with libvirt-3.2.0-4.el7.x86_64 and a qemu-kvm which version < 2.9.0 (qemu-kvm-rhev-2.6.0-28.el7_3.9.x86_64). Test with the same steps in comment 10, the only different is there is no "l3-cache=off" and "host-cache-info=off" in qemu command line.

Test with libvirt-3.2.0-4.el7.x86_64 and a qemu-kvm unsupported l3-cache and host-cache-info:

1. l3-cache

# virsh start r7
error: Failed to start domain r7
error: internal error: qemu unexpectedly closed the monitor: 2017-05-09T05:57:57.519970Z qemu-kvm: unrecognized feature l3-cache

2. host-cache-info

# virsh start r7
error: Failed to start domain r7
error: internal error: qemu unexpectedly closed the monitor: 2017-05-09T05:58:32.526720Z qemu-kvm: unrecognized feature host-cache-info

3. disable

# virsh start r7
Domain r7 started
Comment 12 Luyao Huang 2017-05-09 06:00:43 UTC 
Verify this bug with the test result in comment 10 and comment 11.
Comment 13 errata-xmlrpc 2017-08-01 17:24:15 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Comment 14 errata-xmlrpc 2017-08-02 00:03:43 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Note You need to log in before you can comment on or make changes to this bug.