Bug 1428952

In addition to the l3-cache property, we should consider this RFE to cover exposure of the 'host-cache-info' property too.

Patches sent upstream for review: https://www.redhat.com/archives/libvir-list/2017-April/msg01190.html

Implemented upstream by

commit e841a41169dd726af13f29cb57fbafe658b7011a
Refs: v3.2.0-283-ge841a4116
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Apr 24 15:38:41 2017 +0200
Commit:     Jiri Denemark <jdenemar>
CommitDate: Thu Apr 27 21:50:29 2017 +0200

    conf: Rename mode parameter in virCPUDefParseXML

    The type of this parameter is virCPUType so calling it 'mode' is pretty
    strange, 'type' is a much better name.

    Signed-off-by: Jiri Denemark <jdenemar>

commit a646a6016a7c62688988f7cfaa22e1340c2ce761
Refs: v3.2.0-284-ga646a6016
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Apr 24 15:40:07 2017 +0200
Commit:     Jiri Denemark <jdenemar>
CommitDate: Thu Apr 27 22:07:14 2017 +0200

    Add support for CPU cache specification

    This patch introduces

        <cache level='N' mode='emulate'/>
       	<cache mode='passthrough'/>
       	<cache mode='disable'/>

    sub element of /domain/cpu. Currently only a single <cache> element is
    allowed.

    Signed-off-by: Jiri Denemark <jdenemar>

commit df13c0b477ffda460eed259c3b8aab7255f11199
Refs: v3.2.0-285-gdf13c0b47
Author:     Jiri Denemark <jdenemar>
AuthorDate: Tue Apr 25 19:07:19 2017 +0200
Commit:     Jiri Denemark <jdenemar>
CommitDate: Thu Apr 27 22:41:10 2017 +0200

    qemu: Add support for guest CPU cache

    This patch maps /domain/cpu/cache element into -cpu parameters:

    - <cache mode='passthrough'/> is translated to host-cache-info=on
    - <cache level='3' mode='emulate'/> is transformed into l3-cache=on
    - <cache mode='disable'/> is turned in host-cache-info=off,l3-cache=off

    Any other <cache> element is forbidden.

    The tricky part is detecting whether QEMU supports the CPU properties.

    The 'host-cache-info' property is introduced in v2.4.0-1389-ge265e3e480,
    earlier QEMU releases enabled host-cache-info by default and had no way
    to disable it. If the property is present, it defaults to 'off' for any
    QEMU until at least 2.9.0.

    The 'l3-cache' property was introduced later by v2.7.0-200-g14c985cffa.
    Earlier versions worked as if l3-cache=off was passed. For any QEMU
    until at least 2.9.0 l3-cache is 'off' by default.

    QEMU 2.9.0 was the first release which supports probing both properties
    by running device-list-properties with typename=host-x86_64-cpu. Older
    QEMU releases did not support device-list-properties command for CPU
    devices. Thus we can't really rely on probing them and we can just use
    query-cpu-model-expansion QMP command as a witness.

    Because the cache property probing is only reliable for QEMU >= 2.9.0
    when both are already supported for quite a few releases, we let QEMU
    report an error if a specific cache mode is explicitly requested. The
    other mode (or both if a user requested CPU cache to be disabled) is
    explicitly turned off for QEMU >= 2.9.0 to avoid any surprises in case
    the QEMU defaults change. Any older QEMU already turns them off so not
    doing so explicitly does not make any harm.

    Signed-off-by: Jiri Denemark <jdenemar>

Test with libvirt-3.2.0-4.el7.x86_64, qemu-kvm-rhev-2.9.0-3.el7.x86_64:

S1: l3 + host-model

1. prepare a guest which enable l3 cache:

  <cpu mode='host-model'>
    <model fallback='allow'/>
    <cache level='3' mode='emulate'/>
  </cpu>

2. start guest and recheck guest xml and qemu cmdline:

# virsh start r7
Domain r7 started

# virsh dumpxml r7 |grep  'cache'
    <cache level='3' mode='emulate'/>

# ps aux|grep qemu
qemu     12079 38.1  1.0 2165764 357216 ?      Sl   22:53   0:25 ...... -cpu Opteron_G5,vme=on,x2apic=on,tsc-deadline=on,hypervisor=on,arat=on,tsc_adjust=on,bmi1=on,mmxext=on,fxsr_opt=on,cmp_legacy=on,cr8legacy=on,osvw=on,rdtscp=off,svm=off,l3-cache=on

3. restart libvirtd and recheck guest

4. login guest and check cache:

# lscpu |grep cache
L1d cache:             64K
L1i cache:             64K
L2 cache:              512K
L3 cache:              16384K


S2: disable + host-model

1. modify the guest xml like this:

  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
    <cache mode='disable'/>
  </cpu>

2.  start guest and recheck guest xml and qemu cmdline:

# virsh start r7
Domain r7 started

# virsh dumpxml r7 |grep -a4 cache
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Opteron_G5</model>
    <vendor>AMD</vendor>
    <cache mode='disable'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='tsc-deadline'/>
    <feature policy='require' name='hypervisor'/>

# ps aux|grep qemu
qemu     12517 23.2  1.1 2170900 365120 ?      Sl   23:14   0:26 ... -cpu Opteron_G5,vme=on,x2apic=on,tsc-deadline=on,hypervisor=on,arat=on,tsc_adjust=on,bmi1=on,mmxext=on,fxsr_opt=on,cmp_legacy=on,cr8legacy=on,osvw=on,rdtscp=off,svm=off,l3-cache=off

3. restart libvirtd and recheck guest

4. login guest and check cache:

# lscpu |grep cache
L1d cache:             64K
L1i cache:             64K
L2 cache:              512K


S3 host-passthrough + l3

1. modify the guest xml like this:

  <cpu mode='host-passthrough' check='partial'>
    <cache level='3' mode='emulate'/>
  </cpu>

2.  start guest and recheck guest xml and qemu cmdline:

# virsh start r7
Domain r7 started

# virsh dumpxml r7 |grep -a1 cache
  <cpu mode='host-passthrough' check='partial'>
    <cache level='3' mode='emulate'/>
  </cpu>

# ps aux|grep qemu
qemu     12834 44.4  1.1 2204684 360968 ?      Sl   23:24   0:25 ..... -cpu host,l3-cache=on,host-cache-info=off

3. restart libvirtd and recheck guest

4. login guest and check cache:

# lscpu |grep cache
L1d cache:             64K
L1i cache:             64K
L2 cache:              512K
L3 cache:              16384K

S4 host-passthrough + passthrough

1. modify the guest xml like this:
  <cpu mode='host-passthrough' check='partial'>
    <cache mode='passthrough'/>
  </cpu>

2. start guest and recheck guest xml & qemu cmdline:

# virsh start r7
Domain r7 started

# virsh dumpxml r7 |grep -a1 cache
  <cpu mode='host-passthrough' check='partial'>
    <cache mode='passthrough'/>
  </cpu>

# ps aux|grep qemu
qemu     13158 95.5  1.1 2146228 387888 ?      Sl   23:30   0:25 ..... -cpu host,host-cache-info=on,l3-cache=off

3. restart libvirtd and recheck guest

4. login guest and check cache info:

# lscpu |grep cache
L1d cache:             16K
L1i cache:             64K
L2 cache:              2048K
L3 cache:              12288K


S5: host-passthrough + disable

1. modify the guest xml like this:

  <cpu mode='host-passthrough' check='partial'>
    <cache mode='disable'/>
  </cpu>

2. start guest and recheck guest xml & qemu cmdline:

# virsh start r7
Domain r7 started

# virsh dumpxml r7 |grep -a1 cache
  <cpu mode='host-passthrough' check='partial'>
    <cache mode='disable'/>
  </cpu>

# ps aux|grep qemu
qemu     13479 84.7  1.1 2159544 366840 ?      Sl   23:37   0:26 ... -cpu host,host-cache-info=off,l3-cache=off

3. restart libvirtd and recheck guest

4. login guest and check cache info:

# lscpu |grep cache
L1d cache:             64K
L1i cache:             64K
L2 cache:              512K


S6 default cpu mode + l3

1. modify guest xml like this:

  <cpu>
    <cache level='3' mode='emulate'/>
  </cpu>

2. start guest and recheck guest xml & qemu cmdline:
# virsh dumpxml r7 |grep -a1 cache
  <cpu>
    <cache level='3' mode='emulate'/>
  </cpu>

# ps aux |grep qemu
qemu     13779 76.1  1.2 2156472 395268 ?      Sl   23:42   0:23 .... -cpu qemu64,l3-cache=on 

3. restart libvirtd and recheck guest

4. login guest and check cache info:

# lscpu |grep cache
L1d cache:             64K
L1i cache:             64K
L2 cache:              512K
L3 cache:              16384K


S7 default cpu mode + disable

1. modify guest xml like this:

  <cpu>
    <cache mode='disable'/>
  </cpu>

2. start guest and recheck guest xml & qemu cmdline:

# virsh dumpxml r7 |grep -a1 cache
  <cpu>
    <cache mode='disable'/>
  </cpu>

# ps aux |grep qemu
qemu     14107 92.8  0.1 2120612 44808 ?       Sl   23:48   0:04 ...... -cpu qemu64,l3-cache=off

3. restart libvirtd and recheck guest

4. login guest and check cache info:

# lscpu |grep cache
L1d cache:             64K
L1i cache:             64K
L2 cache:              512K


S8 invalid cases

1. passthrough + host-model

# virsh edit r7
error: unsupported configuration: CPU cache mode 'passthrough' can only be used with 'host-passthrough' CPUs
Failed. Try again? [y,n,i,f,?]: 

2. cache level 2

# virsh edit r7
error: unsupported configuration: CPU cache mode 'emulate' can only be used with level='3'
Failed. Try again? [y,n,i,f,?]: 

3. cache level 3 + disable

# virsh edit r7
error: unsupported configuration: unsupported CPU cache level for mode 'disable'
Failed. Try again? [y,n,i,f,?]:

Test with libvirt-3.2.0-4.el7.x86_64 and a qemu-kvm which version < 2.9.0 (qemu-kvm-rhev-2.6.0-28.el7_3.9.x86_64). Test with the same steps in comment 10, the only different is there is no "l3-cache=off" and "host-cache-info=off" in qemu command line.

Test with libvirt-3.2.0-4.el7.x86_64 and a qemu-kvm unsupported l3-cache and host-cache-info:

1. l3-cache

# virsh start r7
error: Failed to start domain r7
error: internal error: qemu unexpectedly closed the monitor: 2017-05-09T05:57:57.519970Z qemu-kvm: unrecognized feature l3-cache

2. host-cache-info

# virsh start r7
error: Failed to start domain r7
error: internal error: qemu unexpectedly closed the monitor: 2017-05-09T05:58:32.526720Z qemu-kvm: unrecognized feature host-cache-info

3. disable

# virsh start r7
Domain r7 started

Verify this bug with the test result in comment 10 and comment 11.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846