Description of problem: l3 cache not seen on vm even though cpu_mode=host-passthrough is set. https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/9/html/configuration_reference_guide/ch_configuring-openstack-compute#kvm [jmelvin@collab-shell 01814317]$ grep log_opt sosreport-20170320-234709/overcloud-compute-2.localdomain/var/log/nova/nova-compute.log |grep cpu_mode 2017-03-20 23:23:53.282 533020 DEBUG oslo_service.service [req-56209127-876e-44ba-818a-1e61af07b281 - - - - -] libvirt.cpu_mode = host-passthrough log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2630 ###Not working Guest VM lscpu: [root@vsbcSystem-vsbc1 ~]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Stepping: 2 CPU MHz: 2596.992 BogoMIPS: 5193.98 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 4096K NUMA node0 CPU(s): 0-3 [root@vsbcSystem-vsbc1 ~]# ###Not working Compute Host lscpu: [root@overcloud-compute-2 ~]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 48 On-line CPU(s) list: 0-47 Thread(s) per core: 2 Core(s) per socket: 12 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz Stepping: 2 CPU MHz: 2600.000 BogoMIPS: 5208.41 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 30720K NUMA node0 CPU(s): 0-11,24-35 NUMA node1 CPU(s): 12-23,36-47 [root@overcloud-compute-2 ~]# Version-Release number of selected component (if applicable): qemu-kvm-rhev-2.6.0-27.el7.x86_64 libvirt-2.0.0-10.el7_3.2.x86_64 How reproducible: 100% Steps to Reproduce: 1. enable cpu_mode=host-passthrough in /etc/nova.conf on compute node 2. notice vms lanuch do not show l3 cache. 3. Actual results: no l3 cache on vm Expected results: l3 cache on vms
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.
(In reply to Jeremy from comment #0) > Description of problem: > l3 cache not seen on vm even though cpu_mode=host-passthrough is set. > > https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/9/ > html/configuration_reference_guide/ch_configuring-openstack-compute#kvm > ... > Version-Release number of selected component (if applicable): > qemu-kvm-rhev-2.6.0-27.el7.x86_64 This version of qemu-kvm-rhev did not advertise a virtual L3 cache to the guest. Bug 1430802 added it to qemu-kvm-rhev-2.6.0-28.el7_3.8. Is it possible for the customer to try that version ? > libvirt-2.0.0-10.el7_3.2.x86_64 Also see the related libvirt bug 1428952
Meanwhile, here is the in-progress upstream libvirt design discussion for this attribute https://www.redhat.com/archives/libvir-list/2017-April/msg00815.html [RFC] Add support for QEMU's l3-cache CPU property
I have just tested in a RHEL 7.3 environment. I can confirm that using the libvirt's <qemu:commandline> [...] </qemu:commandline> namespace[#] to supply the "-cpu host,l3-cache=on" command-line parameter results in "L3 cache" being visible in the guest. [#] https://libvirt.org/drvqemu.html#qemucommand -- this useful for testing QEMU features that don't have official libvirt interface(s) yet. NOTE: this is *unsupported* for production use. * * * Before * * * ~~~~~~~~~~~~~~~~~~ Running `lscpu` in a RHEL 7.3 guest with qemu-kvm-rhev-2.6.0-28.el7_3.9.x86_64, by default, there's no "L3 cache" attribute: ----------------------------------------------------------------------- $ virsh console l2-rhel7vm [...] [root@l2-rhel7vm ~]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 1 On-line CPU(s) list: 0 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 60 Model name: Intel Core Processor (Haswell, no TSX) Stepping: 1 CPU MHz: 2294.630 BogoMIPS: 4589.26 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 4096K NUMA node0 CPU(s): 0 ----------------------------------------------------------------------- * * * After * * * ~~~~~~~~~~~~~~~~~ Use the libvirt's (unsupported for production) <qemu:commandline> [...] </qemu:commandline> namespace to supply the "-cpu host,l3-cache=on" command-line parameter to the guest. (1) Edit the guest XML: $ virsh edit l2-rhel7vm (2) Declare the XML namespace: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> Then add the below lines at the end of the XML: <qemu:commandline> <qemu:arg value='-cpu'/> <qemu:arg value='host,l3-cache=on'/> </qemu:commandline> (3) Start the guest: $ virsh start l2-rhel7vm (4) Log in and re-run `lscpu`, to observe the 16384K "L3 cache" attribute of size 16384K. ----------------------------------------------------------------------- $ virsh console l2-rhel7vm [...] [root@el7 images]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 1 On-line CPU(s) list: 0 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 60 Model name: Intel(R) Core(TM) i5-4670T CPU @ 2.30GHz Stepping: 3 CPU MHz: 2294.694 BogoMIPS: 4589.38 Virtualization: VT-x Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 4096K L3 cache: 16384K NUMA node0 CPU(s): 0 -----------------------------------------------------------------------
(In reply to Kashyap Chamarthy from comment #35) [...] > [...] > [root@el7 images]# lscpu > Architecture: x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 1 > On-line CPU(s) list: 0 > Thread(s) per core: 1 > Core(s) per socket: 1 > Socket(s): 1 > NUMA node(s): 1 > Vendor ID: GenuineIntel > CPU family: 6 > Model: 60 > Model name: Intel(R) Core(TM) i5-4670T CPU @ 2.30GHz > Stepping: 3 > CPU MHz: 2294.694 > BogoMIPS: 4589.38 > Virtualization: VT-x > Hypervisor vendor: KVM > Virtualization type: full > L1d cache: 32K > L1i cache: 32K > L2 cache: 4096K > L3 cache: 16384K > NUMA node0 CPU(s): 0 > ----------------------------------------------------------------------- In step (4) above, I inadvertently posted the output of `lscpu` from the *host* (the eagle-eyed will notice from the shell prompt that the name of the host is "el7", which is different from the guest, "l2-rhel7vm", as you see below). Here's output of `lscpu` from the *guest* (which is identical to the host): ----------------------------------------------------------------------- $ virsh console l2-rhel7vm [...] [root@l2-rhel7vm ~]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 1 On-line CPU(s) list: 0 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 60 Model name: Intel(R) Core(TM) i5-4670T CPU @ 2.30GHz Stepping: 3 CPU MHz: 2294.630 BogoMIPS: 4589.26 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 4096K L3 cache: 16384K NUMA node0 CPU(s): 0 -----------------------------------------------------------------------
Is host passthrough going to be required when this is fixed completely (ie OSP, libvirt, qemu)? Or will we always need host passthrough to expose the L3 cache?
(In reply to Benjamin Schmaus from comment #37) > Is host passthrough going to be required when this is fixed completely (ie > OSP, libvirt, qemu)? Or will we always need host passthrough to expose the > L3 cache? Nope, host passthrough isn't required for the virtual L3 cache.
(In reply to Benjamin Schmaus from comment #37) > Is host passthrough going to be required when this is fixed completely (ie > OSP, libvirt, qemu)? Or will we always need host passthrough to expose the > L3 cache? As Bandan noted in comment#38, it's not required. I should have used a different example, like below. If the libvirt guest XML snippet is as below for CPU model: [...] <cpu mode='custom' match='exact'> <model>Haswell-noTSX</model> <cache level='3' mode='emulate'/> </cpu> [...] That would be translated to QEMU command-line: [...] -cpu Haswell-noTSX,l3-cache=on [...]
Created attachment 1275962 [details] Instructions to enable the Nova (workaround) config attribute 'l3_cache'
Created attachment 1281136 [details] Test notes for setting up RHEL 7.2 QEMU machine type on the Compute nodes running on RHEL 7.3 If we want to go with the option of recommending them to use RHEL 7.2 machine type (which needs also the CPU mode to be 'host-passthrough' for L3 cache to be exposed to the guest) on the Nova Compute nodes, there are two main test cases to be performed: (1) Boot a Nova instance, and verify that L3 cache is exposed in the instance. (NB: The size of the L3 cache here is whatever the host has; and _not_ a fixed size of 16 MB.) (2) Live migrate this Nova instance (running on RHEL 7.3 host) to a RHEL 7.4 host. Upon completion of migration, ensure that L3 cache is still exposed to the instance (verify by running `lscpu` *inside* the guest) running on the RHEl 7.4 host. And the CPU L3 cache size inside the guest should be 16 MB.)
I just finished some testing with configuring the Nova instances (running on RHEL 7.3) with RHEL 7.2 QEMU machine type, and here are the results. Setup ----- On the source Compute node, which is of RHEL 7.3, set these config attributes in /etc/nova/nova.conf [libvirt] hw_machine_type=x86_64=pc-i440fx-rhel7.2.0 cpu_mode=host-passthrough And restart the Nova Compute service on the Compute node: $ systemctl restart openstack-nova-compute Prepare a destination Compute node with RHEL 7.4 (pre-release bits), as a target host. Test results ------------ The following are the four important scenarios that were tested. .==============================================================. | | | | L3 cache | | Guest state | Host | Machine Type | value in | | | | | the guest | '==============================================================' | Pre-migration, | | RHEL 7.2 | Inherited | | boot a guest | RHEL 7.3 |(pc-i440fx-rhel7.2.0)| from | | | | | host cache | .--------------------------------------------------------------. | Immediately | | RHEL 7.2 | Inherited | | *after* | RHEL 7.4 |(pc-i440fx-rhel7.2.0)| from | | migration | | | host cache | .--------------------------------------------------------------. | Reboot | | RHEL 7.2 | Inherited | | the guest | RHEL 7.4 |(pc-i440fx-rhel7.2.0)| from | | post-migration | | | host cache | .--------------------------------------------------------------. | Start & stop | | RHEL 7.4 | 16 MB | | the guest | RHEL 7.4 | pc-i440fx-rhel7.4.0 | (16384K | | post-migration | | | | '--------------------------------------------------------------' Versions -------- NOTE: It is important use libvirt version libvirt-3.2.0-6.el7.x86_64 on the target host (RHEL 7.4), to avoid hitting this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1449346#c10 Baremetal (RHEL 7.4): $ uname -r; rpm -q libvirt qemu-kvm-rhev 3.10.0-514.el7.x86_64 libvirt-3.2.0-6.el7.x86_64 qemu-kvm-rhev-2.9.0-6.el7.x86_64 Destination Compute node (RHEL 7.4): $ uname -r; rpm -q libvirt qemu-kvm-rhev 3.10.0-663.el7.x86_64 libvirt-3.2.0-6.el7.x86_64 qemu-kvm-rhev-2.9.0-6.el7.x86_64 Source Compute node (RHEL 7.3): $ uname -r; rpm -q libvirt qemu-kvm-rhev 3.10.0-514.10.2.el7.x86_64 libvirt-2.0.0-10.el7_3.5.x86_64 qemu-kvm-rhev-2.6.0-28.el7_3.6.x86_64
(In reply to Kashyap Chamarthy from comment #47) [...] > | Reboot | | RHEL 7.2 | Inherited | > | the guest | RHEL 7.4 |(pc-i440fx-rhel7.2.0)| from | > | post-migration | | | host cache | > .--------------------------------------------------------------. > | Start & stop | | RHEL 7.4 | 16 MB | Small correction: Obviously, the order above must be: *Stop*, followed by start. [...]
(In reply to Kashyap Chamarthy from comment #47) > I just finished some testing with configuring the Nova instances (running on > RHEL 7.3) with RHEL 7.2 QEMU machine type, and here are the results. > > Setup > ----- > > On the source Compute node, which is of RHEL 7.3, set these config > attributes in /etc/nova/nova.conf > > [libvirt] > hw_machine_type=x86_64=pc-i440fx-rhel7.2.0 > cpu_mode=host-passthrough For future readers, please also be aware that hw:machine_type can instead be set in the guest's image properties rather than for the whole compute node.
Closing this bug as WORKSFORME (seems to be the most reasonable from the list, for this scenario), based on comments #47, #48, and #49, where customer went with adjusting machine types appropriately. The full procedure is noted here: https://bugzilla.redhat.com/attachment.cgi?id=1281136