Bug 1570605 - [RHOS-14] Lift the restriction on choices for `cpu_model_extra_flags` config attribute
Summary: [RHOS-14] Lift the restriction on choices for `cpu_model_extra_flags` config ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: Upstream M2
: 14.0 (Rocky)
Assignee: Kashyap Chamarthy
QA Contact: nova-maint
URL:
Whiteboard:
Depends On:
Blocks: 1577088
TreeView+ depends on / blocked
 
Reported: 2018-04-23 10:27 UTC by Kashyap Chamarthy
Modified: 2019-09-09 16:42 UTC (History)
10 users (show)

Fixed In Version: openstack-nova-18.0.0-0.20180710150340.8469fa7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1577088 1577093 1577101 1577133 (view as bug list)
Environment:
Last Closed: 2019-01-11 11:49:38 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1766208 0 None None None 2018-04-23 10:30:36 UTC
OpenStack gerrit 563926 0 'None' MERGED libvirt: Lift the restriction of choices for `cpu_model_extra_flags` 2020-08-26 06:57:08 UTC
Red Hat Product Errata RHEA-2019:0045 0 None None None 2019-01-11 11:49:58 UTC

Description Kashyap Chamarthy 2018-04-23 10:27:31 UTC
(This bug is a follow-up item to the:
https://bugzilla.redhat.com/show_bug.cgi?id=1398343 -- RFE: add ability
to configure extra CPU flags for named CPU models)

In the patch[*] that introduced the Nova configuration attribute
`[libvirt]/cpu_model_extra_flags`, we have restricted the choices to be
only 'PCID' -- to alleviate the immediate guest performance degradation
as a result of applying the "Meltdown" CVE fixes.

Now remove that restriction to allow adding and removing multiple CPU
flags, making way for other use cases.

Use cases:

  - Ability to use 1GB huge pages with Haswell model as one use case for
    extra flags:

        cpu_model=Haswell-noTSX-IBRS 
        cpu_model_extra_flags="pdpe1gb"

  - Nested Virtualization -- an operator can specify the Intel 'vmx' or
    AMD 'svm' flags in the level-1 Nova guest.

[*] http://git.openstack.org/cgit/openstack/nova/commit/?h=master&id=6b601b7
    -- libvirt: Allow to specify granular CPU feature flags


A potential example of specifying multiple CPU feature flags
------------------------------------------------------------

If you specify:

    [libvirt]
    cpu_mode=custom
    cpu_model=IvyBridge
    cpu_model_extra_flags="+pcid,-mtrr,pdpe1gb"

Then, Nova should generate the below XML:

    <cpu match='exact'> 
      <model fallback='forbid'>IvyBridge</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='pcid'/>
      <feature policy='require' name='pdpe1gb'/>
    </cpu>

The +/- for individual flags in nova.conf will be optional.  If nothing
is specified, assume '+'.

You might ask: "Why would you want to remove a CPU flag though?"  One
scenario for that is: An Operator wants to generate a baseline CPU
config.  And a certain CPU flag is causing performance issue or other
nuisance, and if the Operator isolated the problem to _that_ specific
CPU flag, then she may want to remove the flag.

Comment 3 Kashyap Chamarthy 2018-04-27 14:38:47 UTC
(In reply to Kashyap Chamarthy from comment #0)

[...]

> 
> A potential example of specifying multiple CPU feature flags
> ------------------------------------------------------------
> 
> If you specify:
> 
>     [libvirt]
>     cpu_mode=custom
>     cpu_model=IvyBridge
>     cpu_model_extra_flags="+pcid,-mtrr,pdpe1gb"
> 
> Then, Nova should generate the below XML:
> 
>     <cpu match='exact'> 
>       <model fallback='forbid'>IvyBridge</model>
>       <vendor>Intel</vendor>
>       <feature policy='require' name='pcid'/>
>       <feature policy='require' name='pdpe1gb'/>
>     </cpu>
> 
> The +/- for individual flags in nova.conf will be optional.  If nothing
> is specified, assume '+'.
> 
> You might ask: "Why would you want to remove a CPU flag though?"  One
> scenario for that is: An Operator wants to generate a baseline CPU
> config.  And a certain CPU flag is causing performance issue or other
> nuisance, and if the Operator isolated the problem to _that_ specific
> CPU flag, then she may want to remove the flag.

The existing patch only allows *adding* of CPU flags:

    https://review.openstack.org/#/c/563926/ -- libvirt: Lift the 
    restriction of choices for `cpu_model_extra_flags`

(Removal of CPU flags is to be added in a future patch.)

Comment 4 Kashyap Chamarthy 2018-05-09 14:07:15 UTC
The patch is now merged in upstream "Rocky":

commit cc27a2007f314df18812d54a087f6192d6aed3fb
Author: Kashyap Chamarthy <kchamart@redhat.com>
Date:   Sat Apr 28 20:04:36 2018 +0200

    libvirt: Lift the restriction of choices for `cpu_model_extra_flags`
    
    Commit 6b601b7 (libvirt: Allow to specify granular CPU feature flags)
    added support for allowing to specify individual CPU feature flags, but
    restricted the options only to "PCID" (refer to its commit message for
    why).
    
    In this change we lift the restriction of choices, and allow to specify
    multiple CPU feature flags for all three CPU modes for the libvirt
    driver: 'custom', 'host-model', and 'host-passthrough'.
    
    For example:
    
         [libvirt]
         cpu_mode = custom
         cpu_model = IvyBridge
         cpu_model_extra_flags = pcid, vmx, pdpe1gb
    
    This will allow additional use cases such as:
    
      - Ability to use 1GB huge pages with models that don't provide it
        (such as Intel "Haswell" variants) as one use case for extra flags:
    
            cpu_mode = custom
            cpu_model = Haswell-noTSX-IBRS
            cpu_model_extra_flags = pdpe1gb
    
      - Nested Virtualization -- an operator can specify the Intel 'vmx' (or
        AMD 'svm') flags for the level-1 Nova guest CPU models.  (Assuming
        the 'nested' flag is enabled on the level-0 / bare-metal kernel.)
    
    (A future Nova patch will also allow ability to remove CPU flags.)
    
    Change-Id: I9a862619f379057bb48cb85a84dfc50d763030a6
    Signed-off-by: Kashyap Chamarthy <kchamart@redhat.com>
    BluePrint: libvirt-cpu-model-extra-flags

Comment 5 Kashyap Chamarthy 2018-05-09 15:21:41 UTC
Test Evidence.

Environment: I tested this on Fedora in a DevStack environment with
nested KVM.  My Compute node is a level-1 guest, with
'host-passthrough' as its CPU mode.  And the level-2 guest, the Nova
instance has a custom model with specific CPU flags enabled.

Test
----

On the Compute node, in /etc/nova/nova.conf, configure a custom CPU
model:

    [libvirt]
    cpu_mode = custom
    cpu_model = Haswell-noTSX
    cpu_model_extra_flags = vmx, pdpe1gb
    virt_type = kvm
    [...]

Then start a CirrOS guest:

    [ComputeNode]$ openstack server create testvm1 \
        --image cirros-0.3.5-x86_64-disk --flavor 2

Ensure the guest is running

    [ComputeNode]$ openstack server list -f json
    [
      {
        "Status": "ACTIVE", 
        "Name": "testvm1", 
        "Image": "cirros-0.3.5-x86_64-disk", 
        "ID": "eb1ba291-93eb-4cde-b5e9-3386bfe5fa3e", 
        "Flavor": "m1.small", 
        "Networks": "private=fda0:77af:6ab0:0:f816:3eff:fe50:f6a5, 10.0.0.7"
      }

Then, ensure that the 'vmx' and 'pdpe1gb' CPU features are available in
the CPU configuration of the just-booted Nova instance
("instance-00000001"):

    $ sudo virsh dumpxml instance-00000001 | grep "cpu mode" -A12
      <cpu mode='custom' match='exact' check='full'>
        <model fallback='forbid'>Haswell-noTSX</model>
        <topology sockets='1' cores='1' threads='1'/>
        <feature policy='require' name='pdpe1gb'/>
        <feature policy='disable' name='vmx'/>
        [...]
      </cpu>

As a final confirmation, ensure that the 'vmx' and 'pdpe1gb' CPU
features show up as "-cpu Haswell-noTSX,pdpe1gb=on,vmx=on" as part of
the Nova instance's QEMU command-line:

[ComputeNode]$ ps -ef | grep qemu
qemu     17738     1 11 17:06 ?        00:00:03 /usr/bin/qemu-system-x86_64 -machine accel=kvm -name guest=instance-00000001,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-instance-00000001/master-key.aes -machine pc-i440fx-2.9,accel=kvm,usb=off,dump-guest-core=off -cpu Haswell-noTSX,pdpe1gb=on,vmx=on -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid eb1ba291-93eb-4cde-b5e9-3386bfe5fa3e -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=18.0.0,serial=5cfd0699-ad8c-be09-a3a8-a4446a0aff89,uuid=eb1ba291-93eb-4cde-b5e9-3386bfe5fa3e,family=Virtual Machine -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-instance-00000001/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/stack/src/cloud/data/nova/instances/eb1ba291-93eb-4cde-b5e9-3386bfe5fa3e/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=31,id=hostnet0,vhost=on,vhostfd=33 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:50:f6:a5,bus=pci.0,addr=0x3 -add-fd set=2,fd=35 -chardev pty,id=charserial0,logfile=/dev/fdset/2,logappend=on -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on

Comment 9 errata-xmlrpc 2019-01-11 11:49:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045


Note You need to log in before you can comment on or make changes to this bug.