Bug 1525599

Initial patches posted upstream. They don't cover the entire feature.

  https://www.redhat.com/archives/libvir-list/2018-January/msg00779.html

David, which of the optional capabilities do you think should be
exposed through libvirt? The ones I'm aware of are

  cap-vsx=bool (Allow Vector Scalar Extensions (VSX))
  cap-htm=bool (Allow Hardware Transactional Memory (HTM))
  cap-dfp=bool (Allow Decimal Floating Point (DFP))
  cap-sbbc=string (Speculation Barrier Bounds Checking (broken, workaround, fixed))
  cap-cfpc=string (Cache Flush on Privilege Change (broken, workaround, fixed))
  cap-ibs=string (Indirect Branch Serialisation (broken, workaround, fixed))

HTM definitely needs to be in, as it's going to be disabled by
default and people might reasonably want to enable it.

SBBC, CFPC and IBS should probably also be exposed? I'm actually
not clear on how they would be used in practice.

VSX and DFP look like they could be skipped, since IIUC they're
going to be enabled by default and there aren't any compelling
reasons to ever disable them.

I concur.  I guess you want HTM so it can be enabled (at least until we get HTM-without-suspend sorted out, which could be a while).

You need SBBC, CFPC and IBS for taking explicit control of spectre/workaround mitigation.  The only likely use case I see is that if you definitely want the mitigations in your VM you should set them all to "workaround", as well as update your guest kernel.

Second iteration posted upstream.

  https://www.redhat.com/archives/libvir-list/2018-February/msg00310.html

First non-RFC version posted upstream. Only support for the HTM
feature is implemented.

  https://www.redhat.com/archives/libvir-list/2018-March/msg00474.html

I've discussed this with Andrea, and the current scope of this BZ is pretty broad: there are a bunch of capabilities flags that have been added to pseries on the qemu side, and we probably don't need support for all (or even most) of them in libvirt.

Refocussing this bug on the HTM capability.  We're not sure if we'll need that one in libvirt either, but it's the most likely.

Hi Andrea,

I talked about the HTM emulation stuff to Suraj today.  The kernel emulation is now merged upstream, and Suraj is going to do a backport shortly (see bug 1517546).

We still have some decisions to make about how to handle this at the higher levels, though.

The HTM capability is turned off in qemu by default for the released 2.12 upstream and rhel7.5.0 downstream machine types.  So, I think we basically have two options:

A)
    * Change qemu default back to cap_htm=on the 3.0 and rhel7.6.0 machine types
    * No support for setting the flag for libvirt
    * The new machine type won't work on POWER9 unless you have a newer kernel with the emulation code in (this is already of the <=2.11/rhel7.4.0 machine types)
        * To run with POWER9 on an older kernel you'd need to either explicitly set the qemu option or use the 2.12 / rhel7.5.0 machine type.

B)
    * Leave the qemu default as it is - htm available by default only on <=2.11/rhel7.4.0 machine types
    * libvirt adds support for the cap_htm flag and decides on some policy for setting it


Opinions?

(In reply to David Gibson from comment #13)
> Hi Andrea,
> 
> I talked about the HTM emulation stuff to Suraj today.  The kernel emulation
> is now merged upstream, and Suraj is going to do a backport shortly (see bug
> 1517546).
> 
> We still have some decisions to make about how to handle this at the higher
> levels, though.
> 
> The HTM capability is turned off in qemu by default for the released 2.12
> upstream and rhel7.5.0 downstream machine types.  So, I think we basically
> have two options:
> 
> A)
>     * Change qemu default back to cap_htm=on the 3.0 and rhel7.6.0 machine
> types
>     * No support for setting the flag for libvirt
>     * The new machine type won't work on POWER9 unless you have a newer
> kernel with the emulation code in (this is already of the <=2.11/rhel7.4.0
> machine types)
>         * To run with POWER9 on an older kernel you'd need to either
> explicitly set the qemu option or use the 2.12 / rhel7.5.0 machine type.

I think it's fair to expect POWER9 machines to run a recent kernel:
it's recent hardware after all :)

I also expect that, if someone tried to migrate a guest with HTM
enabled to a POWER9 hosts that is running an older kernel, they
would get back a reasonable error message, since that's a situation
you can already find yourself in by trying to migrate a RHEL 7.4
guest to a RHEL 7.5 POWER9 host.

Overall, choosing this path would keep things slightly awkward in
the short term, with the corresponding workaround mirroring the
one we already provided and documented for RHEL 7.5 (use a
different machine type), but make sure they Just Work™ in the
longer run.

> B)
>     * Leave the qemu default as it is - htm available by default only on
> <=2.11/rhel7.4.0 machine types
>     * libvirt adds support for the cap_htm flag and decides on some policy
> for setting it

libvirt only provides mechanisms, not policies, so it would be up
to the user or the management application to enable the correct
knob; in other words, going down this road would mean anyone
needing HTM will have to explicitly opt-in to it.

On the other hand, part of the reason we disabled HTM by default
in RHEL 7.5 is that we had a hunch not many people actually used
it in practice, and the lack of loud complaints so far seems to
validate the theory, so perhaps it's not that big of a deal if
opt-in is required? Plus it would avoid flip-flopping on the QEMU
default, which can certainly cause confusion.


At the end of the day, neither solution is completely free of
drawbacks. Perhaps the first one would be preferrable since it is
easier to understand: "HTM is enabled by default, except for RHEL
7.5", as opposed to "HTM is enabled by default until RHEL 7.4,
disabled by default with no way to turn it on in RHEL 7.5, and
disabled by default but can be turned on with a knob in RHEL 7.6
and forward".

(In reply to David Gibson from comment #13)
> The HTM capability is turned off in qemu by default for the released 2.12
> upstream and rhel7.5.0 downstream machine types.  So, I think we basically
> have two options:
> 
> A)
>     * Change qemu default back to cap_htm=on the 3.0 and rhel7.6.0 machine
> types
>     * No support for setting the flag for libvirt
>     * The new machine type won't work on POWER9 unless you have a newer
> kernel with the emulation code in (this is already of the <=2.11/rhel7.4.0
> machine types)
>         * To run with POWER9 on an older kernel you'd need to either
> explicitly set the qemu option or use the 2.12 / rhel7.5.0 machine type.

IIUC from today's meeting emulated MT needs not only a recent
kernel, but also DD2.2 hardware with up-to-date firmware; given how
much pre-DD2.2 hardware is still around and actively used at least
for development, that would tip the scale towards leaving MT
disabled for the time being.

Yeah, that makes sense to me.  Especially since we don't seem to have heard any screams for HTM support.

This is the do-nothing option from the qemu perspective, what does it mean from the libvirt perspective?

(In reply to David Gibson from comment #17)
> Yeah, that makes sense to me.  Especially since we don't seem to have heard
> any screams for HTM support.
> 
> This is the do-nothing option from the qemu perspective, what does it mean
> from the libvirt perspective?

Implementing a knob that allows users to enable HTM :)

I have code for that that I posted as RFC a while ago, I'll brush
it up and post it in earnest this time around.

v2 series posted upstream.

  https://www.redhat.com/archives/libvir-list/2018-June/msg01374.html

v3 series posted upstream.

  https://www.redhat.com/archives/libvir-list/2018-June/msg01655.html

Patches merged upstream.

commit d4c11171076edfb2e603804e79edf7ccc3cce5dc
Author: Andrea Bolognani <abologna>
Date:   Mon Jul 2 10:37:09 2018 +0200

    qemu: Format the HTM pSeries feature
    
    This makes the feature fully operational.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1525599
    
    Signed-off-by: Andrea Bolognani <abologna>
    Reviewed-by: John Ferlan <jferlan>

v4.5.0-19-gd4c1117107

Package:
libvirt-4.5.0-6.el7.ppc64le
qemu-kvm-rhev-2.12.0-9.el7.ppc64le
kernel-4.14.0-101.el7a.ppc64le


Case 1: Set 'htm' feature to 'on' and 'off' respectively on ppc64le host and start guest  --- PASS
XML:
  <features>
    <htm state='on'/>
  </features>
Check qemu command line:
-machine pseries-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off,cap-htm=on

XML:
  <features>
    <htm state='off'/>
  </features>
Check qemu command line:
-machine pseries-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off,cap-htm=off

Case 2: Set 'htm' feature to 'on' on x86_64 host  --- PASS
# virsh edit guest
unsupported configuration: The 'htm' feature is not supported for architecture 'x86_64' or machine type 'pc-i440fx-rhel7.6.0'

Case 3: Set invalid syntax for htm feature        --- PASS
Use 'htmm' 'offf' 'sstate' to modify the guest xml for <htm state='on'/>, the invalid syntax is not allowed.

error: XML document failed to validate against schema: Unable to validate doc against /usr/share/libvirt/schemas/domain.rng
Extra element features in interleave
Element domain failed to validate content


Case 4: Migrate guest with htm enabled
Prepare nfs shared storage environment
Edit guest with   <features> <htm state='on'/> </features>
Start guest and do migration
# virsh migrate --live --p2p guest qemu+ssh://<target_host>/system --verbose

The migration should be successful
Check qemu command line on target host, which should include 'cap-htm=on'

For Case 4, the test result will be updated after we get two Power machines. @junli will help on it when I am on vacation.

Andrea, What other deep checking or test scenarios do you think as necessary?

(In reply to Dan Zheng from comment #25)
> Andrea, What other deep checking or test scenarios do you think as necessary?

Not really, once you have tested migration I'd say we're all set :)

Case 4 test pass with the enviroment:
# rpm -q kernel libvirt qemu-kvm-rhev
kernel-3.10.0-933.el7.ppc64le
libvirt-4.5.0-7.el7.ppc64le
qemu-kvm-rhev-2.12.0-11.el7.ppc64le

Based on comment 24 ~ 27, I mark it verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3113