Description of problem: Cannot run any ppc64le VM on IBM Power9 AC922 hardware with default RHV 4.4 cluster and settings. When a plain VM is created RHV refuses to boot the machine with the following error: VM ppc64le-dc44 is down with error. Exit message: internal error: qemu unexpectedly closed the monitor: 2020-08-28T18:22:15.390930Z qemu-kvm: -numa node,nodeid=0,cpus=0-15,mem=4096: warning: Parameter -numa node,mem is deprecated, use -numa node,memdev instead 2020-08-28T18:22:15.412920Z qemu-kvm: Requested count cache flush assist capability level not supported by kvm, try appending -machine cap-ccf-assist=off. This issue was present on RHV 4.3 too, but on that time an workaround that I've used to do was run the machine with custom Emulated Machine to pseries-rhel7.6.0 (without SXXM). But now on RHV 4.4 there's no options without SXXM, and even on 4.3 the default was to run with SXXM and not without. pseries-rhel8.2.0 is the default now, and I cannot start any VM with this emulated machine. If I fallback from pseries-rhel8.2.0 to pseries-rhel7.6.0, I can boot the VM, on pseries-rhel7.6.0-sxxm I cannot, with the same error as pseries-rhel8.2.0. The hardware is an standard IBM AC922 POWER9 machine, with the following firmware: FIRMWARE VERSION IBM-witherspoon-ibm-OP9-v2.4-4.49-prod FIRMWARE VERSION op940.10-5-0-g22edca685 HARDWARE REVISION cpu : POWER9, altivec supported clock : 3683.000000MHz revision : 2.2 (pvr 004e 1202) timebase : 512000000 platform : PowerNV model : 8335-GTH machine : PowerNV 8335-GTH firmware : OPAL MMU : Radix I've already contacted IBM to get the complete updates and feature sets to the hardware to remove the surface of the problem, there's nothing else that I can do on hardware side. PS: I was seeking help on the oVirt Mailing List, because I tried it with oVirt too. Same issue. Some folks are helping out there but we didn't find any solution either. Here's the thread: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FNNAUKOD2XFU2KRLHO7VV3PNJL5AZ3A4/ Version-Release number of selected component (if applicable): RHV 4.4.1 fully updated. How reproducible: 100% Steps to Reproduce: 1. Create a ppc64le VM 2. Try to launch it and fail due to missing capabilities on kvm Actual results: Not able to run any VM with default settings. Expected results: Be able to proper launch a VM. Additional info: *** I don't have Red Hat support on RHV. We are on the technology partner program since we have an opensource project, and we are trying to port it to ppc64le architecture. In theory this machine came with a RHEL subscription, but we are using RHV from the tecnhology partner program, I'm reporting this to enhance the quality of the product, and I'm not expecting SLA or things like that from Red Hat ***
Polina, can you please check what we get in our PPC environment?
The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.
I can reproduce the problem also on POWER8. When I add <features><ccf-assist state='off'/></features> to the domain XML, the VM starts.
Well, it works for me after upgrading the firmware. However, the machine in Comment 0 has an up-to-date firmware. Unless nested (or generally non-hvm) virtualization is used, which is apparently not the case, it should work. So it looks like a platform bug; we may want to check whether it could be something that has been fixed in RHEL/AV 8.3.
I'm going to file a QEMU bug. Vinícius, what's the qemu-kvm version on your host? And can you confirm that you run your VMs on bare metal (no nested virtualization)?
Hi all. I'm not running nested virtualisation, it's bare metal, and here is anything with qemu on the machine: [root@rhvpower ~]# rpm -qa | grep -i qemu qemu-kvm-common-4.2.0-29.module+el8.2.1+7712+3c3fe332.2.ppc64le qemu-kvm-block-ssh-4.2.0-29.module+el8.2.1+7712+3c3fe332.2.ppc64le qemu-kvm-4.2.0-29.module+el8.2.1+7712+3c3fe332.2.ppc64le ipxe-roms-qemu-20181214-5.git133f4c47.el8.noarch qemu-img-4.2.0-29.module+el8.2.1+7712+3c3fe332.2.ppc64le qemu-kvm-block-iscsi-4.2.0-29.module+el8.2.1+7712+3c3fe332.2.ppc64le libvirt-daemon-driver-qemu-6.0.0-25.2.module+el8.2.1+7722+a9e38cf3.ppc64le qemu-kvm-block-curl-4.2.0-29.module+el8.2.1+7712+3c3fe332.2.ppc64le qemu-kvm-core-4.2.0-29.module+el8.2.1+7712+3c3fe332.2.ppc64le qemu-kvm-block-rbd-4.2.0-29.module+el8.2.1+7712+3c3fe332.2.ppc64le Just for completeness the engine runs on x86_64, and it is self-hosted if this means anything to help out. If there's need for sosreport or something like this please let me know.
Thank you for the information, QEMU bug created: https://bugzilla.redhat.com/1886803
Hi Vinícius, could you please check Risk Level setting of your firmware, as described by David in https://bugzilla.redhat.com/show_bug.cgi?id=1886803#c1 ? (You can respond to that comment directly.)
Hi Milan, I've answered there already. There's other questions, I'll respond now. For whatever reasons, I'm not receiving messages from Bugzilla anymore. Since it went down a couple days ago.
Hi Vinícius, yes, thank you. I watch both the bugs and receive e-mail notifications from Bugzilla normally, no idea why they don't work for you.
As discussed in BZ 1886803, the reported problem disappeared after switching to Risk Level 0 in the firmware. It's probably worth to have it documented some way, so moving this bug to documentation. In summary, to be able to run a VM with pseries-rhel8* emulated machine on POWER 9, the host firmware must be up to date and the firmware risk level (Spectre mitigations etc.) must be set to 0. See BZ 1886803 for details.
Didn't get in to 4.4.4, moving to 4.4.5
What exactly is the request for documentation?
Hi Steve, please get in touch with David Gibson who should know what would be good to document regarding firmware configuration on POWER to avoid user confusion -- see https://bugzilla.redhat.com/show_bug.cgi?id=1886803#c9 .
Closing documentation bug. This content is best suited for a KCS article.
What is a KCS article?
(In reply to Vinícius Ferrão from comment #20) > What is a KCS article? It was documented here: https://access.redhat.com/solutions/6042541