This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1464913 - KVM PR does not work under PowerVM
KVM PR does not work under PowerVM
Status: ASSIGNED
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev (Show other bugs)
7.4-Alt
ppc64le Linux
low Severity high
: rc
: ---
Assigned To: Laurent Vivier
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-26 04:49 EDT by Pingfan Liu
Modified: 2017-10-18 02:28 EDT (History)
16 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Pingfan Liu 2017-06-26 04:49:46 EDT
Description of problem:
Using RHEL-ALT-7.4-20170622.n.0-Server-ppc64le as both host and guest. After anaconda finishes installing the guest, the guest can not boot up. 

Version-Release number of selected component (if applicable):
Host:
kernel-4.11.0-7.el7.ppc64le
qemu-kvm-common-rhev-2.9.0-12.el7.ppc64le
qemu-img-rhev-2.9.0-12.el7.ppc64le
qemu-kvm-rhev-2.9.0-12.el7.ppc64le
SLOF-20170303-4.git66d250e.el7.noarch
seavgabios-bin-1.10.2-3.el7.noarch

Guest:
RHEL-ALT-7.4-20170622.n.0-Server-ppc64le-dvd1.iso


How reproducible:
100%

Steps to Reproduce:
1.
2.
3.

Actual results:

After booting, the guest hangs at Booting Linux via __start() @ 0x0000000002000000 ...

Expected results:

Booting up.

Additional info:
Comment 2 Pingfan Liu 2017-06-26 05:00:04 EDT
As a guest OS, the RHEL-ALT-7.4-20170622.n.0-Server-ppc64le works well on a beaker machine ibm-p8-kvm-10-guest-08.lab4.eng.bos.redhat.com. But with the following qemu cmdline, it can not work on the above environment.
The same qemu cmdline can work well when using Fedora-Server-dvd-ppc64le-26-20170622.n.0.iso as both host and guest.

qemu cmdline
-------------------
/usr/libexec/qemu-kvm -name avocado-vt-vm1 -enable-kvm -sandbox off -nodefaults -machine pseries -m 4G -smp 4 -device virtio-serial-pci,id=virtio_serial_pci0,bus=pci.0,addr=03 -device virtio-scsi-pci,id=scsi1,bus=pci.0,addr=0x4 -drive file=./root.img,format=raw,if=none,id=blk1 -device scsi-hd,bus=scsi1.0,drive=blk1,id=blk-disk1,bootindex=1 -device nec-usb-xhci,id=xhci,bus=pci.0 -drive id=drive_cd1,if=none,aio=native,cache=none,media=cdrom,file=/build/Fedora-Server-dvd-ppc64le-26-20170625.n.0.iso -device scsi-cd,id=cd1,drive=drive_cd1,bootindex=2 -boot order=cdn,once=c,menu=on,strict=off -device usb-kbd -device usb-tablet -msg timestamp=on -rtc base=localtime,clock=vm,driftfix=slew -device spapr-vty,chardev=charserial0,reg=0x30000000 -chardev stdio,mux=on,id=charserial0 -net nic,model=virtio,macaddr=1a:46:0b:ca:bc:7b -net tap,fd=3 -net nic,macaddr=52:54:00:12:34:56,model=virtio -net tap,ifname=tap1,script=no -vga none -nographic


SLOF message
--------
SLOF **********************************************************************
QEMU Starting
 Build Date = May 19 2017 07:43:48
 FW Version = mockbuild@ release 20170303
 Press "s" to enter Open Firmware.

Press F12 for boot menu.

Populating /vdevice methods
Populating /vdevice/vty@30000000
Populating /vdevice/nvram@71000000
Populating /pci@800000020000000
                     00 0000 (D) : 1af4 1000    virtio [ net ]
                     00 0800 (D) : 1af4 1000    virtio [ net ]
                     00 1000 (D) : 1033 0194    serial bus [ usb-xhci ]
                     00 1800 (D) : 1af4 1003    virtio [ serial ]
                     00 2000 (D) : 1af4 1004    virtio [ scsi ]
Populating /pci@800000020000000/scsi@4
       SCSI: Looking for devices
          100000000000000 DISK     : "QEMU     QEMU HARDDISK    2.5+"
          101000000000000 CD-ROM   : "QEMU     QEMU CD-ROM      2.5+"
No NVRAM common partition, re-initializing...
Scanning USB 
  XHCI: Initializing
    USB Keyboard 
Using default console: /vdevice/vty@30000000
     


  Welcome to Open Firmware

  Copyright (c) 2004, 2017 IBM Corporation All rights reserved.
  This program and the accompanying materials are made available
  under the terms of the BSD License available at
  http://www.opensource.org/licenses/bsd-license.php



kernel message
------
OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 4.11.0-9.el7.ppc64le (mockbuild@ppc-048.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Fri Jun 16 14:43:03 EDT 2017
Detected machine type: 0000000000000101
command line: BOOT_IMAGE=/vmlinuz-4.11.0-9.el7.ppc64le root=/dev/mapper/rhelaa-root ro crashkernel=auto rd.lvm.lv=rhelaa/root rd.lvm.lv=rhelaa/swap LANG=en_US.UTF-8
Max number of cores passed to firmware: 2048 (NR_CPUS = 2048)
Calling ibm,client-architecture-support...2017-06-26T08:39:11.010367Z qemu-kvm: Unable to set CPU compatibility mode in KVM: Invalid argument

WARNING: ibm,client-architecture-support call FAILED!
 done
memory layout at init:
  memory_limit : 0000000000000000 (16 MB aligned)
  alloc_bottom : 0000000004cf0000
  alloc_top    : 0000000030000000
  alloc_top_hi : 0000000100000000
  rmo_top      : 0000000030000000
  ram_top      : 0000000100000000
instantiating rtas at 0x000000002fff0000... done
prom_hold_cpus: skipped
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings 0x0000000004d00000 -> 0x0000000004d00b0b
Device tree struct  0x0000000004d10000 -> 0x0000000004d20000
Quiescing Open Firmware ...
Booting Linux via __start() @ 0x0000000002000000 ...
Comment 3 Pingfan Liu 2017-06-27 03:33:57 EDT
The host which I produced this bug is LPAR(ibm-p8-alpine-01-lp8.lab4.eng.bos.redhat.com), not a powerNV(not tested).

Regards,
Pingfan
Comment 4 Laurent Vivier 2017-06-27 04:41:56 EDT
(In reply to Pingfan Liu from comment #3)
> The host which I produced this bug is
> LPAR(ibm-p8-alpine-01-lp8.lab4.eng.bos.redhat.com), not a powerNV(not
> tested).

So I guess you are using kvm_pr, not kvm_hv.

Could you check that with: lsmod|grep kvm

Thanks,
Laurent
Comment 5 Pingfan Liu 2017-06-28 01:38:33 EDT
Yes, it is kvm_pr.

Regards,
Pingfan
Comment 6 Laurent Vivier 2017-06-28 04:26:22 EDT
Could you execute this test case on a machine with KVM HV (on a not LPAR machine)?
Red Hat does not support KVM PR on RHEL.
Comment 7 David Gibson 2017-06-28 04:48:43 EDT
Moving to rhel-7.5-alt? (pegas1.1).  This is way too late for rhel 7.4, and we don't support KVM PR there anyway.  And the bug was reported on Pegas, not mainline RHEL.  This is not a priority for pegas 1.0 (PR is useful for OpenStack testing, but not supported for general use).

Althugh KVM PR can work nested under KVM KV, I believe KVM PR won't quite work under PowerVM [L2 guest hypercalls will always be directed to the L0 host.  A KVM HV L0 host will redirect hypercalls from L1 userspace to the L1 kernel, where KVM PR can process them, but PowerVM will simply fail then IIRC].  So you're going to need to find a PowerNV host to do this.
Comment 8 Pingfan Liu 2017-06-28 21:47:42 EDT
Hi David and Laurent,

On powerNV, the latest pegas can work. If redhat does not support nested KVM PR, then please feel free to close the bug. But from a technique view, the fedora26 can work fine with nested KVM PR, it does not require L0 to be powerNV.


Best regards,
Pingfan
Comment 9 David Gibson 2017-06-30 00:06:43 EDT
Hm, ok.  I'm not quite sure how F26 is doing this.  My guess is it needs some L2 guest support (using an alternative hypercall mechanism which the L1 host can intercept).

It would be good to look into that so we can do PR on PowerVM, but given that's not a supported configuration, it's obviously not going to be something we have time for for a while.
Comment 10 Thomas Huth 2017-10-09 08:03:02 EDT
David,(In reply to David Gibson from comment #9)
> It would be good to look into that so we can do PR on PowerVM [...]

By the way, according to comment #2, the problem also occurs with KVM running as bare-metal hypervisor, so this issue seems to be related to KVM-PR in general, not only to KVM-PR running under PowerVM.

Also looking at the error message in comment #2 ("Calling ibm,client-architecture-support...2017-06-26T08:39:11.010367Z qemu-kvm: Unable to set CPU compatibility mode in KVM: Invalid argument" and "WARNING: ibm,client-architecture-support call FAILED!"), it looks like we at least are missing the following patch to run a guest here successfully:

https://git.qemu.org/?p=qemu.git;a=commitdiff;h=cc7b35b169e96600c09947a31c610c84a3eda3ff
("spapr: fallback to raw mode if best compat mode cannot be set during CAS")
Comment 11 David Gibson 2017-10-09 20:52:16 EDT
Ah, comment 2 seems ambiguous to me, I read it differently.

But, yes, that error does strongly suggest that missing patch is the problem.  Good catch.

I'm brewing a test qemu with that patch at:

https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=14227692

Pingfan, can you test that and see if it addresses the problem?
Comment 12 Pingfan Liu 2017-10-17 01:27:06 EDT
Hi David,

Sorry to reply late. I tried the test with the new qemu rpm, but it did not work.

The L2 guest hangs as:
OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 3.10.0-693.el7.ppc64le (mockbuild@ppc-058.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Thu Jul 6 19:59:44 EDT 2017
Detected machine type: 0000000000000101
Max number of cores passed to firmware: 2048 (NR_CPUS = 2048)
Calling ibm,client-architecture-support... done
command line: BOOT_IMAGE=/vmlinuz-3.10.0-693.el7.ppc64le root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap LANG=en_US.UTF-8
memory layout at init:
  memory_limit : 0000000000000000 (16 MB aligned)
  alloc_bottom : 0000000004bd0000
  alloc_top    : 0000000030000000
  alloc_top_hi : 0000000100000000
  rmo_top      : 0000000030000000
  ram_top      : 0000000100000000
instantiating rtas at 0x000000002fff0000...

Additional info:
L1 guest is installed from RHEL-7.4-20170711.0-Server-ppc64le-dvd1, with the qemu package update.

Thanks and regards,
Pingfan
Comment 13 David Gibson 2017-10-17 23:39:55 EDT
Drat.  That's rather harder to debug.

Does the same L1/L2 combination work if the L0 is a RHEL/KVM system rather than PowerVM?
Comment 14 Pingfan Liu 2017-10-18 02:28:23 EDT
In my test, L0/L1/L2 all are rhel7, both L0 and L1 installed with new qemu package. The L2 failed to bootup.
But if using fedora, it succeeded to boot up L2 guest.

Thanks and regards,
Pingfan

Note You need to log in before you can comment on or make changes to this bug.