Bug 1652845
Summary: | VM installation hangs with error "Fixing recursive fault but reboot is needed!" on F29 ppc64le box | ||||||
---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Sinny Kumari <skumari> | ||||
Component: | qemu | Assignee: | Fedora Virtualization Maintainers <virt-maint> | ||||
Status: | CLOSED WONTFIX | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||
Severity: | unspecified | Docs Contact: | |||||
Priority: | unspecified | ||||||
Version: | 29 | CC: | airlied, amit, berrange, bskeggs, cfergeau, dan, dgibson, dwmw2, ewk, hdegoede, ichavero, itamar, jarodwilson, jglisse, john.j5live, jonathan, josef, kernel-maint, linville, lvivier, mchehab, mjg59, normand, pbonzini, pmac, rjones, skumari, steved, surajjs, virt-maint | ||||
Target Milestone: | --- | ||||||
Target Release: | --- | ||||||
Hardware: | ppc64le | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2019-02-21 02:01:05 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Attachments: |
|
Description
Sinny Kumari
2018-11-23 09:15:40 UTC
Created attachment 1508237 [details]
virt-install log
Which version of qemu? Which host? for the record - works fine with F-28 ppc64le host and latest virt stack from https://copr.fedorainfracloud.org/coprs/g/virtmaint-sig/virt-preview/ (qemu-system-ppc-3.1.0-0.1.rc1.fc28.ppc64le) qemu version - 3.0.0-1.fc29 Kernel - 4.18.16-300.fc29.ppc64le I hope by host you meant kernel version on which virt-install is running I was also thinking about whether it's POWER 8/9/etc. (In reply to Richard W.M. Jones from comment #5) > I was also thinking about whether it's POWER 8/9/etc. It's Power 8 (In reply to Sinny Kumari from comment #0) ... > Connected to domain fah29-20181123 > Escape character is ^] > 26 > ERROR: Unhandled relocation (A) type 26 Looks like a SLOF bug fixed by: commit 031cb1b921694f0c8676e0b478a3dccbae9d1639 Author: Thomas Huth <thuth> Date: Wed Jun 13 17:58:16 2018 +0200 libelf: Add REL32 to the list of ignored relocations When compiling SLOF with GCC 8.1, I currently get a lot of these errors: ERROR: Unhandled relocation (A) type 26 Type 26 is the "relative 32-bit" relocation. We can simply ignore it (like the other relative relocations - REL14, REL24 and REL64) to shut up these error messages. Signed-off-by: Thomas Huth <thuth> Reviewed-by: Laurent Vivier <lvivier> Signed-off-by: Alexey Kardashevskiy <aik> (In reply to Laurent Vivier from comment #7) > (In reply to Sinny Kumari from comment #0) > ... > > Connected to domain fah29-20181123 > > Escape character is ^] > > 26 > > ERROR: Unhandled relocation (A) type 26 > > > Looks like a SLOF bug fixed by: > > commit 031cb1b921694f0c8676e0b478a3dccbae9d1639 > Author: Thomas Huth <thuth> > Date: Wed Jun 13 17:58:16 2018 +0200 > > libelf: Add REL32 to the list of ignored relocations But this should not fix the crash as this commit only mutes the error message. But perhaps you could try at least a more up-to-date SLOF. (In reply to Laurent Vivier from comment #8) > (In reply to Laurent Vivier from comment #7) > > (In reply to Sinny Kumari from comment #0) > > ... > > > Connected to domain fah29-20181123 > > > Escape character is ^] > > > 26 > > > ERROR: Unhandled relocation (A) type 26 > > > > > > Looks like a SLOF bug fixed by: > > > > commit 031cb1b921694f0c8676e0b478a3dccbae9d1639 > > Author: Thomas Huth <thuth> > > Date: Wed Jun 13 17:58:16 2018 +0200 > > > > libelf: Add REL32 to the list of ignored relocations > > But this should not fix the crash as this commit only mutes the error > message. > > But perhaps you could try at least a more up-to-date SLOF. Did a scratch build of SLOF from latest master branch (0198ba7) - https://koji.fedoraproject.org/koji/taskinfo?taskID=31121042 and installed it. It only suppresses relocation (A) type 26 error. VM creation still hangs at "[ 3.027284] Fixing recursive fault but reboot is needed!" (In reply to Dan Horák from comment #3) > for the record - works fine with F-28 ppc64le host and latest virt stack > from https://copr.fedorainfracloud.org/coprs/g/virtmaint-sig/virt-preview/ > (qemu-system-ppc-3.1.0-0.1.rc1.fc28.ppc64le) Using virt stack from copr repo doesn't help on F29 Can this be reproduced with an upstream kernel? The real problem seems to be: [ 0.021906] kernel tried to execute exec-protected page (c000000001600a44) -exploit attempt? (uid: 0) [ 0.021972] Unable to handle kernel paging request for instruction fetch [ 0.022018] Faulting instruction address: 0xc000000001600a44 and several similar faults following. No idea why that would be happening, though. I've not been able to reproduce the problem with the same ISO and the same command line. virt-install --name fah29-20181123 --ram 2048 --vcpus 1 --os-type=linux \ --os-variant=fedora26 \ --disk path=/var/lib/libvirt/images/f29_20181123.qcow2,size=6,bus=virtio,format=qcow2 \ --network bridge=virbr0 --nographic \ -c /tmp/Fedora-AtomicHost-ostree-ppc64le-29-20181123.0.iso qemu-system-ppc-core-3.0.0-0.2.rc3.fc29.ppc64le SLOF-0.1.git20180621-1.fc29.noarch kernel-4.19.4-300.fc29.ppc64le virt-install-1.6.0-0.3.git3bc7ff24c.fc29.noarch # lscpu Architecture: ppc64le Byte Order: Little Endian CPU(s): 88 On-line CPU(s) list: 0,8,16,24,32,48,56,64,72,80,88 Off-line CPU(s) list: 1-7,9-15,17-23,25-31,33-39,49-55,57-63,65-71,73-79,81-87,89-95 Thread(s) per core: 1 Core(s) per socket: 5 Socket(s): 2 NUMA node(s): 2 Model: 2.1 (pvr 004b 0201) Model name: POWER8E (raw), altivec supported CPU max MHz: 3325.0000 CPU min MHz: 2061.0000 L1d cache: 64K L1i cache: 32K L2 cache: 512K L3 cache: 8192K NUMA node0 CPU(s): 0,8,16,24,32 NUMA node1 CPU(s): 48,56,64,72,80,88 The problem happens only if I use kvm_pr instead of kvm_hv. and kvm_pr is used for nested virt, which is what Sinny used AFAIK. (In reply to Dan Horák from comment #15) > and kvm_pr is used for nested virt, which is what Sinny used AFAIK. Thanks Dan! yes it is using kvm_pr It seems the problem happens with kvm_pr but only with qemu 3.0.0 not with qemu-2.12.0 (In reply to Laurent Vivier from comment #18) > It seems the problem happens with kvm_pr but only with qemu 3.0.0 not with > qemu-2.12.0 yeah, downgrading to 2.12.0 seems to work fine for me as well Bisected to: commit 9dceda5fc34a5868012260ee7271c7a6f36cc1f4 Author: David Gibson <david.id.au> Date: Mon Apr 16 16:47:19 2018 +1000 spapr: Limit available pagesizes to provide a consistent guest environment KVM HV has some limitations (deriving from the hardware) that mean not all host-cpu supported pagesizes may be usable in the guest. At present this means that KVM guests and TCG guests may see different available page sizes even if they notionally have the same vcpu model. This is confusing and also prevents migration between TCG and KVM. This patch makes the environment consistent by always allowing the same set of pagesizes. Since we can't remove the KVM limitations, we do this by always applying the same limitations it has, even to TCG guests. Signed-off-by: David Gibson <david.id.au> Reviewed-by: Cédric Le Goater <clg> Reviewed-by: Greg Kurz <groug> Ah, right. I'd forgotten about that. Basically, I sacrificed KVM PR support in order to get consistent guest behaviour. To fix this we'd have to fix PR in a bunch of places - which means finding someone with the time to do it. Removing needinfo since we know that this issue was introduced with qemu changes in commit 9dceda . Testing with upstream kernel shouldn't be necessary now. This bug is also affecting our ppc64le Fedora imagebuilder - https://pagure.io/fedora-infrastructure/issue/7463 It will be nice to have this issue fixed sooner. It'd be nice, yes, but no-one is currently working on it AFAIK, since people with the knowledge to work on KVM PR are few, and those with the time as well basically non-existent. David, do you know that status of nested KVM HV? I saw a presentation recently saying it's coming very soon :-) Nested HV is already merged upstream (as of 4.19, though some important bug fixes may still only be in rcs), and is in RHEL8. Note however that it will only work on POWER9 hosts, and only supports RPT at all levels. There are plans to support HPT (and thereby POWER8 compat mode) guests at the bottom level, but every other level will need to have a POWER9 RPT kernel. Correction, nested HV was merged upstream in 4.20, not 4.19. The Power9 requirement should be OK as Fedora infra has Power9 HW. Kernel 4.20 will go the stable releases at some point, qemu 3.1 is in Rawhide/F-30 only, but it could be upgraded on the builders similarly as it's downgraded now. I think we are almost ready to switch :-) Opened https://pagure.io/fedora-infrastructure/issue/7475 for the infra team. So with kernel 4.20 and qemu 3.1 (from the virt preview repo) in both host and guest I see [dan@localhost ~]$ dmesg | grep kvm [ 0.780384] systemd[1]: Detected virtualization kvm. [ 6.031295] systemd[1]: Detected virtualization kvm. [ 26.595043] kvm-hv: Parent hypervisor does not support nesting (rc=-2) in the guest. Should the comment #26 read as "important fixes still only in 4.21/5.0 rc"? So, I think there are some fairly important bugs which might only be fixed later than 4.20, but those are mostly with dirty map / migration and some edge cases. It should basically work in 4.20. I think what's probably going on here is that you need to enable nesting for each specific L1 guest when you start it on the L0. Specifically you need to add -machine cap-nested-hv=on to the qemu command line for your L1 (or any level you want to be able to run nested guests). Without that the L0 won't advertise capability to run nested guests to the L1, which I think is what you're seeing here. yes, that was it :-) I've added the following snippet to the L1 domain XML and then it boots an L2 guest <features> <nested-hv state='on'/> </features> Dan, sounds like we could close this with NOTABUG, or maybe WONTFIX, yes? I would say either WONTFIX or CANTFIX would be appropriate. At least we should add some documentation to F29 for the restriction, where to do it ? If you boot the KVM-PR guest with the following added to the qemu command line: -machine pseries,cap-hpt-max-page-size=16777216 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Does it make any difference? |