Bug 1528173
| Summary: | Hot-unplug memory during booting early stage induced qemu-kvm coredump | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Min Deng <mdeng> | ||||
| Component: | qemu-kvm-rhev | Assignee: | Serhii Popovych <spopovyc> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Min Deng <mdeng> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 7.5 | CC: | bugproxy, dgibson, dzheng, fnovak, hannsj_uhl, knoel, lmiksik, mdeng, michen, mtessun, qzhang, spopovyc, virt-maint | ||||
| Target Milestone: | rc | ||||||
| Target Release: | 7.5 | ||||||
| Hardware: | ppc64le | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | qemu-kvm-rhev-2.10.0-17.el7 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-04-11 00:55:27 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 1533857 | ||||||
| Bug Blocks: | 1399177 | ||||||
| Attachments: |
|
||||||
|
Description
Min Deng
2017-12-21 07:54:27 UTC
Serhii, Can you take a look at this one please. I think it is probably a race between multiple attempts to unplug from the qemu side, if delays on the guest side been the early attempts don't complete quickly. It might even be an equivalent bug on the qemu side to one of the ones you tracked down on the host kernel side. We definitely need commit 2a129767ebb1 (hw/ppc/spapr.c: abort unplug_request if previous unplug isn't done) from master. Applied, tested using steps from comment 0 and issue no longer reproducible. Unfortunately have some issues with brew job submission: investigating. Also while analyzing case have additional question: is following race is possible even after picking commit from upstream: hw/ppc/spapr.c: --------------- spapr_memory_unplug_request() { .... /* (1) False for 1st attemt. True for next. */ if (spapr_pending_dimm_unplugs_find(spapr, dimm)) return; /* (2) Guard against multiple unplug request via */ spapr_pending_dimm_unplugs_add(); for (i = 0; i < nr_lmbs; i++) { /* (3) Lookup for DRC */ drc = spapr_drc_by_id(); spapr_drc_detach(drc); } } spapr_lmb_release() { /* Wait for all LMBs */ if (--ds->nr_lmbs) { return; .... /* (4) This opens (1) while @drc is still * available to spapr_drc_by_id() in (3) due * to (5). * * This causes reentrancy of spapr_drc_detach(drc) * in (3) while both @dev and @drc can be released. */ spapr_pending_dimm_unplugs_remove(); } hw/ppc/spapr_drc.c: ------------------- spapr_drc_lmb_class_init() { .... drck->release = spapr_lmb_release(); } spapr_drc_detach() { .... drck->release(drc->dev); /* spapr_lmb_release() */ .... /* (5) This should be done before calling * spapr_lmb_release() to hide @drc from * spapr_drc_by_id() in (3). */ object_property_del(OBJECT(drc), "device", &error_abort); drck->dev = NULL; } Also while working on the issue I contiguously spot following traces from guest kernel (3.10.0-826.el7.ppc64): [ 0.040090] IOMMU table initialized, virtual merging enabled [ 0.040141] iommu: Adding device 0000:00:00.0 to group 0 [ 0.040210] iommu: Adding device 0000:00:01.0 to group 0 [ 0.040273] iommu: Adding device 0000:00:02.0 to group 0 [ 0.040337] iommu: Adding device 0000:00:03.0 to group 0 [ 0.040571] Unable to handle kernel paging request for data at address 0x00000100 [ 0.040619] Faulting instruction address: 0xc0000000001159c8 [ 0.040660] Oops: Kernel access of bad area, sig: 11 [#1] [ 0.040692] SMP NR_CPUS=2048 NUMA pSeries [ 0.040725] Modules linked in: [ 0.040760] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-826.el7.ppc64le #1 [ 0.040807] task: c0000001fd7c0000 ti: c0000001fffe8000 task.ti: c0000001fd800000 [ 0.040855] NIP: c0000000001159c8 LR: c0000000001160e4 CTR: 0000000000000000 [ 0.040903] REGS: c0000001fffeb950 TRAP: 0300 Not tainted (3.10.0-826.el7.ppc64le) [ 0.040950] MSR: 8000000100009033 <SF,EE,ME,IR,DR,RI,LE> CR: 28000042 XER: 00000000 [ 0.041063] CFAR: 000000000000aad0 DAR: 0000000000000100 DSISR: 40000000 SOFTE: 0 GPR00: c0000000001160e4 c0000001fffebbd0 c000000001273f00 0000000000000800 GPR04: 0000000000000000 c0000000fca72e00 0000000000000000 0000000fffffffe1 GPR08: 0000000fffffffe0 0000000000000000 0000000fffffffe0 0000000100001001 GPR12: 0000000000000000 c000000007b80000 c00000000000cb08 0000000000000000 GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 GPR20: 0000000000000000 0000000000000001 0000000000000002 0000000000000015 GPR24: c0000001fd8585a0 0000000000000800 c0000001fd858400 0000000000000000 GPR28: 0000000000000800 c0000000015f0f98 0000000000000000 c0000000fca72e00 [ 0.041702] NIP [c0000000001159c8] __queue_work+0x68/0x6d0 [ 0.041736] LR [c0000000001160e4] queue_work_on+0xb4/0xc0 [ 0.041768] Call Trace: [ 0.041786] [c0000001fffebca0] [c0000000001160e4] queue_work_on+0xb4/0xc0 [ 0.041835] [c0000001fffebcd0] [c0000000000a5e90] queue_hotplug_event+0xd0/0x150 [ 0.041891] [c0000001fffebd20] [c0000000000a3620] ras_hotplug_interrupt+0x130/0x150 [ 0.041947] [c0000001fffebdb0] [c0000000001e26d0] __handle_irq_event_percpu+0x90/0x2e0 [ 0.042003] [c0000001fffebe80] [c0000000001e2a18] handle_irq_event+0x68/0xf0 [ 0.042052] [c0000001fffebec0] [c0000000001e8040] handle_fasteoi_irq+0xd0/0x230 [ 0.042108] [c0000001fffebef0] [c0000000001e18d0] generic_handle_irq+0x50/0x80 [ 0.042164] [c0000001fffebf20] [c000000000014a88] __do_irq+0x88/0x190 [ 0.042213] [c0000001fffebf90] [c000000000028e00] call_do_irq+0x14/0x24 [ 0.042261] [c0000001fd8035a0] [c000000000014c24] do_IRQ+0x94/0x110 [ 0.042310] [c0000001fd8035f0] [c000000000002794] hardware_interrupt_common+0x114/0x180 [ 0.042367] --- Exception: 501 at arch_local_irq_restore+0xf0/0x140 [ 0.042367] LR = arch_local_irq_restore+0xf0/0x140 [ 0.042439] [c0000001fd8038e0] [c0000000001e78f0] irq_startup+0x60/0x100 (unreliable) [ 0.042496] [c0000001fd803900] [c000000000a0ccac] _raw_spin_unlock_irqrestore+0x3c/0x80 [ 0.042553] [c0000001fd803920] [c0000000001e5558] __setup_irq+0x488/0x6b0 [ 0.042601] [c0000001fd803ac0] [c0000000001e5950] request_threaded_irq+0xf0/0x1f0 [ 0.042657] [c0000001fd803b20] [c0000000000a33e0] request_event_sources_irqs+0x110/0x220 [ 0.042713] [c0000001fd803c30] [c000000000d3aff0] init_ras_IRQ+0x98/0xfc [ 0.042762] [c0000001fd803c60] [c00000000000c28c] do_one_initcall+0x12c/0x2c0 [ 0.042811] [c0000001fd803cf0] [c000000000d240d4] kernel_init_freeable+0x244/0x324 [ 0.042868] [c0000001fd803db0] [c00000000000cb2c] kernel_init+0x2c/0x220 [ 0.042917] [c0000001fd803e30] [c00000000000a4b8] ret_from_kernel_thread+0x5c/0xa4 [ 0.042972] Instruction dump: [ 0.042997] faa1ffa8 fac1ffb0 fae1ffb8 f8010010 fb01ffc0 fb41ffd0 fba1ffe8 fbc1fff0 [ 0.043078] f821ff31 892d02a2 2fa90000 40de04c0 <813b0100> 792887e3 40c205fc 7be79364 [ 0.043161] ---[ end trace ed727bb1f359e245 ]--- [ 0.044422] [ 2.044473] Kernel panic - not syncing: Fatal exception in interrupt If I hot add memory before kernel loads (e.g. during the grub2 time) then machine restarts (not qemu process) instead of kernel crash. May be this happens because on development P8 machine I have ppc64 (BE) host environment. Serhii, 1) Please backport 2a129767ebb1 for this BZ ASAP. 2) I'm still analyzing the race you suggest in comment 3 - let's handle that as a different BZ 3) The kernel crash in comment 4 looks like a different bug, let's handle that separately 4) "If I hot add memory before kernel loads (e.g. during the grub2 time) then machine restarts (not qemu process) instead of kernel crash." This is expected - hotplugs before CAS (guest/qemu negotiation phase) cause a reboot (and the guest will pick up the new device on the reboot). It's not ideal but avoiding it would introduce a whole heap of other complexities. 5) "May be this happens because on development P8 machine I have ppc64 (BE) host environment." That sounds unlikely to me, but let's continue to invesigate after we've fixed the known qemu bug. Min, is this a regression from RHEL7.4? (In reply to David Gibson from comment #6) > Min, is this a regression from RHEL7.4? David, In my opinions,it wasn't a regression issue since some issues could also be found in the following builds. kernel-3.10.0-693.el7.ppc64le qemu-kvm-rhev-2.9.0-16.el7_4.1.ppc64le QE got two types of error for those builds after several tries. 1. qemu-kvm quit directly 2. (qemu)object_add memory-backend-ram,id=mem1,size=10G (qemu) device_add pc-dimm,id=dimm1,memdev=mem1 (qemu) device_del dimm1 Memory hot unplug not supported for this guest (qemu) device_del dimm1 (qemu) device_del dimm1 Message from syslogd@localhost at Jan 11 01:32:36 ... kernel:NMI watchdog: BUG: soft lockup - CPU#56 stuck for 22s! [CPU 0/KVM:53310] Message from syslogd@localhost at Jan 11 01:32:36 ... kernel:NMI watchdog: BUG: soft lockup - CPU#72 stuck for 22s! [CPU 1/KVM:53311] Message from syslogd@localhost at Jan 11 01:32:36 ... kernel:NMI watchdog: BUG: soft lockup - CPU#160 stuck for 22s! [CPU 3/KVM:53313] Fix included in qemu-kvm-rhev-2.10.0-17.el7 Reproduced on build kernel-3.10.0-823.el7.ppc64le (host and guest) qemu-kvm-rhev-2.10.0-13.el7.ppc64le QE re-tested the bug on the following build kernel-3.10.0-827.el7.ppc64le - host kernel-3.10.0-828.el7.ppc64le - guest qemu-kvm-rhev-2.10.0-17.el7.ppc64le Steps please refer to comment0 Actual results, The original issue could not be reproduced *but* reproduced bug 1533857 - > see comment4 and bug 1528178 reported by QE before. David, What do you think,is it proper for us to mark it as verified ? At the same time,QE uploaded the entire log for comparing with 1533857.Any issues please let me know. Thanks a lot Min Created attachment 1382313 [details]
comparinglog1533857
if necessary,please comparing it with 1533857,thanks a lot
Reproduced on build kernel-3.10.0-823.el7.ppc64le (host and guest) qemu-kvm-rhev-2.10.0-13.el7.ppc64le Verified the bug on the following build kernel-3.10.0-842.el7.ppc64le (both host and guest) qemu-kvm-rhev-2.10.0-18.el7.ppc64le Steps please refer to comment0 The bug has been fixed.Thanks a lot. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:1104 |