Bug 1528178
| Summary: | qemu-kvm quits with early memory unplug | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Min Deng <mdeng> |
| Component: | qemu-kvm-rhev | Assignee: | Serhii Popovych <spopovyc> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Min Deng <mdeng> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 7.5 | CC: | ailan, dgibson, mdeng, michen, qzhang, virt-maint |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | ppc64le | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-06-13 03:12:05 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Min Deng
2017-12-21 08:20:35 UTC
QE will test it on x86 and update results as soon as possible. What exactly is the symptom in the guest? The messages you show look as if you're running qemu under gdb. What happens if you don't use gdb? (In reply to Min Deng from comment #2) > QE will test it on x86 and update results as soon as possible. QE couldn't reproduce the issue on x86 platform kernel-3.10.0-823.el7.x86_64 qemu-kvm-rhev-2.10.0-12.el7.x86_64 > The messages you show look as if you're running qemu under gdb. What
> happens if you don't use gdb?
QE just wanted to grab more information for this issue so use gdb,and there wasn't any difference between them in my opinions.
Because this only happens sometimes, with repeated unplug during an early boot stage, this isn't vital enough to get the exception or blocker flags. Therefore, postponing to 7.6. Retesting conditions from comment 0 on current components: Host : kernel-3.10.0-862.el7.ppc64le Qemu : qemu-kvm-rhev-2.12.0-3.el7.ppc64le Guest: kernel-3.10.0-897.el7.ppc64le and kernel-4.14.0-66.el7a.ppc64le I'm not able to reproduce described issue reliably. However I can reproduce most of times (2/3) memory hot-unplug failures that applied to both 4.14 and 3.10 kernels. They look as bz1432302, except for error message, that is duplicate of bz1245892. After guest reboot qemu gets notified about DEVICE_DELETE and dimm1 device removed. Both guest and qemu via HMP "info numa" reports that memory removed. So after initial hot plug memory is used by guest kernel for some reason. ------------------------------------------------------------------------------ QEMU 2.12.0 monitor - type 'help' for more information (qemu) Red Hat Enterprise Linux Server 7.6 Beta (Maipo) Kernel 4.14.0-66.el7a.ppc64le on an ppc64le localhost login: (qemu) (qemu) object_add memory-backend-ram,id=mem1,size=10G (qemu) device_add pc-dimm,id=dimm1,memdev=mem1 (qemu) [ 15.113583] pseries-hotplug-mem: Attempting to hot-add 40 LMB(s) at index 80000020 [ 15.162212] lpar: Attempting to resize HPT to shift 24 [ 15.358036] lpar: Hash collision while resizing HPT [ 15.358156] Unable to resize hash page table to target order 24: -28 [ 15.396661] Built 2 zonelists, mobility grouping on. Total pages: 126077 [ 15.396718] Policy zone: Normal [ 15.396976] lpar: Attempting to resize HPT to shift 24 [ 15.570240] lpar: Hash collision while resizing HPT [ 15.570373] Unable to resize hash page table to target order 24: -28 [ 15.608050] lpar: Attempting to resize HPT to shift 24 [ 15.766930] lpar: Hash collision while resizing HPT [ 15.767051] Unable to resize hash page table to target order 24: -28 [ 15.804494] lpar: Attempting to resize HPT to shift 24 [ 15.984056] lpar: Hash collision while resizing HPT [ 15.984172] Unable to resize hash page table to target order 24: -28 [ 16.021835] lpar: Attempting to resize HPT to shift 24 [ 16.199397] lpar: Hash collision while resizing HPT [ 16.199505] Unable to resize hash page table to target order 24: -28 [ 16.237830] lpar: Attempting to resize HPT to shift 24 [ 16.388882] lpar: Hash collision while resizing HPT [ 16.388997] Unable to resize hash page table to target order 24: -28 [ 16.427306] lpar: Attempting to resize HPT to shift 24 [ 16.597805] lpar: Hash collision while resizing HPT [ 16.597919] Unable to resize hash page table to target order 24: -28 [ 16.635939] lpar: Attempting to resize HPT to shift 24 [ 16.794841] lpar: Hash collision while resizing HPT [ 16.794954] Unable to resize hash page table to target order 24: -28 [ 16.832693] lpar: Attempting to resize HPT to shift 24 [ 17.013998] lpar: Hash collision while resizing HPT [ 17.014112] Unable to resize hash page table to target order 24: -28 [ 17.051680] lpar: Attempting to resize HPT to shift 24 [ 17.222303] lpar: Hash collision while resizing HPT [ 17.222428] Unable to resize hash page table to target order 24: -28 [ 17.259540] lpar: Attempting to resize HPT to shift 24 [ 17.412989] lpar: Hash collision while resizing HPT [ 17.413502] Unable to resize hash page table to target order 24: -28 [ 17.450431] lpar: Attempting to resize HPT to shift 24 [ 17.612244] lpar: Hash collision while resizing HPT [ 17.612397] Unable to resize hash page table to target order 24: -28 [ 17.649777] lpar: Attempting to resize HPT to shift 24 [ 17.802082] lpar: Hash collision while resizing HPT [ 17.802193] Unable to resize hash page table to target order 24: -28 [ 17.840878] lpar: Attempting to resize HPT to shift 24 [ 17.997120] lpar: Hash collision while resizing HPT [ 17.997235] Unable to resize hash page table to target order 24: -28 [ 18.035753] lpar: Attempting to resize HPT to shift 24 [ 18.195585] lpar: Hash collision while resizing HPT [ 18.195706] Unable to resize hash page table to target order 24: -28 [ 18.234865] lpar: Attempting to resize HPT to shift 24 [ 18.396867] lpar: Hash collision while resizing HPT [ 18.396986] Unable to resize hash page table to target order 24: -28 [ 18.434821] lpar: Attempting to resize HPT to shift 24 [ 18.612423] lpar: Hash collision while resizing HPT [ 18.612514] Unable to resize hash page table to target order 24: -28 [ 18.649944] lpar: Attempting to resize HPT to shift 24 [ 18.796310] lpar: Hash collision while resizing HPT [ 18.796408] Unable to resize hash page table to target order 24: -28 [ 18.834245] lpar: Attempting to resize HPT to shift 24 [ 18.984514] lpar: Hash collision while resizing HPT [ 18.984616] Unable to resize hash page table to target order 24: -28 [ 19.022863] lpar: Attempting to resize HPT to shift 24 [ 19.173941] lpar: Hash collision while resizing HPT [ 19.174047] Unable to resize hash page table to target order 24: -28 [ 19.212657] lpar: Attempting to resize HPT to shift 24 [ 19.363605] lpar: Hash collision while resizing HPT [ 19.363703] Unable to resize hash page table to target order 24: -28 [ 19.402957] lpar: Attempting to resize HPT to shift 24 [ 19.569392] lpar: Hash collision while resizing HPT [ 19.569512] Unable to resize hash page table to target order 24: -28 [ 19.607091] lpar: Attempting to resize HPT to shift 24 [ 19.769003] lpar: Hash collision while resizing HPT [ 19.769112] Unable to resize hash page table to target order 24: -28 [ 19.806755] lpar: Attempting to resize HPT to shift 24 [ 19.959782] lpar: Hash collision while resizing HPT [ 19.959901] Unable to resize hash page table to target order 24: -28 [ 19.997405] lpar: Attempting to resize HPT to shift 24 [ 20.149505] lpar: Hash collision while resizing HPT [ 20.149632] Unable to resize hash page table to target order 24: -28 [ 20.187113] lpar: Attempting to resize HPT to shift 24 [ 20.341638] lpar: Hash collision while resizing HPT [ 20.341744] Unable to resize hash page table to target order 24: -28 [ 20.378898] lpar: Attempting to resize HPT to shift 24 [ 20.527972] lpar: Hash collision while resizing HPT [ 20.528072] Unable to resize hash page table to target order 24: -28 [ 20.564706] lpar: Attempting to resize HPT to shift 24 [ 20.720252] lpar: Hash collision while resizing HPT [ 20.720401] Unable to resize hash page table to target order 24: -28 [ 20.758219] lpar: Attempting to resize HPT to shift 24 [ 20.916414] lpar: Hash collision while resizing HPT [ 20.916543] Unable to resize hash page table to target order 24: -28 [ 20.954624] lpar: Attempting to resize HPT to shift 24 [ 21.106385] lpar: Hash collision while resizing HPT [ 21.106509] Unable to resize hash page table to target order 24: -28 [ 21.144648] lpar: Attempting to resize HPT to shift 24 [ 21.310382] lpar: Hash collision while resizing HPT [ 21.310489] Unable to resize hash page table to target order 24: -28 [ 21.348800] lpar: Attempting to resize HPT to shift 24 [ 21.497243] lpar: Hash collision while resizing HPT [ 21.497375] Unable to resize hash page table to target order 24: -28 [ 21.837664] pseries-hotplug-mem: Memory at 200000000 (drc index 80000020) was hot-added [ 21.837745] pseries-hotplug-mem: Memory at 210000000 (drc index 80000021) was hot-added [ 21.837807] pseries-hotplug-mem: Memory at 220000000 (drc index 80000022) was hot-added [ 21.837868] pseries-hotplug-mem: Memory at 230000000 (drc index 80000023) was hot-added [ 21.837930] pseries-hotplug-mem: Memory at 240000000 (drc index 80000024) was hot-added [ 21.837990] pseries-hotplug-mem: Memory at 250000000 (drc index 80000025) was hot-added [ 21.838051] pseries-hotplug-mem: Memory at 260000000 (drc index 80000026) was hot-added [ 21.838383] pseries-hotplug-mem: Memory at 270000000 (drc index 80000027) was hot-added [ 21.838452] pseries-hotplug-mem: Memory at 280000000 (drc index 80000028) was hot-added [ 21.838515] pseries-hotplug-mem: Memory at 290000000 (drc index 80000029) was hot-added [ 21.838576] pseries-hotplug-mem: Memory at 2a0000000 (drc index 8000002a) was hot-added [ 21.838638] pseries-hotplug-mem: Memory at 2b0000000 (drc index 8000002b) was hot-added [ 21.838700] pseries-hotplug-mem: Memory at 2c0000000 (drc index 8000002c) was hot-added [ 21.838764] pseries-hotplug-mem: Memory at 2d0000000 (drc index 8000002d) was hot-added [ 21.838824] pseries-hotplug-mem: Memory at 2e0000000 (drc index 8000002e) was hot-added [ 21.838886] pseries-hotplug-mem: Memory at 2f0000000 (drc index 8000002f) was hot-added [ 21.838947] pseries-hotplug-mem: Memory at 300000000 (drc index 80000030) was hot-added [ 21.839008] pseries-hotplug-mem: Memory at 310000000 (drc index 80000031) was hot-added [ 21.839285] pseries-hotplug-mem: Memory at 320000000 (drc index 80000032) was hot-added [ 21.839349] pseries-hotplug-mem: Memory at 330000000 (drc index 80000033) was hot-added [ 21.839410] pseries-hotplug-mem: Memory at 340000000 (drc index 80000034) was hot-added [ 21.839470] pseries-hotplug-mem: Memory at 350000000 (drc index 80000035) was hot-added [ 21.839531] pseries-hotplug-mem: Memory at 360000000 (drc index 80000036) was hot-added [ 21.839592] pseries-hotplug-mem: Memory at 370000000 (drc index 80000037) was hot-added [ 21.839653] pseries-hotplug-mem: Memory at 380000000 (drc index 80000038) was hot-added [ 21.839713] pseries-hotplug-mem: Memory at 390000000 (drc index 80000039) was hot-added [ 21.839774] pseries-hotplug-mem: Memory at 3a0000000 (drc index 8000003a) was hot-added [ 21.839834] pseries-hotplug-mem: Memory at 3b0000000 (drc index 8000003b) was hot-added [ 21.839895] pseries-hotplug-mem: Memory at 3c0000000 (drc index 8000003c) was hot-added [ 21.839956] pseries-hotplug-mem: Memory at 3d0000000 (drc index 8000003d) was hot-added [ 21.840017] pseries-hotplug-mem: Memory at 3e0000000 (drc index 8000003e) was hot-added [ 21.840077] pseries-hotplug-mem: Memory at 3f0000000 (drc index 8000003f) was hot-added [ 21.840138] pseries-hotplug-mem: Memory at 400000000 (drc index 80000040) was hot-added [ 21.840199] pseries-hotplug-mem: Memory at 410000000 (drc index 80000041) was hot-added [ 21.840259] pseries-hotplug-mem: Memory at 420000000 (drc index 80000042) was hot-added [ 21.840320] pseries-hotplug-mem: Memory at 430000000 (drc index 80000043) was hot-added [ 21.840381] pseries-hotplug-mem: Memory at 440000000 (drc index 80000044) was hot-added [ 21.840442] pseries-hotplug-mem: Memory at 450000000 (drc index 80000045) was hot-added [ 21.840503] pseries-hotplug-mem: Memory at 460000000 (drc index 80000046) was hot-added [ 21.840563] pseries-hotplug-mem: Memory at 470000000 (drc index 80000047) was hot-added (qemu) (qemu) (qemu) (qemu) (qemu) info numa 2 nodes node 0 cpus: 0 1 node 0 size: 14336 MB node 0 plugged: 10240 MB node 1 cpus: 2 3 node 1 size: 4096 MB node 1 plugged: 0 MB (qemu) device_del dimm1 (qemu) [ 41.438153] pseries-hotplug-mem: Attempting to hot-remove 40 LMB(s) at 80000020 [ 41.438253] pseries-hotplug-mem: Memory indexed-count-remove failed, adding any removed LMBs (qemu) (qemu) (qemu) (qemu) info numa 2 nodes node 0 cpus: 0 1 node 0 size: 14336 MB node 0 plugged: 10240 MB node 1 cpus: 2 3 node 1 size: 4096 MB node 1 plugged: 0 MB (qemu) device_del dimm1 Memory unplug already in progress for device dimm1 (qemu) (qemu) (qemu) device_add pc-dimm,id=dimm1,memdev=mem1 Duplicate ID 'dimm1' for device Try "help device_add" for more information (qemu) *** after guest reboot **** Red Hat Enterprise Linux Server 7.6 Beta (Maipo) Kernel 4.14.0-66.el7a.ppc64le on an ppc64le localhost log (qemu) (qemu) device_del dimm1 Device 'dimm1' not found (qemu) info numa 2 nodes node 0 cpus: 0 1 node 0 size: 4096 MB node 0 plugged: 0 MB node 1 cpus: 2 3 node 1 size: 4096 MB node 1 plugged: 0 MB (qemu) device_add pc-dimm,id=dimm1,memdev=mem1 Hi, Min Any tips to reproduce this bug in comment 0? Thanks. Hi Serhii, It looks like you used the newer builds so could you please try the following old builds mentioned in comment0 ? Thanks,any issues please let me know. kernel-3.10.0-823.el7.ppc64le (host and guest) qemu-kvm-rhev-2.10.0-13.el7.ppc64le Min (In reply to Min Deng from comment #10) > Hi Serhii, > It looks like you used the newer builds so could you please try the > following old builds mentioned in comment0 ? Thanks,any issues please let me > know. > kernel-3.10.0-823.el7.ppc64le (host and guest) > qemu-kvm-rhev-2.10.0-13.el7.ppc64le > Min Besides that,the original problem do happen with the old builds.But now,I have to say it could *not* be reproducible with the latest builds any more.Thanks,any problem let me know. kernel-3.10.0-901.el7.ppc64le - guest kernel- 3.10.0-900.el7.ppc64le - host qemu-kvm-rhev-2.12.0-3.el7.ppc64le Hi developer. QE reported a bug which was similar with comment 8's and please feel free to update it,Bug 1589647,thanks. Min Based on comment 11, this no longer reproduces. It's probably been fixed upstream, and we've collected the fix with our rebase. |