1. Please describe the problem: kdump kernel hangs in guest KVM VM with POWER9 as host. I've done a bisection and it's a regression caused by "[PATCH 00/31] powerpc: Modernize the PCI/MSI support". So this issue doesn't bother kernel 5.14. Btw, If we checkout commit a5f3d2c17b07 ("powerpc/pseries/pci: Add MSI domains"), even the 1st kernel will hang. 2. What is the Version-Release number of the kernel: kernel >= 5.15, e.g. 5.19.3-300.fc37.ppc64le 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at Yes, at least it works for 5.14 e.g. kernel-core-5.14.0-60.fc36.ppc64le.rpm 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: a. Provision a guest KVM VM with vcpus>=16 b. install kexec-tools c. using "kdumpctl reset-crashkernel" to set up the crashkernel kernel parameter d. reboot e. systemctl start kdump f. trigger sysrq 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: Yes. 6. Are you running any modules that not shipped with directly Fedora's kernel?: no 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag. localhost login: [ 26.637286] sysrq: Trigger a crash [ 26.637328] Kernel panic - not syncing: sysrq triggered crash [ 26.637358] CPU: 26 PID: 4249 Comm: bash Kdump: loaded Not tainted 6.1.0-rc3+ #3 .... [ 0.000000] Linux version 6.1.0-rc3+ (root.lab.eng.bos.redhat.com) (gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-15), GNU ld version 2.30-117.el8) #3 SMP Fri Nov 4 03:31:06 EDT 2022 ... [ 0.000000] Kernel command line: elfcorehdr=0x22c00000 no_timer_check net.ifnames=0 console=tty0 console=hvc0,115200n8 irqpoll maxcpus=1 noirqdistrib reset_devices cgroup_disable=memory numa=off udev.children-max=2 ehea.use_mcs=0 panic=10 kvm_cma_resv_ratio=0 transparent_hugepage=never novmcoredd hugetlb_cma=0 ... [ 7.763260] virtio_blk virtio2: 32/0/0 default/read/poll queues [ 7.771391] virtio_blk virtio2: [vda] 20971520 512-byte logical blocks (10.7 GB/10.0 GiB) [ 68.398234] systemd-udevd[187]: virtio2: Worker [190] processing SEQNUM=1193 is taking a long time [ 188.398258] systemd-udevd[187]: virtio2: Worker [190] processing SEQNUM=1193 killed
Maybe relevant with https://bugzilla.redhat.com/show_bug.cgi?id=2129726 Using kernel param srcutree.big_cpu_lim=0 to see whether this issue is gone
(In reply to Pingfan Liu from comment #1) > Maybe relevant with https://bugzilla.redhat.com/show_bug.cgi?id=2129726 > > Using kernel param srcutree.big_cpu_lim=0 to see whether this issue is gone Unfortunately, kdump kernel still hang with the above kernel parameter, localhost login: [ 29.410603] sysrq: Trigger a crash [ 29.410679] Kernel panic - not syncing: sysrq triggered crash [ 29.410718] CPU: 0 PID: 1535 Comm: bash Kdump: loaded Not tainted 6.1.0-0.rc2.21.fc38.ppc64le #1 ... [ 0.000000] Kernel command line: elfcorehdr=0x22cd0000 BOOT_IMAGE=(ieee1275/disk,msdos2)/vmlinuz-6.1.0-0.rc2.21.fc38.ppc64le no_timer_check net.ifnames=0 console=tty0 console=hvc0,115200n8 irqpoll maxcpus=1 noirqdistrib reset_devices cgroup_disable=memory numa=off udev.children-max=2 ehea.use_mcs=0 panic=10 kvm_cma_resv_ratio=0 transparent_hugepage=never novmcoredd hugetlb_cma=0 srcutree.big_cpu_lim=0 ... [ 5.871341] Console: switching to colour frame buffer device 128x48 [ 5.872591] bochs-drm 0000:00:07.0: [drm] fb0: bochs-drmdrmfb frame buffer device [ 65.387497] systemd-udevd[162]: virtio2: Worker [166] processing SEQNUM=1114 is taking a long time [ 185.386427] systemd-udevd[162]: virtio2: Worker [166] processing SEQNUM=1114 killed But with maxcpus=2, kdump works again (for both vcpus=4 and vcpus=32), [ 0.000000] Kernel command line: elfcorehdr=0x22cd0000 BOOT_IMAGE=(ieee1275/disk,msdos2)/vmlinuz-6.1.0-0.rc2.21.fc38.ppc64le no_timer_check net.ifnames=0 console=tty0 console=hvc0,115200n8 irqpoll maxcpus=2 noirqdistrib reset_devices cgroup_disable=memory numa=off udev.children-max=2 ehea.use_mcs=0 panic=10 kvm_cma_resv_ratio=0 transparent_hugepage=never novmcoredd hugetlb_cma=0 ... [ 7.202898] kdump.sh[331]: The dumpfile is saved to /sysroot/var/crash/127.0.0.1-2022-11-04-02:24:05//vmcore-incomplete. [ 7.203061] kdump.sh[331]: makedumpfile Completed.
And for 6.1.0-0.rc3.20221101git5aaef24b5c6d.28.eln123.ppc64le, when kdump kernel hangs with maxcpus=1, the logs are slight different from 6.1.0-0.rc2.21.fc38.ppc64le, [ 0.000000] Kernel command line: elfcorehdr=0x27640000 no_timer_check net.ifnames=0 console=tty0 console=hvc0,115200n8 irqpoll maxcpus=1 noirqdistrib reset_devices cgroup_disable=memory numa=off udev.children-max=2 ehea.use_mcs=0 panic=10 kvm_cma_resv_ratio=0 transparent_hugepage=never novmcoredd hugetlb_cma=0 ... [ 10.119485] systemd[1]: Detected architecture ppc64-le. [ 10.122314] systemd[1]: Running in initial RAM disk. Welcome to Red Hat Enterprise Linux 9.2 Beta (Plow) dracut-057-13.git20220816.el9 (Initramfs)! [ 10.126963] systemd[1]: No hostname configured, using default hostname. [ 10.130279] systemd[1]: Hostname set to <localhost>. [ 243.791472] INFO: task systemd:1 blocked for more than 122 seconds. [ 243.793463] Tainted: G W ------- --- 6.1.0-0.rc3.20221101git5aaef24b5c6d.28.eln123.ppc64le #1 [ 243.798381] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 243.801054] task:systemd state:D stack:20032 pid:1 ppid:0 flags:0x00042000 [ 243.803821] Call Trace: [ 243.806419] [c0000000297876b0] [c000000020022f00] __switch_to+0x200/0x380 [ 243.809169] [c000000029787720] [c00000002169b4a0] __schedule+0x4a0/0xc80 [ 243.811905] [c000000029787800] [c00000002169bd4c] schedule+0xcc/0x1c0 [ 243.814631] [c000000029787880] [c0000000216a74b4] schedule_timeout+0x144/0x1c0 [ 243.817331] [c000000029787970] [c00000002169d628] __wait_for_common+0x2d8/0x3a0 [ 243.819977] [c000000029787a20] [c0000000201ed528] __flush_work+0x108/0x150 [ 243.822611] [c000000029787b20] [c0000000208e06d8] fsnotify_wait_marks_destroyed+0x28/0x40 [ 243.825244] [c000000029787b40] [c0000000208dfb64] fsnotify_destroy_group+0xc4/0x1e0 [ 243.827825] [c000000029787bb0] [c0000000208e5820] inotify_release+0x40/0xb0 [ 243.830332] [c000000029787c20] [c00000002082da04] __fput+0x174/0x530 [ 243.832805] [c000000029787ca0] [c0000000201f722c] task_work_run+0x12c/0x1e0 [ 243.835206] [c000000029787d10] [c000000020026e44] do_notify_resume+0x1a4/0x1b0 [ 243.837560] [c000000029787d50] [c000000020038260] interrupt_exit_user_prepare_main+0x2d0/0x370 [ 243.839961] [c000000029787dc0] [c0000000200383d4] syscall_exit_prepare+0xd4/0x250 [ 243.842364] [c000000029787e10] [c00000002000c17c] system_call_vectored_common+0xfc/0x280 [ 243.844746] --- interrupt: 3000 at 0x7fff9c995dc4 [ 243.847057] NIP: 00007fff9c995dc4 LR: 0000000000000000 CTR: 0000000000000000 [ 243.849398] REGS: c000000029787e80 TRAP: 3000 Tainted: G W ------- --- (6.1.0-0.rc3.20221101git5aaef24b5c6d.28.eln123.ppc64le) [ 243.854025] MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 42044440 XER: 00000000 [ 243.856432] IRQMASK: 0 [ 243.856432] GPR00: 0000000000000006 00007fffd4973390 00007fff9caa7200 0000000000000000 [ 243.856432] GPR04: 0000000000000002 000000000000000a 0000000000000000 0000000000000001 [ 243.856432] GPR08: 0000000136574700 0000000000000000 0000000000000000 0000000000000000 [ 243.856432] GPR12: 0000000000000000 00007fff9d1b4b80 0000000000000000 0000000000000000 [ 243.856432] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 243.856432] GPR20: 000000010c6ddc50 000000000000000e 000000000097c954 0000000000000000 [ 243.856432] GPR24: 00007fffd4973760 0000000000000005 0000000136584c50 0000000136572c50 [ 243.856432] GPR28: 0000000136573520 0000000136574240 0000000136574730 00007fff9d1add58 [ 243.876723] NIP [00007fff9c995dc4] 0x7fff9c995dc4 [ 243.878301] LR [0000000000000000] 0x0 [ 243.879826] --- interrupt: 3000 [ 243.881706] INFO: task kworker/u64:0:9 blocked for more than 122 seconds. [ 243.883705] Tainted: G W ------- --- 6.1.0-0.rc3.20221101git5aaef24b5c6d.28.eln123.ppc64le #1 [ 243.887729] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 243.889916] task:kworker/u64:0 state:D stack:25712 pid:9 ppid:2 flags:0x00000800 [ 243.892210] Workqueue: events_unbound fsnotify_mark_destroy_workfn [ 243.894446] Call Trace: [ 243.896568] [c000000029897510] [c0000000298975b0] 0xc0000000298975b0 (unreliable) [ 243.898889] [c000000029897700] [c000000020022f00] __switch_to+0x200/0x380 [ 243.901213] [c000000029897770] [c00000002169b4a0] __schedule+0x4a0/0xc80 [ 243.903517] [c000000029897850] [c00000002169bd4c] schedule+0xcc/0x1c0 [ 243.905797] [c0000000298978d0] [c0000000216a74b4] schedule_timeout+0x144/0x1c0 [ 243.908059] [c0000000298979c0] [c00000002169d628] __wait_for_common+0x2d8/0x3a0 [ 243.910327] [c000000029897a70] [c000000020301d04] __synchronize_srcu+0x1d4/0x260 [ 243.912613] [c000000029897b30] [c0000000208e0c10] fsnotify_mark_destroy_workfn+0xb0/0x1f0 [ 243.914876] [c000000029897bb0] [c0000000201eabc4] process_one_work+0x644/0xd80 [ 243.917070] [c000000029897ce0] [c0000000201eb3fc] worker_thread+0xfc/0x770 [ 243.919202] [c000000029897db0] [c0000000201fef1c] kthread+0x1ec/0x200 [ 243.921318] [c000000029897e10] [c00000002000cf6c] ret_from_kernel_thread+0x5c/0x64 [ 243.923502] [ 243.923502] Showing all locks held in the system: [ 243.927258] 2 locks held by kworker/u64:0/9: [ 243.929196] #0: c000000029117148 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x530/0xd80 [ 243.933404] #1: c000000029897c18 ((reaper_work).work){+.+.}-{0:0}, at: process_one_work+0x530/0xd80 [ 243.947768] 1 lock held by rcu_tasks_rude_/11: [ 243.950084] #0: c0000000245351a0 (rcu_tasks_rude.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x58/0x330 [ 243.954913] 1 lock held by rcu_tasks_trace/12: [ 243.957322] #0: c000000024534ec0 (rcu_tasks_trace.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x58/0x330 [ 243.962399] 1 lock held by khungtaskd/22: [ 243.964859] #0: c000000024534c40 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire.constprop.0+0x8/0x30 [ 243.969816] [ 243.972177] ============================================= [ 243.972177]
To clarify, I mixed two issues into this bug report although by setting maxcpus=2 in /etc/sysconfig/kdump makes kdump work again in both cases. One issue is is as hown in comment 3 and there is an upstream patch [1] sent by my college. The other issue is about kdump kernel hanging with logs like [ 68.398234] systemd-udevd[187]: virtio2: Worker [190] processing SEQNUM=1193 is taking a long time [ 188.398258] systemd-udevd[187]: virtio2: Worker [190] processing SEQNUM=1193 killed" I will use this bug to exclusively track the latter issue. [1] https://lore.kernel.org/rcu/20221026032716.78674-1-kernelfans@gmail.com/T/#u
I tried(In reply to Coiby from comment #4) > To clarify, I mixed two issues into this bug report although by setting > maxcpus=2 in /etc/sysconfig/kdump makes kdump work again in both cases. One > issue is is as hown in comment 3 and there is an upstream patch [1] sent by > my college. The other issue is about kdump kernel hanging with logs like > [ 68.398234] systemd-udevd[187]: virtio2: Worker [190] processing > SEQNUM=1193 is taking a long time > [ 188.398258] systemd-udevd[187]: virtio2: Worker [190] processing > SEQNUM=1193 killed" > I suspect this is caused by virtio driver (at least, related). I have tried the xml in comment#5 with the following changes 28,29c28 < <target dev="vda" bus="virtio"/> < <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/> --- > <target dev="sda" bus="scsi"/> Then using a 6.1-rc4 kernel with srcu fix, the kdump can save vmcore in the three times tests.
Hi Jason, This issue may be related with virtio. Could you give some suggestion? Thanks
I have opened an upstream bug to trace it https://bugzilla.kernel.org/show_bug.cgi?id=217350
This message is a reminder that Fedora Linux 36 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 36 on 2023-05-16. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '36'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see it. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 36 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed.
Memo from pingfan's comment in other places: https://lore.kernel.org/linuxppc-dev/CAFgQCTvfFW57nKvFpUwtQO-HozfFRf2XqBFcQ0rGqrXQS=seig@mail.gmail.com/T/#t
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days