Bug 1978032
| Summary: | RHEL9[crash]-live analysis will cause kernel panic sometimes | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Jie Li <jieli> |
| Component: | crash | Assignee: | Philipp Rudo <prudo> |
| Status: | CLOSED WORKSFORME | QA Contact: | Jie Li <jieli> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 9.0 | CC: | dhowells, kazuhito.hagio, lijiang, ruyang, xiawu, yiyan |
| Target Milestone: | beta | Keywords: | Reopened, Triaged |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-11-24 08:58:02 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1971841 | ||
|
Comment 5
lijiang
2021-07-06 06:43:47 UTC
Adding David and Jeremy maybe they have an idea how to interpret my findings.
Unfortunately the structure of the kernel-ark/rhel9 tree doesn't allow
bisecting between kernel-5.13.0-0.rc4.33.el9..kernel-5.13.0-0.rc7.51.el9. Thus
I bisected the rhel9 tree between v5.13-rc7..kernel-5.13.0-0.rc7.51.el9, i.e.
the os-build branch, which led to
93a35c313b1b Merge 'master' into 'os-build'
right before
d600a62d61c1 [redhat] kernel-5.13.0-0.rc5.20210611git06af8679449d.40
All in all bisecting was quite painful as there where quite some build
failures and missing/new config options. But the result seems reasonable to me
as I can easily reproduce the panic with kernel-5.13.0-0.rc6.45 but not with
kernel-5.13.0-0.rc5.38 from kernel-ark.
After that I tried to reproduce the panic with an upstream kernel and failed.
Thus I looked at the differences between kernel-ark/rhel9 and upstream. This
led to the kernel lockdown patches, i.e.
5850c93175b9 ("efi: Lock down the kernel if booted in secure boot mode")
53250b991f84 ("efi: Add an EFI_SECURE_BOOT flag to indicate secure boot mode")
72223fd1241c ("security: lockdown: expose a hook to lock the kernel down")
After applying them to the upstream kernel I managed to reproduce the panic.
To be honest I don't think that the lock down patches introduced the panic but
merely changes timing and/or memory layout such that the panic occurs more
reliably.
NOTE1: While investigating I sometimes encountered panics with a slightly
different back trace. All had in common that they happened after the scheduler
did some reweighting, i.e. they started to differ after
[ 62.734966] ? __update_load_avg_cfs_rq+0x275/0x2e0
[ 62.735942] ? reweight_entity+0x128/0x140
NOTE2: I tried to get a dump but once I reserved a crashkernel memory I could
no longer trigger the panic. (I managed to trigger it once but then the kdump
kernel ran oom....)
(In reply to Philipp Rudo from comment #8) [...] > After that I tried to reproduce the panic with an upstream kernel and failed. > Thus I looked at the differences between kernel-ark/rhel9 and upstream. This > led to the kernel lockdown patches, i.e. > > 5850c93175b9 ("efi: Lock down the kernel if booted in secure boot mode") > 53250b991f84 ("efi: Add an EFI_SECURE_BOOT flag to indicate secure boot mode") > 72223fd1241c ("security: lockdown: expose a hook to lock the kernel down") I had an other look and it turned out that the lock down patches were a false lead. I had an untracked change that disabled CONFIG_IO_URING, which alone was sufficient to reproduce the bug. (There's a RHEL-only patch in kernel-ark 50f8c7fb77c9 ("RHEL: disable io_uring support") that tampers with the Kconfig. If you simply copy the rhel9/ark config over to upstream and run 'make oldconfig' IO_URING will be silently enabled.) During my tests I made the observation that on the same kernel the panic can sometimes be triggered on the first try and sometimes not at all. Which one is the case seems to be determined during boot. With that in mind I was able to reproduce the panic on upstream v5.13-rc4 but _not_ on v5.12. Finally I took a closer look at what crash does when the panic occurs. It turns out the panic is always triggered by the same read. In particular when crash tries to read the per_cpu cpu_number for an non existing cpu i.e. when __per_cpu_offset == __per_cpu_load. This additional read is a bug in crash I've got a provisional fix for. With that applied the panic is no longer triggered and crash starts as expected. But that's of course no proper solution for the panic... BTW, in the meantime i was able to get a dump. Furthermore, in my tests I also encountered a panic with quite a different back trace, especially without reweighting done by the scheduler. [ 591.979833] general protection fault, probably for non-canonical address 0xf8b96d21ff068: 0000 [#1] SMP PTI [ 591.985304] CPU: 0 PID: 1267 Comm: crash Tainted: G X --------- --- 5.13.0-0.rc7.51.el9.x86_64 #1 [ 591.987053] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [ 591.988036] RIP: 0010:kern_addr_valid+0x160/0x1d0 [ 591.988858] Code: 00 e0 ff f6 c2 80 48 0f 45 f1 48 89 f1 48 f7 d1 48 21 d1 81 e1 80 00 00 00 75 38 48 c1 e8 09 48 21 f2 25 f8 0f 00 00 48 01 f8 <48> 8b 04 10 48 a9 9f ff ff ff 0f 84 e0 fe ff ff 48 89 c7 83 e7 01 [ 591.992048] RSP: 0018:ffffb573c04bfc38 EFLAGS: 00010282 [ 591.992957] RAX: ffff8b9700000068 RBX: 0000000000000004 RCX: 0000000000000000 [ 591.994184] RDX: 000fffffd21ff000 RSI: 000ffffffffff000 RDI: ffff8b9700000000 [ 591.995402] RBP: ffffb573c04bfea8 R08: 000fffffffffffff R09: 000ffffffffff000 [ 591.996633] R10: 0000000000000000 R11: ffffb573c04bfc88 R12: 00007ffd2f6cfbb4 [ 591.997855] R13: 0000000000000004 R14: ffffffffaff87b20 R15: ffffffffafa0d568 [ 591.999080] FS: 00007fe285c31740(0000) GS:ffff8b9830000000(0000) knlGS:0000000000000000 [ 592.000649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 592.001799] CR2: 00007fe283087e4b CR3: 0000000102a8c004 CR4: 00000000007706f0 [ 592.003223] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 592.004644] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 592.005950] PKRU: 55555554 [ 592.006439] Call Trace: [ 592.006910] read_kcore+0x3a9/0x840 [ 592.008103] ? set_next_entity+0xad/0x1e0 [ 592.008937] ? __queue_work+0x1d1/0x3e0 [ 592.009729] ? queue_work_on+0x20/0x30 [ 592.010399] ? pty_write+0x86/0x90 [ 592.011096] ? __wake_up_common_lock+0x8a/0xc0 [ 592.012003] ? do_tty_write+0x1a3/0x260 [ 592.012784] ? update_load_avg+0x7a/0x590 [ 592.013590] ? set_next_entity+0xad/0x1e0 [ 592.014401] ? __switch_to+0x393/0x500 [ 592.015107] ? __cond_resched+0x16/0x40 [ 592.015802] ? inode_security+0x22/0x60 [ 592.016492] proc_reg_read+0x55/0xa0 [ 592.017146] ? vfs_read+0x65/0x180 [ 592.017768] vfs_read+0x97/0x180 [ 592.018348] ksys_read+0x5f/0xe0 [ 592.018941] do_syscall_64+0x40/0x80 [ 592.019608] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 592.020485] RIP: 0033:0x7fe285d3c8a2 [ 592.021167] Code: c0 e9 b2 fe ff ff 50 48 8d 3d 42 ae 0b 00 e8 45 f6 01 00 0f 1f 44 00 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 0f 05 <48> 3d 00 f0 ff ff 77 56 c3 0f 1f 44 00 00 48 83 ec 28 48 89 54 24 [ 592.024411] RSP: 002b:00007ffd2f6cfaf8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 592.025747] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007fe285d3c8a2 [ 592.026974] RDX: 0000000000000004 RSI: 00007ffd2f6cfbb4 RDI: 0000000000000004 [ 592.028208] RBP: 0000000000000004 R08: ffff8b972de0d568 R09: ffffffffb0230000 [ 592.029445] R10: 0000000000000004 R11: 0000000000000246 R12: 0000000000000004 [ 592.030678] R13: 00007ffd2f6cfbb4 R14: 00007fffafa11568 R15: 0000000000000004 [ 592.032422] Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore rfkill intel_rapl_msr intel_rapl_common isst_if_common nfit libnvdimm kvm_intel kvm irqbypass i2c_piix4 joydev virtio_balloon pcspkr drm fuse xfs libcrc32c ata_generic ata_piix libata virtio_net crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel net_failover serio_raw virtio_blk virtio_console failover dm_mirror dm_region_hash dm_log dm_mod [ 592.042990] ---[ end trace 36ef134bb8acdf98 ]--- [ 592.043817] RIP: 0010:kern_addr_valid+0x160/0x1d0 [ 592.044647] Code: 00 e0 ff f6 c2 80 48 0f 45 f1 48 89 f1 48 f7 d1 48 21 d1 81 e1 80 00 00 00 75 38 48 c1 e8 09 48 21 f2 25 f8 0f 00 00 48 01 f8 <48> 8b 04 10 48 a9 9f ff ff ff 0f 84 e0 fe ff ff 48 89 c7 83 e7 01 [ 592.047851] RSP: 0018:ffffb573c04bfc38 EFLAGS: 00010282 [ 592.048767] RAX: ffff8b9700000068 RBX: 0000000000000004 RCX: 0000000000000000 [ 592.050009] RDX: 000fffffd21ff000 RSI: 000ffffffffff000 RDI: ffff8b9700000000 [ 592.051242] RBP: ffffb573c04bfea8 R08: 000fffffffffffff R09: 000ffffffffff000 [ 592.052534] R10: 0000000000000000 R11: ffffb573c04bfc88 R12: 00007ffd2f6cfbb4 [ 592.053791] R13: 0000000000000004 R14: ffffffffaff87b20 R15: ffffffffafa0d568 [ 592.055023] FS: 00007fe285c31740(0000) GS:ffff8b9830000000(0000) knlGS:0000000000000000 [ 592.056419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 592.057423] CR2: 00007fe283087e4b CR3: 0000000102a8c004 CR4: 00000000007706f0 [ 592.058666] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 592.059898] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 592.061131] PKRU: 55555554 [ 592.061620] Kernel panic - not syncing: Fatal exception [ 592.063238] Kernel Offset: 0x2c600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 592.065386] ---[ end Kernel panic - not syncing: Fatal exception ]--- could no longer reproduce with kernel-5.14.0-0.rc4.35.el9.x86_64 from RHEL-9.0.0-20210809.7 |