Bug 1880502
Summary: | optimization for delaying the freeing of empty slubs causes a NULL pointer dereference [rhel-8.2.0.z] | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 8 | Reporter: | Nitesh Narayan Lal <nilal> |
Component: | kernel-rt | Assignee: | Nitesh Narayan Lal <nilal> |
kernel-rt sub component: | Memory Management | QA Contact: | Pei Zhang <pezhang> |
Status: | CLOSED ERRATA | Docs Contact: | |
Severity: | medium | ||
Priority: | unspecified | CC: | aquini, bhu, chwhite, jinzhao, juri.lelli, lcapitulino, lgoncalv, llong, mm-maint, peterx, pezhang, rt-maint, rt-qe, virt-maint, williams |
Version: | 8.2 | Keywords: | ZStream |
Target Milestone: | rc | ||
Target Release: | 8.0 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | kernel-rt-4.18.0-193.31.1.rt13.81.el8_2 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2020-12-15 08:33:04 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 1825271 |
Description
Nitesh Narayan Lal
2020-09-18 17:10:31 UTC
The reported issue has been introduced after the introduction of the patch: "mm/SLUB: delay giving back empty slubs to IRQ enabled regions" This particular issue gets triggered because of a check that was introduced with the above-mentioned patch and is meant to verify if the slub is present on the CPU before invoking the free_delayed(). In situations where a CPU already flushes the slub, this check can cause de-referencing of already released kmem_cache object, that leads to the issue that has been reported in this Bug. The patch that is meant to fix this issue is: "mm: slub: Always flush the delayed empty slubs in flush_all()" This patch is already present in 8.3 RT hence the issue is not reproducible with it. The problematic patch is not available in rhel and hence the issue is not present in that. Did a quick test and it seems the issue is not reproducible with the above-mentioned fix included in the latest 8.2.z. I will run some more tests to confirm. The kernel panic that indicated the root cause of the issue: [ 1184.592053] BUG: unable to handle kernel NULL pointer dereference at 0000000000000b28 [ 1184.592055] PGD 0 P4D 0 [ 1184.592057] Oops: 0000 [#1] PREEMPT SMP PTI [ 1184.592059] CPU: 1 PID: 2888 Comm: libvirtd Kdump: loaded Not tainted 4.18.0-193.19.1.rt13.70.el8_2.x86_64 #1 [ 1184.592059] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.5.4 10/05/2015 [ 1184.592064] RIP: 0010:__free_slab+0x1a1/0x470 [ 1184.592065] Code: 00 8b 15 be 87 04 01 48 c1 e8 36 48 8b 04 c5 60 a6 04 86 4c 8d b0 c0 9f 02 00 85 d2 7e 14 48 63 90 70 9e 02 00 48 8b 74 24 08 <4c> 8b b4 d6 20 0b 00 00 49 39 86 80 00 00 00 0f 85 82 02 00 00 41 [ 1184.592066] RSP: 0018:ffff995447e479e8 EFLAGS: 00010202 [ 1184.592067] RAX: ffff891fbffd4000 RBX: 0000000000000008 RCX: 0000000000000003 [ 1184.592067] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff891f99336600 [ 1184.592068] RBP: fffffffffffffff8 R08: ffffe924d8e78b48 R09: 00000000f0000080 [ 1184.592068] R10: 0000000000000000 R11: 0000000000000001 R12: ffff891f99336600 [ 1184.592069] R13: ffffe924e1647800 R14: ffff891fbfffdfc0 R15: ffff891ba256f200 [ 1184.592070] FS: 00007f8417720700(0000) GS:ffff891fafa00000(0000) knlGS:0000000000000000 [ 1184.592071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1184.592071] CR2: 0000000000000b28 CR3: 000000085928a006 CR4: 00000000001606e0 [ 1184.592072] Call Trace: [ 1184.592076] free_delayed+0x61/0x80 [ 1184.592079] ? __alloc_file+0x2a/0x130 [ 1184.592081] __slab_alloc.isra.82+0x87/0xb0 [ 1184.592082] ? __alloc_file+0x2a/0x130 [ 1184.592084] kmem_cache_alloc+0x111/0x1d0 [ 1184.592085] __alloc_file+0x2a/0x130 [ 1184.592087] alloc_empty_file+0x43/0xc0 [ 1184.592090] ? atime_needs_update+0x77/0xe0 Steps: 1. Start RT guest 2. Shutdown RT guest 3. Re-load kvm_intel with disabling PREEMPTION_TIMER # modprobe -r kvm_intel # modprobe -r kvm # modprobe kvm # modprobe kvm_intel PREEMPTION_TIMER=0 4. Start RT guest == Reproduced with 4.18.0-193.23.1.rt13.73.el8_2.x86_64: After step 4, RT host crash as below: [ 372.842343] BUG: unable to handle kernel NULL pointer dereference at 0000000000000b28 [ 372.842345] PGD 0 P4D 0 [ 372.842348] Oops: 0000 [#1] PREEMPT SMP PTI [ 372.842350] CPU: 8 PID: 2002 Comm: in:imjournal Kdump: loaded Not tainted 4.18.0-193.23.1.rt13.73.el8_2.x86_64 #1 [ 372.842351] Hardware name: Dell Inc. PowerEdge R430/0CN7X8, BIOS 2.0.1 04/11/2016 [ 372.842358] RIP: 0010:__free_slab+0x1a1/0x470 [ 372.842360] Code: 00 8b 15 be 87 04 01 48 c1 e8 36 48 8b 04 c5 60 a6 44 87 4c 8d b0 c0 9f 02 00 85 d2 7e 14 48 63 90 70 9e 02 00 48 8b 74 24 08 <4c> 8b b4 d6 20 0b 00 00 49 39 86 80 00 00 00 0f 85 82 02 00 00 41 [ 372.842361] RSP: 0018:ffffb176cceb3b98 EFLAGS: 00010202 [ 372.842362] RAX: ffff98a97ffd4000 RBX: 0000000000000001 RCX: dead000000000200 [ 372.842362] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff989a027c1600 [ 372.842363] RBP: ffffffffffffffff R08: ffffd86841648108 R09: 0000000000000006 [ 372.842364] R10: ffff98a13e52c180 R11: 0000000000000000 R12: ffff989a027c1600 [ 372.842364] R13: ffffd86841667500 R14: ffff98a97fffdfc0 R15: ffff98a15a201200 [ 372.842366] FS: 00007effb34b4700(0000) GS:ffff98a15f800000(0000) knlGS:0000000000000000 [ 372.842366] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 372.842367] CR2: 0000000000000b28 CR3: 0000000843474003 CR4: 00000000001606e0 [ 372.842368] Call Trace: [ 372.842374] free_delayed+0x61/0x80 [ 372.842379] ? __alloc_file+0x2a/0x130 [ 372.842380] __slab_alloc.isra.82+0x87/0xb0 [ 372.842382] ? __alloc_file+0x2a/0x130 [ 372.842384] kmem_cache_alloc+0x111/0x1d0 [ 372.842386] __alloc_file+0x2a/0x130 [ 372.842388] alloc_empty_file+0x43/0xc0 [ 372.842392] path_openat+0x53/0x14d0 [ 372.842397] ? ttwu_do_wakeup+0x19/0x1a0 [ 372.842400] ? _raw_spin_unlock_irqrestore+0x20/0x60 [ 372.842402] ? try_to_wake_up+0x227/0x6c0 [ 372.842404] ? migrate_enable+0x123/0x3a0 [ 372.842406] do_filp_open+0x93/0x100 [ 372.842409] ? preempt_count_add+0x5a/0xb0 [ 372.842411] ? migrate_enable+0x123/0x3a0 [ 372.842412] ? rt_spin_unlock+0x23/0x40 [ 372.842415] ? inotify_read+0x1d6/0x440 [ 372.842418] ? __check_object_size+0xae/0x166 [ 372.842419] ? rt_spin_unlock+0x23/0x40 [ 372.842422] do_sys_open+0x184/0x220 [ 372.842427] do_syscall_64+0x87/0x1a0 [ 372.842428] entry_SYSCALL_64_after_hwframe+0x65/0xca [ 372.842431] RIP: 0033:0x7effb76fe386 [ 372.842433] Code: 89 54 24 08 e8 7b f4 ff ff 8b 74 24 0c 48 8b 3c 24 41 89 c0 44 8b 54 24 08 b8 01 01 00 00 89 f2 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 30 44 89 c7 89 44 24 08 e8 a6 f4 ff ff 8b 44 [ 372.842433] RSP: 002b:00007effb34b3700 EFLAGS: 00000293 ORIG_RAX: 0000000000000101 [ 372.842435] RAX: ffffffffffffffda RBX: 00000000ffffffff RCX: 00007effb76fe386 [ 372.842435] RDX: 0000000000080800 RSI: 00007effb34b3860 RDI: 00000000ffffff9c [ 372.842436] RBP: 00007effb34b3930 R08: 0000000000000000 R09: 0000000000000002 [ 372.842437] R10: 0000000000000000 R11: 0000000000000293 R12: 00007effac0083e0 [ 372.842437] R13: 00007effb34b3860 R14: 00000000b34b3800 R15: 00007effb34b38d0 [ 372.842439] Modules linked in: kvm_intel kvm irqbypass vhost_net vhost tap xt_CHECKSUM ipt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 nft_compat nft_chain_route_ipv6 nft_chain_nat_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nft_counter nft_chain_route_ipv4 nft_chain_nat_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack nf_tables nfnetlink tun bridge stp llc intel_rapl_msr iTCO_wdt iTCO_vendor_support dcdbas intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp intel_cstate intel_uncore intel_rapl_perf pcspkr ipmi_ssif ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter mei_me lpc_ich mei ip_tables xfs libcrc32c sd_mod sg mxm_wmi crct10dif_pclmul crc32_pclmul crc32c_intel mgag200 drm_vram_helper i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ghash_clmulni_intel drm tg3 megaraid_sas ahci libahci libata wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod [last unloaded: irqbypass] [ 372.842470] CR2: 0000000000000b28 So this issue has been reproduced. == Verified with 4.18.0-193.32.1.rt13.82.el8_2.x86_64: After step 4, RT host keeps working well. So this bug has been fixed very well. Move to 'Verified'. (In reply to Pei Zhang from comment #18) > Steps: > 1. Start RT guest > > 2. Shutdown RT guest > > 3. Re-load kvm_intel with disabling PREEMPTION_TIMER > # modprobe -r kvm_intel > # modprobe -r kvm > # modprobe kvm > # modprobe kvm_intel PREEMPTION_TIMER=0 If just reload kvm_intel like below can also reproduce this issue. # modprobe kvm_intel > > 4. Start RT guest Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: kernel-rt security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5428 |