RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1880502 - optimization for delaying the freeing of empty slubs causes a NULL pointer dereference [rhel-8.2.0.z]
Summary: optimization for delaying the freeing of empty slubs causes a NULL pointer de...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: kernel-rt
Version: 8.2
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: rc
: 8.0
Assignee: Nitesh Narayan Lal
QA Contact: Pei Zhang
URL:
Whiteboard:
Depends On:
Blocks: 1825271
TreeView+ depends on / blocked
 
Reported: 2020-09-18 17:10 UTC by Nitesh Narayan Lal
Modified: 2023-08-08 02:59 UTC (History)
15 users (show)

Fixed In Version: kernel-rt-4.18.0-193.31.1.rt13.81.el8_2
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-12-15 08:33:04 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Nitesh Narayan Lal 2020-09-18 17:10:31 UTC 
Description of problem:
Disabling PREEMPTION TIMER and restarting a VM after the initial shutdown
somehow ends up restarting the entire host.

Version-Release number of selected component (if applicable):

How reproducible:
Every time


Steps to Reproduce:
1.Boot a host with the latest 8.2.z rt kernel
2.Install a VM and bring it up.
3.Shutdown the VM and reload the kvm_intel module with PREEMPTION_TIMER
  disabled.
4.Start the VM again.

Actual results:
Host restart.

Expected results:
VM should come up and the host should continue without any issues.

Additional info:
* This issue is not triggered if I don't start the VM after host bringup and
  just reload kvm_intel with disabled preemption timer followed by the VM 
  start.

* I haven't been able to reproduce this with rhel. It would be worthwhile to
  see if this issue is also reproducible with the latest 8.3 kernel-rt.

Host cmdline:
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-193.23.1.rt13.73.el8_2.x86_64 root=/dev/mapper/rhel_virtlab500-root ro crashkernel=auto resume=/dev/mapper/rhel_virtlab500-swap rd.lvm.lv=rhel_virtlab500/root rd.lvm.lv=rhel_virtlab500/swap console=ttyS1,115200

Virt-install cmd:
virt-install -n RHEL8-RT --os-variant=rhel8.0    --memory=4000 --vcpus=3,cpuset=1,2,3 --disk path=/home/VMs/rhel8-rt.img,bus=virtio,cache=none,format=raw,io=threads,size=30   --graphics type=spice,listen=0.0.0.0  -l http://download.eng.bos.redhat.com/rel-eng/rhel-8/RHEL-8/latest-RHEL-8.2/compose/BaseOS/x86_64/os/
Comment 1 Nitesh Narayan Lal 2020-09-22 00:48:52 UTC 
The reported issue has been introduced after the introduction of the patch:
"mm/SLUB: delay giving back empty slubs to IRQ enabled regions"

This particular issue gets triggered because of a check that was introduced
with the above-mentioned patch and is meant to verify if the slub is present
on the CPU before invoking the free_delayed().
In situations where a CPU already flushes the slub, this check can cause
de-referencing of already released kmem_cache object, that leads to the issue
that has been reported in this Bug.

The patch that is meant to fix this issue is:
"mm: slub: Always flush the delayed empty slubs in flush_all()"

This patch is already present in 8.3 RT hence the issue is not reproducible 
with it.

The problematic patch is not available in rhel and hence the issue is not
present in that. Did a quick test and it seems the issue is not reproducible
with the above-mentioned fix included in the latest 8.2.z.

I will run some more tests to confirm.

The kernel panic that indicated the root cause of the issue:

[ 1184.592053] BUG: unable to handle kernel NULL pointer dereference at 0000000000000b28
[ 1184.592055] PGD 0 P4D 0
[ 1184.592057] Oops: 0000 [#1] PREEMPT SMP PTI
[ 1184.592059] CPU: 1 PID: 2888 Comm: libvirtd Kdump: loaded Not tainted 4.18.0-193.19.1.rt13.70.el8_2.x86_64 #1
[ 1184.592059] Hardware name: Dell Inc. PowerEdge R430/03XKDV, BIOS 1.5.4 10/05/2015
[ 1184.592064] RIP: 0010:__free_slab+0x1a1/0x470
[ 1184.592065] Code: 00 8b 15 be 87 04 01 48 c1 e8 36 48 8b 04 c5 60 a6 04 86 4c 8d b0 c0 9f 02 00 85 d2 7e 14 48 63 90 70 9e 02 00 48 8b 74 24 08 <4c> 8b b4 d6 20 0b 00 00 49 39 86 80 00 00 00 0f 85 82 02 00 00 41
[ 1184.592066] RSP: 0018:ffff995447e479e8 EFLAGS: 00010202
[ 1184.592067] RAX: ffff891fbffd4000 RBX: 0000000000000008 RCX: 0000000000000003
[ 1184.592067] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff891f99336600
[ 1184.592068] RBP: fffffffffffffff8 R08: ffffe924d8e78b48 R09: 00000000f0000080
[ 1184.592068] R10: 0000000000000000 R11: 0000000000000001 R12: ffff891f99336600
[ 1184.592069] R13: ffffe924e1647800 R14: ffff891fbfffdfc0 R15: ffff891ba256f200
[ 1184.592070] FS:  00007f8417720700(0000) GS:ffff891fafa00000(0000) knlGS:0000000000000000
[ 1184.592071] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1184.592071] CR2: 0000000000000b28 CR3: 000000085928a006 CR4: 00000000001606e0
[ 1184.592072] Call Trace:
[ 1184.592076]  free_delayed+0x61/0x80
[ 1184.592079]  ? __alloc_file+0x2a/0x130
[ 1184.592081]  __slab_alloc.isra.82+0x87/0xb0
[ 1184.592082]  ? __alloc_file+0x2a/0x130
[ 1184.592084]  kmem_cache_alloc+0x111/0x1d0
[ 1184.592085]  __alloc_file+0x2a/0x130
[ 1184.592087]  alloc_empty_file+0x43/0xc0
[ 1184.592090]  ? atime_needs_update+0x77/0xe0
Comment 18 Pei Zhang 2020-11-15 11:42:18 UTC 
Steps:
1. Start RT guest

2. Shutdown RT guest

3. Re-load kvm_intel with disabling PREEMPTION_TIMER
# modprobe -r kvm_intel
# modprobe -r kvm
# modprobe kvm
# modprobe kvm_intel PREEMPTION_TIMER=0

4. Start RT guest


== Reproduced with 4.18.0-193.23.1.rt13.73.el8_2.x86_64:

After step 4, RT host crash as below:

[  372.842343] BUG: unable to handle kernel NULL pointer dereference at 0000000000000b28
[  372.842345] PGD 0 P4D 0 
[  372.842348] Oops: 0000 [#1] PREEMPT SMP PTI
[  372.842350] CPU: 8 PID: 2002 Comm: in:imjournal Kdump: loaded Not tainted 4.18.0-193.23.1.rt13.73.el8_2.x86_64 #1
[  372.842351] Hardware name: Dell Inc. PowerEdge R430/0CN7X8, BIOS 2.0.1 04/11/2016
[  372.842358] RIP: 0010:__free_slab+0x1a1/0x470
[  372.842360] Code: 00 8b 15 be 87 04 01 48 c1 e8 36 48 8b 04 c5 60 a6 44 87 4c 8d b0 c0 9f 02 00 85 d2 7e 14 48 63 90 70 9e 02 00 48 8b 74 24 08 <4c> 8b b4 d6 20 0b 00 00 49 39 86 80 00 00 00 0f 85 82 02 00 00 41
[  372.842361] RSP: 0018:ffffb176cceb3b98 EFLAGS: 00010202
[  372.842362] RAX: ffff98a97ffd4000 RBX: 0000000000000001 RCX: dead000000000200
[  372.842362] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff989a027c1600
[  372.842363] RBP: ffffffffffffffff R08: ffffd86841648108 R09: 0000000000000006
[  372.842364] R10: ffff98a13e52c180 R11: 0000000000000000 R12: ffff989a027c1600
[  372.842364] R13: ffffd86841667500 R14: ffff98a97fffdfc0 R15: ffff98a15a201200
[  372.842366] FS:  00007effb34b4700(0000) GS:ffff98a15f800000(0000) knlGS:0000000000000000
[  372.842366] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  372.842367] CR2: 0000000000000b28 CR3: 0000000843474003 CR4: 00000000001606e0
[  372.842368] Call Trace:
[  372.842374]  free_delayed+0x61/0x80
[  372.842379]  ? __alloc_file+0x2a/0x130
[  372.842380]  __slab_alloc.isra.82+0x87/0xb0
[  372.842382]  ? __alloc_file+0x2a/0x130
[  372.842384]  kmem_cache_alloc+0x111/0x1d0
[  372.842386]  __alloc_file+0x2a/0x130
[  372.842388]  alloc_empty_file+0x43/0xc0
[  372.842392]  path_openat+0x53/0x14d0
[  372.842397]  ? ttwu_do_wakeup+0x19/0x1a0
[  372.842400]  ? _raw_spin_unlock_irqrestore+0x20/0x60
[  372.842402]  ? try_to_wake_up+0x227/0x6c0
[  372.842404]  ? migrate_enable+0x123/0x3a0
[  372.842406]  do_filp_open+0x93/0x100
[  372.842409]  ? preempt_count_add+0x5a/0xb0
[  372.842411]  ? migrate_enable+0x123/0x3a0
[  372.842412]  ? rt_spin_unlock+0x23/0x40
[  372.842415]  ? inotify_read+0x1d6/0x440
[  372.842418]  ? __check_object_size+0xae/0x166
[  372.842419]  ? rt_spin_unlock+0x23/0x40
[  372.842422]  do_sys_open+0x184/0x220
[  372.842427]  do_syscall_64+0x87/0x1a0
[  372.842428]  entry_SYSCALL_64_after_hwframe+0x65/0xca
[  372.842431] RIP: 0033:0x7effb76fe386
[  372.842433] Code: 89 54 24 08 e8 7b f4 ff ff 8b 74 24 0c 48 8b 3c 24 41 89 c0 44 8b 54 24 08 b8 01 01 00 00 89 f2 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 77 30 44 89 c7 89 44 24 08 e8 a6 f4 ff ff 8b 44
[  372.842433] RSP: 002b:00007effb34b3700 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[  372.842435] RAX: ffffffffffffffda RBX: 00000000ffffffff RCX: 00007effb76fe386
[  372.842435] RDX: 0000000000080800 RSI: 00007effb34b3860 RDI: 00000000ffffff9c
[  372.842436] RBP: 00007effb34b3930 R08: 0000000000000000 R09: 0000000000000002
[  372.842437] R10: 0000000000000000 R11: 0000000000000293 R12: 00007effac0083e0
[  372.842437] R13: 00007effb34b3860 R14: 00000000b34b3800 R15: 00007effb34b38d0
[  372.842439] Modules linked in: kvm_intel kvm irqbypass vhost_net vhost tap xt_CHECKSUM ipt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 nft_compat nft_chain_route_ipv6 nft_chain_nat_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nft_counter nft_chain_route_ipv4 nft_chain_nat_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack nf_tables nfnetlink tun bridge stp llc intel_rapl_msr iTCO_wdt iTCO_vendor_support dcdbas intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp intel_cstate intel_uncore intel_rapl_perf pcspkr ipmi_ssif ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter mei_me lpc_ich mei ip_tables xfs libcrc32c sd_mod sg mxm_wmi crct10dif_pclmul crc32_pclmul crc32c_intel mgag200 drm_vram_helper i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ghash_clmulni_intel drm tg3 megaraid_sas ahci libahci libata wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod [last unloaded: irqbypass]
[  372.842470] CR2: 0000000000000b28


So this issue has been reproduced.

== Verified with 4.18.0-193.32.1.rt13.82.el8_2.x86_64:

After step 4, RT host keeps working well.

So this bug has been fixed very well.

Move to 'Verified'.
Comment 19 Pei Zhang 2020-11-15 11:50:29 UTC 
(In reply to Pei Zhang from comment #18)
> Steps:
> 1. Start RT guest
> 
> 2. Shutdown RT guest
> 
> 3. Re-load kvm_intel with disabling PREEMPTION_TIMER
> # modprobe -r kvm_intel
> # modprobe -r kvm
> # modprobe kvm
> # modprobe kvm_intel PREEMPTION_TIMER=0

If just reload kvm_intel like below can also reproduce this issue.

# modprobe kvm_intel 

> 
> 4. Start RT guest
Comment 23 errata-xmlrpc 2020-12-15 08:33:04 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: kernel-rt security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5428
Note You need to log in before you can comment on or make changes to this bug.