Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1717528

Summary: rmmod kvdo trace: remove_proc_entry: removing non-empty directory '/proc/vdo', leaking at least 'vdo0'
Product: Red Hat Enterprise Linux 8 Reporter: Jakub Krysl <jkrysl>
Component: kmod-kvdoAssignee: Ken Raeburn <raeburn>
Status: CLOSED DUPLICATE QA Contact: Filip Suba <fsuba>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.0CC: awalsh, corwin
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-09-29 18:04:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
command sequence showing the ioctl error none

Description Jakub Krysl 2019-06-05 18:00:09 UTC 
Created attachment 1577645 [details]
command sequence showing the ioctl error

Description of problem:

Removing kvdo module after reproducer in BZ 1678761 #c1 results in following trace:
[ 2292.362845] ------------[ cut here ]------------
[ 2292.386548] remove_proc_entry: removing non-empty directory '/proc/vdo', leaking at least 'vdo0'
[ 2292.431327] WARNING: CPU: 3 PID: 4712 at fs/proc/generic.c:681 remove_proc_entry+0x140/0x160
[ 2292.469944] Modules linked in: kvdo(OE-) uds(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc dm_multipath intel_rapl sb_edac x86_pkg_temp_thermal iTCO_wdt intel_powerclamp iTCO_vendor_support coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ipmi_ssif ghash_clmulni_intel intel_cstate intel_uncore intel_rapl_perf pcspkr i2c_i801 sg hpilo lpc_ich hpwdt ipmi_si wmi ioatdma ipmi_devintf ipmi_msghandler acpi_power_meter acpi_tad xfs libcrc32c sd_mod mgag200 i2c_algo_bit drm_kms_helper ahci libahci syscopyarea sysfillrect sysimgblt fb_sys_fops libata ttm crc32c_intel drm serio_raw ixgbe tg3 mdio dca dm_mirror dm_region_hash dm_log dm_mod [last unloaded: kvdo]
[ 2292.756856] CPU: 3 PID: 4712 Comm: rmmod Kdump: loaded Tainted: G        W  OE    --------- -  - 4.18.0-100.el8.x86_64 #1
[ 2292.808631] Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 10/25/2017
[ 2292.850294] RIP: 0010:remove_proc_entry+0x140/0x160
[ 2292.874947] Code: 97 48 c7 c7 b8 45 0a 98 48 85 c0 48 8d 50 80 48 0f 45 c2 48 8b 53 70 4c 8b 80 98 00 00 00 48 8b 92 98 00 00 00 e8 da 26 d8 ff <0f> 0b eb 89 48 89 ee 48 c7 c7 fd 44 0a 98 e8 c7 26 d8 ff 0f 0b e9
[ 2292.968401] RSP: 0018:ffffc2afc191fea0 EFLAGS: 00010286
[ 2292.993940] RAX: 0000000000000000 RBX: ffff9f937549a800 RCX: 0000000000000000
[ 2293.029149] RDX: ffff9f9377aded80 RSI: ffff9f9377ad6958 RDI: ffff9f9377ad6958
[ 2293.064812] RBP: ffff9f937549a878 R08: 00000000000004f7 R09: 0000000000000005
[ 2293.098762] R10: 0000000000000000 R11: ffffffff98a23b0d R12: ffffffff98306d00
[ 2293.132602] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 2293.166952] FS:  00007f98c63d1740(0000) GS:ffff9f9377ac0000(0000) knlGS:0000000000000000
[ 2293.204706] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2293.231105] CR2: 00007f98c5300f50 CR3: 0000000238432002 CR4: 00000000001606e0
[ 2293.263877] Call Trace:
[ 2293.275368]  vdoDestroyProcfs+0x13/0x20 [kvdo]
[ 2293.295912]  vdoDestroy+0x3c/0x73 [kvdo]
[ 2293.313804]  __x64_sys_delete_module+0x139/0x270
[ 2293.334891]  do_syscall_64+0x5b/0x1b0
[ 2293.351884]  entry_SYSCALL_64_after_hwframe+0x65/0xca
[ 2293.375092] RIP: 0033:0x7f98c53c1c2b
[ 2293.391488] Code: 73 01 c3 48 8b 0d 5d 02 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 2d 02 2c 00 f7 d8 64 89 01 48
[ 2293.483472] RSP: 002b:00007ffcbc17a038 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0
[ 2293.518282] RAX: ffffffffffffffda RBX: 00005575ab2cc800 RCX: 00007f98c53c1c2b
[ 2293.551312] RDX: 000000000000000a RSI: 0000000000000800 RDI: 00005575ab2cc868
[ 2293.584107] RBP: 0000000000000000 R08: 00007ffcbc178fb1 R09: 0000000000000000
[ 2293.617335] R10: 00007f98c5434020 R11: 0000000000000206 R12: 00007ffcbc17a260
[ 2293.650085] R13: 00007ffcbc17b3ac R14: 00005575ab2cc260 R15: 00005575ab2cc800
[ 2293.683097] ---[ end trace 6c014f6f2577f5f8 ]---

The first time reproduced BZ 1678761, any subsequent run of the reproducer failed to reproduce BZ 1678761 and instead failed with ioctl error and rmmod was required, which reproduced this trace.

Attaching command sequence to show how the interruption of part 3) of reproducer causes the trouble. I can crete the device as many times as I like, but as long as I remove it and create anew while getting its status, I get stuck.

Version-Release number of selected component (if applicable):
kernel-4.18.0-100.el8.x86_64
kmod-kvdo-6.2.1.48-53.el8.x86_64

How reproducible:
100%

Steps to Reproduce:
0) vdoformat /dev/mapper/rhel_storageqe--74-home
1) sudo dmsetup create vdo0 -n
2) sudo dmsetup load vdo0 --table "0 853475328 vdo V2 /dev/mapper/rhel_storageqe--74-home 106684416 4096 32768 16380 off sync vdo0 hash 1 logical 3 physical 2"
3) sleep 1 && sudo dmsetup load vdo0 --table "0 41943040 vdo V2 /dev/mapper/rhel_storageqe--74-home 106684416 4096 32768 16380 off sync vdo0 hash 1 logical 3 physical 2" & for i in {1..1000}; do sudo dmsetup status vdo0 --inactive; done;
4) Interrupt (CTRL+C) after ~10 seconds so the 'dmsetup status vdo0 --inactive' is running
5) sudo dmsetup remove vdo0
6) same as 1)
7) same as 2)
<crash, BZ 1678761>
8) repeat from 1) to 7)
<mentioned call trace, 8) is repeatable>

Actual results:
call trace
ioctl error when trying to load the table

Expected results:
no call trace
being able to load the table again

Additional info:
Comment 1 Andy Walsh 2019-06-06 14:31:26 UTC 
I was able to reproduce the stack trace mentioned in the description when I run 'modprobe -r kvdo', but I also get another stack trace.

[  132.474751] kvdo: modprobe: unloaded version 6.2.1.48
[  132.475699] uds: modprobe: assertion "kmalloc memory used (40 bytes in 2 blocks) is returned to the kernel" (memoryStats.kmallocBytes == 0) failed at /builddir/build/BUILD/kvdo-8834a38068c5258584513541b965538fecdf6b80/obj/./uds/memoryLinuxKernel.c:413
[  132.479788] uds: modprobe: [backtrace]
[  132.480323] CPU: 0 PID: 1530 Comm: modprobe Kdump: loaded Tainted: G        W  OE    --------- -  - 4.18.0-100.el8.x86_64 #1
[  132.481922] Hardware name: Red Hat KVM, BIOS 1.10.2-3.el7_4.1 04/01/2014
[  132.482896] Call Trace:
[  132.483275]  dump_stack+0x5c/0x80
[  132.483771]  assertionFailedLogOnly+0x49/0x70 [uds]
[  132.484544]  ? kobject_del.part.5+0x2b/0x40
[  132.485183]  memoryExit+0xb5/0xf0 [uds]
[  132.485738]  dedupeExit+0xf/0x174 [uds]
[  132.486432]  __x64_sys_delete_module+0x139/0x270
[  132.487092]  do_syscall_64+0x5b/0x1b0
[  132.487596]  entry_SYSCALL_64_after_hwframe+0x65/0xca
[  132.488141] RIP: 0033:0x7f74ea1e53bb
[  132.488509] Code: 73 01 c3 48 8b 0d cd 0a 2c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 9d 0a 2c 00 f7 d8 64 89 01 48
[  132.490248] RSP: 002b:00007ffd8cc450d8 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0
[  132.490924] RAX: ffffffffffffffda RBX: 00005594d5492dd0 RCX: 00007f74ea1e53bb
[  132.491842] RDX: 0000000000000000 RSI: 0000000000000800 RDI: 00005594d5492e38
[  132.492781] RBP: 00005594d5492e38 R08: 00007ffd8cc44081 R09: 0000000000000000
[  132.493719] R10: 00007f74ea2579c0 R11: 0000000000000206 R12: 00005594d5493240
[  132.494662] R13: 0000000000000000 R14: 00005594d5492c68 R15: 00007ffd8cc46500
[  132.495676] uds: modprobe: unloaded version 7.0.0.15


If I modprobe the kvdo kernel back in, I'm able to start over again on the reproduction steps.
Comment 4 Jakub Krysl 2019-10-15 14:38:54 UTC 
Mass migration to Filip.
Comment 5 corwin 2020-09-29 18:04:55 UTC 

*** This bug has been marked as a duplicate of bug 1678761 ***
Comment 7 Red Hat Bugzilla 2023-09-15 00:17:02 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days