Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Well, first off, you can't remove a partition device with
# multipath -f
Those aren't multipath devices. They are supposed to be linear devices. Multipath should automatically remove them when you try to remove the multipath device that they are part of. However, that won't work here, because your device isn't a linear device..
Looking at the dmsetup table output, I see.
mpathdtp1: 0 571078492 error
It also clearly doesn't depend on the mpathdt device anymore, and you can see by noting that mpathdt's open count is 0
mpathdt 253 67 L--w 0 1 9 mpath-360000970000192606259533031444338
So, clearly, something outside of multipath has been mucking around with this device table. Also, the open count for a large number of your devices is huge. Do you have any idea what could be holding these devices open thousands of times? Clearly this isn't a multipath problem. It might be a kernel issue, if these devices really havn't been opened thousands of times. But unless the kernel says that the device is not in-use, userspace device-mapper and multipath can do nothing.
Created attachment 1200447[details]
dmsetup info
Copy from pastebin.
Note very large open count number i.e.:
mpathtp1 253 244 L--w 4743 1 0 part1-mpath-360000970000192606259533031343736
(In reply to Ben Marzinski from comment #6)
> Well, first off, you can't remove a partition device with
>
> # multipath -f
>
> Those aren't multipath devices. They are supposed to be linear devices.
> Multipath should automatically remove them when you try to remove the
> multipath device that they are part of. However, that won't work here,
> because your device isn't a linear device..
>
> Looking at the dmsetup table output, I see.
>
> mpathdtp1: 0 571078492 error
This is the result for the 'dmsetup remove -f' operation which is not subject of this BZ (I've already tried to explain them in other BZ to not do that)
It should only focus on too large open count which effectively will prevent any possible device removal.
> thousands of times? Clearly this isn't a multipath problem. It might be a
> kernel issue, if these devices really havn't been opened thousands of times.
> But unless the kernel says that the device is not in-use, userspace
> device-mapper and multipath can do nothing.
Yep - I'd expect kernel device driver issue.
But as already requested - I'd like to see reproduction with last available RHEL6 kernel - whether it had been already fixed or the issue is still present.
Comment 11nikhil kshirsagar
2016-09-15 03:17:23 UTC
Although sysrq dump looks fairly incomplete - from the end it looks like
there is a possible use of some 'block-device' mapped over the file-system on a block devices ?
Is there such use by customer ?
Red Hat Enterprise Linux 6 transitioned to the Production 3 Phase on May 10, 2017. During the Production 3 Phase, Critical impact Security Advisories (RHSAs) and selected Urgent Priority Bug Fix Advisories (RHBAs) may be released as they become available.
The official life cycle policy can be reviewed here:
http://redhat.com/rhel/lifecycle
This issue does not appear to meet the inclusion criteria for the Production Phase 3 and will be marked as CLOSED/WONTFIX. If this remains a critical requirement, please contact Red Hat Customer Support to request a re-evaluation of the issue, citing a clear business justification. Red Hat Customer Support can be contacted via the Red Hat Customer Portal at the following URL:
https://access.redhat.com
Comment 17Red Hat Bugzilla
2023-09-14 03:30:47 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days
Description of problem: device-mapper doesnt remove a device even though nothings holding it. Open count is a huge number (3531) Version-Release number of selected component (if applicable): cisco-device-mapper-config-1.1-8.rhel6.noarch Wed Aug 20 16:02:48 2014 device-mapper-1.02.79-8.el6.x86_64 Sun Feb 8 11:51:57 2015 device-mapper-event-1.02.79-8.el6.x86_64 Sun Feb 8 11:52:02 2015 device-mapper-event-libs-1.02.79-8.el6.x86_64 Sun Feb 8 11:51:57 2015 device-mapper-libs-1.02.79-8.el6.i686 Sun Feb 8 11:54:07 2015 device-mapper-libs-1.02.79-8.el6.x86_64 Sun Feb 8 11:51:57 2015 device-mapper-multipath-0.4.9-72.el6_5.2.x86_64 Sun Feb 8 11:53:39 2015 device-mapper-multipath-libs-0.4.9-72.el6_5.2.x86_64 Sun Feb 8 11:52:05 2015 device-mapper-persistent-data-0.2.8-4.el6_5.x86_64 Sun Feb 8 11:51:48 2015 How reproducible: Checking with customer how they got into this situation and if they can reproduce this. Additional info: I have an active case where the customer is trying [root@lnxdb-prd-890 ~]# dmsetup remove -f mpathdtp1 device-mapper: remove ioctl on mpathdtp1 failed: Device or resource busy Command failed We're trying to figure what is keeping the device busy. The device has been released from userspace by asm, but devicemapper still holds a reference to it and its marked as live. [root@lnxdb-prd-891 ~]# dmsetup info -C | grep mpathdt mpathdt 253 67 L--w 0 1 9 mpath-360000970000192606259533031444338 mpathdtp1 253 149 L--w 3531 1 1 part1-mpath-360000970000192606259533031444338 lsof shows nothing [nkshirsa@foobar lnxdb-prd-891-2016081517031471305837]$ cat lsof |grep mpathdt [nkshirsa@foobar lnxdb-prd-891-2016081517031471305837]$ But its present in the lvm cache [nkshirsa@foobar lnxdb-prd-891-2016081517031471305837]$ grep -ri mpathdtp1 * etc/lvm/cache/.cache: "/dev/mapper/mpathdtp1", etc/lvm/cache/.cache: "/dev/disk/by-id/dm-name-mpathdtp1", sos_commands/filesys/lsblk:mpathdtp1 (dm-149) 253:149 0 272.3G 0 part sos_commands/devicemapper/dmsetup_table:mpathdtp1: 0 571078492 error sos_commands/devicemapper/dmsetup_status:mpathdtp1: 0 571078492 error We tried asking them to run # dmsetup clear mpathdtp1 # dmsetup remove -f mpathdtp1 --verbose but no good, [root@lnxdb-prd-891 ~]# dmsetup info -c mpathdtp1 Name Maj Min Stat Open Targ Event UUID mpathdtp1 253 149 L--w 3531 1 1 part1-mpath-360000970000192606259533031444338 [root@lnxdb-prd-891 ~]# dmsetup clear mpathdtp1 [root@lnxdb-prd-891 ~]# dmsetup remove -f mpathdtp1 -v device-mapper: remove ioctl on mpathdtp1 failed: Device or resource busy Command failed I'm attaching the output of # strace dmsetup remove -f mpathdtp1 -v as well as lvs -o +devices and /etc/lvm/cache/.cache to the bz. Also, more info if it helps, # cat sos_commands/devicemapper/dmsetup_info_-c |grep mpathdt mpathdt 253 67 L--w 0 1 9 mpath-360000970000192606259533031444338 mpathdtp1 253 149 L--w 3531 1 1 part1-mpath-360000970000192606259533031444338 # grep -ri 360000970000192606259533031444338 * etc/multipath/wwids:/360000970000192606259533031444338/ etc/multipath/bindings:mpathdt 360000970000192606259533031444338 etc/lvm/cache/.cache: "/dev/disk/by-id/dm-uuid-part1-mpath-360000970000192606259533031444338", ------------------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------------------ strace output: [root@lnxdb-prd-891 ~]# strace dmsetup remove -f mpathdtp1 -v execve("/sbin/dmsetup", ["dmsetup", "remove", "-f", "mpathdtp1", "-v"], [/* 24 vars */]) = 0 brk(0) = 0x1b44000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172ff5000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=81470, ...}) = 0 mmap(NULL, 81470, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f1172fe1000 close(3) = 0 open("/lib64/libdevmapper.so.1.02", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@n@\245;\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=245488, ...}) = 0 mmap(0x3ba5400000, 2342304, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3ba5400000 mprotect(0x3ba5438000, 2097152, PROT_NONE) = 0 mmap(0x3ba5638000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x38000) = 0x3ba5638000 mmap(0x3ba563b000, 3488, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x3ba563b000 close(3) = 0 open("/lib64/libudev.so.0", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000)\0\245;\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=57552, ...}) = 0 mmap(0x3ba5000000, 2152544, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3ba5000000 mprotect(0x3ba500c000, 2097152, PROT_NONE) = 0 mmap(0x3ba520c000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xc000) = 0x3ba520c000 close(3) = 0 open("/lib64/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\356\1\244;\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=1926520, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172fe0000 mmap(0x3ba4000000, 3750152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3ba4000000 mprotect(0x3ba418a000, 2097152, PROT_NONE) = 0 mmap(0x3ba438a000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18a000) = 0x3ba438a000 mmap(0x3ba438f000, 18696, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x3ba438f000 close(3) = 0 open("/lib64/libselinux.so.1", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320X\200\245;\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=124624, ...}) = 0 mmap(0x3ba5800000, 2221912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3ba5800000 mprotect(0x3ba581d000, 2093056, PROT_NONE) = 0 mmap(0x3ba5a1c000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c000) = 0x3ba5a1c000 mmap(0x3ba5a1e000, 1880, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x3ba5a1e000 close(3) = 0 open("/lib64/libsepol.so.1", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200>\300\244;\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=251112, ...}) = 0 mmap(0x3ba4c00000, 2345600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3ba4c00000 mprotect(0x3ba4c3b000, 2097152, PROT_NONE) = 0 mmap(0x3ba4e3b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3b000) = 0x3ba4e3b000 close(3) = 0 open("/lib64/libdl.so.2", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\r\200\244;\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=22536, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172fdf000 mmap(0x3ba4800000, 2109696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3ba4800000 mprotect(0x3ba4802000, 2097152, PROT_NONE) = 0 mmap(0x3ba4a02000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x3ba4a02000 close(3) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172fde000 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172fdc000 arch_prctl(ARCH_SET_FS, 0x7f1172fdc7a0) = 0 mprotect(0x3ba520c000, 4096, PROT_READ) = 0 mprotect(0x3ba438a000, 16384, PROT_READ) = 0 mprotect(0x3ba5a1c000, 4096, PROT_READ) = 0 mprotect(0x3ba4e3b000, 4096, PROT_READ) = 0 mprotect(0x3ba3e1f000, 4096, PROT_READ) = 0 mprotect(0x3ba4a02000, 4096, PROT_READ) = 0 munmap(0x7f1172fe1000, 81470) = 0 statfs("/selinux", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=3096364, f_bfree=1942016, f_bavail=1784730, f_files=786432, f_ffree=690935, f_fsid={-57432873, -1917899891}, f_namelen=255, f_frsize=4096}) = 0 brk(0) = 0x1b44000 brk(0x1b65000) = 0x1b65000 open("/proc/filesystems", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172ff4000 read(3, "nodev\tsysfs\nnodev\trootfs\nnodev\tb"..., 1024) = 393 read(3, "", 1024) = 0 close(3) = 0 munmap(0x7f1172ff4000, 4096) = 0 open("/usr/lib/locale/locale-archive", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=99158576, ...}) = 0 mmap(NULL, 99158576, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f116d14b000 close(3) = 0 uname({sys="Linux", node="lnxdb-prd-891", ...}) = 0 open("/proc/devices", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172ff4000 read(3, "Character devices:\n 1 mem\n 4 /"..., 1024) = 535 close(3) = 0 munmap(0x7f1172ff4000, 4096) = 0 open("/proc/misc", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172ff4000 read(3, "220 mptctl\n 56 autofs\n184 microc"..., 1024) = 194 close(3) = 0 munmap(0x7f1172ff4000, 4096) = 0 stat("/dev/mapper/control", {st_mode=S_IFCHR|0660, st_rdev=makedev(10, 58), ...}) = 0 open("/dev/mapper/control", O_RDWR) = 3 open("/proc/devices", O_RDONLY) = 4 fstat(4, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172ff4000 read(4, "Character devices:\n 1 mem\n 4 /"..., 1024) = 535 close(4) = 0 munmap(0x7f1172ff4000, 4096) = 0 ioctl(3, DM_VERSION, 0x1b451b0) = 0 ioctl(3, DM_TABLE_STATUS, 0x1b450e0) = 0 ioctl(3, DM_TABLE_LOAD, 0x1b450e0) = 0 semctl(0, 0, SEM_INFO, 0x7fffb76bc890) = 22 open("/etc/udev/udev.conf", O_RDONLY) = 4 fstat(4, {st_mode=S_IFREG|0644, st_size=218, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172ff4000 read(4, "# The initial syslog(3) priority"..., 4096) = 218 read(4, "", 4096) = 0 close(4) = 0 munmap(0x7f1172ff4000, 4096) = 0 open("/dev/.udev/queue.bin", O_RDONLY) = 4 fstat(4, {st_mode=S_IFREG|0644, st_size=155, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172ff4000 read(4, "QZ,\0\0\0\0\0RZ,\0\0\0\0\0\35\0/devices/virtu"..., 4096) = 155 close(4) = 0 munmap(0x7f1172ff4000, 4096) = 0 open("/dev/urandom", O_RDONLY) = 4 read(4, "o7", 2) = 2 semget(0xd4d376f, 1, IPC_CREAT|IPC_EXCL|0600) = 19300375 semctl(19300375, 0, SETVAL, 0x1) = 0 semctl(19300375, 0, GETVAL, 0xffffffffffffffff) = 1 close(4) = 0 semop(19300375, {{0, 1, 0}}, 1) = 0 semctl(19300375, 0, GETVAL, 0xffffffffffffffff) = 2 ioctl(3, DM_DEV_SUSPEND, 0x1b49250) = 0 semget(0xd4d376f, 1, 0) = 19300375 semctl(19300375, 0, GETVAL, 0xffffffffffffffff) = 2 semop(19300375, {{0, -1, IPC_NOWAIT}}, 1) = 0 semop(19300375, {{0, 0, 0}}, 1) = 0 semctl(19300375, 0, IPC_RMID, 0) = 0 open("/dev/urandom", O_RDONLY) = 4 read(4, "h\316", 2) = 2 semget(0xd4dce68, 1, IPC_CREAT|IPC_EXCL|0600) = 19333143 semctl(19333143, 0, SETVAL, 0x1) = 0 semctl(19333143, 0, GETVAL, 0xffffffffffffffff) = 1 close(4) = 0 semop(19333143, {{0, 1, 0}}, 1) = 0 semctl(19333143, 0, GETVAL, 0xffffffffffffffff) = 2 ioctl(3, DM_DEV_REMOVE, 0x1b45010) = -1 EBUSY (Device or resource busy) open("/usr/share/locale/locale.alias", O_RDONLY) = 4 fstat(4, {st_mode=S_IFREG|0644, st_size=2512, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172ff4000 read(4, "# Locale name alias data base.\n#"..., 4096) = 2512 read(4, "", 4096) = 0 close(4) = 0 munmap(0x7f1172ff4000, 4096) = 0 open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) write(2, "device-mapper: remove ioctl on m"..., 72device-mapper: remove ioctl on mpathdtp1 failed: Device or resource busy) = 72 write(2, "\n", 1 ) = 1 semget(0xd4dce68, 1, 0) = 19333143 semctl(19333143, 0, GETVAL, 0xffffffffffffffff) = 2 semop(19333143, {{0, -1, IPC_NOWAIT}}, 1) = 0 semget(0xd4dce68, 1, 0) = 19333143 semctl(19333143, 0, GETVAL, 0xffffffffffffffff) = 1 semop(19333143, {{0, -1, IPC_NOWAIT}}, 1) = 0 semop(19333143, {{0, 0, 0}}, 1) = 0 semctl(19333143, 0, IPC_RMID, 0) = 0 write(2, "Command failed\n", 15Command failed ) = 15 close(3) = 0 exit_group(1) = ? [root@lnxdb-prd-891 ~]# fuser -kuc /dev/mapper/mpathdtp1 [root@lnxdb-prd-891 ~]#