| Summary: | unable to remove a device using dmsetup remove, huge open count | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | nikhil kshirsagar <nkshirsa> | ||||||||||||
| Component: | device-mapper-multipath | Assignee: | Ben Marzinski <bmarzins> | ||||||||||||
| Status: | CLOSED WONTFIX | QA Contact: | Lin Li <lilin> | ||||||||||||
| Severity: | medium | Docs Contact: | |||||||||||||
| Priority: | unspecified | ||||||||||||||
| Version: | 6.4 | CC: | agk, bmarzins, cww, heinzm, lilin, msnitzer, nkshirsa, prajnoha, rbalakri, zkabelac | ||||||||||||
| Target Milestone: | rc | Flags: | zkabelac:
needinfo?
(nkshirsa) |
||||||||||||
| Target Release: | --- | ||||||||||||||
| Hardware: | Unspecified | ||||||||||||||
| OS: | Unspecified | ||||||||||||||
| Whiteboard: | |||||||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||||||
| Doc Text: | Story Points: | --- | |||||||||||||
| Clone Of: | Environment: | ||||||||||||||
| Last Closed: | 2017-06-07 22:15:15 UTC | Type: | Bug | ||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||
| Documentation: | --- | CRM: | |||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||
| Attachments: |
|
||||||||||||||
Created attachment 1200145 [details]
lvm cache
Created attachment 1200174 [details]
lsof gzipped
Well, first off, you can't remove a partition device with # multipath -f Those aren't multipath devices. They are supposed to be linear devices. Multipath should automatically remove them when you try to remove the multipath device that they are part of. However, that won't work here, because your device isn't a linear device.. Looking at the dmsetup table output, I see. mpathdtp1: 0 571078492 error It also clearly doesn't depend on the mpathdt device anymore, and you can see by noting that mpathdt's open count is 0 mpathdt 253 67 L--w 0 1 9 mpath-360000970000192606259533031444338 So, clearly, something outside of multipath has been mucking around with this device table. Also, the open count for a large number of your devices is huge. Do you have any idea what could be holding these devices open thousands of times? Clearly this isn't a multipath problem. It might be a kernel issue, if these devices really havn't been opened thousands of times. But unless the kernel says that the device is not in-use, userspace device-mapper and multipath can do nothing. Created attachment 1200446 [details]
dmsetup table
(just copy from pastebin)
Created attachment 1200447 [details]
dmsetup info
Copy from pastebin.
Note very large open count number i.e.:
mpathtp1 253 244 L--w 4743 1 0 part1-mpath-360000970000192606259533031343736
(In reply to Ben Marzinski from comment #6) > Well, first off, you can't remove a partition device with > > # multipath -f > > Those aren't multipath devices. They are supposed to be linear devices. > Multipath should automatically remove them when you try to remove the > multipath device that they are part of. However, that won't work here, > because your device isn't a linear device.. > > Looking at the dmsetup table output, I see. > > mpathdtp1: 0 571078492 error This is the result for the 'dmsetup remove -f' operation which is not subject of this BZ (I've already tried to explain them in other BZ to not do that) It should only focus on too large open count which effectively will prevent any possible device removal. > thousands of times? Clearly this isn't a multipath problem. It might be a > kernel issue, if these devices really havn't been opened thousands of times. > But unless the kernel says that the device is not in-use, userspace > device-mapper and multipath can do nothing. Yep - I'd expect kernel device driver issue. But as already requested - I'd like to see reproduction with last available RHEL6 kernel - whether it had been already fixed or the issue is still present. Created attachment 1201078 [details]
sysrq-trigger output
Although sysrq dump looks fairly incomplete - from the end it looks like there is a possible use of some 'block-device' mapped over the file-system on a block devices ? Is there such use by customer ? Red Hat Enterprise Linux 6 transitioned to the Production 3 Phase on May 10, 2017. During the Production 3 Phase, Critical impact Security Advisories (RHSAs) and selected Urgent Priority Bug Fix Advisories (RHBAs) may be released as they become available. The official life cycle policy can be reviewed here: http://redhat.com/rhel/lifecycle This issue does not appear to meet the inclusion criteria for the Production Phase 3 and will be marked as CLOSED/WONTFIX. If this remains a critical requirement, please contact Red Hat Customer Support to request a re-evaluation of the issue, citing a clear business justification. Red Hat Customer Support can be contacted via the Red Hat Customer Portal at the following URL: https://access.redhat.com |
Description of problem: device-mapper doesnt remove a device even though nothings holding it. Open count is a huge number (3531) Version-Release number of selected component (if applicable): cisco-device-mapper-config-1.1-8.rhel6.noarch Wed Aug 20 16:02:48 2014 device-mapper-1.02.79-8.el6.x86_64 Sun Feb 8 11:51:57 2015 device-mapper-event-1.02.79-8.el6.x86_64 Sun Feb 8 11:52:02 2015 device-mapper-event-libs-1.02.79-8.el6.x86_64 Sun Feb 8 11:51:57 2015 device-mapper-libs-1.02.79-8.el6.i686 Sun Feb 8 11:54:07 2015 device-mapper-libs-1.02.79-8.el6.x86_64 Sun Feb 8 11:51:57 2015 device-mapper-multipath-0.4.9-72.el6_5.2.x86_64 Sun Feb 8 11:53:39 2015 device-mapper-multipath-libs-0.4.9-72.el6_5.2.x86_64 Sun Feb 8 11:52:05 2015 device-mapper-persistent-data-0.2.8-4.el6_5.x86_64 Sun Feb 8 11:51:48 2015 How reproducible: Checking with customer how they got into this situation and if they can reproduce this. Additional info: I have an active case where the customer is trying [root@lnxdb-prd-890 ~]# dmsetup remove -f mpathdtp1 device-mapper: remove ioctl on mpathdtp1 failed: Device or resource busy Command failed We're trying to figure what is keeping the device busy. The device has been released from userspace by asm, but devicemapper still holds a reference to it and its marked as live. [root@lnxdb-prd-891 ~]# dmsetup info -C | grep mpathdt mpathdt 253 67 L--w 0 1 9 mpath-360000970000192606259533031444338 mpathdtp1 253 149 L--w 3531 1 1 part1-mpath-360000970000192606259533031444338 lsof shows nothing [nkshirsa@foobar lnxdb-prd-891-2016081517031471305837]$ cat lsof |grep mpathdt [nkshirsa@foobar lnxdb-prd-891-2016081517031471305837]$ But its present in the lvm cache [nkshirsa@foobar lnxdb-prd-891-2016081517031471305837]$ grep -ri mpathdtp1 * etc/lvm/cache/.cache: "/dev/mapper/mpathdtp1", etc/lvm/cache/.cache: "/dev/disk/by-id/dm-name-mpathdtp1", sos_commands/filesys/lsblk:mpathdtp1 (dm-149) 253:149 0 272.3G 0 part sos_commands/devicemapper/dmsetup_table:mpathdtp1: 0 571078492 error sos_commands/devicemapper/dmsetup_status:mpathdtp1: 0 571078492 error We tried asking them to run # dmsetup clear mpathdtp1 # dmsetup remove -f mpathdtp1 --verbose but no good, [root@lnxdb-prd-891 ~]# dmsetup info -c mpathdtp1 Name Maj Min Stat Open Targ Event UUID mpathdtp1 253 149 L--w 3531 1 1 part1-mpath-360000970000192606259533031444338 [root@lnxdb-prd-891 ~]# dmsetup clear mpathdtp1 [root@lnxdb-prd-891 ~]# dmsetup remove -f mpathdtp1 -v device-mapper: remove ioctl on mpathdtp1 failed: Device or resource busy Command failed I'm attaching the output of # strace dmsetup remove -f mpathdtp1 -v as well as lvs -o +devices and /etc/lvm/cache/.cache to the bz. Also, more info if it helps, # cat sos_commands/devicemapper/dmsetup_info_-c |grep mpathdt mpathdt 253 67 L--w 0 1 9 mpath-360000970000192606259533031444338 mpathdtp1 253 149 L--w 3531 1 1 part1-mpath-360000970000192606259533031444338 # grep -ri 360000970000192606259533031444338 * etc/multipath/wwids:/360000970000192606259533031444338/ etc/multipath/bindings:mpathdt 360000970000192606259533031444338 etc/lvm/cache/.cache: "/dev/disk/by-id/dm-uuid-part1-mpath-360000970000192606259533031444338", ------------------------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------------------------ strace output: [root@lnxdb-prd-891 ~]# strace dmsetup remove -f mpathdtp1 -v execve("/sbin/dmsetup", ["dmsetup", "remove", "-f", "mpathdtp1", "-v"], [/* 24 vars */]) = 0 brk(0) = 0x1b44000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172ff5000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=81470, ...}) = 0 mmap(NULL, 81470, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f1172fe1000 close(3) = 0 open("/lib64/libdevmapper.so.1.02", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@n@\245;\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=245488, ...}) = 0 mmap(0x3ba5400000, 2342304, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3ba5400000 mprotect(0x3ba5438000, 2097152, PROT_NONE) = 0 mmap(0x3ba5638000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x38000) = 0x3ba5638000 mmap(0x3ba563b000, 3488, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x3ba563b000 close(3) = 0 open("/lib64/libudev.so.0", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000)\0\245;\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=57552, ...}) = 0 mmap(0x3ba5000000, 2152544, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3ba5000000 mprotect(0x3ba500c000, 2097152, PROT_NONE) = 0 mmap(0x3ba520c000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xc000) = 0x3ba520c000 close(3) = 0 open("/lib64/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\356\1\244;\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=1926520, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172fe0000 mmap(0x3ba4000000, 3750152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3ba4000000 mprotect(0x3ba418a000, 2097152, PROT_NONE) = 0 mmap(0x3ba438a000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18a000) = 0x3ba438a000 mmap(0x3ba438f000, 18696, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x3ba438f000 close(3) = 0 open("/lib64/libselinux.so.1", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320X\200\245;\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=124624, ...}) = 0 mmap(0x3ba5800000, 2221912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3ba5800000 mprotect(0x3ba581d000, 2093056, PROT_NONE) = 0 mmap(0x3ba5a1c000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c000) = 0x3ba5a1c000 mmap(0x3ba5a1e000, 1880, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x3ba5a1e000 close(3) = 0 open("/lib64/libsepol.so.1", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200>\300\244;\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=251112, ...}) = 0 mmap(0x3ba4c00000, 2345600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3ba4c00000 mprotect(0x3ba4c3b000, 2097152, PROT_NONE) = 0 mmap(0x3ba4e3b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3b000) = 0x3ba4e3b000 close(3) = 0 open("/lib64/libdl.so.2", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\r\200\244;\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=22536, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172fdf000 mmap(0x3ba4800000, 2109696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3ba4800000 mprotect(0x3ba4802000, 2097152, PROT_NONE) = 0 mmap(0x3ba4a02000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x3ba4a02000 close(3) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172fde000 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172fdc000 arch_prctl(ARCH_SET_FS, 0x7f1172fdc7a0) = 0 mprotect(0x3ba520c000, 4096, PROT_READ) = 0 mprotect(0x3ba438a000, 16384, PROT_READ) = 0 mprotect(0x3ba5a1c000, 4096, PROT_READ) = 0 mprotect(0x3ba4e3b000, 4096, PROT_READ) = 0 mprotect(0x3ba3e1f000, 4096, PROT_READ) = 0 mprotect(0x3ba4a02000, 4096, PROT_READ) = 0 munmap(0x7f1172fe1000, 81470) = 0 statfs("/selinux", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=3096364, f_bfree=1942016, f_bavail=1784730, f_files=786432, f_ffree=690935, f_fsid={-57432873, -1917899891}, f_namelen=255, f_frsize=4096}) = 0 brk(0) = 0x1b44000 brk(0x1b65000) = 0x1b65000 open("/proc/filesystems", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172ff4000 read(3, "nodev\tsysfs\nnodev\trootfs\nnodev\tb"..., 1024) = 393 read(3, "", 1024) = 0 close(3) = 0 munmap(0x7f1172ff4000, 4096) = 0 open("/usr/lib/locale/locale-archive", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=99158576, ...}) = 0 mmap(NULL, 99158576, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f116d14b000 close(3) = 0 uname({sys="Linux", node="lnxdb-prd-891", ...}) = 0 open("/proc/devices", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172ff4000 read(3, "Character devices:\n 1 mem\n 4 /"..., 1024) = 535 close(3) = 0 munmap(0x7f1172ff4000, 4096) = 0 open("/proc/misc", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172ff4000 read(3, "220 mptctl\n 56 autofs\n184 microc"..., 1024) = 194 close(3) = 0 munmap(0x7f1172ff4000, 4096) = 0 stat("/dev/mapper/control", {st_mode=S_IFCHR|0660, st_rdev=makedev(10, 58), ...}) = 0 open("/dev/mapper/control", O_RDWR) = 3 open("/proc/devices", O_RDONLY) = 4 fstat(4, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172ff4000 read(4, "Character devices:\n 1 mem\n 4 /"..., 1024) = 535 close(4) = 0 munmap(0x7f1172ff4000, 4096) = 0 ioctl(3, DM_VERSION, 0x1b451b0) = 0 ioctl(3, DM_TABLE_STATUS, 0x1b450e0) = 0 ioctl(3, DM_TABLE_LOAD, 0x1b450e0) = 0 semctl(0, 0, SEM_INFO, 0x7fffb76bc890) = 22 open("/etc/udev/udev.conf", O_RDONLY) = 4 fstat(4, {st_mode=S_IFREG|0644, st_size=218, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172ff4000 read(4, "# The initial syslog(3) priority"..., 4096) = 218 read(4, "", 4096) = 0 close(4) = 0 munmap(0x7f1172ff4000, 4096) = 0 open("/dev/.udev/queue.bin", O_RDONLY) = 4 fstat(4, {st_mode=S_IFREG|0644, st_size=155, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172ff4000 read(4, "QZ,\0\0\0\0\0RZ,\0\0\0\0\0\35\0/devices/virtu"..., 4096) = 155 close(4) = 0 munmap(0x7f1172ff4000, 4096) = 0 open("/dev/urandom", O_RDONLY) = 4 read(4, "o7", 2) = 2 semget(0xd4d376f, 1, IPC_CREAT|IPC_EXCL|0600) = 19300375 semctl(19300375, 0, SETVAL, 0x1) = 0 semctl(19300375, 0, GETVAL, 0xffffffffffffffff) = 1 close(4) = 0 semop(19300375, {{0, 1, 0}}, 1) = 0 semctl(19300375, 0, GETVAL, 0xffffffffffffffff) = 2 ioctl(3, DM_DEV_SUSPEND, 0x1b49250) = 0 semget(0xd4d376f, 1, 0) = 19300375 semctl(19300375, 0, GETVAL, 0xffffffffffffffff) = 2 semop(19300375, {{0, -1, IPC_NOWAIT}}, 1) = 0 semop(19300375, {{0, 0, 0}}, 1) = 0 semctl(19300375, 0, IPC_RMID, 0) = 0 open("/dev/urandom", O_RDONLY) = 4 read(4, "h\316", 2) = 2 semget(0xd4dce68, 1, IPC_CREAT|IPC_EXCL|0600) = 19333143 semctl(19333143, 0, SETVAL, 0x1) = 0 semctl(19333143, 0, GETVAL, 0xffffffffffffffff) = 1 close(4) = 0 semop(19333143, {{0, 1, 0}}, 1) = 0 semctl(19333143, 0, GETVAL, 0xffffffffffffffff) = 2 ioctl(3, DM_DEV_REMOVE, 0x1b45010) = -1 EBUSY (Device or resource busy) open("/usr/share/locale/locale.alias", O_RDONLY) = 4 fstat(4, {st_mode=S_IFREG|0644, st_size=2512, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1172ff4000 read(4, "# Locale name alias data base.\n#"..., 4096) = 2512 read(4, "", 4096) = 0 close(4) = 0 munmap(0x7f1172ff4000, 4096) = 0 open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) open("/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) write(2, "device-mapper: remove ioctl on m"..., 72device-mapper: remove ioctl on mpathdtp1 failed: Device or resource busy) = 72 write(2, "\n", 1 ) = 1 semget(0xd4dce68, 1, 0) = 19333143 semctl(19333143, 0, GETVAL, 0xffffffffffffffff) = 2 semop(19333143, {{0, -1, IPC_NOWAIT}}, 1) = 0 semget(0xd4dce68, 1, 0) = 19333143 semctl(19333143, 0, GETVAL, 0xffffffffffffffff) = 1 semop(19333143, {{0, -1, IPC_NOWAIT}}, 1) = 0 semop(19333143, {{0, 0, 0}}, 1) = 0 semctl(19333143, 0, IPC_RMID, 0) = 0 write(2, "Command failed\n", 15Command failed ) = 15 close(3) = 0 exit_group(1) = ? [root@lnxdb-prd-891 ~]# fuser -kuc /dev/mapper/mpathdtp1 [root@lnxdb-prd-891 ~]#