Bugzilla will be upgraded to version 5.0 on December 2, 2018. The outage period for the upgrade will start at 0:00 UTC and have a duration of 12 hours
Bug 986767 - multipathd deamon process exit with segfault
multipathd deamon process exit with segfault
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: device-mapper-multipath (Show other bugs)
x86_64 Linux
unspecified Severity urgent
: rc
: ---
Assigned To: Ben Marzinski
Depends On:
  Show dependency treegraph
Reported: 2013-07-21 23:38 EDT by yangjun
Modified: 2013-12-02 02:18 EST (History)
13 users (show)

See Also:
Fixed In Version: device-mapper-multipath-0.4.9-69.el6
Doc Type: Bug Fix
Doc Text:
Cause: Multipath wasn't blacklisting tapdev, which cannot be multipathed. It tried to multipath them, but their sysfs format is different from expected path devices. Consequence: Multipathd could crash if multiple tadpdev devices were removed from the system. Fix: Multipath now blacklists /dev/td[a-z].* device by default. Result: Multipath will no longer try to setup multipath devices on tapdev devices, and will not crash when they are removed.
Story Points: ---
Clone Of:
: 1036503 (view as bug list)
Last Closed: 2013-11-21 02:50:58 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
the core dump file (323.29 KB, application/x-zip-compressed)
2013-07-21 23:38 EDT, yangjun
no flags Details

External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2013:1574 normal SHIPPED_LIVE device-mapper-multipath bug fix and enhancement update 2013-11-20 19:40:46 EST

  None (edit)
Description yangjun 2013-07-21 23:38:41 EDT
Created attachment 776672 [details]
the core dump file

Description of problem:

the multipathd deamon process exit with segfault

1.create the block devices(/dev/td*)
2.delete the block devices(/dev/td*)
3.after minutes,the multipathd exit with segfault
cat /var/log/messages:
kernel: multipathd[32369]: segfault at 7f57a9dd6ff8 ip 0000003574a2d3c9 sp 00007f57a9dd7000 error 6 in libmultipath.so[3574a00000+3b000]
Comment 2 yangjun 2013-07-22 00:43:25 EDT
the rpm version:
[root@localhost ~]# rpm -qa|grep multipath
Comment 4 Jose Castillo 2013-07-24 09:18:05 EDT
Could you let us know the specific steps you followed to create these block devices, and how reproducible is the problem?
Comment 5 Ben Marzinski 2013-07-24 13:05:21 EDT
I have a dumb question. What kind of devices are td devices?  Are they supposed to be multipathed? multipath certainly doesn't blacklist them by default, but perhaps it should.

The issue is that these td devices all have the sysfs parent of /devices/virtual/block/.  All devices that multipath currently works with have unique parents. What's happening here is that for all paths devices, multipath stores their sysfs devices and their parents in a cache, since it needs to reference them to grab sysfs values.  When the second td* path device is added by multipath, it uses the cached version of the parent from the first device instead of creating it's own device in the cache.  When multipath removes the first device, it deletes that cached parent device.  When it removes the second device, it again tries to delete the cached parent device, but it already has done so.

If these devices aren't supposed to be multipathed, then the workaround is to add

blacklist {
        devnode "^td"

to /etc/multipath.conf, and I can add this to the default blacklist, so that starting in RHEL-6.5, multipath will automatically blacklist these devices.

If these devices really are supposed to be blacklisted, that's a trickier problem.  Multipath would need to either cache separate parents for these devices, or it would need to keep reference counts and only delete the device when the last user drops it.  Unfortunately, the way multipath uses these cached devices, neither solution is particularly straightforward.
Comment 6 yangjun 2013-08-04 22:20:22 EDT
I'm sorry. last few days, I'm on holidays .so didn't reply the bug.

/dev/td* are tapdev devices:
[root@localhost ~]# cat /proc/devices |grep tap
251 blktap2
253 tapdev
[root@localhost ~]# ll /dev/tda 
brw-rw---- 1 root disk 253, 0 Aug  5 09:19 /dev/tda
brw-rw---- 1 root disk 253, 1 Aug  5 09:19 /dev/tdb

/dev/td* did not support the multipath.
Comment 7 Ben Marzinski 2013-08-05 14:28:21 EDT
So, I'll go ahead and make multipath automatically blacklist these devices. Can you verify that the workaround of adding

blacklist {
        devnode "^td"

to /etc/multipath.conf solves the issue for you?
Comment 8 Ben Marzinski 2013-08-13 11:45:57 EDT
I've updated the default blacklist to exclude tapdev devices.
Comment 10 yanfu,wang 2013-10-11 02:56:13 EDT
Reproduced on device-mapper-multipath-0.4.9-64.el6:
[root@storageqe-17 ~]# multipathd show conf|grep td
[root@storageqe-17 ~]# echo $?

Verified on device-mapper-multipath-0.4.9-71.el6, "td[a-z]" devices are now blacklisted by default.
[root@storageqe-17 ~]# multipathd show conf|grep td
	devnode "^(td|hd)[a-z]"
Comment 12 errata-xmlrpc 2013-11-21 02:50:58 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.