Bug 446947 - multipath[d] seg faulting on startup
multipath[d] seg faulting on startup
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: device-mapper-multipath (Show other bugs)
9
x86_64 Linux
medium Severity medium
: ---
: ---
Assigned To: Ben Marzinski
Fedora Extras Quality Assurance
:
Depends On: 447220
Blocks:
  Show dependency treegraph
 
Reported: 2008-05-16 14:35 EDT by Zing
Modified: 2008-06-22 19:56 EDT (History)
5 users (show)

See Also:
Fixed In Version: 0.4.7-15.fc9
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-06-12 22:21:39 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
multipathd strace (23.78 KB, text/plain)
2008-05-16 14:37 EDT, Zing
no flags Details
full dmesg log (23.83 KB, text/plain)
2008-05-16 14:43 EDT, Zing
no flags Details
possible gdb backtrace of seg fault? (2.06 KB, text/plain)
2008-05-16 14:51 EDT, Zing
no flags Details
multipath dmesg (294 bytes, text/plain)
2008-06-20 20:05 EDT, Paulo Rodrigues
no flags Details
strace multipath -ll (16.36 KB, text/plain)
2008-06-20 20:11 EDT, Paulo Rodrigues
no flags Details
gdb backtrace multipath -ll (489 bytes, text/plain)
2008-06-20 20:18 EDT, Paulo Rodrigues
no flags Details

  None (edit)
Description Zing 2008-05-16 14:35:09 EDT
Description of problem:
machine upgraded from f8->f9, both multipath and multipathd seg fault on startup.

Version-Release number of selected component (if applicable):
device-mapper-multipath-0.4.7-11.fc7.x86_64

How reproducible:
every time

Steps to Reproduce:
1. f8 upgraded with preupgrade to f9
2. run multipathd
  
Actual results:
Segmentation fault

Expected results:
no seg fault

Additional info:

dmesg:
=====
multipath[2474]: segfault at 188 ip ad4136 sp 7fffed9afb18 error 4 in
libc-2.8.so[a53000+162000]
multipath[2545]: segfault at 188 ip ad4136 sp 7fff5f85e9c8 error 4 in
libc-2.8.so[a53000+162000]
multipathd[2602]: segfault at 188 ip 41352a sp 7fffe0d85500 error 4 in
multipathd[400000+38000]
multipathd[2660]: segfault at 188 ip 41352a sp 7ffff9e26f40 error 4 in
multipathd[400000+38000]
multipathd[2669]: segfault at 188 ip 41352a sp 7fff43638730 error 4 in
multipathd[400000+38000]
multipathd[2832]: segfault at 188 ip 41352a sp 7fff80b2db80 error 4 in
multipathd[400000+38000]
multipathd[2867]: segfault at 188 ip 41352a sp 7fff5313e190 error 4 in
multipathd[400000+38000]
multipath[2872]: segfault at 188 ip 7f5972ee3136 sp 7fff7b1db2b8 error 4 in
libc-2.8.so[7f5972e62000+162000]

/var/log/messages:
May 16 14:07:09 amd64 kernel: multipath[2474]: segfault at 188 ip ad4136 sp
7fffed9afb18 error 4 in libc-2.8.so[a53000+162000]
May 16 14:13:22 amd64 kernel: multipath[2545]: segfault at 188 ip ad4136 sp
7fff5f85e9c8 error 4 in libc-2.8.so[a53000+162000]
May 16 14:14:01 amd64 kernel: multipathd[2602]: segfault at 188 ip 41352a sp
7fffe0d85500 error 4 in multipathd[400000+38000]
May 16 14:15:38 amd64 kernel: multipathd[2660]: segfault at 188 ip 41352a sp
7ffff9e26f40 error 4 in multipathd[400000+38000]
May 16 14:16:34 amd64 kernel: multipathd[2669]: segfault at 188 ip 41352a sp
7fff43638730 error 4 in multipathd[400000+38000]
May 16 14:25:01 amd64 kernel: multipathd[2832]: segfault at 188 ip 41352a sp
7fff80b2db80 error 4 in multipathd[400000+38000]
May 16 14:27:10 amd64 kernel: multipathd[2867]: segfault at 188 ip 41352a sp
7fff5313e190 error 4 in multipathd[400000+38000]
May 16 14:29:29 amd64 kernel: multipath[2872]: segfault at 188 ip 7f5972ee3136
sp 7fff7b1db2b8 error 4 in libc-2.8.so[7f5972e62000+162000]

#multipathd -d -v 9
May 16 14:25:01 | --------start up--------
May 16 14:25:01 | read /etc/multipath.conf
May 16 14:25:01 | dm-0: blacklisted
May 16 14:25:01 | dm-1: blacklisted
May 16 14:25:01 | dm-2: blacklisted
May 16 14:25:01 | dm-3: blacklisted
May 16 14:25:01 | dm-4: blacklisted
May 16 14:25:01 | dm-5: blacklisted
May 16 14:25:01 | dm-6: blacklisted
May 16 14:25:01 | dm-7: blacklisted
May 16 14:25:01 | loop0: blacklisted
May 16 14:25:01 | loop1: blacklisted
May 16 14:25:01 | loop2: blacklisted
May 16 14:25:01 | loop3: blacklisted
May 16 14:25:01 | loop4: blacklisted
May 16 14:25:01 | loop5: blacklisted
May 16 14:25:01 | loop6: blacklisted
May 16 14:25:01 | loop7: blacklisted
May 16 14:25:01 | ram0: blacklisted
May 16 14:25:01 | ram1: blacklisted
May 16 14:25:01 | ram10: blacklisted
May 16 14:25:01 | ram11: blacklisted
May 16 14:25:01 | ram12: blacklisted
May 16 14:25:01 | ram13: blacklisted
May 16 14:25:01 | ram14: blacklisted
May 16 14:25:01 | ram15: blacklisted
May 16 14:25:01 | ram2: blacklisted
May 16 14:25:01 | ram3: blacklisted
May 16 14:25:01 | ram4: blacklisted
May 16 14:25:01 | ram5: blacklisted
May 16 14:25:01 | ram6: blacklisted
May 16 14:25:01 | ram7: blacklisted
May 16 14:25:01 | ram8: blacklisted
May 16 14:25:01 | ram9: blacklisted
May 16 14:25:01 | sda: blacklisted
May 16 14:25:01 | sdb: not found in pathvec
May 16 14:25:01 | sdb: mask = 0x1f
Segmentation fault
Comment 1 Zing 2008-05-16 14:37:53 EDT
Created attachment 305720 [details]
multipathd strace
Comment 2 Zing 2008-05-16 14:43:23 EDT
Created attachment 305721 [details]
full dmesg log
Comment 3 Zing 2008-05-16 14:51:50 EDT
Created attachment 305723 [details]
possible gdb backtrace of seg fault?

not sure if I did this backtrace right...
Comment 4 Zing 2008-05-18 17:28:10 EDT
Here is what I did to get running:

1. Patch sysfsutils with the patch located here (this solves the seg fault):
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=481015
2. multipath (or maybe libsysfs) can't find the "bus"... hack libmultipath to
use "subsystem" instead.

at least my paths are setup now and queuing io correctly for now:
# multipath -ll
mpath0 () dm-0 NETAPP,LUN
[size=1.0T][features=1 queue_if_no_path][hwhandler=0]
\_ round-robin 0 [prio=100][active]
 \_ 3:0:1:0 sdc 8:32  [active][ready]
 \_ 4:0:1:0 sde 8:64  [active][ready]
\_ round-robin 0 [prio=20][enabled]
 \_ 3:0:0:0 sdb 8:16  [active][ready]
 \_ 4:0:0:0 sdd 8:48  [active][ready]

I've put my hacked up packages here:
http://shishz.fedorapeople.org/
Comment 5 Ben Marzinski 2008-05-21 12:58:19 EDT
Patch applied. Thanks.
Comment 6 Fedora Update System 2008-05-21 13:28:13 EDT
device-mapper-multipath-0.4.7-15.fc9 has been submitted as an update for Fedora 9
Comment 7 Fedora Update System 2008-05-22 16:38:22 EDT
device-mapper-multipath-0.4.7-15.fc9 has been pushed to the Fedora 9 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update device-mapper-multipath'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F9/FEDORA-2008-4375
Comment 8 Fedora Update System 2008-06-12 22:21:34 EDT
device-mapper-multipath-0.4.7-15.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 9 Paulo Rodrigues 2008-06-20 20:05:48 EDT
Created attachment 309955 [details]
multipath dmesg
Comment 10 Paulo Rodrigues 2008-06-20 20:11:05 EDT
Created attachment 309956 [details]
strace multipath -ll
Comment 11 Paulo Rodrigues 2008-06-20 20:18:46 EDT
Created attachment 309957 [details]
gdb backtrace multipath -ll
Comment 12 Paulo Rodrigues 2008-06-20 20:20:53 EDT
yum upgrade from f8 to f9, device-mapper-multipath-0.4.7-15.fc9.i386

multipath, multipathd and multipath.static all segfault

are there packages with debugging information?
Comment 13 Zing 2008-06-20 20:27:24 EDT
Paulo could you try:

yum --enablerepo=updates-testing update libsysfs

it looks like libsysfs-2.1.0-4.fc9 hasn't hit stable yet.  If this fixes your
problem could you increment karma on:

https://admin.fedoraproject.org/updates/F9/FEDORA-2008-4290
Comment 14 Paulo Rodrigues 2008-06-22 19:56:39 EDT
Zing thanks, that did it! Will do!

Note You need to log in before you can comment on or make changes to this bug.