Description of problem: machine upgraded from f8->f9, both multipath and multipathd seg fault on startup. Version-Release number of selected component (if applicable): device-mapper-multipath-0.4.7-11.fc7.x86_64 How reproducible: every time Steps to Reproduce: 1. f8 upgraded with preupgrade to f9 2. run multipathd Actual results: Segmentation fault Expected results: no seg fault Additional info: dmesg: ===== multipath[2474]: segfault at 188 ip ad4136 sp 7fffed9afb18 error 4 in libc-2.8.so[a53000+162000] multipath[2545]: segfault at 188 ip ad4136 sp 7fff5f85e9c8 error 4 in libc-2.8.so[a53000+162000] multipathd[2602]: segfault at 188 ip 41352a sp 7fffe0d85500 error 4 in multipathd[400000+38000] multipathd[2660]: segfault at 188 ip 41352a sp 7ffff9e26f40 error 4 in multipathd[400000+38000] multipathd[2669]: segfault at 188 ip 41352a sp 7fff43638730 error 4 in multipathd[400000+38000] multipathd[2832]: segfault at 188 ip 41352a sp 7fff80b2db80 error 4 in multipathd[400000+38000] multipathd[2867]: segfault at 188 ip 41352a sp 7fff5313e190 error 4 in multipathd[400000+38000] multipath[2872]: segfault at 188 ip 7f5972ee3136 sp 7fff7b1db2b8 error 4 in libc-2.8.so[7f5972e62000+162000] /var/log/messages: May 16 14:07:09 amd64 kernel: multipath[2474]: segfault at 188 ip ad4136 sp 7fffed9afb18 error 4 in libc-2.8.so[a53000+162000] May 16 14:13:22 amd64 kernel: multipath[2545]: segfault at 188 ip ad4136 sp 7fff5f85e9c8 error 4 in libc-2.8.so[a53000+162000] May 16 14:14:01 amd64 kernel: multipathd[2602]: segfault at 188 ip 41352a sp 7fffe0d85500 error 4 in multipathd[400000+38000] May 16 14:15:38 amd64 kernel: multipathd[2660]: segfault at 188 ip 41352a sp 7ffff9e26f40 error 4 in multipathd[400000+38000] May 16 14:16:34 amd64 kernel: multipathd[2669]: segfault at 188 ip 41352a sp 7fff43638730 error 4 in multipathd[400000+38000] May 16 14:25:01 amd64 kernel: multipathd[2832]: segfault at 188 ip 41352a sp 7fff80b2db80 error 4 in multipathd[400000+38000] May 16 14:27:10 amd64 kernel: multipathd[2867]: segfault at 188 ip 41352a sp 7fff5313e190 error 4 in multipathd[400000+38000] May 16 14:29:29 amd64 kernel: multipath[2872]: segfault at 188 ip 7f5972ee3136 sp 7fff7b1db2b8 error 4 in libc-2.8.so[7f5972e62000+162000] #multipathd -d -v 9 May 16 14:25:01 | --------start up-------- May 16 14:25:01 | read /etc/multipath.conf May 16 14:25:01 | dm-0: blacklisted May 16 14:25:01 | dm-1: blacklisted May 16 14:25:01 | dm-2: blacklisted May 16 14:25:01 | dm-3: blacklisted May 16 14:25:01 | dm-4: blacklisted May 16 14:25:01 | dm-5: blacklisted May 16 14:25:01 | dm-6: blacklisted May 16 14:25:01 | dm-7: blacklisted May 16 14:25:01 | loop0: blacklisted May 16 14:25:01 | loop1: blacklisted May 16 14:25:01 | loop2: blacklisted May 16 14:25:01 | loop3: blacklisted May 16 14:25:01 | loop4: blacklisted May 16 14:25:01 | loop5: blacklisted May 16 14:25:01 | loop6: blacklisted May 16 14:25:01 | loop7: blacklisted May 16 14:25:01 | ram0: blacklisted May 16 14:25:01 | ram1: blacklisted May 16 14:25:01 | ram10: blacklisted May 16 14:25:01 | ram11: blacklisted May 16 14:25:01 | ram12: blacklisted May 16 14:25:01 | ram13: blacklisted May 16 14:25:01 | ram14: blacklisted May 16 14:25:01 | ram15: blacklisted May 16 14:25:01 | ram2: blacklisted May 16 14:25:01 | ram3: blacklisted May 16 14:25:01 | ram4: blacklisted May 16 14:25:01 | ram5: blacklisted May 16 14:25:01 | ram6: blacklisted May 16 14:25:01 | ram7: blacklisted May 16 14:25:01 | ram8: blacklisted May 16 14:25:01 | ram9: blacklisted May 16 14:25:01 | sda: blacklisted May 16 14:25:01 | sdb: not found in pathvec May 16 14:25:01 | sdb: mask = 0x1f Segmentation fault
Created attachment 305720 [details] multipathd strace
Created attachment 305721 [details] full dmesg log
Created attachment 305723 [details] possible gdb backtrace of seg fault? not sure if I did this backtrace right...
Here is what I did to get running: 1. Patch sysfsutils with the patch located here (this solves the seg fault): http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=481015 2. multipath (or maybe libsysfs) can't find the "bus"... hack libmultipath to use "subsystem" instead. at least my paths are setup now and queuing io correctly for now: # multipath -ll mpath0 () dm-0 NETAPP,LUN [size=1.0T][features=1 queue_if_no_path][hwhandler=0] \_ round-robin 0 [prio=100][active] \_ 3:0:1:0 sdc 8:32 [active][ready] \_ 4:0:1:0 sde 8:64 [active][ready] \_ round-robin 0 [prio=20][enabled] \_ 3:0:0:0 sdb 8:16 [active][ready] \_ 4:0:0:0 sdd 8:48 [active][ready] I've put my hacked up packages here: http://shishz.fedorapeople.org/
Patch applied. Thanks.
device-mapper-multipath-0.4.7-15.fc9 has been submitted as an update for Fedora 9
device-mapper-multipath-0.4.7-15.fc9 has been pushed to the Fedora 9 testing repository. If problems still persist, please make note of it in this bug report. If you want to test the update, you can install it with su -c 'yum --enablerepo=updates-testing update device-mapper-multipath'. You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F9/FEDORA-2008-4375
device-mapper-multipath-0.4.7-15.fc9 has been pushed to the Fedora 9 stable repository. If problems still persist, please make note of it in this bug report.
Created attachment 309955 [details] multipath dmesg
Created attachment 309956 [details] strace multipath -ll
Created attachment 309957 [details] gdb backtrace multipath -ll
yum upgrade from f8 to f9, device-mapper-multipath-0.4.7-15.fc9.i386 multipath, multipathd and multipath.static all segfault are there packages with debugging information?
Paulo could you try: yum --enablerepo=updates-testing update libsysfs it looks like libsysfs-2.1.0-4.fc9 hasn't hit stable yet. If this fixes your problem could you increment karma on: https://admin.fedoraproject.org/updates/F9/FEDORA-2008-4290
Zing thanks, that did it! Will do!