Description of problem: I test the IB-SRP initiator (RHEL 5.4 x86_64) The multipath can't remap some disk when total number of usable paths was from failover to active.And fdisk -l was hanging. I didn't know why the disk was in system,but multipath can't remap the disk to dm-8. [root@localhost sdw]# multipath -l 26134613837363132 dm-6 _INSPUR_,vg0_lv03 [size=400G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][active] \_ 744:0:0:4 sde 8:64 [active][undef] \_ 9:0:0:4 sdu 65:64 [active][undef] 23233313238646237 dm-3 _INSPUR_,vg19_lv02 [size=500G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][active] \_ 744:0:0:7 sdk 8:160 [active][undef] \_ 9:0:0:7 sdx 65:112 [active][undef] 2343537626435392d dm-5 _INSPUR_,vg19_lv00 [size=500G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][active] \_ 744:0:0:5 sdf 8:80 [active][undef] \_ 9:0:0:5 sdv 65:80 [active][undef] 26237306165653635 dm-4 _INSPUR_,vg0_lv02 [size=400G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][active] \_ 744:0:0:3 sdd 8:48 [active][undef] \_ 9:0:0:3 sdt 65:48 [active][undef] 23330623031356330 dm-7 _INSPUR_,vg19_lv03 [size=500G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][active] \_ 744:0:0:8 sdl 8:176 [active][undef] \_ 9:0:0:8 sdy 65:128 [active][undef] 23930346664653862 dm-2 _INSPUR_,vg0_lv00 [size=400G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][active] \_ 744:0:0:1 sdb 8:16 [active][undef] \_ 9:0:0:1 sdr 65:16 [active][undef] 23833656434366663 dm-9 _INSPUR_,vg0_lv01 [size=400G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][active] \_ 744:0:0:2 sdc 8:32 [active][undef] \_ 9:0:0:2 sds 65:32 [active][undef] 23137663532353265 dm-8 , [size=500G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][enabled] \_ #:#:#:# - #:# [failed][undef] \_ #:#:#:# - #:# [failed][undef] The dm-8 can't remap the disk(sdj sdw).But the node was exist in /dev/sdj /dev/sdw.And I can R/W the disk(dd).The sdg and sdo before dm-8 failover was still residual(sdg and sdo were not exist in /dev/sdg /dev/sdo).There were some bio requeued in the dm-8.But because the dm-8 was not recovery,IO would always requeue. [root@localhost slaves]# pwd /sys/block/dm-8/slaves [root@localhost slaves]# ls -l lrwxrwxrwx 1 root root 0 07-24 20:38 sdg -> ../../../block/sdg lrwxrwxrwx 1 root root 0 07-25 09:31 sdj -> ../../../block/sdj lrwxrwxrwx 1 root root 0 07-24 20:38 sdo -> ../../../block/sdo lrwxrwxrwx 1 root root 0 07-24 20:38 sdw -> ../../../block/sdw The multipath.conf : defaults { # user_friendly_names yes udev_dir /dev polling_interval 10 selector "round-robin 0" path_grouping_policy multibus getuid_callout "/sbin/scsi_id -g -u -s /block/%n" prio_callout /bin/true features "1 queue_if_no_path" path_checker tur rr_min_io 1000 max_fds 8192 rr_weight priorities failback immediate no_path_retry queue # queue_without_daemon no } Version-Release number of selected component (if applicable): Name : kpartx Arch : x86_64 Version : 0.4.7 Release : 48.el5_8.1 Name : device-mapper-multipath Arch : x86_64 Version : 0.4.7 Release : 48.el5_8.1 How reproducible: Steps to Reproduce: 1.Using IB-srp to map some disk. 2.Mount the disk with ext3 and dd if=/dev/zero of=./testfile bs=1M count=100000 2.Cut all the connection .It cause the multipath-device faulty. 3. srp_daemon.sh will reconnect the disk var IB-SRP later. Actual results: some disk maybe not in dm device Expected results: All the disk were active and ready Additional info:
I used multipath -F.It was hanging. The udev config: [root@localhost rules.d]# pwd /etc/udev/rules.d [root@localhost rules.d]# cat 40-multipath.rules # multipath wants the devmaps presented as meaninglful device names # so name them after their devmap name SUBSYSTEM!="block", GOTO="end_mpath" KERNEL!="dm-[0-9]*", GOTO="end_mpath" PROGRAM!="/sbin/mpath_wait %M %m", GOTO="end_mpath" PROGRAM!="/sbin/dmsetup info -c --noheadings -j %M -m %m", GOTO="end_mpath" RESULT!="*:*:*:*:*:*:*:mpath-*", GOTO="kpartx_check" PROGRAM!="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", GOTO="end_mpath" RESULT!="?*", GOTO="end_mpath" NAME="%k", SYMLINK="mpath/%c" ACTION!="change", GOTO="skip_kpartx" RUN+="/bin/bash -c '/sbin/mpath_wait /dev/mapper/%c; /sbin/kpartx -a -p p /dev/mapper/%c'" LABEL="skip_kpartx" OPTIONS="last_rule" LABEL="kpartx_check" RESULT!="*:*:*:*:*:*:*:part*-mpath-*", GOTO="end_mpath" PROGRAM=="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", RESULT=="?*", NAME="%k", SYMLINK="mpath/%c" OPTIONS="last_rule" LABEL="end_mpath" [root@localhost rules.d]# cat 91-srp.rules ACTION=="add", KERNEL=="sd*[!0-9]", RUN+="/sbin/multipath %M:%m"
[root@localhost ~]# sg_map /dev/sg0 /dev/sda /dev/sg1 /dev/scd0 /dev/sg2 /dev/sdb /dev/sg3 /dev/sdc /dev/sg4 /dev/sdd /dev/sg5 /dev/sde /dev/sg6 /dev/sdf /dev/sg7 /dev/sdj /dev/sg8 /dev/sdk /dev/sg9 /dev/sdl /dev/sg10 /dev/sdr /dev/sg11 /dev/sds /dev/sg12 /dev/sdt /dev/sg13 /dev/sdu /dev/sg14 /dev/sdv /dev/sg15 /dev/sdw /dev/sg16 /dev/sdx /dev/sg17 /dev/sdy [root@localhost rules.d]# multipath -v4 dm-0: blacklisted dm-1: blacklisted dm-2: blacklisted dm-3: blacklisted dm-4: blacklisted dm-5: blacklisted dm-6: blacklisted dm-7: blacklisted dm-8: blacklisted dm-9: blacklisted md0: blacklisted ram0: blacklisted ram10: blacklisted ram11: blacklisted ram12: blacklisted ram13: blacklisted ram14: blacklisted ram15: blacklisted ram1: blacklisted ram2: blacklisted ram3: blacklisted ram4: blacklisted ram5: blacklisted ram6: blacklisted ram7: blacklisted ram8: blacklisted ram9: blacklisted sda: blacklisted sdb: not found in pathvec sdb: mask = 0x1f sdb: bus = 1 sdb: dev_t = 8:16 sdb: size = 838860800 sdb: vendor = _INSPUR_ sdb: product = vg0_lv00 sdb: rev = 220 sdb: h:b:t:l = 744:0:0:1 sdb: serial = 904fde8b sdb: path checker = tur (config file default) sdb: checker timeout = 60000 ms (sysfs setting) sdb: state = 2 sdb: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default) sdb: uid = 23930346664653862 (callout) sdb: getprio = /bin/true (config file default) sdb: prio = 0 sdc: not found in pathvec sdc: mask = 0x1f sdc: bus = 1 sdc: dev_t = 8:32 sdc: size = 838860800 sdc: vendor = _INSPUR_ sdc: product = vg0_lv01 sdc: rev = 220 sdc: h:b:t:l = 744:0:0:2 sdc: serial = 83ed46fc sdc: path checker = tur (config file default) sdc: checker timeout = 60000 ms (sysfs setting) sdc: state = 2 sdc: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default) sdc: uid = 23833656434366663 (callout) sdc: getprio = /bin/true (config file default) sdc: prio = 0 sdd: not found in pathvec sdd: mask = 0x1f sdd: bus = 1 sdd: dev_t = 8:48 sdd: size = 838860800 sdd: vendor = _INSPUR_ sdd: product = vg0_lv02 sdd: rev = 220 sdd: h:b:t:l = 744:0:0:3 sdd: serial = b70aee65 sdd: path checker = tur (config file default) sdd: checker timeout = 60000 ms (sysfs setting) sdd: state = 2 sdd: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default) sdd: uid = 26237306165653635 (callout) sdd: getprio = /bin/true (config file default) sdd: prio = 0 sde: not found in pathvec sde: mask = 0x1f sde: bus = 1 sde: dev_t = 8:64 sde: size = 838860800 sde: vendor = _INSPUR_ sde: product = vg0_lv03 sde: rev = 220 sde: h:b:t:l = 744:0:0:4 sde: serial = a4a87612 sde: path checker = tur (config file default) sde: checker timeout = 60000 ms (sysfs setting) sde: state = 2 sde: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default) sde: uid = 26134613837363132 (callout) sde: getprio = /bin/true (config file default) sde: prio = 0 sdf: not found in pathvec sdf: mask = 0x1f sdf: bus = 1 sdf: dev_t = 8:80 sdf: size = 1048576000 sdf: vendor = _INSPUR_ sdf: product = vg19_lv00 sdf: rev = 220 sdf: h:b:t:l = 744:0:0:5 sdf: serial = 457bd59 sdf: path checker = tur (config file default) sdf: checker timeout = 60000 ms (sysfs setting) sdf: state = 2 sdf: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default) sdf: uid = 2343537626435392d (callout) sdf: getprio = /bin/true (config file default) sdf: prio = 0 sdj: not found in pathvec sdj: mask = 0x1f sdj: bus = 1 sdj: dev_t = 8:144 sdj: size = 1048576000 sdj: vendor = _INSPUR_ sdj: product = vg19_lv01 sdj: rev = 220 sdj: h:b:t:l = 744:0:0:6 sdj: serial = 17f5252e sdj: path checker = tur (config file default) sdj: checker timeout = 60000 ms (sysfs setting) sdj: state = 2 sdj: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default) sdj: uid = 23137663532353265 (callout) sdj: getprio = /bin/true (config file default) sdj: prio = 0 sdk: not found in pathvec sdk: mask = 0x1f sdk: bus = 1 sdk: dev_t = 8:160 sdk: size = 1048576000 sdk: vendor = _INSPUR_ sdk: product = vg19_lv02 sdk: rev = 220 sdk: h:b:t:l = 744:0:0:7 sdk: serial = 23128db7 sdk: path checker = tur (config file default) sdk: checker timeout = 60000 ms (sysfs setting) sdk: state = 2 sdk: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default) sdk: uid = 23233313238646237 (callout) sdk: getprio = /bin/true (config file default) sdk: prio = 0 sdl: not found in pathvec sdl: mask = 0x1f sdl: bus = 1 sdl: dev_t = 8:176 sdl: size = 1048576000 sdl: vendor = _INSPUR_ sdl: product = vg19_lv03 sdl: rev = 220 sdl: h:b:t:l = 744:0:0:8 sdl: serial = 30b015c0 sdl: path checker = tur (config file default) sdl: checker timeout = 60000 ms (sysfs setting) sdl: state = 2 sdl: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default) sdl: uid = 23330623031356330 (callout) sdl: getprio = /bin/true (config file default) sdl: prio = 0 sdr: not found in pathvec sdr: mask = 0x1f sdr: bus = 1 sdr: dev_t = 65:16 sdr: size = 838860800 sdr: vendor = _INSPUR_ sdr: product = vg0_lv00 sdr: rev = 220 sdr: h:b:t:l = 9:0:0:1 sdr: serial = 904fde8b sdr: path checker = tur (config file default) sdr: checker timeout = 60000 ms (sysfs setting) sdr: state = 2 sdr: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default) sdr: uid = 23930346664653862 (callout) sdr: getprio = /bin/true (config file default) sdr: prio = 0 sds: not found in pathvec sds: mask = 0x1f sds: bus = 1 sds: dev_t = 65:32 sds: size = 838860800 sds: vendor = _INSPUR_ sds: product = vg0_lv01 sds: rev = 220 sds: h:b:t:l = 9:0:0:2 sds: serial = 83ed46fc sds: path checker = tur (config file default) sds: checker timeout = 60000 ms (sysfs setting) sds: state = 2 sds: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default) sds: uid = 23833656434366663 (callout) sds: getprio = /bin/true (config file default) sds: prio = 0 sdt: not found in pathvec sdt: mask = 0x1f sdt: bus = 1 sdt: dev_t = 65:48 sdt: size = 838860800 sdt: vendor = _INSPUR_ sdt: product = vg0_lv02 sdt: rev = 220 sdt: h:b:t:l = 9:0:0:3 sdt: serial = b70aee65 sdt: path checker = tur (config file default) sdt: checker timeout = 60000 ms (sysfs setting) sdt: state = 2 sdt: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default) sdt: uid = 26237306165653635 (callout) sdt: getprio = /bin/true (config file default) sdt: prio = 0 sdu: not found in pathvec sdu: mask = 0x1f sdu: bus = 1 sdu: dev_t = 65:64 sdu: size = 838860800 sdu: vendor = _INSPUR_ sdu: product = vg0_lv03 sdu: rev = 220 sdu: h:b:t:l = 9:0:0:4 sdu: serial = a4a87612 sdu: path checker = tur (config file default) sdu: checker timeout = 60000 ms (sysfs setting) sdu: state = 2 sdu: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default) sdu: uid = 26134613837363132 (callout) sdu: getprio = /bin/true (config file default) sdu: prio = 0 sdv: not found in pathvec sdv: mask = 0x1f sdv: bus = 1 sdv: dev_t = 65:80 sdv: size = 1048576000 sdv: vendor = _INSPUR_ sdv: product = vg19_lv00 sdv: rev = 220 sdv: h:b:t:l = 9:0:0:5 sdv: serial = 457bd59 sdv: path checker = tur (config file default) sdv: checker timeout = 60000 ms (sysfs setting) sdv: state = 2 sdv: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default) sdv: uid = 2343537626435392d (callout) sdv: getprio = /bin/true (config file default) sdv: prio = 0 sdw: not found in pathvec sdw: mask = 0x1f sdw: bus = 1 sdw: dev_t = 65:96 sdw: size = 1048576000 sdw: vendor = _INSPUR_ sdw: product = vg19_lv01 sdw: rev = 220 sdw: h:b:t:l = 9:0:0:6 sdw: serial = 17f5252e sdw: path checker = tur (config file default) sdw: checker timeout = 60000 ms (sysfs setting) sdw: state = 2 sdw: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default) sdw: uid = 23137663532353265 (callout) sdw: getprio = /bin/true (config file default) sdw: prio = 0 sdx: not found in pathvec sdx: mask = 0x1f sdx: bus = 1 sdx: dev_t = 65:112 sdx: size = 1048576000 sdx: vendor = _INSPUR_ sdx: product = vg19_lv02 sdx: rev = 220 sdx: h:b:t:l = 9:0:0:7 sdx: serial = 23128db7 sdx: path checker = tur (config file default) sdx: checker timeout = 60000 ms (sysfs setting) sdx: state = 2 sdx: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default) sdx: uid = 23233313238646237 (callout) sdx: getprio = /bin/true (config file default) sdx: prio = 0 sdy: not found in pathvec sdy: mask = 0x1f sdy: bus = 1 sdy: dev_t = 65:128 sdy: size = 1048576000 sdy: vendor = _INSPUR_ sdy: product = vg19_lv03 sdy: rev = 220 sdy: h:b:t:l = 9:0:0:8 sdy: serial = 30b015c0 sdy: path checker = tur (config file default) sdy: checker timeout = 60000 ms (sysfs setting) sdy: state = 2 sdy: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default) sdy: uid = 23330623031356330 (callout) sdy: getprio = /bin/true (config file default) sdy: prio = 0 sr0: blacklisted ===== paths list ===== uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev 23930346664653862 744:0:0:1 sdb 8:16 0 [undef][ready] _INSPUR_,vg0_lv00 23833656434366663 744:0:0:2 sdc 8:32 0 [undef][ready] _INSPUR_,vg0_lv01 26237306165653635 744:0:0:3 sdd 8:48 0 [undef][ready] _INSPUR_,vg0_lv02 26134613837363132 744:0:0:4 sde 8:64 0 [undef][ready] _INSPUR_,vg0_lv03 2343537626435392d 744:0:0:5 sdf 8:80 0 [undef][ready] _INSPUR_,vg19_lv00 23137663532353265 744:0:0:6 sdj 8:144 0 [undef][ready] _INSPUR_,vg19_lv01 23233313238646237 744:0:0:7 sdk 8:160 0 [undef][ready] _INSPUR_,vg19_lv02 23330623031356330 744:0:0:8 sdl 8:176 0 [undef][ready] _INSPUR_,vg19_lv03 23930346664653862 9:0:0:1 sdr 65:16 0 [undef][ready] _INSPUR_,vg0_lv00 23833656434366663 9:0:0:2 sds 65:32 0 [undef][ready] _INSPUR_,vg0_lv01 26237306165653635 9:0:0:3 sdt 65:48 0 [undef][ready] _INSPUR_,vg0_lv02 26134613837363132 9:0:0:4 sdu 65:64 0 [undef][ready] _INSPUR_,vg0_lv03 2343537626435392d 9:0:0:5 sdv 65:80 0 [undef][ready] _INSPUR_,vg19_lv00 23137663532353265 9:0:0:6 sdw 65:96 0 [undef][ready] _INSPUR_,vg19_lv01 23233313238646237 9:0:0:7 sdx 65:112 0 [undef][ready] _INSPUR_,vg19_lv02 23330623031356330 9:0:0:8 sdy 65:128 0 [undef][ready] _INSPUR_,vg19_lv03 params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:64 1000 65:64 1000 status = 2 0 0 0 1 1 A 0 2 0 8:64 A 0 65:64 A 0 *word = 1, len = 1 *word = queue_if_no_path, len = 16 *word = 0, len = 1 *word = 1, len = 1 *word = 1, len = 1 *word = round-robin, len = 11 *word = 0, len = 1 *word = 2, len = 1 *word = 1, len = 1 *word = 8:64, len = 4 *word = 1000, len = 4 *word = 65:64, len = 5 *word = 1000, len = 4 sde: mask = 0x8 sde: prio = 0 sdu: mask = 0x8 sdu: prio = 0 *word = 2, len = 1 *word = 0, len = 1 *word = 0, len = 1 *word = 1, len = 1 *word = A, len = 1 *word = 2, len = 1 *word = 0, len = 1 *word = A, len = 1 *word = 0, len = 1 *word = A, len = 1 *word = 0, len = 1 params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:160 1000 65:112 1000 status = 2 0 0 0 1 1 A 0 2 0 8:160 A 0 65:112 A 0 *word = 1, len = 1 *word = queue_if_no_path, len = 16 *word = 0, len = 1 *word = 1, len = 1 *word = 1, len = 1 *word = round-robin, len = 11 *word = 0, len = 1 *word = 2, len = 1 *word = 1, len = 1 *word = 8:160, len = 5 *word = 1000, len = 4 *word = 65:112, len = 6 *word = 1000, len = 4 sdk: mask = 0x8 sdk: prio = 0 sdx: mask = 0x8 sdx: prio = 0 *word = 2, len = 1 *word = 0, len = 1 *word = 0, len = 1 *word = 1, len = 1 *word = A, len = 1 *word = 2, len = 1 *word = 0, len = 1 *word = A, len = 1 *word = 0, len = 1 *word = A, len = 1 *word = 0, len = 1 params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:80 1000 65:80 1000 status = 2 0 0 0 1 1 A 0 2 0 8:80 A 0 65:80 A 0 *word = 1, len = 1 *word = queue_if_no_path, len = 16 *word = 0, len = 1 *word = 1, len = 1 *word = 1, len = 1 *word = round-robin, len = 11 *word = 0, len = 1 *word = 2, len = 1 *word = 1, len = 1 *word = 8:80, len = 4 *word = 1000, len = 4 *word = 65:80, len = 5 *word = 1000, len = 4 sdf: mask = 0x8 sdf: prio = 0 sdv: mask = 0x8 sdv: prio = 0 *word = 2, len = 1 *word = 0, len = 1 *word = 0, len = 1 *word = 1, len = 1 *word = A, len = 1 *word = 2, len = 1 *word = 0, len = 1 *word = A, len = 1 *word = 0, len = 1 *word = A, len = 1 *word = 0, len = 1 params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:48 1000 65:48 1000 status = 2 0 0 0 1 1 A 0 2 0 8:48 A 0 65:48 A 0 *word = 1, len = 1 *word = queue_if_no_path, len = 16 *word = 0, len = 1 *word = 1, len = 1 *word = 1, len = 1 *word = round-robin, len = 11 *word = 0, len = 1 *word = 2, len = 1 *word = 1, len = 1 *word = 8:48, len = 4 *word = 1000, len = 4 *word = 65:48, len = 5 *word = 1000, len = 4 sdd: mask = 0x8 sdd: prio = 0 sdt: mask = 0x8 sdt: prio = 0 *word = 2, len = 1 *word = 0, len = 1 *word = 0, len = 1 *word = 1, len = 1 *word = A, len = 1 *word = 2, len = 1 *word = 0, len = 1 *word = A, len = 1 *word = 0, len = 1 *word = A, len = 1 *word = 0, len = 1 params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:176 1000 65:128 1000 status = 2 0 0 0 1 1 A 0 2 0 8:176 A 0 65:128 A 0 *word = 1, len = 1 *word = queue_if_no_path, len = 16 *word = 0, len = 1 *word = 1, len = 1 *word = 1, len = 1 *word = round-robin, len = 11 *word = 0, len = 1 *word = 2, len = 1 *word = 1, len = 1 *word = 8:176, len = 5 *word = 1000, len = 4 *word = 65:128, len = 6 *word = 1000, len = 4 sdl: mask = 0x8 sdl: prio = 0 sdy: mask = 0x8 sdy: prio = 0 *word = 2, len = 1 *word = 0, len = 1 *word = 0, len = 1 *word = 1, len = 1 *word = A, len = 1 *word = 2, len = 1 *word = 0, len = 1 *word = A, len = 1 *word = 0, len = 1 *word = A, len = 1 *word = 0, len = 1 params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:16 1000 65:16 1000 status = 2 0 0 0 1 1 A 0 2 0 8:16 A 0 65:16 A 0 *word = 1, len = 1 *word = queue_if_no_path, len = 16 *word = 0, len = 1 *word = 1, len = 1 *word = 1, len = 1 *word = round-robin, len = 11 *word = 0, len = 1 *word = 2, len = 1 *word = 1, len = 1 *word = 8:16, len = 4 *word = 1000, len = 4 *word = 65:16, len = 5 *word = 1000, len = 4 sdb: mask = 0x8 sdb: prio = 0 sdr: mask = 0x8 sdr: prio = 0 *word = 2, len = 1 *word = 0, len = 1 *word = 0, len = 1 *word = 1, len = 1 *word = A, len = 1 *word = 2, len = 1 *word = 0, len = 1 *word = A, len = 1 *word = 0, len = 1 *word = A, len = 1 *word = 0, len = 1 params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:32 1000 65:32 1000 status = 2 0 0 0 1 1 A 0 2 0 8:32 A 0 65:32 A 0 *word = 1, len = 1 *word = queue_if_no_path, len = 16 *word = 0, len = 1 *word = 1, len = 1 *word = 1, len = 1 *word = round-robin, len = 11 *word = 0, len = 1 *word = 2, len = 1 *word = 1, len = 1 *word = 8:32, len = 4 *word = 1000, len = 4 *word = 65:32, len = 5 *word = 1000, len = 4 sdc: mask = 0x8 sdc: prio = 0 sds: mask = 0x8 sds: prio = 0 *word = 2, len = 1 *word = 0, len = 1 *word = 0, len = 1 *word = 1, len = 1 *word = A, len = 1 *word = 2, len = 1 *word = 0, len = 1 *word = A, len = 1 *word = 0, len = 1 *word = A, len = 1 *word = 0, len = 1 params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:96 1000 8:224 1000 status = 2 18546 0 0 1 1 E 0 2 0 8:96 F 18 8:224 F 18 *word = 1, len = 1 *word = queue_if_no_path, len = 16 *word = 0, len = 1 *word = 1, len = 1 *word = 1, len = 1 *word = round-robin, len = 11 *word = 0, len = 1 *word = 2, len = 1 *word = 1, len = 1 *word = 8:96, len = 4 8:96: not found in pathvec *word = 1000, len = 4 *word = 8:224, len = 5 8:224: not found in pathvec *word = 1000, len = 4 *word = 2, len = 1 *word = 0, len = 1 *word = 0, len = 1 *word = 1, len = 1 *word = E, len = 1 *word = 2, len = 1 *word = 0, len = 1 *word = F, len = 1 *word = 18, len = 2 *word = F, len = 1 *word = 18, len = 2 sdb: ownership set to 23930346664653862 sdb: not found in pathvec sdb: mask = 0xc sdb: path checker = tur (config file default) sdb: checker timeout = 60000 ms (sysfs setting) sdb: state = 2 sdb: prio = 0 sdr: ownership set to 23930346664653862 sdr: not found in pathvec sdr: mask = 0xc sdr: path checker = tur (config file default) sdr: checker timeout = 60000 ms (sysfs setting) sdr: state = 2 sdr: prio = 0 23930346664653862: pgfailback = -2 (config file default) 23930346664653862: pgpolicy = multibus (config file default) 23930346664653862: selector = round-robin 0 (internal default) 23930346664653862: features = 1 queue_if_no_path (internal default) 23930346664653862: hwhandler = 0 (internal default) 23930346664653862: rr_weight = 2 (config file default) 23930346664653862: minio = 1000 (config file default) 23930346664653862: no_path_retry = -2 (config file default) pg_timeout = NONE (internal default) 23930346664653862: set ACT_NOTHING (map unchanged) sdc: ownership set to 23833656434366663 sdc: not found in pathvec sdc: mask = 0xc sdc: path checker = tur (config file default) sdc: checker timeout = 60000 ms (sysfs setting) sdc: state = 2 sdc: prio = 0 sds: ownership set to 23833656434366663 sds: not found in pathvec sds: mask = 0xc sds: path checker = tur (config file default) sds: checker timeout = 60000 ms (sysfs setting) sds: state = 2 sds: prio = 0 23833656434366663: pgfailback = -2 (config file default) 23833656434366663: pgpolicy = multibus (config file default) 23833656434366663: selector = round-robin 0 (internal default) 23833656434366663: features = 1 queue_if_no_path (internal default) 23833656434366663: hwhandler = 0 (internal default) 23833656434366663: rr_weight = 2 (config file default) 23833656434366663: minio = 1000 (config file default) 23833656434366663: no_path_retry = -2 (config file default) pg_timeout = NONE (internal default) 23833656434366663: set ACT_NOTHING (map unchanged) sdd: ownership set to 26237306165653635 sdd: not found in pathvec sdd: mask = 0xc sdd: path checker = tur (config file default) sdd: checker timeout = 60000 ms (sysfs setting) sdd: state = 2 sdd: prio = 0 sdt: ownership set to 26237306165653635 sdt: not found in pathvec sdt: mask = 0xc sdt: path checker = tur (config file default) sdt: checker timeout = 60000 ms (sysfs setting) sdt: state = 2 sdt: prio = 0 26237306165653635: pgfailback = -2 (config file default) 26237306165653635: pgpolicy = multibus (config file default) 26237306165653635: selector = round-robin 0 (internal default) 26237306165653635: features = 1 queue_if_no_path (internal default) 26237306165653635: hwhandler = 0 (internal default) 26237306165653635: rr_weight = 2 (config file default) 26237306165653635: minio = 1000 (config file default) 26237306165653635: no_path_retry = -2 (config file default) pg_timeout = NONE (internal default) 26237306165653635: set ACT_NOTHING (map unchanged) sde: ownership set to 26134613837363132 sde: not found in pathvec sde: mask = 0xc sde: path checker = tur (config file default) sde: checker timeout = 60000 ms (sysfs setting) sde: state = 2 sde: prio = 0 sdu: ownership set to 26134613837363132 sdu: not found in pathvec sdu: mask = 0xc sdu: path checker = tur (config file default) sdu: checker timeout = 60000 ms (sysfs setting) sdu: state = 2 sdu: prio = 0 26134613837363132: pgfailback = -2 (config file default) 26134613837363132: pgpolicy = multibus (config file default) 26134613837363132: selector = round-robin 0 (internal default) 26134613837363132: features = 1 queue_if_no_path (internal default) 26134613837363132: hwhandler = 0 (internal default) 26134613837363132: rr_weight = 2 (config file default) 26134613837363132: minio = 1000 (config file default) 26134613837363132: no_path_retry = -2 (config file default) pg_timeout = NONE (internal default) 26134613837363132: set ACT_NOTHING (map unchanged) sdf: ownership set to 2343537626435392d sdf: not found in pathvec sdf: mask = 0xc sdf: path checker = tur (config file default) sdf: checker timeout = 60000 ms (sysfs setting) sdf: state = 2 sdf: prio = 0 sdv: ownership set to 2343537626435392d sdv: not found in pathvec sdv: mask = 0xc sdv: path checker = tur (config file default) sdv: checker timeout = 60000 ms (sysfs setting) sdv: state = 2 sdv: prio = 0 2343537626435392d: pgfailback = -2 (config file default) 2343537626435392d: pgpolicy = multibus (config file default) 2343537626435392d: selector = round-robin 0 (internal default) 2343537626435392d: features = 1 queue_if_no_path (internal default) 2343537626435392d: hwhandler = 0 (internal default) 2343537626435392d: rr_weight = 2 (config file default) 2343537626435392d: minio = 1000 (config file default) 2343537626435392d: no_path_retry = -2 (config file default) pg_timeout = NONE (internal default) 2343537626435392d: set ACT_NOTHING (map unchanged) sdj: ownership set to 23137663532353265 sdj: not found in pathvec sdj: mask = 0xc sdj: path checker = tur (config file default) sdj: checker timeout = 60000 ms (sysfs setting) sdj: state = 2 sdj: prio = 0 sdw: ownership set to 23137663532353265 sdw: not found in pathvec sdw: mask = 0xc sdw: path checker = tur (config file default) sdw: checker timeout = 60000 ms (sysfs setting) sdw: state = 2 sdw: prio = 0 : ownership set to 23137663532353265 : not found in pathvec : mask = 0xc : path checker = tur (config file default) : checker timeout = 300000 ms (internal default) : state = 1 : checker msg is "tur checker reports path is down" : ownership set to 23137663532353265 : mask = 0xc : path checker = tur (config file default) : checker timeout = 300000 ms (internal default) : state = 1 : checker msg is "tur checker reports path is down" 23137663532353265: removing path 8:96 with no devname 23137663532353265: pgfailback = -2 (config file default) 23137663532353265: pgpolicy = multibus (config file default) 23137663532353265: selector = round-robin 0 (internal default) 23137663532353265: features = 1 queue_if_no_path (internal default) 23137663532353265: hwhandler = 0 (internal default) 23137663532353265: rr_weight = 2 (config file default) 23137663532353265: minio = 1000 (config file default) 23137663532353265: no_path_retry = -2 (config file default) pg_timeout = NONE (internal default) 23137663532353265: set ACT_RELOAD (path group topology change)
[root@localhost slaves]# multipath -v2 : checker msg is "tur checker reports path is down" : checker msg is "tur checker reports path is down"
root@localhost ~]# lsscsi [0:0:0:0] disk ATA WDC WD3200AAJS-2 3E01 /dev/sda [5:0:0:0] cd/dvd ATAPI DVD D DH16D2P HP56 /dev/scd0 [9:0:0:1] disk _INSPUR_ vg0_lv00 220 /dev/sdr [9:0:0:2] disk _INSPUR_ vg0_lv01 220 /dev/sds [9:0:0:3] disk _INSPUR_ vg0_lv02 220 /dev/sdt [9:0:0:4] disk _INSPUR_ vg0_lv03 220 /dev/sdu [9:0:0:5] disk _INSPUR_ vg19_lv00 220 /dev/sdv [9:0:0:6] disk _INSPUR_ vg19_lv01 220 /dev/sdw [9:0:0:7] disk _INSPUR_ vg19_lv02 220 /dev/sdx [9:0:0:8] disk _INSPUR_ vg19_lv03 220 /dev/sdy [744:0:0:1] disk _INSPUR_ vg0_lv00 220 /dev/sdb [744:0:0:2] disk _INSPUR_ vg0_lv01 220 /dev/sdc [744:0:0:3] disk _INSPUR_ vg0_lv02 220 /dev/sdd [744:0:0:4] disk _INSPUR_ vg0_lv03 220 /dev/sde [744:0:0:5] disk _INSPUR_ vg19_lv00 220 /dev/sdf [744:0:0:6] disk _INSPUR_ vg19_lv01 220 /dev/sdj [744:0:0:7] disk _INSPUR_ vg19_lv02 220 /dev/sdk [744:0:0:8] disk _INSPUR_ vg19_lv03 220 /dev/sdl
[root@localhost ~]# multipath /dev/sdj : checker msg is "tur checker reports path is down" : checker msg is "tur checker reports path is down" [root@localhost ~]# multipath /dev/sdw : checker msg is "tur checker reports path is down" : checker msg is "tur checker reports path is down"
[root@localhost slaves]# dd if=/dev/sdw of=/dev/null count=1 iflag=direct 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.000432 seconds, 1.2 MB/s [root@localhost slaves]# dd if=/dev/sdj of=/dev/null count=1 iflag=direct 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.000146 seconds, 3.5 MB/s The problem like this : http://www.spinics.net/lists/dm-devel/msg16857.html
I add the verbosity 3 to multipath.conf : [root@localhost ~]# dmsetup table 26134613837363132: 0 838860800 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:64 1000 65:64 1000 23233313238646237: 0 1048576000 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:160 1000 65:112 1000 2343537626435392d: 0 1048576000 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:80 1000 65:80 1000 26237306165653635: 0 838860800 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:48 1000 65:48 1000 VolGroup01-LogVol01: 0 12189696 linear 8:2 612696448 VolGroup01-LogVol00: 0 612696064 linear 8:2 384 23330623031356330: 0 1048576000 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:176 1000 65:128 1000 23930346664653862: 0 838860800 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:16 1000 65:16 1000 23833656434366663: 0 838860800 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:32 1000 65:32 1000 23137663532353265: 0 1048576000 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:96 1000 8:224 1000 [root@localhost ~]# multipath -ll dm-0: blacklisted dm-1: blacklisted dm-2: blacklisted dm-3: blacklisted dm-4: blacklisted dm-5: blacklisted dm-6: blacklisted dm-7: blacklisted dm-8: blacklisted dm-9: blacklisted md0: blacklisted ram0: blacklisted ram10: blacklisted ram11: blacklisted ram12: blacklisted ram13: blacklisted ram14: blacklisted ram15: blacklisted ram1: blacklisted ram2: blacklisted ram3: blacklisted ram4: blacklisted ram5: blacklisted ram6: blacklisted ram7: blacklisted ram8: blacklisted ram9: blacklisted sda: blacklisted sdb: not found in pathvec sdb: mask = 0x5 sdb: bus = 1 sdb: dev_t = 8:16 sdb: size = 838860800 sdb: vendor = _INSPUR_ sdb: product = vg0_lv00 sdb: rev = 220 sdb: h:b:t:l = 744:0:0:1 sdb: path checker = tur (config file default) sdb: checker timeout = 60000 ms (sysfs setting) sdb: state = 2 sdc: not found in pathvec sdc: mask = 0x5 sdc: bus = 1 sdc: dev_t = 8:32 sdc: size = 838860800 sdc: vendor = _INSPUR_ sdc: product = vg0_lv01 sdc: rev = 220 sdc: h:b:t:l = 744:0:0:2 sdc: path checker = tur (config file default) sdc: checker timeout = 60000 ms (sysfs setting) sdc: state = 2 sdd: not found in pathvec sdd: mask = 0x5 sdd: bus = 1 sdd: dev_t = 8:48 sdd: size = 838860800 sdd: vendor = _INSPUR_ sdd: product = vg0_lv02 sdd: rev = 220 sdd: h:b:t:l = 744:0:0:3 sdd: path checker = tur (config file default) sdd: checker timeout = 60000 ms (sysfs setting) sdd: state = 2 sde: not found in pathvec sde: mask = 0x5 sde: bus = 1 sde: dev_t = 8:64 sde: size = 838860800 sde: vendor = _INSPUR_ sde: product = vg0_lv03 sde: rev = 220 sde: h:b:t:l = 744:0:0:4 sde: path checker = tur (config file default) sde: checker timeout = 60000 ms (sysfs setting) sde: state = 2 sdf: not found in pathvec sdf: mask = 0x5 sdf: bus = 1 sdf: dev_t = 8:80 sdf: size = 1048576000 sdf: vendor = _INSPUR_ sdf: product = vg19_lv00 sdf: rev = 220 sdf: h:b:t:l = 744:0:0:5 sdf: path checker = tur (config file default) sdf: checker timeout = 60000 ms (sysfs setting) sdf: state = 2 sdj: not found in pathvec sdj: mask = 0x5 sdj: bus = 1 sdj: dev_t = 8:144 sdj: size = 1048576000 sdj: vendor = _INSPUR_ sdj: product = vg19_lv01 sdj: rev = 220 sdj: h:b:t:l = 744:0:0:6 sdj: path checker = tur (config file default) sdj: checker timeout = 60000 ms (sysfs setting) sdj: state = 2 sdk: not found in pathvec sdk: mask = 0x5 sdk: bus = 1 sdk: dev_t = 8:160 sdk: size = 1048576000 sdk: vendor = _INSPUR_ sdk: product = vg19_lv02 sdk: rev = 220 sdk: h:b:t:l = 744:0:0:7 sdk: path checker = tur (config file default) sdk: checker timeout = 60000 ms (sysfs setting) sdk: state = 2 sdl: not found in pathvec sdl: mask = 0x5 sdl: bus = 1 sdl: dev_t = 8:176 sdl: size = 1048576000 sdl: vendor = _INSPUR_ sdl: product = vg19_lv03 sdl: rev = 220 sdl: h:b:t:l = 744:0:0:8 sdl: path checker = tur (config file default) sdl: checker timeout = 60000 ms (sysfs setting) sdl: state = 2 sdr: not found in pathvec sdr: mask = 0x5 sdr: bus = 1 sdr: dev_t = 65:16 sdr: size = 838860800 sdr: vendor = _INSPUR_ sdr: product = vg0_lv00 sdr: rev = 220 sdr: h:b:t:l = 9:0:0:1 sdr: path checker = tur (config file default) sdr: checker timeout = 60000 ms (sysfs setting) sdr: state = 2 sds: not found in pathvec sds: mask = 0x5 sds: bus = 1 sds: dev_t = 65:32 sds: size = 838860800 sds: vendor = _INSPUR_ sds: product = vg0_lv01 sds: rev = 220 sds: h:b:t:l = 9:0:0:2 sds: path checker = tur (config file default) sds: checker timeout = 60000 ms (sysfs setting) sds: state = 2 sdt: not found in pathvec sdt: mask = 0x5 sdt: bus = 1 sdt: dev_t = 65:48 sdt: size = 838860800 sdt: vendor = _INSPUR_ sdt: product = vg0_lv02 sdt: rev = 220 sdt: h:b:t:l = 9:0:0:3 sdt: path checker = tur (config file default) sdt: checker timeout = 60000 ms (sysfs setting) sdt: state = 2 sdu: not found in pathvec sdu: mask = 0x5 sdu: bus = 1 sdu: dev_t = 65:64 sdu: size = 838860800 sdu: vendor = _INSPUR_ sdu: product = vg0_lv03 sdu: rev = 220 sdu: h:b:t:l = 9:0:0:4 sdu: path checker = tur (config file default) sdu: checker timeout = 60000 ms (sysfs setting) sdu: state = 2 sdv: not found in pathvec sdv: mask = 0x5 sdv: bus = 1 sdv: dev_t = 65:80 sdv: size = 1048576000 sdv: vendor = _INSPUR_ sdv: product = vg19_lv00 sdv: rev = 220 sdv: h:b:t:l = 9:0:0:5 sdv: path checker = tur (config file default) sdv: checker timeout = 60000 ms (sysfs setting) sdv: state = 2 sdw: not found in pathvec sdw: mask = 0x5 sdw: bus = 1 sdw: dev_t = 65:96 sdw: size = 1048576000 sdw: vendor = _INSPUR_ sdw: product = vg19_lv01 sdw: rev = 220 sdw: h:b:t:l = 9:0:0:6 sdw: path checker = tur (config file default) sdw: checker timeout = 60000 ms (sysfs setting) sdw: state = 2 sdx: not found in pathvec sdx: mask = 0x5 sdx: bus = 1 sdx: dev_t = 65:112 sdx: size = 1048576000 sdx: vendor = _INSPUR_ sdx: product = vg19_lv02 sdx: rev = 220 sdx: h:b:t:l = 9:0:0:7 sdx: path checker = tur (config file default) sdx: checker timeout = 60000 ms (sysfs setting) sdx: state = 2 sdy: not found in pathvec sdy: mask = 0x5 sdy: bus = 1 sdy: dev_t = 65:128 sdy: size = 1048576000 sdy: vendor = _INSPUR_ sdy: product = vg19_lv03 sdy: rev = 220 sdy: h:b:t:l = 9:0:0:8 sdy: path checker = tur (config file default) sdy: checker timeout = 60000 ms (sysfs setting) sdy: state = 2 sr0: blacklisted ===== paths list ===== uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev 744:0:0:1 sdb 8:16 0 [undef][ready] _INSPUR_,vg0_lv00 744:0:0:2 sdc 8:32 0 [undef][ready] _INSPUR_,vg0_lv01 744:0:0:3 sdd 8:48 0 [undef][ready] _INSPUR_,vg0_lv02 744:0:0:4 sde 8:64 0 [undef][ready] _INSPUR_,vg0_lv03 744:0:0:5 sdf 8:80 0 [undef][ready] _INSPUR_,vg19_lv00 744:0:0:6 sdj 8:144 0 [undef][ready] _INSPUR_,vg19_lv01 744:0:0:7 sdk 8:160 0 [undef][ready] _INSPUR_,vg19_lv02 744:0:0:8 sdl 8:176 0 [undef][ready] _INSPUR_,vg19_lv03 9:0:0:1 sdr 65:16 0 [undef][ready] _INSPUR_,vg0_lv00 9:0:0:2 sds 65:32 0 [undef][ready] _INSPUR_,vg0_lv01 9:0:0:3 sdt 65:48 0 [undef][ready] _INSPUR_,vg0_lv02 9:0:0:4 sdu 65:64 0 [undef][ready] _INSPUR_,vg0_lv03 9:0:0:5 sdv 65:80 0 [undef][ready] _INSPUR_,vg19_lv00 9:0:0:6 sdw 65:96 0 [undef][ready] _INSPUR_,vg19_lv01 9:0:0:7 sdx 65:112 0 [undef][ready] _INSPUR_,vg19_lv02 9:0:0:8 sdy 65:128 0 [undef][ready] _INSPUR_,vg19_lv03 params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:64 1000 65:64 1000 status = 2 0 0 0 1 1 E 0 2 0 8:64 A 0 65:64 A 0 sde: mask = 0x8 sde: getprio = /bin/true (config file default) sde: prio = 0 sdu: mask = 0x8 sdu: getprio = /bin/true (config file default) sdu: prio = 0 26134613837363132 dm-6 _INSPUR_,vg0_lv03 [size=400G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][enabled] \_ 744:0:0:4 sde 8:64 [active][ready] \_ 9:0:0:4 sdu 65:64 [active][ready] params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:160 1000 65:112 1000 status = 2 0 0 0 1 1 A 0 2 0 8:160 A 0 65:112 A 0 sdk: mask = 0x8 sdk: getprio = /bin/true (config file default) sdk: prio = 0 sdx: mask = 0x8 sdx: getprio = /bin/true (config file default) sdx: prio = 0 23233313238646237 dm-3 _INSPUR_,vg19_lv02 [size=500G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][active] \_ 744:0:0:7 sdk 8:160 [active][ready] \_ 9:0:0:7 sdx 65:112 [active][ready] params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:80 1000 65:80 1000 status = 2 0 0 0 1 1 E 0 2 0 8:80 A 0 65:80 A 0 sdf: mask = 0x8 sdf: getprio = /bin/true (config file default) sdf: prio = 0 sdv: mask = 0x8 sdv: getprio = /bin/true (config file default) sdv: prio = 0 2343537626435392d dm-5 _INSPUR_,vg19_lv00 [size=500G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][enabled] \_ 744:0:0:5 sdf 8:80 [active][ready] \_ 9:0:0:5 sdv 65:80 [active][ready] params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:48 1000 65:48 1000 status = 2 0 0 0 1 1 E 0 2 0 8:48 A 0 65:48 A 0 sdd: mask = 0x8 sdd: getprio = /bin/true (config file default) sdd: prio = 0 sdt: mask = 0x8 sdt: getprio = /bin/true (config file default) sdt: prio = 0 26237306165653635 dm-4 _INSPUR_,vg0_lv02 [size=400G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][enabled] \_ 744:0:0:3 sdd 8:48 [active][ready] \_ 9:0:0:3 sdt 65:48 [active][ready] params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:176 1000 65:128 1000 status = 2 0 0 0 1 1 A 0 2 0 8:176 A 0 65:128 A 0 sdl: mask = 0x8 sdl: getprio = /bin/true (config file default) sdl: prio = 0 sdy: mask = 0x8 sdy: getprio = /bin/true (config file default) sdy: prio = 0 23330623031356330 dm-7 _INSPUR_,vg19_lv03 [size=500G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][active] \_ 744:0:0:8 sdl 8:176 [active][ready] \_ 9:0:0:8 sdy 65:128 [active][ready] params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:16 1000 65:16 1000 status = 2 0 0 0 1 1 E 0 2 0 8:16 A 0 65:16 A 0 sdb: mask = 0x8 sdb: getprio = /bin/true (config file default) sdb: prio = 0 sdr: mask = 0x8 sdr: getprio = /bin/true (config file default) sdr: prio = 0 23930346664653862 dm-2 _INSPUR_,vg0_lv00 [size=400G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][enabled] \_ 744:0:0:1 sdb 8:16 [active][ready] \_ 9:0:0:1 sdr 65:16 [active][ready] params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:32 1000 65:32 1000 status = 2 0 0 0 1 1 E 0 2 0 8:32 A 0 65:32 A 0 sdc: mask = 0x8 sdc: getprio = /bin/true (config file default) sdc: prio = 0 sds: mask = 0x8 sds: getprio = /bin/true (config file default) sds: prio = 0 23833656434366663 dm-9 _INSPUR_,vg0_lv01 [size=400G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][enabled] \_ 744:0:0:2 sdc 8:32 [active][ready] \_ 9:0:0:2 sds 65:32 [active][ready] params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:96 1000 8:224 1000 status = 2 18546 0 0 1 1 E 0 2 0 8:96 F 28 8:224 F 28 8:96: not found in pathvec 8:224: not found in pathvec 23137663532353265 dm-8 , [size=500G][features=1 queue_if_no_path][hwhandler=0][rw] \_ round-robin 0 [prio=0][enabled] \_ #:#:#:# - #:# [failed][faulty] \_ #:#:#:# - #:# [failed][faulty]
I test again and find some problem,the disk which showed faulty was hanging with multipath %M:%m. [root@localhost slaves]# ps -ef |grep multi root 10603 1 0 17:07 ? 00:00:08 multipathd -d root 11301 1 0 17:20 ? 00:00:06 /sbin/multipath 8:96 root 11991 1 0 17:21 ? 00:00:00 /sbin/multipath 65:16 Maybe scsi_error_handler hang the cmd ? [root@localhost slaves]#echo t >/proc/sysrq-trigger Jul 26 18:57:05 localhost kernel: multipath D ffff8100cca1c7a0 0 11991 1 12483 11301 (NOTLB) Jul 26 18:57:05 localhost kernel: ffff8100a775fbe8 0000000000000086 0000000000000100 ffff8101159cc2c0 Jul 26 18:57:05 localhost kernel: 00000000000000d0 0000000000000002 ffff8100cca1c7a0 ffff81000ab9e820 Jul 26 18:57:05 localhost kernel: 00000418b1ad4052 0000000000106bb4 ffff8100cca1c988 0000000188949183 Jul 26 18:57:05 localhost kernel: Call Trace: Jul 26 18:57:05 localhost kernel: [<ffffffff801566c6>] __next_cpu+0x19/0x28 Jul 26 18:57:05 localhost kernel: [<ffffffff882d86a5>] :dm_mod:dev_suspend+0x0/0x18d Jul 26 18:57:05 localhost kernel: [<ffffffff80064a1b>] __down+0xc3/0xd8 Jul 26 18:57:05 localhost kernel: [<ffffffff8008ee79>] default_wake_function+0x0/0xe Jul 26 18:57:05 localhost kernel: [<ffffffff800646d9>] __down_failed+0x35/0x3a Jul 26 18:57:05 localhost kernel: [<ffffffff882d5afd>] :dm_mod:.text.lock.dm+0x2d/0x3c Jul 26 18:57:05 localhost kernel: [<ffffffff8008ee79>] default_wake_function+0x0/0xe Jul 26 18:57:05 localhost kernel: [<ffffffff882d879d>] :dm_mod:dev_suspend+0xf8/0x18d Jul 26 18:57:05 localhost kernel: [<ffffffff882d9064>] :dm_mod:ctl_ioctl+0x210/0x25b Jul 26 18:57:05 localhost kernel: [<ffffffff80041eba>] do_ioctl+0x55/0x6b Jul 26 18:57:05 localhost kernel: [<ffffffff8002ff14>] vfs_ioctl+0x457/0x4b9 Jul 26 18:57:05 localhost kernel: [<ffffffff800ba76c>] audit_syscall_entry+0x1a8/0x1d3 Jul 26 18:57:05 localhost kernel: [<ffffffff8004c288>] sys_ioctl+0x59/0x78 Jul 26 18:57:05 localhost kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0 Jul 26 18:57:05 localhost kernel: scsi_eh_26 S 0000000000000282 0 11783 55 12699 10881 (L-TLB) Jul 26 18:57:05 localhost kernel: ffff81004711be60 0000000000000046 ffff81004711bdd0 ffffffff8008e88f Jul 26 18:57:05 localhost kernel: 0000041571d83579 0000000000000009 ffff8100a4f340c0 ffff81011e58c080 Jul 26 18:57:05 localhost kernel: 0000041571dd044f 00000000000007a1 ffff8100a4f342a8 00000002a4f340f8 Jul 26 18:57:05 localhost kernel: Call Trace: Jul 26 18:57:05 localhost kernel: [<ffffffff8008e88f>] __activate_task+0x56/0x6d Jul 26 18:57:05 localhost kernel: [<ffffffff80063002>] thread_return+0x62/0xfe Jul 26 18:57:05 localhost kernel: [<ffffffff800a3295>] keventd_create_kthread+0x0/0xc4 Jul 26 18:57:05 localhost kernel: [<ffffffff880792d6>] :scsi_mod:scsi_error_handler+0x6a/0x4ce Jul 26 18:57:05 localhost kernel: [<ffffffff800a3295>] keventd_create_kthread+0x0/0xc4 Jul 26 18:57:05 localhost kernel: [<ffffffff8807926c>] :scsi_mod:scsi_error_handler+0x0/0x4ce Jul 26 18:57:05 localhost kernel: [<ffffffff800a3295>] keventd_create_kthread+0x0/0xc4 Jul 26 18:57:05 localhost kernel: [<ffffffff80032660>] kthread+0xfe/0x132 Jul 26 18:57:05 localhost kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11 Jul 26 18:57:05 localhost kernel: [<ffffffff800a3295>] keventd_create_kthread+0x0/0xc4 Jul 26 18:57:05 localhost kernel: [<ffffffff80152b3e>] cfq_latter_request+0x0/0x1e Jul 26 18:57:05 localhost kernel: [<ffffffff8804d50d>] :ext3:bput_one+0x0/0x7 Jul 26 18:57:05 localhost kernel: [<ffffffff80032562>] kthread+0x0/0x132 Jul 26 18:57:05 localhost kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11
I put the command: [root@localhost ~]# dmsetup message /dev/dm-8 1 fail_if_no_path And all requeue io in dm-8 returned with -EIO(function do_end_io err=-5 m->queue_if_no_path 0) The command /sbin/multipath 65:16 would not hang and successful. The path changed to active. I didn't know why add disk to path was not successful when there were some requeue io? Was it a bug?
When occur this bug,I used gdb -p pid or strace -p pid to dbg the hanging process of /sbin/multipath %M:%m.The process will not hang and return.The path will become active. gdb) where #0 0x00000032612cc557 in ioctl () from /lib64/libc.so.6 #1 0x0000003f6f61c256 in ?? () from /lib64/libdevmapper.so.1.02 #2 0x0000003f6f619d0b in dm_task_run () from /lib64/libdevmapper.so.1.02 #3 0x00000000004074c4 in dm_simplecmd (task=5, name=0x1e59ce30 "23930346664653862", no_flush=1) at devmapper.c:179 #4 0x00000000004061f2 in domap (mpp=0x1e597250) at configure.c:385 #5 0x0000000000406853 in coalesce_paths (vecs=0x7fff54b2e490, newmp=0x0, refwwid=0x0) at configure.c:560 #6 0x0000000000403419 in configure () at main.c:295 #7 0x0000000000403912 in main (argc=2, argv=0x7fff54b2e5f8) at main.c:
Are you really running this on RHEL 5.4? There have been a large number of kernel changes that may have fixed this. Would it be possible to try and reproduce this on RHEL 5.8?
I test the RHEL 6.2.Everything is well.I don't know whether 5.8 is well.
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux release for currently deployed products. This request is not yet committed for inclusion in a release.
I can't hit this with a recent kernel. If you can please let me know.
This is the last RHEL5 release, and I can't fix this without being able to reproduce it.