Bug 843309 - multipath can't remap some disk [NEEDINFO]
multipath can't remap some disk
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: device-mapper-multipath (Show other bugs)
5.4
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Ben Marzinski
Storage QE
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-07-25 22:56 EDT by 25641463
Modified: 2014-03-24 19:34 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-03-24 19:34:53 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
bmarzins: needinfo? (yangfanlinux)


Attachments (Terms of Use)

  None (edit)
Description 25641463 2012-07-25 22:56:13 EDT
Description of problem:
I test the IB-SRP initiator (RHEL 5.4 x86_64)
The multipath can't remap some disk when total number of usable paths  was from failover to active.And fdisk -l was hanging. I didn't know why the disk was in system,but multipath can't remap the disk to dm-8.

[root@localhost sdw]# multipath -l
26134613837363132 dm-6 _INSPUR_,vg0_lv03
[size=400G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 744:0:0:4 sde 8:64   [active][undef]
 \_ 9:0:0:4   sdu 65:64  [active][undef]
23233313238646237 dm-3 _INSPUR_,vg19_lv02
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 744:0:0:7 sdk 8:160  [active][undef]
 \_ 9:0:0:7   sdx 65:112 [active][undef]
2343537626435392d dm-5 _INSPUR_,vg19_lv00
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 744:0:0:5 sdf 8:80   [active][undef]
 \_ 9:0:0:5   sdv 65:80  [active][undef]
26237306165653635 dm-4 _INSPUR_,vg0_lv02
[size=400G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 744:0:0:3 sdd 8:48   [active][undef]
 \_ 9:0:0:3   sdt 65:48  [active][undef]
23330623031356330 dm-7 _INSPUR_,vg19_lv03
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 744:0:0:8 sdl 8:176  [active][undef]
 \_ 9:0:0:8   sdy 65:128 [active][undef]
23930346664653862 dm-2 _INSPUR_,vg0_lv00
[size=400G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 744:0:0:1 sdb 8:16   [active][undef]
 \_ 9:0:0:1   sdr 65:16  [active][undef]
23833656434366663 dm-9 _INSPUR_,vg0_lv01
[size=400G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 744:0:0:2 sdc 8:32   [active][undef]
 \_ 9:0:0:2   sds 65:32  [active][undef]
23137663532353265 dm-8 ,
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][enabled]
 \_ #:#:#:#   -   #:#    [failed][undef]
 \_ #:#:#:#   -   #:#    [failed][undef]

The dm-8 can't remap the disk(sdj sdw).But the node was exist in /dev/sdj /dev/sdw.And I can R/W the disk(dd).The sdg and sdo  before dm-8 failover was still residual(sdg and sdo were not exist in /dev/sdg /dev/sdo).There were some bio requeued in the dm-8.But because the dm-8 was not recovery,IO would always requeue.

[root@localhost slaves]# pwd
/sys/block/dm-8/slaves
[root@localhost slaves]# ls -l
lrwxrwxrwx 1 root root 0 07-24 20:38 sdg -> ../../../block/sdg
lrwxrwxrwx 1 root root 0 07-25 09:31 sdj -> ../../../block/sdj
lrwxrwxrwx 1 root root 0 07-24 20:38 sdo -> ../../../block/sdo
lrwxrwxrwx 1 root root 0 07-24 20:38 sdw -> ../../../block/sdw

The multipath.conf :
defaults {
#       user_friendly_names     yes
        udev_dir                /dev
        polling_interval        10
        selector                "round-robin 0"
        path_grouping_policy    multibus
        getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
        prio_callout            /bin/true
        features                "1 queue_if_no_path"
        path_checker            tur
        rr_min_io               1000
        max_fds                 8192
        rr_weight               priorities
        failback                immediate
        no_path_retry           queue
#        queue_without_daemon    no
}



Version-Release number of selected component (if applicable):
Name       : kpartx
Arch       : x86_64
Version    : 0.4.7
Release    : 48.el5_8.1

Name       : device-mapper-multipath
Arch       : x86_64
Version    : 0.4.7
Release    : 48.el5_8.1

How reproducible:


Steps to Reproduce:
1.Using IB-srp to map some disk.
2.Mount the disk with ext3 and dd if=/dev/zero of=./testfile bs=1M count=100000
2.Cut all the connection .It cause the multipath-device faulty.
3. srp_daemon.sh will reconnect the disk var IB-SRP later.
  
Actual results:
some disk maybe not in dm device

Expected results:
All the disk were active and ready

Additional info:
Comment 1 25641463 2012-07-25 23:09:52 EDT
I used multipath -F.It was hanging.
The udev config:

[root@localhost rules.d]# pwd
/etc/udev/rules.d
[root@localhost rules.d]# cat 40-multipath.rules 
# multipath wants the devmaps presented as meaninglful device names
# so name them after their devmap name
SUBSYSTEM!="block", GOTO="end_mpath"
KERNEL!="dm-[0-9]*", GOTO="end_mpath"
PROGRAM!="/sbin/mpath_wait %M %m", GOTO="end_mpath"
PROGRAM!="/sbin/dmsetup info -c --noheadings -j %M -m %m", GOTO="end_mpath"
RESULT!="*:*:*:*:*:*:*:mpath-*", GOTO="kpartx_check"
PROGRAM!="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", GOTO="end_mpath"
RESULT!="?*", GOTO="end_mpath"
NAME="%k", SYMLINK="mpath/%c"
ACTION!="change", GOTO="skip_kpartx"
RUN+="/bin/bash -c '/sbin/mpath_wait /dev/mapper/%c; /sbin/kpartx -a -p p /dev/mapper/%c'"
LABEL="skip_kpartx"
OPTIONS="last_rule"
LABEL="kpartx_check"
RESULT!="*:*:*:*:*:*:*:part*-mpath-*", GOTO="end_mpath"
PROGRAM=="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", RESULT=="?*", NAME="%k", SYMLINK="mpath/%c"
OPTIONS="last_rule"
LABEL="end_mpath"

[root@localhost rules.d]# cat 91-srp.rules 
ACTION=="add", KERNEL=="sd*[!0-9]", RUN+="/sbin/multipath %M:%m"
Comment 2 25641463 2012-07-25 23:13:19 EDT
[root@localhost ~]# sg_map
/dev/sg0  /dev/sda
/dev/sg1  /dev/scd0
/dev/sg2  /dev/sdb
/dev/sg3  /dev/sdc
/dev/sg4  /dev/sdd
/dev/sg5  /dev/sde
/dev/sg6  /dev/sdf
/dev/sg7  /dev/sdj
/dev/sg8  /dev/sdk
/dev/sg9  /dev/sdl
/dev/sg10  /dev/sdr
/dev/sg11  /dev/sds
/dev/sg12  /dev/sdt
/dev/sg13  /dev/sdu
/dev/sg14  /dev/sdv
/dev/sg15  /dev/sdw
/dev/sg16  /dev/sdx
/dev/sg17  /dev/sdy

[root@localhost rules.d]# multipath -v4
dm-0: blacklisted
dm-1: blacklisted
dm-2: blacklisted
dm-3: blacklisted
dm-4: blacklisted
dm-5: blacklisted
dm-6: blacklisted
dm-7: blacklisted
dm-8: blacklisted
dm-9: blacklisted
md0: blacklisted
ram0: blacklisted
ram10: blacklisted
ram11: blacklisted
ram12: blacklisted
ram13: blacklisted
ram14: blacklisted
ram15: blacklisted
ram1: blacklisted
ram2: blacklisted
ram3: blacklisted
ram4: blacklisted
ram5: blacklisted
ram6: blacklisted
ram7: blacklisted
ram8: blacklisted
ram9: blacklisted
sda: blacklisted
sdb: not found in pathvec
sdb: mask = 0x1f
sdb: bus = 1
sdb: dev_t = 8:16
sdb: size = 838860800
sdb: vendor = _INSPUR_
sdb: product = vg0_lv00
sdb: rev =  220
sdb: h:b:t:l = 744:0:0:1
sdb: serial = 904fde8b
sdb: path checker = tur (config file default)
sdb: checker timeout = 60000 ms (sysfs setting)
sdb: state = 2
sdb: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default)
sdb: uid = 23930346664653862 (callout)
sdb: getprio = /bin/true (config file default)
sdb: prio = 0
sdc: not found in pathvec
sdc: mask = 0x1f
sdc: bus = 1
sdc: dev_t = 8:32
sdc: size = 838860800
sdc: vendor = _INSPUR_
sdc: product = vg0_lv01
sdc: rev =  220
sdc: h:b:t:l = 744:0:0:2
sdc: serial = 83ed46fc
sdc: path checker = tur (config file default)
sdc: checker timeout = 60000 ms (sysfs setting)
sdc: state = 2
sdc: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default)
sdc: uid = 23833656434366663 (callout)
sdc: getprio = /bin/true (config file default)
sdc: prio = 0
sdd: not found in pathvec
sdd: mask = 0x1f
sdd: bus = 1
sdd: dev_t = 8:48
sdd: size = 838860800
sdd: vendor = _INSPUR_
sdd: product = vg0_lv02
sdd: rev =  220
sdd: h:b:t:l = 744:0:0:3
sdd: serial = b70aee65
sdd: path checker = tur (config file default)
sdd: checker timeout = 60000 ms (sysfs setting)
sdd: state = 2
sdd: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default)
sdd: uid = 26237306165653635 (callout)
sdd: getprio = /bin/true (config file default)
sdd: prio = 0
sde: not found in pathvec
sde: mask = 0x1f
sde: bus = 1
sde: dev_t = 8:64
sde: size = 838860800
sde: vendor = _INSPUR_
sde: product = vg0_lv03
sde: rev =  220
sde: h:b:t:l = 744:0:0:4
sde: serial = a4a87612
sde: path checker = tur (config file default)
sde: checker timeout = 60000 ms (sysfs setting)
sde: state = 2
sde: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default)
sde: uid = 26134613837363132 (callout)
sde: getprio = /bin/true (config file default)
sde: prio = 0
sdf: not found in pathvec
sdf: mask = 0x1f
sdf: bus = 1
sdf: dev_t = 8:80
sdf: size = 1048576000
sdf: vendor = _INSPUR_
sdf: product = vg19_lv00
sdf: rev =  220
sdf: h:b:t:l = 744:0:0:5
sdf: serial = 457bd59
sdf: path checker = tur (config file default)
sdf: checker timeout = 60000 ms (sysfs setting)
sdf: state = 2
sdf: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default)
sdf: uid = 2343537626435392d (callout)
sdf: getprio = /bin/true (config file default)
sdf: prio = 0
sdj: not found in pathvec
sdj: mask = 0x1f
sdj: bus = 1
sdj: dev_t = 8:144
sdj: size = 1048576000
sdj: vendor = _INSPUR_
sdj: product = vg19_lv01
sdj: rev =  220
sdj: h:b:t:l = 744:0:0:6
sdj: serial = 17f5252e
sdj: path checker = tur (config file default)
sdj: checker timeout = 60000 ms (sysfs setting)
sdj: state = 2
sdj: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default)
sdj: uid = 23137663532353265 (callout)
sdj: getprio = /bin/true (config file default)
sdj: prio = 0
sdk: not found in pathvec
sdk: mask = 0x1f
sdk: bus = 1
sdk: dev_t = 8:160
sdk: size = 1048576000
sdk: vendor = _INSPUR_
sdk: product = vg19_lv02
sdk: rev =  220
sdk: h:b:t:l = 744:0:0:7
sdk: serial = 23128db7
sdk: path checker = tur (config file default)
sdk: checker timeout = 60000 ms (sysfs setting)
sdk: state = 2
sdk: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default)
sdk: uid = 23233313238646237 (callout)
sdk: getprio = /bin/true (config file default)
sdk: prio = 0
sdl: not found in pathvec
sdl: mask = 0x1f
sdl: bus = 1
sdl: dev_t = 8:176
sdl: size = 1048576000
sdl: vendor = _INSPUR_
sdl: product = vg19_lv03
sdl: rev =  220
sdl: h:b:t:l = 744:0:0:8
sdl: serial = 30b015c0
sdl: path checker = tur (config file default)
sdl: checker timeout = 60000 ms (sysfs setting)
sdl: state = 2
sdl: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default)
sdl: uid = 23330623031356330 (callout)
sdl: getprio = /bin/true (config file default)
sdl: prio = 0
sdr: not found in pathvec
sdr: mask = 0x1f
sdr: bus = 1
sdr: dev_t = 65:16
sdr: size = 838860800
sdr: vendor = _INSPUR_
sdr: product = vg0_lv00
sdr: rev =  220
sdr: h:b:t:l = 9:0:0:1
sdr: serial = 904fde8b
sdr: path checker = tur (config file default)
sdr: checker timeout = 60000 ms (sysfs setting)
sdr: state = 2
sdr: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default)
sdr: uid = 23930346664653862 (callout)
sdr: getprio = /bin/true (config file default)
sdr: prio = 0
sds: not found in pathvec
sds: mask = 0x1f
sds: bus = 1
sds: dev_t = 65:32
sds: size = 838860800
sds: vendor = _INSPUR_
sds: product = vg0_lv01
sds: rev =  220
sds: h:b:t:l = 9:0:0:2
sds: serial = 83ed46fc
sds: path checker = tur (config file default)
sds: checker timeout = 60000 ms (sysfs setting)
sds: state = 2
sds: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default)
sds: uid = 23833656434366663 (callout)
sds: getprio = /bin/true (config file default)
sds: prio = 0
sdt: not found in pathvec
sdt: mask = 0x1f
sdt: bus = 1
sdt: dev_t = 65:48
sdt: size = 838860800
sdt: vendor = _INSPUR_
sdt: product = vg0_lv02
sdt: rev =  220
sdt: h:b:t:l = 9:0:0:3
sdt: serial = b70aee65
sdt: path checker = tur (config file default)
sdt: checker timeout = 60000 ms (sysfs setting)
sdt: state = 2
sdt: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default)
sdt: uid = 26237306165653635 (callout)
sdt: getprio = /bin/true (config file default)
sdt: prio = 0
sdu: not found in pathvec
sdu: mask = 0x1f
sdu: bus = 1
sdu: dev_t = 65:64
sdu: size = 838860800
sdu: vendor = _INSPUR_
sdu: product = vg0_lv03
sdu: rev =  220
sdu: h:b:t:l = 9:0:0:4
sdu: serial = a4a87612
sdu: path checker = tur (config file default)
sdu: checker timeout = 60000 ms (sysfs setting)
sdu: state = 2
sdu: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default)
sdu: uid = 26134613837363132 (callout)
sdu: getprio = /bin/true (config file default)
sdu: prio = 0
sdv: not found in pathvec
sdv: mask = 0x1f
sdv: bus = 1
sdv: dev_t = 65:80
sdv: size = 1048576000
sdv: vendor = _INSPUR_
sdv: product = vg19_lv00
sdv: rev =  220
sdv: h:b:t:l = 9:0:0:5
sdv: serial = 457bd59
sdv: path checker = tur (config file default)
sdv: checker timeout = 60000 ms (sysfs setting)
sdv: state = 2
sdv: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default)
sdv: uid = 2343537626435392d (callout)
sdv: getprio = /bin/true (config file default)
sdv: prio = 0
sdw: not found in pathvec
sdw: mask = 0x1f
sdw: bus = 1
sdw: dev_t = 65:96
sdw: size = 1048576000
sdw: vendor = _INSPUR_
sdw: product = vg19_lv01
sdw: rev =  220
sdw: h:b:t:l = 9:0:0:6
sdw: serial = 17f5252e
sdw: path checker = tur (config file default)
sdw: checker timeout = 60000 ms (sysfs setting)
sdw: state = 2
sdw: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default)
sdw: uid = 23137663532353265 (callout)
sdw: getprio = /bin/true (config file default)
sdw: prio = 0
sdx: not found in pathvec
sdx: mask = 0x1f
sdx: bus = 1
sdx: dev_t = 65:112
sdx: size = 1048576000
sdx: vendor = _INSPUR_
sdx: product = vg19_lv02
sdx: rev =  220
sdx: h:b:t:l = 9:0:0:7
sdx: serial = 23128db7
sdx: path checker = tur (config file default)
sdx: checker timeout = 60000 ms (sysfs setting)
sdx: state = 2
sdx: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default)
sdx: uid = 23233313238646237 (callout)
sdx: getprio = /bin/true (config file default)
sdx: prio = 0
sdy: not found in pathvec
sdy: mask = 0x1f
sdy: bus = 1
sdy: dev_t = 65:128
sdy: size = 1048576000
sdy: vendor = _INSPUR_
sdy: product = vg19_lv03
sdy: rev =  220
sdy: h:b:t:l = 9:0:0:8
sdy: serial = 30b015c0
sdy: path checker = tur (config file default)
sdy: checker timeout = 60000 ms (sysfs setting)
sdy: state = 2
sdy: getuid = /sbin/scsi_id -g -u -s /block/%n (config file default)
sdy: uid = 23330623031356330 (callout)
sdy: getprio = /bin/true (config file default)
sdy: prio = 0
sr0: blacklisted
===== paths list =====
uuid              hcil      dev dev_t  pri dm_st  chk_st  vend/prod/rev     
23930346664653862 744:0:0:1 sdb 8:16   0   [undef][ready] _INSPUR_,vg0_lv00 
23833656434366663 744:0:0:2 sdc 8:32   0   [undef][ready] _INSPUR_,vg0_lv01 
26237306165653635 744:0:0:3 sdd 8:48   0   [undef][ready] _INSPUR_,vg0_lv02 
26134613837363132 744:0:0:4 sde 8:64   0   [undef][ready] _INSPUR_,vg0_lv03 
2343537626435392d 744:0:0:5 sdf 8:80   0   [undef][ready] _INSPUR_,vg19_lv00
23137663532353265 744:0:0:6 sdj 8:144  0   [undef][ready] _INSPUR_,vg19_lv01
23233313238646237 744:0:0:7 sdk 8:160  0   [undef][ready] _INSPUR_,vg19_lv02
23330623031356330 744:0:0:8 sdl 8:176  0   [undef][ready] _INSPUR_,vg19_lv03
23930346664653862 9:0:0:1   sdr 65:16  0   [undef][ready] _INSPUR_,vg0_lv00 
23833656434366663 9:0:0:2   sds 65:32  0   [undef][ready] _INSPUR_,vg0_lv01 
26237306165653635 9:0:0:3   sdt 65:48  0   [undef][ready] _INSPUR_,vg0_lv02 
26134613837363132 9:0:0:4   sdu 65:64  0   [undef][ready] _INSPUR_,vg0_lv03 
2343537626435392d 9:0:0:5   sdv 65:80  0   [undef][ready] _INSPUR_,vg19_lv00
23137663532353265 9:0:0:6   sdw 65:96  0   [undef][ready] _INSPUR_,vg19_lv01
23233313238646237 9:0:0:7   sdx 65:112 0   [undef][ready] _INSPUR_,vg19_lv02
23330623031356330 9:0:0:8   sdy 65:128 0   [undef][ready] _INSPUR_,vg19_lv03
params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:64 1000 65:64 1000 
status = 2 0 0 0 1 1 A 0 2 0 8:64 A 0 65:64 A 0 
*word = 1, len = 1
*word = queue_if_no_path, len = 16
*word = 0, len = 1
*word = 1, len = 1
*word = 1, len = 1
*word = round-robin, len = 11
*word = 0, len = 1
*word = 2, len = 1
*word = 1, len = 1
*word = 8:64, len = 4
*word = 1000, len = 4
*word = 65:64, len = 5
*word = 1000, len = 4
sde: mask = 0x8
sde: prio = 0
sdu: mask = 0x8
sdu: prio = 0
*word = 2, len = 1
*word = 0, len = 1
*word = 0, len = 1
*word = 1, len = 1
*word = A, len = 1
*word = 2, len = 1
*word = 0, len = 1
*word = A, len = 1
*word = 0, len = 1
*word = A, len = 1
*word = 0, len = 1
params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:160 1000 65:112 1000 
status = 2 0 0 0 1 1 A 0 2 0 8:160 A 0 65:112 A 0 
*word = 1, len = 1
*word = queue_if_no_path, len = 16
*word = 0, len = 1
*word = 1, len = 1
*word = 1, len = 1
*word = round-robin, len = 11
*word = 0, len = 1
*word = 2, len = 1
*word = 1, len = 1
*word = 8:160, len = 5
*word = 1000, len = 4
*word = 65:112, len = 6
*word = 1000, len = 4
sdk: mask = 0x8
sdk: prio = 0
sdx: mask = 0x8
sdx: prio = 0
*word = 2, len = 1
*word = 0, len = 1
*word = 0, len = 1
*word = 1, len = 1
*word = A, len = 1
*word = 2, len = 1
*word = 0, len = 1
*word = A, len = 1
*word = 0, len = 1
*word = A, len = 1
*word = 0, len = 1
params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:80 1000 65:80 1000 
status = 2 0 0 0 1 1 A 0 2 0 8:80 A 0 65:80 A 0 
*word = 1, len = 1
*word = queue_if_no_path, len = 16
*word = 0, len = 1
*word = 1, len = 1
*word = 1, len = 1
*word = round-robin, len = 11
*word = 0, len = 1
*word = 2, len = 1
*word = 1, len = 1
*word = 8:80, len = 4
*word = 1000, len = 4
*word = 65:80, len = 5
*word = 1000, len = 4
sdf: mask = 0x8
sdf: prio = 0
sdv: mask = 0x8
sdv: prio = 0
*word = 2, len = 1
*word = 0, len = 1
*word = 0, len = 1
*word = 1, len = 1
*word = A, len = 1
*word = 2, len = 1
*word = 0, len = 1
*word = A, len = 1
*word = 0, len = 1
*word = A, len = 1
*word = 0, len = 1
params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:48 1000 65:48 1000 
status = 2 0 0 0 1 1 A 0 2 0 8:48 A 0 65:48 A 0 
*word = 1, len = 1
*word = queue_if_no_path, len = 16
*word = 0, len = 1
*word = 1, len = 1
*word = 1, len = 1
*word = round-robin, len = 11
*word = 0, len = 1
*word = 2, len = 1
*word = 1, len = 1
*word = 8:48, len = 4
*word = 1000, len = 4
*word = 65:48, len = 5
*word = 1000, len = 4
sdd: mask = 0x8
sdd: prio = 0
sdt: mask = 0x8
sdt: prio = 0
*word = 2, len = 1
*word = 0, len = 1
*word = 0, len = 1
*word = 1, len = 1
*word = A, len = 1
*word = 2, len = 1
*word = 0, len = 1
*word = A, len = 1
*word = 0, len = 1
*word = A, len = 1
*word = 0, len = 1
params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:176 1000 65:128 1000 
status = 2 0 0 0 1 1 A 0 2 0 8:176 A 0 65:128 A 0 
*word = 1, len = 1
*word = queue_if_no_path, len = 16
*word = 0, len = 1
*word = 1, len = 1
*word = 1, len = 1
*word = round-robin, len = 11
*word = 0, len = 1
*word = 2, len = 1
*word = 1, len = 1
*word = 8:176, len = 5
*word = 1000, len = 4
*word = 65:128, len = 6
*word = 1000, len = 4
sdl: mask = 0x8
sdl: prio = 0
sdy: mask = 0x8
sdy: prio = 0
*word = 2, len = 1
*word = 0, len = 1
*word = 0, len = 1
*word = 1, len = 1
*word = A, len = 1
*word = 2, len = 1
*word = 0, len = 1
*word = A, len = 1
*word = 0, len = 1
*word = A, len = 1
*word = 0, len = 1
params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:16 1000 65:16 1000 
status = 2 0 0 0 1 1 A 0 2 0 8:16 A 0 65:16 A 0 
*word = 1, len = 1
*word = queue_if_no_path, len = 16
*word = 0, len = 1
*word = 1, len = 1
*word = 1, len = 1
*word = round-robin, len = 11
*word = 0, len = 1
*word = 2, len = 1
*word = 1, len = 1
*word = 8:16, len = 4
*word = 1000, len = 4
*word = 65:16, len = 5
*word = 1000, len = 4
sdb: mask = 0x8
sdb: prio = 0
sdr: mask = 0x8
sdr: prio = 0
*word = 2, len = 1
*word = 0, len = 1
*word = 0, len = 1
*word = 1, len = 1
*word = A, len = 1
*word = 2, len = 1
*word = 0, len = 1
*word = A, len = 1
*word = 0, len = 1
*word = A, len = 1
*word = 0, len = 1
params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:32 1000 65:32 1000 
status = 2 0 0 0 1 1 A 0 2 0 8:32 A 0 65:32 A 0 
*word = 1, len = 1
*word = queue_if_no_path, len = 16
*word = 0, len = 1
*word = 1, len = 1
*word = 1, len = 1
*word = round-robin, len = 11
*word = 0, len = 1
*word = 2, len = 1
*word = 1, len = 1
*word = 8:32, len = 4
*word = 1000, len = 4
*word = 65:32, len = 5
*word = 1000, len = 4
sdc: mask = 0x8
sdc: prio = 0
sds: mask = 0x8
sds: prio = 0
*word = 2, len = 1
*word = 0, len = 1
*word = 0, len = 1
*word = 1, len = 1
*word = A, len = 1
*word = 2, len = 1
*word = 0, len = 1
*word = A, len = 1
*word = 0, len = 1
*word = A, len = 1
*word = 0, len = 1
params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:96 1000 8:224 1000 
status = 2 18546 0 0 1 1 E 0 2 0 8:96 F 18 8:224 F 18 
*word = 1, len = 1
*word = queue_if_no_path, len = 16
*word = 0, len = 1
*word = 1, len = 1
*word = 1, len = 1
*word = round-robin, len = 11
*word = 0, len = 1
*word = 2, len = 1
*word = 1, len = 1
*word = 8:96, len = 4
8:96: not found in pathvec
*word = 1000, len = 4
*word = 8:224, len = 5
8:224: not found in pathvec
*word = 1000, len = 4
*word = 2, len = 1
*word = 0, len = 1
*word = 0, len = 1
*word = 1, len = 1
*word = E, len = 1
*word = 2, len = 1
*word = 0, len = 1
*word = F, len = 1
*word = 18, len = 2
*word = F, len = 1
*word = 18, len = 2
sdb: ownership set to 23930346664653862
sdb: not found in pathvec
sdb: mask = 0xc
sdb: path checker = tur (config file default)
sdb: checker timeout = 60000 ms (sysfs setting)
sdb: state = 2
sdb: prio = 0
sdr: ownership set to 23930346664653862
sdr: not found in pathvec
sdr: mask = 0xc
sdr: path checker = tur (config file default)
sdr: checker timeout = 60000 ms (sysfs setting)
sdr: state = 2
sdr: prio = 0
23930346664653862: pgfailback = -2 (config file default)
23930346664653862: pgpolicy = multibus (config file default)
23930346664653862: selector = round-robin 0 (internal default)
23930346664653862: features = 1 queue_if_no_path (internal default)
23930346664653862: hwhandler = 0 (internal default)
23930346664653862: rr_weight = 2 (config file default)
23930346664653862: minio = 1000 (config file default)
23930346664653862: no_path_retry = -2 (config file default)
pg_timeout = NONE (internal default)
23930346664653862: set ACT_NOTHING (map unchanged)
sdc: ownership set to 23833656434366663
sdc: not found in pathvec
sdc: mask = 0xc
sdc: path checker = tur (config file default)
sdc: checker timeout = 60000 ms (sysfs setting)
sdc: state = 2
sdc: prio = 0
sds: ownership set to 23833656434366663
sds: not found in pathvec
sds: mask = 0xc
sds: path checker = tur (config file default)
sds: checker timeout = 60000 ms (sysfs setting)
sds: state = 2
sds: prio = 0
23833656434366663: pgfailback = -2 (config file default)
23833656434366663: pgpolicy = multibus (config file default)
23833656434366663: selector = round-robin 0 (internal default)
23833656434366663: features = 1 queue_if_no_path (internal default)
23833656434366663: hwhandler = 0 (internal default)
23833656434366663: rr_weight = 2 (config file default)
23833656434366663: minio = 1000 (config file default)
23833656434366663: no_path_retry = -2 (config file default)
pg_timeout = NONE (internal default)
23833656434366663: set ACT_NOTHING (map unchanged)
sdd: ownership set to 26237306165653635
sdd: not found in pathvec
sdd: mask = 0xc
sdd: path checker = tur (config file default)
sdd: checker timeout = 60000 ms (sysfs setting)
sdd: state = 2
sdd: prio = 0
sdt: ownership set to 26237306165653635
sdt: not found in pathvec
sdt: mask = 0xc
sdt: path checker = tur (config file default)
sdt: checker timeout = 60000 ms (sysfs setting)
sdt: state = 2
sdt: prio = 0
26237306165653635: pgfailback = -2 (config file default)
26237306165653635: pgpolicy = multibus (config file default)
26237306165653635: selector = round-robin 0 (internal default)
26237306165653635: features = 1 queue_if_no_path (internal default)
26237306165653635: hwhandler = 0 (internal default)
26237306165653635: rr_weight = 2 (config file default)
26237306165653635: minio = 1000 (config file default)
26237306165653635: no_path_retry = -2 (config file default)
pg_timeout = NONE (internal default)
26237306165653635: set ACT_NOTHING (map unchanged)
sde: ownership set to 26134613837363132
sde: not found in pathvec
sde: mask = 0xc
sde: path checker = tur (config file default)
sde: checker timeout = 60000 ms (sysfs setting)
sde: state = 2
sde: prio = 0
sdu: ownership set to 26134613837363132
sdu: not found in pathvec
sdu: mask = 0xc
sdu: path checker = tur (config file default)
sdu: checker timeout = 60000 ms (sysfs setting)
sdu: state = 2
sdu: prio = 0
26134613837363132: pgfailback = -2 (config file default)
26134613837363132: pgpolicy = multibus (config file default)
26134613837363132: selector = round-robin 0 (internal default)
26134613837363132: features = 1 queue_if_no_path (internal default)
26134613837363132: hwhandler = 0 (internal default)
26134613837363132: rr_weight = 2 (config file default)
26134613837363132: minio = 1000 (config file default)
26134613837363132: no_path_retry = -2 (config file default)
pg_timeout = NONE (internal default)
26134613837363132: set ACT_NOTHING (map unchanged)
sdf: ownership set to 2343537626435392d
sdf: not found in pathvec
sdf: mask = 0xc
sdf: path checker = tur (config file default)
sdf: checker timeout = 60000 ms (sysfs setting)
sdf: state = 2
sdf: prio = 0
sdv: ownership set to 2343537626435392d
sdv: not found in pathvec
sdv: mask = 0xc
sdv: path checker = tur (config file default)
sdv: checker timeout = 60000 ms (sysfs setting)
sdv: state = 2
sdv: prio = 0
2343537626435392d: pgfailback = -2 (config file default)
2343537626435392d: pgpolicy = multibus (config file default)
2343537626435392d: selector = round-robin 0 (internal default)
2343537626435392d: features = 1 queue_if_no_path (internal default)
2343537626435392d: hwhandler = 0 (internal default)
2343537626435392d: rr_weight = 2 (config file default)
2343537626435392d: minio = 1000 (config file default)
2343537626435392d: no_path_retry = -2 (config file default)
pg_timeout = NONE (internal default)
2343537626435392d: set ACT_NOTHING (map unchanged)
sdj: ownership set to 23137663532353265
sdj: not found in pathvec
sdj: mask = 0xc
sdj: path checker = tur (config file default)
sdj: checker timeout = 60000 ms (sysfs setting)
sdj: state = 2
sdj: prio = 0
sdw: ownership set to 23137663532353265
sdw: not found in pathvec
sdw: mask = 0xc
sdw: path checker = tur (config file default)
sdw: checker timeout = 60000 ms (sysfs setting)
sdw: state = 2
sdw: prio = 0
: ownership set to 23137663532353265
: not found in pathvec
: mask = 0xc
: path checker = tur (config file default)
: checker timeout = 300000 ms (internal default)
: state = 1
: checker msg is "tur checker reports path is down"
: ownership set to 23137663532353265
: mask = 0xc
: path checker = tur (config file default)
: checker timeout = 300000 ms (internal default)
: state = 1
: checker msg is "tur checker reports path is down"
23137663532353265: removing path 8:96 with no devname
23137663532353265: pgfailback = -2 (config file default)
23137663532353265: pgpolicy = multibus (config file default)
23137663532353265: selector = round-robin 0 (internal default)
23137663532353265: features = 1 queue_if_no_path (internal default)
23137663532353265: hwhandler = 0 (internal default)
23137663532353265: rr_weight = 2 (config file default)
23137663532353265: minio = 1000 (config file default)
23137663532353265: no_path_retry = -2 (config file default)
pg_timeout = NONE (internal default)
23137663532353265: set ACT_RELOAD (path group topology change)
Comment 3 25641463 2012-07-25 23:18:49 EDT
[root@localhost slaves]# multipath -v2
: checker msg is "tur checker reports path is down"
: checker msg is "tur checker reports path is down"
Comment 4 25641463 2012-07-25 23:27:32 EDT
root@localhost ~]# lsscsi 
[0:0:0:0]    disk    ATA      WDC WD3200AAJS-2 3E01  /dev/sda
[5:0:0:0]    cd/dvd  ATAPI    DVD D  DH16D2P   HP56  /dev/scd0
[9:0:0:1]    disk    _INSPUR_ vg0_lv00          220  /dev/sdr
[9:0:0:2]    disk    _INSPUR_ vg0_lv01          220  /dev/sds
[9:0:0:3]    disk    _INSPUR_ vg0_lv02          220  /dev/sdt
[9:0:0:4]    disk    _INSPUR_ vg0_lv03          220  /dev/sdu
[9:0:0:5]    disk    _INSPUR_ vg19_lv00         220  /dev/sdv
[9:0:0:6]    disk    _INSPUR_ vg19_lv01         220  /dev/sdw
[9:0:0:7]    disk    _INSPUR_ vg19_lv02         220  /dev/sdx
[9:0:0:8]    disk    _INSPUR_ vg19_lv03         220  /dev/sdy
[744:0:0:1]  disk    _INSPUR_ vg0_lv00          220  /dev/sdb
[744:0:0:2]  disk    _INSPUR_ vg0_lv01          220  /dev/sdc
[744:0:0:3]  disk    _INSPUR_ vg0_lv02          220  /dev/sdd
[744:0:0:4]  disk    _INSPUR_ vg0_lv03          220  /dev/sde
[744:0:0:5]  disk    _INSPUR_ vg19_lv00         220  /dev/sdf
[744:0:0:6]  disk    _INSPUR_ vg19_lv01         220  /dev/sdj
[744:0:0:7]  disk    _INSPUR_ vg19_lv02         220  /dev/sdk
[744:0:0:8]  disk    _INSPUR_ vg19_lv03         220  /dev/sdl
Comment 5 25641463 2012-07-25 23:35:17 EDT
[root@localhost ~]# multipath /dev/sdj
: checker msg is "tur checker reports path is down"
: checker msg is "tur checker reports path is down"

[root@localhost ~]# multipath /dev/sdw 
: checker msg is "tur checker reports path is down"
: checker msg is "tur checker reports path is down"
Comment 6 25641463 2012-07-26 02:08:03 EDT
[root@localhost slaves]# dd if=/dev/sdw of=/dev/null count=1 iflag=direct
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.000432 seconds, 1.2 MB/s
[root@localhost slaves]# dd if=/dev/sdj of=/dev/null count=1 iflag=direct
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.000146 seconds, 3.5 MB/s



The problem like this : http://www.spinics.net/lists/dm-devel/msg16857.html
Comment 7 25641463 2012-07-26 02:49:28 EDT
I add the verbosity  3 to multipath.conf :

[root@localhost ~]# dmsetup table
26134613837363132: 0 838860800 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:64 1000 65:64 1000 
23233313238646237: 0 1048576000 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:160 1000 65:112 1000 
2343537626435392d: 0 1048576000 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:80 1000 65:80 1000 
26237306165653635: 0 838860800 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:48 1000 65:48 1000 
VolGroup01-LogVol01: 0 12189696 linear 8:2 612696448
VolGroup01-LogVol00: 0 612696064 linear 8:2 384
23330623031356330: 0 1048576000 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:176 1000 65:128 1000 
23930346664653862: 0 838860800 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:16 1000 65:16 1000 
23833656434366663: 0 838860800 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:32 1000 65:32 1000 
23137663532353265: 0 1048576000 multipath 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:96 1000 8:224 1000 





[root@localhost ~]# multipath -ll
dm-0: blacklisted
dm-1: blacklisted
dm-2: blacklisted
dm-3: blacklisted
dm-4: blacklisted
dm-5: blacklisted
dm-6: blacklisted
dm-7: blacklisted
dm-8: blacklisted
dm-9: blacklisted
md0: blacklisted
ram0: blacklisted
ram10: blacklisted
ram11: blacklisted
ram12: blacklisted
ram13: blacklisted
ram14: blacklisted
ram15: blacklisted
ram1: blacklisted
ram2: blacklisted
ram3: blacklisted
ram4: blacklisted
ram5: blacklisted
ram6: blacklisted
ram7: blacklisted
ram8: blacklisted
ram9: blacklisted
sda: blacklisted
sdb: not found in pathvec
sdb: mask = 0x5
sdb: bus = 1
sdb: dev_t = 8:16
sdb: size = 838860800
sdb: vendor = _INSPUR_
sdb: product = vg0_lv00
sdb: rev =  220
sdb: h:b:t:l = 744:0:0:1
sdb: path checker = tur (config file default)
sdb: checker timeout = 60000 ms (sysfs setting)
sdb: state = 2
sdc: not found in pathvec
sdc: mask = 0x5
sdc: bus = 1
sdc: dev_t = 8:32
sdc: size = 838860800
sdc: vendor = _INSPUR_
sdc: product = vg0_lv01
sdc: rev =  220
sdc: h:b:t:l = 744:0:0:2
sdc: path checker = tur (config file default)
sdc: checker timeout = 60000 ms (sysfs setting)
sdc: state = 2
sdd: not found in pathvec
sdd: mask = 0x5
sdd: bus = 1
sdd: dev_t = 8:48
sdd: size = 838860800
sdd: vendor = _INSPUR_
sdd: product = vg0_lv02
sdd: rev =  220
sdd: h:b:t:l = 744:0:0:3
sdd: path checker = tur (config file default)
sdd: checker timeout = 60000 ms (sysfs setting)
sdd: state = 2
sde: not found in pathvec
sde: mask = 0x5
sde: bus = 1
sde: dev_t = 8:64
sde: size = 838860800
sde: vendor = _INSPUR_
sde: product = vg0_lv03
sde: rev =  220
sde: h:b:t:l = 744:0:0:4
sde: path checker = tur (config file default)
sde: checker timeout = 60000 ms (sysfs setting)
sde: state = 2
sdf: not found in pathvec
sdf: mask = 0x5
sdf: bus = 1
sdf: dev_t = 8:80
sdf: size = 1048576000
sdf: vendor = _INSPUR_
sdf: product = vg19_lv00
sdf: rev =  220
sdf: h:b:t:l = 744:0:0:5
sdf: path checker = tur (config file default)
sdf: checker timeout = 60000 ms (sysfs setting)
sdf: state = 2
sdj: not found in pathvec
sdj: mask = 0x5
sdj: bus = 1
sdj: dev_t = 8:144
sdj: size = 1048576000
sdj: vendor = _INSPUR_
sdj: product = vg19_lv01
sdj: rev =  220
sdj: h:b:t:l = 744:0:0:6
sdj: path checker = tur (config file default)
sdj: checker timeout = 60000 ms (sysfs setting)
sdj: state = 2
sdk: not found in pathvec
sdk: mask = 0x5
sdk: bus = 1
sdk: dev_t = 8:160
sdk: size = 1048576000
sdk: vendor = _INSPUR_
sdk: product = vg19_lv02
sdk: rev =  220
sdk: h:b:t:l = 744:0:0:7
sdk: path checker = tur (config file default)
sdk: checker timeout = 60000 ms (sysfs setting)
sdk: state = 2
sdl: not found in pathvec
sdl: mask = 0x5
sdl: bus = 1
sdl: dev_t = 8:176
sdl: size = 1048576000
sdl: vendor = _INSPUR_
sdl: product = vg19_lv03
sdl: rev =  220
sdl: h:b:t:l = 744:0:0:8
sdl: path checker = tur (config file default)
sdl: checker timeout = 60000 ms (sysfs setting)
sdl: state = 2
sdr: not found in pathvec
sdr: mask = 0x5
sdr: bus = 1
sdr: dev_t = 65:16
sdr: size = 838860800
sdr: vendor = _INSPUR_
sdr: product = vg0_lv00
sdr: rev =  220
sdr: h:b:t:l = 9:0:0:1
sdr: path checker = tur (config file default)
sdr: checker timeout = 60000 ms (sysfs setting)
sdr: state = 2
sds: not found in pathvec
sds: mask = 0x5
sds: bus = 1
sds: dev_t = 65:32
sds: size = 838860800
sds: vendor = _INSPUR_
sds: product = vg0_lv01
sds: rev =  220
sds: h:b:t:l = 9:0:0:2
sds: path checker = tur (config file default)
sds: checker timeout = 60000 ms (sysfs setting)
sds: state = 2
sdt: not found in pathvec
sdt: mask = 0x5
sdt: bus = 1
sdt: dev_t = 65:48
sdt: size = 838860800
sdt: vendor = _INSPUR_
sdt: product = vg0_lv02
sdt: rev =  220
sdt: h:b:t:l = 9:0:0:3
sdt: path checker = tur (config file default)
sdt: checker timeout = 60000 ms (sysfs setting)
sdt: state = 2
sdu: not found in pathvec
sdu: mask = 0x5
sdu: bus = 1
sdu: dev_t = 65:64
sdu: size = 838860800
sdu: vendor = _INSPUR_
sdu: product = vg0_lv03
sdu: rev =  220
sdu: h:b:t:l = 9:0:0:4
sdu: path checker = tur (config file default)
sdu: checker timeout = 60000 ms (sysfs setting)
sdu: state = 2
sdv: not found in pathvec
sdv: mask = 0x5
sdv: bus = 1
sdv: dev_t = 65:80
sdv: size = 1048576000
sdv: vendor = _INSPUR_
sdv: product = vg19_lv00
sdv: rev =  220
sdv: h:b:t:l = 9:0:0:5
sdv: path checker = tur (config file default)
sdv: checker timeout = 60000 ms (sysfs setting)
sdv: state = 2
sdw: not found in pathvec
sdw: mask = 0x5
sdw: bus = 1
sdw: dev_t = 65:96
sdw: size = 1048576000
sdw: vendor = _INSPUR_
sdw: product = vg19_lv01
sdw: rev =  220
sdw: h:b:t:l = 9:0:0:6
sdw: path checker = tur (config file default)
sdw: checker timeout = 60000 ms (sysfs setting)
sdw: state = 2
sdx: not found in pathvec
sdx: mask = 0x5
sdx: bus = 1
sdx: dev_t = 65:112
sdx: size = 1048576000
sdx: vendor = _INSPUR_
sdx: product = vg19_lv02
sdx: rev =  220
sdx: h:b:t:l = 9:0:0:7
sdx: path checker = tur (config file default)
sdx: checker timeout = 60000 ms (sysfs setting)
sdx: state = 2
sdy: not found in pathvec
sdy: mask = 0x5
sdy: bus = 1
sdy: dev_t = 65:128
sdy: size = 1048576000
sdy: vendor = _INSPUR_
sdy: product = vg19_lv03
sdy: rev =  220
sdy: h:b:t:l = 9:0:0:8
sdy: path checker = tur (config file default)
sdy: checker timeout = 60000 ms (sysfs setting)
sdy: state = 2
sr0: blacklisted
===== paths list =====
uuid hcil      dev dev_t  pri dm_st  chk_st  vend/prod/rev     
     744:0:0:1 sdb 8:16   0   [undef][ready] _INSPUR_,vg0_lv00 
     744:0:0:2 sdc 8:32   0   [undef][ready] _INSPUR_,vg0_lv01 
     744:0:0:3 sdd 8:48   0   [undef][ready] _INSPUR_,vg0_lv02 
     744:0:0:4 sde 8:64   0   [undef][ready] _INSPUR_,vg0_lv03 
     744:0:0:5 sdf 8:80   0   [undef][ready] _INSPUR_,vg19_lv00
     744:0:0:6 sdj 8:144  0   [undef][ready] _INSPUR_,vg19_lv01
     744:0:0:7 sdk 8:160  0   [undef][ready] _INSPUR_,vg19_lv02
     744:0:0:8 sdl 8:176  0   [undef][ready] _INSPUR_,vg19_lv03
     9:0:0:1   sdr 65:16  0   [undef][ready] _INSPUR_,vg0_lv00 
     9:0:0:2   sds 65:32  0   [undef][ready] _INSPUR_,vg0_lv01 
     9:0:0:3   sdt 65:48  0   [undef][ready] _INSPUR_,vg0_lv02 
     9:0:0:4   sdu 65:64  0   [undef][ready] _INSPUR_,vg0_lv03 
     9:0:0:5   sdv 65:80  0   [undef][ready] _INSPUR_,vg19_lv00
     9:0:0:6   sdw 65:96  0   [undef][ready] _INSPUR_,vg19_lv01
     9:0:0:7   sdx 65:112 0   [undef][ready] _INSPUR_,vg19_lv02
     9:0:0:8   sdy 65:128 0   [undef][ready] _INSPUR_,vg19_lv03
params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:64 1000 65:64 1000 
status = 2 0 0 0 1 1 E 0 2 0 8:64 A 0 65:64 A 0 
sde: mask = 0x8
sde: getprio = /bin/true (config file default)
sde: prio = 0
sdu: mask = 0x8
sdu: getprio = /bin/true (config file default)
sdu: prio = 0
26134613837363132 dm-6 _INSPUR_,vg0_lv03
[size=400G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][enabled]
 \_ 744:0:0:4 sde 8:64   [active][ready]
 \_ 9:0:0:4   sdu 65:64  [active][ready]
params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:160 1000 65:112 1000 
status = 2 0 0 0 1 1 A 0 2 0 8:160 A 0 65:112 A 0 
sdk: mask = 0x8
sdk: getprio = /bin/true (config file default)
sdk: prio = 0
sdx: mask = 0x8
sdx: getprio = /bin/true (config file default)
sdx: prio = 0
23233313238646237 dm-3 _INSPUR_,vg19_lv02
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 744:0:0:7 sdk 8:160  [active][ready]
 \_ 9:0:0:7   sdx 65:112 [active][ready]
params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:80 1000 65:80 1000 
status = 2 0 0 0 1 1 E 0 2 0 8:80 A 0 65:80 A 0 
sdf: mask = 0x8
sdf: getprio = /bin/true (config file default)
sdf: prio = 0
sdv: mask = 0x8
sdv: getprio = /bin/true (config file default)
sdv: prio = 0
2343537626435392d dm-5 _INSPUR_,vg19_lv00
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][enabled]
 \_ 744:0:0:5 sdf 8:80   [active][ready]
 \_ 9:0:0:5   sdv 65:80  [active][ready]
params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:48 1000 65:48 1000 
status = 2 0 0 0 1 1 E 0 2 0 8:48 A 0 65:48 A 0 
sdd: mask = 0x8
sdd: getprio = /bin/true (config file default)
sdd: prio = 0
sdt: mask = 0x8
sdt: getprio = /bin/true (config file default)
sdt: prio = 0
26237306165653635 dm-4 _INSPUR_,vg0_lv02
[size=400G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][enabled]
 \_ 744:0:0:3 sdd 8:48   [active][ready]
 \_ 9:0:0:3   sdt 65:48  [active][ready]
params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:176 1000 65:128 1000 
status = 2 0 0 0 1 1 A 0 2 0 8:176 A 0 65:128 A 0 
sdl: mask = 0x8
sdl: getprio = /bin/true (config file default)
sdl: prio = 0
sdy: mask = 0x8
sdy: getprio = /bin/true (config file default)
sdy: prio = 0
23330623031356330 dm-7 _INSPUR_,vg19_lv03
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
 \_ 744:0:0:8 sdl 8:176  [active][ready]
 \_ 9:0:0:8   sdy 65:128 [active][ready]
params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:16 1000 65:16 1000 
status = 2 0 0 0 1 1 E 0 2 0 8:16 A 0 65:16 A 0 
sdb: mask = 0x8
sdb: getprio = /bin/true (config file default)
sdb: prio = 0
sdr: mask = 0x8
sdr: getprio = /bin/true (config file default)
sdr: prio = 0
23930346664653862 dm-2 _INSPUR_,vg0_lv00
[size=400G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][enabled]
 \_ 744:0:0:1 sdb 8:16   [active][ready]
 \_ 9:0:0:1   sdr 65:16  [active][ready]
params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:32 1000 65:32 1000 
status = 2 0 0 0 1 1 E 0 2 0 8:32 A 0 65:32 A 0 
sdc: mask = 0x8
sdc: getprio = /bin/true (config file default)
sdc: prio = 0
sds: mask = 0x8
sds: getprio = /bin/true (config file default)
sds: prio = 0
23833656434366663 dm-9 _INSPUR_,vg0_lv01
[size=400G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][enabled]
 \_ 744:0:0:2 sdc 8:32   [active][ready]
 \_ 9:0:0:2   sds 65:32  [active][ready]
params = 1 queue_if_no_path 0 1 1 round-robin 0 2 1 8:96 1000 8:224 1000 
status = 2 18546 0 0 1 1 E 0 2 0 8:96 F 28 8:224 F 28 
8:96: not found in pathvec
8:224: not found in pathvec
23137663532353265 dm-8 ,
[size=500G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][enabled]
 \_ #:#:#:#   -   #:#    [failed][faulty]
 \_ #:#:#:#   -   #:#    [failed][faulty]
Comment 8 25641463 2012-07-26 07:15:25 EDT
I test again and find some problem,the disk which showed faulty was hanging with multipath %M:%m.
[root@localhost slaves]# ps -ef |grep multi
root     10603     1  0 17:07 ?        00:00:08 multipathd -d
root     11301     1  0 17:20 ?        00:00:06 /sbin/multipath 8:96
root     11991     1  0 17:21 ?        00:00:00 /sbin/multipath 65:16
 
Maybe scsi_error_handler hang the cmd ?
[root@localhost slaves]#echo t >/proc/sysrq-trigger


Jul 26 18:57:05 localhost kernel: multipath     D ffff8100cca1c7a0     0 11991      1         12483 11301 (NOTLB)
Jul 26 18:57:05 localhost kernel:  ffff8100a775fbe8 0000000000000086 0000000000000100 ffff8101159cc2c0
Jul 26 18:57:05 localhost kernel:  00000000000000d0 0000000000000002 ffff8100cca1c7a0 ffff81000ab9e820
Jul 26 18:57:05 localhost kernel:  00000418b1ad4052 0000000000106bb4 ffff8100cca1c988 0000000188949183
Jul 26 18:57:05 localhost kernel: Call Trace:
Jul 26 18:57:05 localhost kernel:  [<ffffffff801566c6>] __next_cpu+0x19/0x28
Jul 26 18:57:05 localhost kernel:  [<ffffffff882d86a5>] :dm_mod:dev_suspend+0x0/0x18d
Jul 26 18:57:05 localhost kernel:  [<ffffffff80064a1b>] __down+0xc3/0xd8
Jul 26 18:57:05 localhost kernel:  [<ffffffff8008ee79>] default_wake_function+0x0/0xe
Jul 26 18:57:05 localhost kernel:  [<ffffffff800646d9>] __down_failed+0x35/0x3a
Jul 26 18:57:05 localhost kernel:  [<ffffffff882d5afd>] :dm_mod:.text.lock.dm+0x2d/0x3c
Jul 26 18:57:05 localhost kernel:  [<ffffffff8008ee79>] default_wake_function+0x0/0xe
Jul 26 18:57:05 localhost kernel:  [<ffffffff882d879d>] :dm_mod:dev_suspend+0xf8/0x18d
Jul 26 18:57:05 localhost kernel:  [<ffffffff882d9064>] :dm_mod:ctl_ioctl+0x210/0x25b
Jul 26 18:57:05 localhost kernel:  [<ffffffff80041eba>] do_ioctl+0x55/0x6b
Jul 26 18:57:05 localhost kernel:  [<ffffffff8002ff14>] vfs_ioctl+0x457/0x4b9
Jul 26 18:57:05 localhost kernel:  [<ffffffff800ba76c>] audit_syscall_entry+0x1a8/0x1d3
Jul 26 18:57:05 localhost kernel:  [<ffffffff8004c288>] sys_ioctl+0x59/0x78
Jul 26 18:57:05 localhost kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0


Jul 26 18:57:05 localhost kernel: scsi_eh_26    S 0000000000000282     0 11783     55         12699 10881 (L-TLB)
Jul 26 18:57:05 localhost kernel:  ffff81004711be60 0000000000000046 ffff81004711bdd0 ffffffff8008e88f
Jul 26 18:57:05 localhost kernel:  0000041571d83579 0000000000000009 ffff8100a4f340c0 ffff81011e58c080
Jul 26 18:57:05 localhost kernel:  0000041571dd044f 00000000000007a1 ffff8100a4f342a8 00000002a4f340f8
Jul 26 18:57:05 localhost kernel: Call Trace:
Jul 26 18:57:05 localhost kernel:  [<ffffffff8008e88f>] __activate_task+0x56/0x6d
Jul 26 18:57:05 localhost kernel:  [<ffffffff80063002>] thread_return+0x62/0xfe
Jul 26 18:57:05 localhost kernel:  [<ffffffff800a3295>] keventd_create_kthread+0x0/0xc4
Jul 26 18:57:05 localhost kernel:  [<ffffffff880792d6>] :scsi_mod:scsi_error_handler+0x6a/0x4ce
Jul 26 18:57:05 localhost kernel:  [<ffffffff800a3295>] keventd_create_kthread+0x0/0xc4
Jul 26 18:57:05 localhost kernel:  [<ffffffff8807926c>] :scsi_mod:scsi_error_handler+0x0/0x4ce
Jul 26 18:57:05 localhost kernel:  [<ffffffff800a3295>] keventd_create_kthread+0x0/0xc4
Jul 26 18:57:05 localhost kernel:  [<ffffffff80032660>] kthread+0xfe/0x132
Jul 26 18:57:05 localhost kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jul 26 18:57:05 localhost kernel:  [<ffffffff800a3295>] keventd_create_kthread+0x0/0xc4
Jul 26 18:57:05 localhost kernel:  [<ffffffff80152b3e>] cfq_latter_request+0x0/0x1e
Jul 26 18:57:05 localhost kernel:  [<ffffffff8804d50d>] :ext3:bput_one+0x0/0x7
Jul 26 18:57:05 localhost kernel:  [<ffffffff80032562>] kthread+0x0/0x132
Jul 26 18:57:05 localhost kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Comment 9 25641463 2012-07-30 02:17:05 EDT
I put the command:

[root@localhost ~]# dmsetup message /dev/dm-8 1 fail_if_no_path

And all requeue io in dm-8 returned with -EIO(function do_end_io err=-5 m->queue_if_no_path 0)

The command  /sbin/multipath 65:16 would not hang and successful.
The path changed to active.

I didn't know why add disk to path was not successful when there were some requeue io? Was it a bug?
Comment 10 25641463 2012-08-02 02:35:26 EDT
When occur this bug,I used gdb -p pid or strace -p pid to dbg the hanging process of /sbin/multipath %M:%m.The process will not hang and return.The path will become active.


gdb) where
#0  0x00000032612cc557 in ioctl () from /lib64/libc.so.6
#1  0x0000003f6f61c256 in ?? () from /lib64/libdevmapper.so.1.02
#2  0x0000003f6f619d0b in dm_task_run () from /lib64/libdevmapper.so.1.02
#3  0x00000000004074c4 in dm_simplecmd (task=5, name=0x1e59ce30 "23930346664653862", no_flush=1) at devmapper.c:179
#4  0x00000000004061f2 in domap (mpp=0x1e597250) at configure.c:385
#5  0x0000000000406853 in coalesce_paths (vecs=0x7fff54b2e490, newmp=0x0, refwwid=0x0) at configure.c:560
#6  0x0000000000403419 in configure () at main.c:295
#7  0x0000000000403912 in main (argc=2, argv=0x7fff54b2e5f8) at main.c:
Comment 11 Ben Marzinski 2012-08-08 18:21:23 EDT
Are you really running this on RHEL 5.4? There have been a large number of kernel changes that may have fixed this.  Would it be possible to try and reproduce this on RHEL 5.8?
Comment 12 25641463 2012-08-21 04:03:05 EDT
I test the RHEL 6.2.Everything is well.I don't know whether 5.8 is well.
Comment 13 RHEL Product and Program Management 2014-01-29 05:36:21 EST
This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux release.  Product Management has
requested further review of this request by Red Hat Engineering, for
potential inclusion in a Red Hat Enterprise Linux release for currently
deployed products.  This request is not yet committed for inclusion in
a release.
Comment 14 Ben Marzinski 2014-02-06 14:22:28 EST
I can't hit this with a recent kernel.  If you can please let me know.
Comment 15 Ben Marzinski 2014-03-24 19:34:53 EDT
This is the last RHEL5 release, and I can't fix this without being able to reproduce it.

Note You need to log in before you can comment on or make changes to this bug.