Created attachment 450813 [details]
OSLog file

Created attachment 450814 [details]
Log file (problem case)

Created attachment 450815 [details]
Log file (no problem case)

RSD did not test RHLinux with R600, DF800 or other storage yet.
RSD will test after RHLinux fix.

On the other hand,
RSD tested RHLinux 5.5 with R700, and they confirmed no problem.
Now,
RSD tested RHLinux 6.0 with R700, and they confirmed this issue.

So, RSD thinks that RHLinux 6.0 behavior is strange.

What command did you run?

# multipath -ll

Once we determine the problem and have a patch, it would be a good idea to provide HDS with a test kernel so they can validate the fix.

The strange thing about this is that multipathd doesn't appear to be having any issues.  It is successfully checking the paths at regular intervals, and it is responding to uevents, even it your failed case. Usually "multipath -ll" gets stalled checking the paths, but multipathd doesn't appear to have any problems at all.  A good first step would be to run

# multipath -ll -v3

That should give me a good idea where it is stalling.  This is of-course assuming that the stuck command in multipath -ll, otherwise, just add the "-v3" to whatever multipath command you are running and attach the output to the bugzilla.

This same issue has been seen with our test lab in India as well. More information to come shortly.

Running the #multipath -ll -v3 has not been able to run and hangs.

Created attachment 453575 [details]
#multipath -ll -v3 abort log

Created attachment 453576 [details]
#multipath -ll -v3 SOSReport

Created attachment 453578 [details]
#multipath -ll -v3 abort screenshot

What happens when you run

# multipath -ll -v3

Does anything get printed out at all? If so, please run

# multipath  -ll -v3 > 639037_multipath_ll 2>@1

That should create a file called 639037_multipath_ll

please attach that file to the bugzilla.

Created attachment 453801 [details]
multipath command log

looking at your output, it doesn't look like any of your scsi devices are giving you problems.  It looks like it's the cciss device that's hanging your output.

could you try adding

blacklist {
    device {
        vendor "HP"
        product "LOGICAL VOLUME"
    }
}

To /etc/multipath.conf. Or if you already have a blacklist section, just add the device subsection.  This will blacklist the cciss device.  Then, run:

# service multipathd reload
# multipath -ll -v3

hopefully this time the commands shouldn't get stuck should work. Whatever the outcome, please let me know.

From our other team experienceing similar issue:

Activity : Multipath Daemon is stopped and then Cable unplug/plug being done to see the udev events .
Observation :  #udevadm monitor –kernel

1.	  When Target side cable is unplugged and then plugged we did not notice/see any udev event on the above command .
2.	  When Initiator side cable is unplugged we observed udev events. Also, when the cable is plugged back in we observe udev events.

From our aforementioned India Team. SO this is a bug found in TWO different labs.

1.	Multipath –l is in hang state when we do cable pull on target side for the following configuration . (refer  TOTS #0003515)
Configuration 
------------- 
HP DL380 G5<->Linux AS 6.0 (IA32)(2.6.32-71.el6.i686)<->Emulex LPe12002(Bundled, ver-8.3.5.17)<->Multipath-Bundled<->Brocade 5000(F/W: v6.4.0b)<->R700(M/C: 70-00-50-00/00). 
Alternate Path: 
HP DL380 G5<->Linux AS 6.0 (IA32)(2.6.32-71.el6.i686)<->Emulex LPe12002(Bundled, ver-8.3.5.17)<->Multipath-Bundled<->Brocade 5100(F/W: v6.4.0b)<->R700(M/C: 70-00-50-00/00).

2.	While doing DR on R600  on the following configuration we see that the faulty path comes online after 30 mins + after fault is cleared on the faulty path.
(In the process of Creating TOTS for this)
Configuration 
------------- 
HP DL580 G3<->Linux AS 6.0 (IA32)(2.6.32-71.el6.i686)<->QLogic QLA2462(Bundled, ver- 8.03.01.05.06.0-k8)<->Multipath-Bundled<->Cisco 9124(F/W: 4.2(3))<->R600 .
Alternate Path 
HP DL580 G3<->Linux AS 6.0 (IA32)(2.6.32-71.el6.i686)<->QLogic QLA2462(Bundled, ver- 8.03.01.05.06.0-k8)<->Multipath-Bundled<->Brocade 200E(F/W:   v6.2.1b)<->R600.

We are investigating on the following issues also let us know if we need to provide any further details on this.

Can you please confirm from if this is expected udev behaviour?

Please note the following sequence of events we have performed with respect to Emulex Lpe12k HBA 

1.	Stopped multipath daemon . ( #service multipathd stop)
2.	Flushed multipath devices   .  ( # multipath –F)
3.	Then started monitoring  udev events  ( #udevadm monitor --kernel)

Activity :  
1. Initiator cable unplugged  
                   Observation :   
a)  udev events are noticed  where we see the devices getting removed.
b)  activity are reported on /var/log/messages
                   
2.  Initiator cable plugged 
                   Observation      
a) udev events are noticed where we see the devices getting added.
b)  activity are reported on /var/log/messages  
                  
3. Target cable unplugged
                   Observation      
a) no udev events noticed .
b) no activity reported on /var/log/messages.
                  
4.  Target cable plugged 
                   Observation         
a) no udev events noticed .
b) no activity reported on /var/log/messages.

Can you please confirm from if this is expected udev behaviour?

Please note the following sequence of events we have performed with respect to Emulex Lpe12k HBA 

1.	Stopped multipath daemon . ( #service multipathd stop)
2.	Flushed multipath devices   .  ( # multipath –F)
3.	Then started monitoring  udev events  ( #udevadm monitor --kernel)

Activity :  
1. Initiator cable unplugged  
                   Observation :   
a)  udev events are noticed  where we see the devices getting removed.
b)  activity are reported on /var/log/messages
                   
2.  Initiator cable plugged 
                   Observation      
a) udev events are noticed where we see the devices getting added.
b)  activity are reported on /var/log/messages  
                  
3. Target cable unplugged
                   Observation      
a) no udev events noticed .
b) no activity reported on /var/log/messages.
                  
4.  Target cable plugged 
                   Observation         
a) no udev events noticed .
b) no activity reported on /var/log/messages.

(In reply to comment #20)
> Can you please confirm from if this is expected udev behaviour?

In activity 3 and 4, are the devices actually getting removed and added?  Assuming that they are not, then there shouldn't be any udev events generated, so udev at least is working fine.  I'm not sure why the kernel is removing the devices in the first case but not the second.  How long are you waiting?

Do you see the devices get removed if you are doing IO to them when you pull the target cable?

What values are you using for dev_loss_tmo?

you can look this up at

/sys/class/fc_remote_ports/rport-<H>:<B>-<R>/dev_loss_tmo

Where <H> and <B> are the host and bus from <H>:<B>:<T>:<L>

and <R> is the rport_id, which you can get by looking at

/sys/class/fc_transport/target<H>:<B>:<T>

This is a symlink, and one of the directories in the pathname that it is
pointing to should be  rport-<H>-<B>-<R>, this is where you need to check for
the dev_loss_tmo value.  This value tells how log the system should wait before removing the device from the system.

Thanks Ben. We are waiting for either the Japanese or India team to respond. May not be until tomorrow. I will add it as a comment as usual. Do you think we should set up a concall for speed and ease of use? Or if this is the most comfortable way to troubleshoot then I am fine with that too. Let me know what you would like to do and I will make it happen.

~Paul
408-970-7180 desk
408-250-1469 cell

(In reply to comment #19)
> From our aforementioned India Team. SO this is a bug found in TWO different
> labs.
> 
> 1. Multipath –l is in hang state when we do cable pull on target side for the
> following configuration . (refer  TOTS #0003515)
> Configuration 
> ------------- 
> HP DL380 G5<->Linux AS 6.0 (IA32)(2.6.32-71.el6.i686)<->Emulex
> LPe12002(Bundled, ver-8.3.5.17)<->Multipath-Bundled<->Brocade 5000(F/W:
> v6.4.0b)<->R700(M/C: 70-00-50-00/00). 
> Alternate Path: 
> HP DL380 G5<->Linux AS 6.0 (IA32)(2.6.32-71.el6.i686)<->Emulex
> LPe12002(Bundled, ver-8.3.5.17)<->Multipath-Bundled<->Brocade 5100(F/W:
> v6.4.0b)<->R700(M/C: 70-00-50-00/00).

Is this when I got the multipath -ll output from? If so, can you please try the steps I mentioned in comment 17. If not, can you please please send me the multipath -ll output from this hang, as I mentioned in comment 15.  In addition, if you could attach gdb to the hung multipathd process and get a backtrace, that would be very helpful. For this you need the device-mapper-multipath-debuginfo rpms installed.  Then run

# gdb /sbin/multipath <pid_of_hung_process>

From the gdb prompt run

bt

That will give me a backtrace of the stuck process.
 
> 2. While doing DR on R600  on the following configuration we see that the
> faulty path comes online after 30 mins + after fault is cleared on the faulty
> path.
> (In the process of Creating TOTS for this)
> Configuration 
> ------------- 
> HP DL580 G3<->Linux AS 6.0 (IA32)(2.6.32-71.el6.i686)<->QLogic QLA2462(Bundled,
> ver- 8.03.01.05.06.0-k8)<->Multipath-Bundled<->Cisco 9124(F/W: 4.2(3))<->R600 .
> Alternate Path 
> HP DL580 G3<->Linux AS 6.0 (IA32)(2.6.32-71.el6.i686)<->QLogic QLA2462(Bundled,
> ver- 8.03.01.05.06.0-k8)<->Multipath-Bundled<->Brocade 200E(F/W:  
> v6.2.1b)<->R600.

In this case, I assume that multipathd is running.  multipathd should be printing out warning messages about your failed path every 5 seconds. Do you see these?  Do they continue during the 30+ that it takes for your path to get restored?

> We are investigating on the following issues also let us know if we need to
> provide any further details on this.

(In reply to comment #23)
> Thanks Ben. We are waiting for either the Japanese or India team to respond.
> May not be until tomorrow. I will add it as a comment as usual. Do you think we
> should set up a concall for speed and ease of use? Or if this is the most
> comfortable way to troubleshoot then I am fine with that too. Let me know what
> you would like to do and I will make it happen.
> 
> ~Paul
> 408-970-7180 desk
> 408-250-1469 cell

I think a concall to make sure we're all on the same page would be a good idea. I have a meeting tomorrow morning and I'm on PTO this friday, but tomorrow afternoon or Thursday work fine.  I work in Minneapolis, so I'm in CDT (UTC - 0500).

In answer to:

3. When 1 path cable was pulled, pulled path information was not displayed.
    And only Active path was displayed. RSD is asking if it is RHEL6 spec.

This is correct behaviour on RHEL6.  Previous to RHEL6, failed scsi devices were not automatically deleted from the system.  In RHEL6, if a scsi device has been failed for dev_loss_tmo seconds (See Comment 22 for how to check dev_loss_tmo), the kernel will remove that deviec from the system.  When this happens, that path is removed from from the multiapath device.

Looking at the multipath -v3 -ll output from Comment 16, I see

sda: path checker = readsector0 (controller setting)

I verified with Biswadip that when /etc/multipath.conf does not have a configuration for your device in the devices section, it autoconfigures these devices using the tur path checker.  The readsector0 path checker should not be used.  It will send IO to the device and then wait for a response or a failure.  While it does this, multipathd will be hung.  When you are testing multipath with your devices, please do not use the devices section in your multipath.conf file, unless you are trying test an alterantive configuration.

From the concall this morning, I think it would be useful to try configuring your devices to use the "directio" path checker instead, to see if it will
respond quicker to device failures.

To do this, you can add the following devices section to your /etc/multipath.conf


devices {
        device {
                vendor "(HITACHI|HP)"
                product "OPEN-.*"
                path_checker directio
        }
}

This checker sends asynchronous IO to the scsi devices, and checks whether the IO succeeds or fails.  In order to make this check more responsive, You should
also set fast_io_fail_tmo in your defaults section to a small number of seconds like so

defaults {
        ...
        fast_io_fail_tmo 5
}

This will set the fast_io_fail_tmo sysfs attribute. This attribute can be viewed at

/sys/class/fc_remote_ports/rport-<H>:<B>-<R>/fast_io_fail_tmo

just like the dev_loss_tmo attribute.  Setting this to 5 means that 5 seconds after the scsi layer notices a problem with a device, it will fail all IO to that path.  This will cause the directio checker to report failure after 5 seconds, instead of when the device is removed from the system or set offline.

To use gdb to see why multipathd is hanging, you need to install the device-mapper-multipath-debuginfo rpm.  You can download it at

http://people.redhat.com/~bmarzins/device-mapper-multipath/rpms/RHEL6/x86_64/device-mapper-multipath-debuginfo-0.4.9-31.el6.x86_64.rpm

or

http://people.redhat.com/~bmarzins/device-mapper-multipath/rpms/RHEL6/i686/device-mapper-multipath-debuginfo-0.4.9-31.el6.i686.rpm

depending on your system architecture. Once you have the debuginfo rpm isntalled, run

# multipath -ll

When this hangs, from another terminal, run

# gdb /sbin/multipath <pid_of_hung_process>

From the gdb prompt run

bt

That will give me a backtrace of the stuck process.

There are three ways you can try to blacklist your cciss devices.  It would be helpful for me if you tried all three and reported any that didn't work.  You should note, blacklisting a device won't automatically cause multipath to remove it.  Once you edit your config file, you need to run

# service multipathd reload

To have multipathd reconfigure itself.  This is true for ANY config file change. Not just blacklisting a device.  multipathd must be told to reread the config file and reconfigure itself.

The first way would be to add

blacklist {
    wwid "<wwid_of_the device>"
}

You should be aware that a device with a vendor of "HP" and product of "LOGICAL VOLUME" will use

# /lib/udev/scsi_id --whitelisted --device=/dev/%n

as it's getuid_callout.  So this is the program you should run to get the WWID.  If you blacklist by wwid you will still see the the device when you do

# multipath -ll -v3

in the paths list. This is because the first blacklist messages are only for the devnode blacklists, which keep a device from even being checked at all.

# multipath -ll

should not show any device with the cciss device as a path, and

# multipath -v3

should show a line like

cciss!c0d0: (HP:LOGICAL VOLUME) wwid blacklisted

You could also blacklist the device by adding

blacklist {
    device {
        vendor "HP"
        product "LOGICAL VOLUME"
    }
}

If you use this blacklist (and run "service multipathd reload"), you will again still see the device in the paths list of

# multipath -v3 -ll

But you shouldn't see any mulitpath devices that use it as a path.

and when you run

# mutipath -v3

You should see a like like

cciss!c0d0: (HP:LOGICAL VOLUME) vendor/product blacklisted

The final way you can blacklist it is to add

blacklist {
    devnode "cciss.*"
}

This will blacklist add devices whose name starts with cciss.  When you do this, you should see it listed in the initial blacklists, and you should not see the device in the paths list.

All three of these ways should keep you from having a multipath device with that device as a path.

Created attachment 454901 [details]
mpathconf program that prints debugging information

Biswadip, you mentioned that on one of your machines, running

# mpathconf --enable

did not create /etc/multipath.conf

could you please try running this version of mpathconf instead.  It should be the same as your current  version, except that it will print out everything the script does.

run

# mpathconf --enable 2> mpathconf_output

and attach the mpathconf_output file to this bugzilla.  It will let me see why it is not creating the configuration file.

Multipathing 
-----------------
Devices are mounted as below
[root@DL380-4 ~]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
…. …. …..
/dev/mapper/360060e8006cf18000000cf180000025ap1       2062512   1059844    897896  55% /fs0
/dev/mapper/360060e8006cf18000000cf1800000260p1       2062512   1059844    897896  55% /fs1
[root@DL380-4 ~]#
Multipath configuration file used for this setup is attached here.  Modifications are as below.

Section : defaults
 
  We did not change any value here. We have used the‘user_friendly_names no’ as in defaults section..


         Section : blacklist
              
                We have added an entry  to blacklist HP local disk from multipathing.

blacklist {
       devnode "cciss.*"                       
                                }
        
          Section :  devices

We have added following entries in devices section for RAID storages:



no_path_retry 6 # This parameter has been added

NB : Above entry specify that after 6 number of retries it stops queuing.



failback immediate # This parameter has been added

NB : Above entry  means immediate failback of path group.

Please let me know if more information is required.

Created attachment 457882 [details]
Output of latest testing as per Ben

This is the Multipath conf that is for the Comment added 11/4/10

I don't see anything wrong with those values. Are you asking to have the default
configuration for this device changed to include

no_path_retry 6
failback immediate


Just as a note, as long as you match the vendor and product regular expression exactly, you don't need to write out the entire device config, just what you want changed. Also, you don't need to include the any of defaults values, since you are just using the compiled in defaults. A much smaller, but equivalent multipath.conf file would be

####

blacklist {
        devnode "cciss.*"
}

devices {
        device {
                vendor "(HITACHI|HP)"
                product "OPEN-.*"
                no_path_retry 6
                failback immediate
        }
}

####

However, could you please try to use the following config file instead.

####

defaults {
        fast_io_fail_tmo 5
}

blacklist {
        devnode "cciss.*"
}

devices {
        device {
                vendor "(HITACHI|HP)"
                product "OPEN-.*"
                path_checker directio
                no_path_retry 6
                failback immediate
        }
}


####

This will allow us to see if the directio path checker works around the hang problem.  I assume that the hang problem still exists with your config. Is that correct?  If you cannot see the hang with your config file, then the only issue left in this bugzilla is to add the "no_path_retry" and "failback" settings to the device's default config, if that's what you want. right?

Oh, just to make a record of this.  Biswadip sent me an email stating that he was able to blacklist the cciss device, and that he was no longer able to reproduce the problem where mpathconf was not creating the config file, which just leaves the original issue, which is the hang.

I will add the following changes to your default configuration

no_path_retry 6
failback immediate

Our Japan Engineers are finding the same hang issue with Device Mapper when a cable is pulled from a dual cable failover configuration. The hang sometimes manifests itself as a reboot or total OS hang as well as the original Device Mapper hang. 

When HDS uses our HDS HDLM failover software no issue is seen at all. 

The issues occur with Red Hat 6.0 IA32 or x64 with R700. Both Emulex and QLogic 8Gb HBAs (will verify if 4Gbs HBAs)

We have added both "no_path_retry 6" and "failback immediate" to the Multipath.conf.

More investigation is still forthcoming. Anyone within the distribution please feel free to add more information in case I have missed anything.

【Phenomena】
> −LPe12002−
> ?!Host-FC Switch cable pull:OS Kernel was crushed and Host was rebooted.
> ?"RAID700 CL-1 side CHA (pull out of the board on the storage) removed:Host was rebooted.
> ?#RAID700 CL-2 side CHA (pull out of the board on the storage) removed:Host was hung up.
> ?$Host-RAID700 cable pull: after pulling, multipath -ll command does not respond.
> 　　But, I/O does not stop.
> 
> −QLe2562−
> ?%RAID700 CL-1 side CHA (pull out of the board on the storage) removed:I/O stopped for 20 minutes and started working.
> ?&Host-RAID間 cable pull: after pulling, multipath -ll command does not respond.
> 　　But, I/O does not stop.

> 　→after that, we connected the cable, and reconfigured multipath, then command started to respond.
> ?'Host-RAID cable pull: after pulling, multipath -ll command does not respond.
>     waited 15 hours, but got no response.
> 
> 【Configuration】
> −LPe12002−
> OS：RHEL6.0(x64) GA(kernel--2.6.32.71) ※RUN Level：5
> HBA：LPe12002
> HBA Driver：OS Bundled(8.3.5.17)
> Connection：1path--Direct　1path--FC Switch
> 　1path：Direct   −RAID700 CL-1
> 　1path：FC Switch−RAID700 CL-2
> FC Switch：Brocade5300
> FC Switch F/W：6.2.2d
> Multipath：Device mapper multipath(OS Bundled)
> Storage：RAID700
> LU number：16
------This our I/O generator-------------
> TP name：SeqTP
> TP processes：32/Storage(=2 processes/LUX16LU)
> 
> 
> −QLe2562−
> OS：RHEL6.0(x64) GA(kernel--2.6.32.71) ※RUN Level：5
> HBA：QLe2562
> HBA Driver：OS Bundled(8.03.01.05.06.0-k8)
> Connection：1path--Direct　1path--FC Switch
> 　1path：Direct   −RAID700 CL-1
> 　1path：FC Switch−RAID700 CL-2
> FC Switch：MDS9222i(8Gb)
> FC Switch F/W：4.2(7a)
> Multipath：Device mapper multipath(OS Bundled)
> Storage：RAID700
> LU number：16
------This our I/O generator-------------
> TP name：SeqTP
> TP processes：32/Storage(=2 processes/LUX16LU)
> 
> 
>

Please find the details of our test bed for reproducing the problem. We were able to generate multipath -ll to hang state .
 
HDS-India 
Configuration :
  －QLe2562－
    OS：RHEL6.0(x64) GA(kernel-2.6.32-71.el6.x86_64) ※RUN Level：5
    HBA：QLe2562
    HBA Driver：OS Bundled(8.03.01.05.06.0-k8)
    Connection：1path--FC Switch　,1path--Direct
  　1path：FC Switch    －RAID700 Port 1D
  　1path：Direct       －RAID700 Port 2D
    FC Switch：Cisco MDS 9513(8Gb)
    FC Switch F/W：4.2(7a)
    Multipath：Device mapper multipath(OS Bundled)
    Storage：RAID700
    LU number：16
    TP name：SeqTP
    TP processes：32/Storage(=2 processes/LUX16LU)
 
Attached scr for running the 32 TPs on 16 LUN.
 
Activites performed   
 
1. Sequence done 14 times . 
 
a) Cable is unplugged between Host and FCSW . 
   -------- wait for 4 minutes ----------
b) Cable is plugged back between Host and FCSW.
   ---------wait for 5 minutes ----------
c) Cable is unplugged between Host and R700 (Direct connection)
   -------- wait for 4 minutes ---------- 
d) Cable is plugged back  between Host and R700 (Direct connection)
   ---------wait for 5 minutes ----------
e) Repeat steps a) to d) 
 
Result : multipath -ll worked normally.
 
2. Sequence done 8 times 
 
a) Cable is unplugged between FCSW and Target . 
   -------- wait for 4 minutes ----------
b) Cable plugged back between FCSW and Target.
   -------- wait for 5 minutes -----------
c) Cable is unplugged between Host and R700 (Direct connection)
   -------- wait for 4 minutes ---------- 
d) Cable is plugged back between Host and R700 
   -------- wait for 5 minutes --------------
e) Repeat step a) to d)
 
Result : On 8th time for c)  multipath -ll is in hang state.
We are waiting to see the command response on the server.

To track down what is going on here, a couple of pieces of information would be
really helpful.

Could you please attach the following information gathered from the machine with the hang

1. The output of
# multipath -v3 -ll

from before the hang

2. /etc/multipath.conf


3. /var/log/messages

from after the hang has occurred.  If you could give me a rough time when the hang started, that would help even more.

4. the output of
# multipath -v3 -ll from when the hang is occuring.

Thanks.

The multipath.conf is exactly the same as we recieved for test.
The test team said when the hang occured, OS didn't work any more (did not recover) and they could not get any log messages.

Please find the outputs from the following configuration.

HDS-India
Configuration :
  －QLe2562－
    OS：RHEL6.0(x64) GA(kernel-2.6.32-71.el6.x86_64) ※RUN Level：5
    HBA：QLe2562
    HBA Driver：OS Bundled(8.03.01.05.06.0-k8)
    Connection：1path--FC Switch　,1path--Direct
  　1path：FC Switch    －RAID700 Port 1D
  　1path：Direct       －RAID700 Port 2D
    FC Switch：Cisco MDS 9513(8Gb)
    FC Switch F/W：4.2(7a)
    Multipath：Device mapper multipath(OS Bundled)
    Storage：RAID700
    LU number：16
    TP name：SeqTP
    TP processes：32/Storage(=2 processes/LUX16LU)


Cable Pull started  at Jan  9 00:17:07.

Sequence of Operation .
   a) Cable is unplugged between FCSW and Target .
   b) Cable plugged back between FCSW and Target.
   c) Cable is unplugged between Host and R700 (Direct connection)
   d) Cable is plugged back between Host and R700
   
Output attached 

1.	multipath –v3 –ll taken before cable pull start.( multipath-v3-ll-start.txt).
2.    multipath -v3 -ll taken during cable pull till multipath -ll in hang state. (mpath.bat.log1)  
3.	Message file excerpt when multipath -ll in hang state. (log-messages-excerpt.txt)

Created attachment 472658 [details]
Added 01-10-11

Created attachment 472660 [details]
Added 01-10-11

Created attachment 472662 [details]
Multipath.conf

Created attachment 472663 [details]
multipath-v3-ll-start.txt

Created attachment 472664 [details]
Multipath.bat.log

Please find the output from the following configuration.

－LPe12002－
 OS：RHEL6.0(x64) GA(kernel--2.6.32.71) ※RUN Level：5
 HBA：LPe12002
 HBA Driver：OS Bundled(8.3.5.17)
 Connection：1path--Direct　1path--FC Switch
 1path：FC Switch   －RAID700 Port 5A
 1path：Direct      －RAID700 Port 6A
 FC Switch：Brocade300
 FC Switch F/W：6.2.2d
 Multipath：Device mapper multipath(OS Bundled)
 Storage：RAID700
 LU number：16
 TP name：SeqTP
 TP processes：32/Storage(=2 processes/LUX16LU)

1.	Host Cable Pull started at - Jan 10 11:23:28 .
2.	System crashed at - Jan 10 11:51, when cable between HBA and Storage Pulled (Direct connect), 2nd iteration
3.	Logs taken after rebooting the system .

NB: Logs are taken when system powered up after crash. The zip file contains (/var/log/messages, /etc/multipath.conf, sosreport, multipath –v3 –ll output)

Created attachment 472859 [details]
Output of latest testing January 11th

(In reply to comment #47)

> 2. System crashed at - Jan 10 11:51, when cable between HBA and Storage Pulled
> (Direct connect), 2nd iteration
> 3. Logs taken after rebooting the system .

In order to debug a system crash, we need the stack trace that is output to the console when the ceash occurs, at the very least. Even better would be to set up kdump and get a crash dump. 

Tom

Hi Tom,

The KDUMP is a very large file. A 150MB file to be exact. Not sure how to get it to you. Do you have a file server that my team can place the KDUMP file?

Paul, Is there any place you could put the kdump file where I could download in from?

There's no useful information I could gather from the latest attachment. This isn't surprising since when the machine crashes, it isn't able to write to /var/log/messages.  However, if you are also seeing hangs that aren't crashing the machine, then it is very likely that useful information will be written to /var/log/messages, and if you could upload a copy of /var/log/messages from one of those hangs, it would be very useful to look at. I'm assuming that the machine isn't always crashing during these hangs. If it always is, that would be useful to know as well.

Paul, I sent you instructions on how to upload the kdump file by email. Please let me know once you've uploaded it, and I'll take a look at it.

Created attachment 474666 [details]
messages from recent testing

Created attachment 474673 [details]
Messages for comment 41

Looking at this /var/log/messages output, it looks like this could be a scsi issue.  There are a couple of scsi issues that caused some multipath setups to lockup in a similar way (although not exactly the same, so I'm can't say for sure that this is what we're seeing).  The one that springs to mind is Bug 634589.

It may be possible to work around this issue in the same way that they did.  The kernel fix for that issue is in 2.6.32-92.el6, I believe. It working around this fails. I'd like to try sending you guys a recent kernel to try out, just so that we can see if we're clear for 6.1 at least.

Could you please add the following two lines to the defaults section of your multipath.conf

    fast_io_fail_tmo  5
    dev_loss_tmo      30

fast_io_fail_tmo tells the scsi layer to fail back IO after the scsi device has been down for 5 seconds.  dev_loss_tmo tells the scsi layer to remove the scsi device after it has been down for 30 seconds.  The kernel problem that I'm thinking of happens when fast_io_fail_tmo isn't set, which it isn't by default. In this case, the IO is failed back when the scsi device is deleted.  However in the kernel you are running there are problems in this code path, and it looks like you may be running into them.

Let me know how this works out.  If this doesn't solve you problem, and running a more recent kernel with the scsi fixes doesn't either, the next step would be to grab a sysrq process dump to see exactly where multipathd is stuck.

We have added the parameters as recommended by Ben and attached the multipath.conf and scr file used on both configurations.

Sequence Followed for Target side Cable pull.

a) Cable is unplugged between FCSW and Target .
   -------- wait for 4 minutes ----------
b) Cable plugged back between FCSW and Target.
   -------- wait for 5 minutes -----------
c) Cable is unplugged between Host and R700 (Direct connection)
   -------- wait for 4 minutes ----------
d) Cable is plugged back between Host and R700 (Direct Connection)
   -------- wait for 5 minutes --------------

Sequence Followed for Host side Cable pull.

a) Cable is unplugged between Host and FCSW .
   -------- wait for 4 minutes ----------
b) Cable plugged back between Host and FCSW.
   -------- wait for 5 minutes -----------
c) Cable is unplugged between Host and R700 (Direct connection)
   -------- wait for 4 minutes ----------
d) Cable is plugged back between Host and R700 (Direct connection)
   -------- wait for 5 minutes --------------

Observations :

Configuration : (1)

a) Target side cable Pull ran for 30 iterations, problem was not observed. 
b) Host side cable Pull ran for 20 iterations, problem was not observed.   

Configuration : (2)

a) Target side cable Pull ran for 30 iterations, problem was not observed.
b) Host side cable Pull ran for 20 iterations, problem was not observed.

Please let us know how RSD wants to us to move forward in testing the configuration .

Created attachment 476223 [details]
multipath.conf 1_31_11

Created attachment 476225 [details]
SCR file 1_31_11

Great. I'm going to move this bug to modified, so that I can add 

no_path_retry 6

parameter to the default configuration for this device. I've also uploaded a recent kernel to

http://people.redhat.com/bmarzins/kernels/

so you can try out this kernel, to verify that you no longer need the workaround.  Of course it's usually a good idea to set fast_io_fail_tmo to something small, so that your multipath devices failover quickly.  Setting dev_loss_tmo is mostly a matter of personal preference (Some people want their dead devices to disappear quickly, some want them to never disappear), and the defaults set by the device drivers are usually reasonable (from 30-60 seconds).  So even if the workaround isn't necessary, it (at least the fast_io_fail part) is still a good practice to get the best performance out of you multipath devices.

Questions from Japan team:

(Q1 to RedHat) About setting fast_io_fail_tmo = 5
 
In the previous e-mail from RH, we were told that
"fast_io_fail_tmo tells the scsi layer to fail back IO after the scsi device has been down for 5 seconds.".
 
We need more details about the effect of setting to 5 for this issue, and about why 5sec is a suitable setting?
 
 
(Q2 to RedHat) About setting dev_loss_tmo = 30
 
In the previous e-mail from RH, we were told that
"dev_loss_tmo tells the scsi layer to remove the scsi device after it has been down for 30 seconds."
 
We need more details about the effect of setting to 30 for this issue, and why 30sec is a suitable setting?
 
 
(Q3 to HDS and RH) About setting multipath.conf parameters, how should/will HDS handle parameter distribution?
 
  (Will HDS/RH tell users to check the parameters? Suggestions?)

Q1: fast_io_fail_tmo is the number of seconds that the system waits after a problem has been detected on a scsi device, before failing back all the IO sent to it.  This parameter defaults to "off".  With this parameter set to "off", the system will not fail back IOs until the scsi device gets removed after dev_loss_tmo expires (see Q2).  If this parameter is set too high, multipath must wait for a long time before it can send the IO down another path, reducing its performance during path failures.  If this parameter is set too low, IOs may be failed back because of a temporary scsi issue, unnecessarily causing a failover  scenario, and reducing performance.  In practice 5 seconds seems to be ample time for the scsi system to recover from most temporary issues, and is short enough that multipath doesn't suffer a noticeable performance issue.

Q2. dev_loss_tmo is the number of seconds that the system waits after a problem has been detected on a scsi device, before removing the device completely. This value defaults to 60, but it usually overridden by the device driver.  The device drivers typically set it to between 30 and 60 seconds. I asked you to set this mostly out of paranoia, so that we were guaranteed that it was not set to the same thing as fast_io_fail_tmo.  If you did not set this in mutipath.conf, and you weren't changing this value from the default somewhere else (for example, in /etc/modprobe.d), the default value would work fine for you.  The proper value to set this is largely a matter of personal preference. If you would like your failed devices to disappear quickly, set this value low. If you would like your failed devices to remain on the system for a long time, set this value high.  You should note that once a device has been removed from the system, multipathd will no longer send messages that the device has failed.  You should also note that if you are multipathing your root device, and it gets removed from the system, multipath will not be able to restore it.

Q3. I'm not sure how you want to handle this.  This bug is a race, and it appears that most people do not hit it.  As far as I know, this is the third
report of the issue.  The race is also independent of the storage hardware. Red hat has been advising people to set these paramters, but largely for performance reasons.  Hopefully the RHEL 6.1 multipath manual will have specific recommendations about this. Of course, in RHEL 6.1 the kernel bug that is causing this hang will be fix, but it's still a good idea to set these parameters.

Activity 

a) Reset each alternate path switches every 30 mins,  repeatedly  over a minimum of 10 iterations (5/per path.).
b) Power off each active alternate path switch(es) a minimum of   10 iterations (5/per path.).
c) Dummy Replace RAID CHT (2 times)

Observations
Configuration :(1)
  －QLe2562－

a) No issues observed.
b) No issues observed.
c) No issues observed.
Configuration :(2)
  －Lpe12002－

a) No issues observed.
b) No issues observed.
c) 2 iterations for channel board in Path-1, problem did not occur . (path：FC Switch   －RAID700 Port 5A)
   2 iterations for channel board in Path-2, Problem occurred .      (path：Direct      －RAID700 Port 6A
   (NB: System crashed during  of DR in direct path . The problem occurs in Direct path.)

Please let us know if you need any further clarifications

Paul

Are the above results with the 2.6.32-71.16.1.el6 kernel from http://people.redhat.com/bmarzins/kernels/ ?

Johnray, I just made that kernel available yesterday, so those are with it.  I talked to Paul over email, and confirmed that the system crash from Comment 63 is the same one that they have had in the past.  It is due to a emulex driver issue.  I haven't been able to find a bug number for it, but looking at the code, it appears to have been resolved in the kernel I provided.

Hello All, 

We have confirmed that the test kernel made available to us has indeed "fixed" all open issues for HDS. We would like to know what specific "changes" or "fixes" have been added for HDS and which "patches" have been added to the Emulex and QLogic drivers, if possible.

Also, HDS would like to know the expected GA date and the kernel version for the above test kernel fixes

Tanks to everyone who helped and worked to support uswith these items. We appreciate your superb assistance and valuable time.

Regards,

Paul

As comment #68 mentioned,

The only change for this bug is adding 'no_path_retry 6' into multipath default configuration:
devices {
#snip
        device {
                vendor "HITACHI"
                product "OPEN-.*"
                path_grouping_policy multibus
                getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
                path_selector "round-robin 0"
                path_checker tur
                features "0"
                hardware_handler "0"
                prio const
                rr_weight uniform
                no_path_retry 6      #<-------
                rr_min_io 1000
        }
#snip
}

This change has been applied into device-mapper-multipath-0.4.9-38.el6

Kernel issue has been fixed by other bug.

Paul,
For the RHEL 6.1 release date or z-steam issue, I think Redhat PM could answer your qustion.

Ben,
Can you answer Paul's qustion about kernel patch or provide the bug number for kernel?

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Previously, DM-Multipath did not set a default value for the no_path_retry parameter for Hitachi R700 devices. With this update, the parameter value for the devices is set to 6 by default.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0725.html