=Comment: #0=================================================
Venkateswarara Jujjuri <jvrao.com> - 
System become not usable when the paths were offline/online'd  in quick successions. 
This appears to be an issue in the most recent kernels. Chandra says he has seen it
in main-line even on 2.6.27.

This defect is to track this problem. This is not a R2 blocker at this point.. It may be an iFix or
 R2-SR1.

When this happened, system is pingable.. but nothing useful beyond that.. Here is what I collected
from the console.

--------------

Buffer I/O error on device sdt, logical block 0^M
Dev sdt: unable to read RDB block 0^M
 unable to read partition table^M
sd 0:0:0:0: [sdt] Attached SCSI disk^M
sd 0:0:0:0: Attached scsi generic sg0 type 0^M
scsi 0:0:0:0: Direct-Access     IBM      1815      FAStT  0914 PQ: 0 ANSI: 5^M
sysfs: duplicate filename '0:0:0:0' can not be created^M
WARNING: at fs/sysfs/dir.c:424 sysfs_add_one()^M
Pid: 713, comm: scsi_wq_0 Not tainted 2.6.24-ibmrt2.10-pf #1^M
^M
Call Trace:^M
 [<ffffffff810f7b4d>] ? sysfs_find_dirent+0x1c/0x31^M
 [<ffffffff810f7bbe>] sysfs_add_one+0x5c/0xc9^M
 [<ffffffff810f81b5>] create_dir+0x58/0x93^M
 [<ffffffff810f8228>] sysfs_create_dir+0x38/0x4f^M
 [<ffffffff811368ec>] ? kobject_get+0x1a/0x21^M
 [<ffffffff81136eba>] kobject_add+0x100/0x1ba^M
 [<ffffffff811b0010>] device_add+0x9f/0x61d^M
 [<ffffffff8805e5f1>] :scsi_mod:scsi_sysfs_add_sdev+0x3c/0x1c9^M
 [<ffffffff8805c310>] :scsi_mod:scsi_probe_and_add_lun+0x9ed/0xb16^M
 [<ffffffff8805cdcb>] :scsi_mod:__scsi_scan_target+0x438/0x601^M
 [<ffffffff8805d486>] :scsi_mod:scsi_sc^M
=Comment: #1=================================================
Venkateswarara Jujjuri <jvrao.com> - 
I figured an easy way to reproduce this. 

This is a race condition of removing/adding a sysfs entry.

I checked on elm3c24 in "/sys/class/fc_remote_ports/rport-0:0-0" that the dev_loss_tmo value is set
to 35 seconds...which means once the port goes offline/not-available for 35 sec, system starts
delete those sysfs entries. 

So to reproduce this...

1. Start blast/IO tests on the system.
2. Take one of the ports offline.
3. Wait for 35 secs.
4. Bring that port online.

Since this is a race, we may have to repeat this couple of times..

Here is the latest trace dump:

 sdt:end_request: I/O error, dev sdt, sector 0
Buffer I/O error on device sdt, logical block 0
Dev sdt: unable to read RDB block 0
 unable to read partition table
sd 0:0:0:0: [sdt] Attached SCSI disk
sd 0:0:0:0: Attached scsi generic sg0 type 0
scsi 0:0:0:0: Direct-Access     IBM      1815      FAStT  0914 PQ: 0 ANSI: 5
sysfs: duplicate filename '0:0:0:0' can not be created
WARNING: at fs/sysfs/dir.c:424 sysfs_add_one()
Pid: 713, comm: scsi_wq_0 Not tainted 2.6.24-ibmrt2.10-pf #1

Call Trace:
 [<ffffffff810f7b4d>] ? sysfs_find_dirent+0x1c/0x31
 [<ffffffff810f7bbe>] sysfs_add_one+0x5c/0xc9
 [<ffffffff810f81b5>] create_dir+0x58/0x93
 [<ffffffff810f8228>] sysfs_create_dir+0x38/0x4f
 [<ffffffff811368ec>] ? kobject_get+0x1a/0x21
 [<ffffffff81136eba>] kobject_add+0x100/0x1ba
 [<ffffffff811b0010>] device_add+0x9f/0x61d
 [<ffffffff8805e5f1>] :scsi_mod:scsi_sysfs_add_sdev+0x3c/0x1c9
 [<ffffffff8805c310>] :scsi_mod:scsi_probe_and_add_lun+0x9ed/0xb16
 [<ffffffff8805cdcb>] :scsi_mod:__scsi_scan_target+0x438/0x601
 [<ffffffff8805d486>] :scsi_mod:scsi_scan_target+0x9a/0xaf
 [<ffffffff88095b08>] :scsi_transport_fc:fc_scsi_scan_rport+0x7a/0x9d
 [<ffffffff8104d5d3>] run_workqueue+0x108/0x1d8
 [<ffffffff88095a8e>] ? :scsi_transport_fc:fc_scsi_scan_rport+0x0/0x9d
 [<ffffffff8104e36e>] ? worker_thread+0x0/0xea
 [<ffffffff8104e44d>] worker_thread+0xdf/0xea
 [<ffffffff8105159b>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff81051477>] kthread+0x49/0x76
 [<ffffffff8100d048>] child_rip+0xa/0x12
 [<ffffffff8105142e>] ? kthread+0x0/0x76
 [<ffffffff8100d03e>] ? child_rip+0x0/0x12

kobject_add failed for 0:0:0:0 with -EEXIST, don't try to register things with the same name in the
same directory.
Pid: 713, comm: scsi_wq_0 Not tainted 2.6.24-ibmrt2.10-pf #1

Call Trace:
 [<ffffffff81136f3f>] kobject_add+0x185/0x1ba
 [<ffffffff811b0010>] device_add+0x9f/0x61d
 [<ffffffff8805e5f1>] :scsi_mod:scsi_sysfs_add_sdev+0x3c/0x1c9
 [<ffffffff8805c310>] :scsi_mod:scsi_probe_and_add_lun+0x9ed/0xb16
 [<ffffffff8805cdcb>] :scsi_mod:__scsi_scan_target+0x438/0x601
 [<ffffffff8805d486>] :scsi_mod:scsi_scan_target+0x9a/0xaf
 [<ffffffff88095b08>] :scsi_transport_fc:fc_scsi_scan_rport+0x7a/0x9d
 [<ffffffff8104d5d3>] run_workqueue+0x108/0x1d8
 [<ffffffff88095a8e>] ? :scsi_transport_fc:fc_scsi_scan_rport+0x0/0x9d
 [<ffffffff8104e36e>] ? worker_thread+0x0/0xea
 [<ffffffff8104e44d>] worker_thread+0xdf/0xea
 [<ffffffff8105159b>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff81051477>] kthread+0x49/0x76
 [<ffffffff8100d048>] child_rip+0xa/0x12
 [<ffffffff8105142e>] ? kthread+0x0/0x76
 [<ffffffff8100d03e>] ? child_rip+0x0/0x12

error 1
scsi 0:0:0:0: Unexpected response from lun 0 while scanning, scan aborted
device-mapper: multipath: Failing path 8:128.
Unable to handle kernel NULL pointer dereference at 0000000000000020 RIP: 
 [<ffffffff881c199e>] :dm_multipath:activate_path+0x4b/0x1b0
PGD 0 
Oops: 0000 [1] PREEMPT SMP 
CPU 0 
Modules linked in: autofs4 hidp rfcomm l2cap bluetooth sunrpc nf_conntrack_netbios_ns ipt_REJECT
nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables ip6t_REJECT xt_tcpudp
ip6table_filter ip6_tables x_tables video output sbs sbshc battery ac ipv6 parport_pc lp parport sg
mptsas bnx2 mptscsih mptbase netxen_nic scsi_transport_sas button serio_raw shpchp pcspkr
dm_round_robin dm_multipath dm_snapshot dm_zero dm_mirror dm_mod qla2xxx scsi_transport_fc scsi_tgt
scsi_dh_rdac scsi_dh sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd
Pid: 776, comm: kmpath_handlerd Not tainted 2.6.24-ibmrt2.10-pf #1
RIP: 0010:[<ffffffff881c199e>]  [<ffffffff881c199e>] :dm_multipath:activate_path+0x4b/0x1b0
RSP: 0018:ffff81014d5cde60  EFLAGS: 00010202
RAX: ffff81014d5c4140 RBX: ffff81014e00ce18 RCX: 0000000000000000
RDX: ffff81014d5c4140 RSI: ffff81014e00ce68 RDI: ffff81014e00ce18
RBP: ffff81014d5cde90 R08: ffff81014e00ce68 R09: ffff81000900b9e0
R10: 0000000000000000 R11: ffff81014fa9fe90 R12: ffff81014e00ce00
R13: 0000000000000020 R14: ffff81014e0a1838 R15: ffff81014e00ce58
FS:  000000004009c940(0000) GS:ffffffff813ef100(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000020 CR3: 0000000000201000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kmpath_handlerd (pid: 776, threadinfo ffff81014d5cc000, task ffff81014d5c4140)
Stack:  ffff81014d5c4488 ffff81014e0a1800 ffff81014e00ce78 ffff81014e0a1838
 ffff81014e0a1838 ffff81014e00ce58 ffff81014d5cded0 ffffffff8104d5d3
 ffffffff881c1953 ffff81014e0a1800 ffffffff8104e36e ffff81014d595c18
Call Trace:
 [<ffffffff8104d5d3>] run_workqueue+0x108/0x1d8
 [<ffffffff881c1953>] ? :dm_multipath:activate_path+0x0/0x1b0
 [<ffffffff8104e36e>] ? worker_thread+0x0/0xea
 [<ffffffff8104e44d>] worker_thread+0xdf/0xea
 [<ffffffff8105159b>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff81051477>] kthread+0x49/0x76
 [<ffffffff8100d048>] child_rip+0xa/0x12
 [<ffffffff8105142e>] ? kthread+0x0/0x76
 [<ffffffff8100d03e>] ? child_rip+0x0/0x12


Code: 0c f9 4d 8b ac 24 90 00 00 00 48 89 df 49 c7 84 24 90 00 00 00 00 00 00 00 e8 8c 59 0c f9 49
83 c5 20 4d 85 ed 0f 84 58 01 00 00 <49> 8b 45 00 4d 8d 75 e0 48 8b 40 18 48 8b 80 d0 00 00 00 48 8b 
RIP  [<ffffffff881c199e>] :dm_multipath:activate_path+0x4b/0x1b0
 RSP <ffff81014d5cde60>
CR2: 0000000000000020
---[ end trace 3b17eae69e6aaa51 ]---

=Comment: #2=================================================
Venkateswarara Jujjuri <jvrao.com> - 
This is a upstream defect.
Problem is there in the mainline too.

=Comment: #3=================================================
Venkateswarara Jujjuri <jvrao.com> - 
Apparently this got fixed in the upstream under:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit
h=32aeef605aa01e1fee45e052eceffb00e72ba2b0

diff --git a/drivers/scsi/scsi.c b/drivers/scsi/scsi.c
index 42e72a2..cbcd3f6 100644 (file)
--- a/drivers/scsi/scsi.c
+++ b/drivers/scsi/scsi.c
@@ -1095,7 +1095,8 @@ EXPORT_SYMBOL(__starget_for_each_device);
  * Description: Looks up the scsi_device with the specified @lun for a given
  * @starget.  The returned scsi_device does not have an additional
  * reference.  You must hold the host's host_lock over this call and
- * any access to the returned scsi_device.
+ * any access to the returned scsi_device. A scsi_device in state
+ * SDEV_DEL is skipped.
  *
  * Note:  The only reason why drivers should use this is because
  * they need to access the device list in irq context.  Otherwise you
@@ -1107,6 +1108,8 @@ struct scsi_device *__scsi_device_lookup_by_target(struct scsi_target *starget,
        struct scsi_device *sdev;
 
        list_for_each_entry(sdev, &starget->devices, same_target_siblings) {
+               if (sdev->sdev_state == SDEV_DEL)
+                       continue;
                if (sdev->lun ==lun)
                        return sdev;
        }


I will back port this patch and test out our R2.
=Comment: #4=================================================
Venkateswarara Jujjuri <jvrao.com> - 

This patch is checked into R2-SR1 to fix the problem.