Bug 497076 - [SAN] System becomes unusable after quick qlogic path offline/online
[SAN] System becomes unusable after quick qlogic path offline/online
Status: CLOSED NOTABUG
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: realtime-kernel (Show other bugs)
1.1
x86_64 All
low Severity high
: ---
: ---
Assigned To: Red Hat Real Time Maintenance
David Sommerseth
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-04-22 05:40 EDT by IBM Bug Proxy
Modified: 2016-05-22 19:28 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-09-12 15:19:13 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
This patch is checked into R2-SR1 to fix the problem. (2.78 KB, text/plain)
2009-04-22 05:40 EDT, IBM Bug Proxy
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
IBM Linux Technology Center 49277 None None None Never

  None (edit)
Description IBM Bug Proxy 2009-04-22 05:40:44 EDT
=Comment: #0=================================================
Venkateswarara Jujjuri <jvrao@us.ibm.com> - 
System become not usable when the paths were offline/online'd  in quick successions. 
This appears to be an issue in the most recent kernels. Chandra says he has seen it
in main-line even on 2.6.27.

This defect is to track this problem. This is not a R2 blocker at this point.. It may be an iFix or
 R2-SR1.

When this happened, system is pingable.. but nothing useful beyond that.. Here is what I collected
from the console.

--------------

Buffer I/O error on device sdt, logical block 0^M
Dev sdt: unable to read RDB block 0^M
 unable to read partition table^M
sd 0:0:0:0: [sdt] Attached SCSI disk^M
sd 0:0:0:0: Attached scsi generic sg0 type 0^M
scsi 0:0:0:0: Direct-Access     IBM      1815      FAStT  0914 PQ: 0 ANSI: 5^M
sysfs: duplicate filename '0:0:0:0' can not be created^M
WARNING: at fs/sysfs/dir.c:424 sysfs_add_one()^M
Pid: 713, comm: scsi_wq_0 Not tainted 2.6.24-ibmrt2.10-pf #1^M
^M
Call Trace:^M
 [<ffffffff810f7b4d>] ? sysfs_find_dirent+0x1c/0x31^M
 [<ffffffff810f7bbe>] sysfs_add_one+0x5c/0xc9^M
 [<ffffffff810f81b5>] create_dir+0x58/0x93^M
 [<ffffffff810f8228>] sysfs_create_dir+0x38/0x4f^M
 [<ffffffff811368ec>] ? kobject_get+0x1a/0x21^M
 [<ffffffff81136eba>] kobject_add+0x100/0x1ba^M
 [<ffffffff811b0010>] device_add+0x9f/0x61d^M
 [<ffffffff8805e5f1>] :scsi_mod:scsi_sysfs_add_sdev+0x3c/0x1c9^M
 [<ffffffff8805c310>] :scsi_mod:scsi_probe_and_add_lun+0x9ed/0xb16^M
 [<ffffffff8805cdcb>] :scsi_mod:__scsi_scan_target+0x438/0x601^M
 [<ffffffff8805d486>] :scsi_mod:scsi_sc^M
=Comment: #1=================================================
Venkateswarara Jujjuri <jvrao@us.ibm.com> - 
I figured an easy way to reproduce this. 

This is a race condition of removing/adding a sysfs entry.

I checked on elm3c24 in "/sys/class/fc_remote_ports/rport-0:0-0" that the dev_loss_tmo value is set
to 35 seconds...which means once the port goes offline/not-available for 35 sec, system starts
delete those sysfs entries. 

So to reproduce this...

1. Start blast/IO tests on the system.
2. Take one of the ports offline.
3. Wait for 35 secs.
4. Bring that port online.

Since this is a race, we may have to repeat this couple of times..

Here is the latest trace dump:

 sdt:end_request: I/O error, dev sdt, sector 0
Buffer I/O error on device sdt, logical block 0
Dev sdt: unable to read RDB block 0
 unable to read partition table
sd 0:0:0:0: [sdt] Attached SCSI disk
sd 0:0:0:0: Attached scsi generic sg0 type 0
scsi 0:0:0:0: Direct-Access     IBM      1815      FAStT  0914 PQ: 0 ANSI: 5
sysfs: duplicate filename '0:0:0:0' can not be created
WARNING: at fs/sysfs/dir.c:424 sysfs_add_one()
Pid: 713, comm: scsi_wq_0 Not tainted 2.6.24-ibmrt2.10-pf #1

Call Trace:
 [<ffffffff810f7b4d>] ? sysfs_find_dirent+0x1c/0x31
 [<ffffffff810f7bbe>] sysfs_add_one+0x5c/0xc9
 [<ffffffff810f81b5>] create_dir+0x58/0x93
 [<ffffffff810f8228>] sysfs_create_dir+0x38/0x4f
 [<ffffffff811368ec>] ? kobject_get+0x1a/0x21
 [<ffffffff81136eba>] kobject_add+0x100/0x1ba
 [<ffffffff811b0010>] device_add+0x9f/0x61d
 [<ffffffff8805e5f1>] :scsi_mod:scsi_sysfs_add_sdev+0x3c/0x1c9
 [<ffffffff8805c310>] :scsi_mod:scsi_probe_and_add_lun+0x9ed/0xb16
 [<ffffffff8805cdcb>] :scsi_mod:__scsi_scan_target+0x438/0x601
 [<ffffffff8805d486>] :scsi_mod:scsi_scan_target+0x9a/0xaf
 [<ffffffff88095b08>] :scsi_transport_fc:fc_scsi_scan_rport+0x7a/0x9d
 [<ffffffff8104d5d3>] run_workqueue+0x108/0x1d8
 [<ffffffff88095a8e>] ? :scsi_transport_fc:fc_scsi_scan_rport+0x0/0x9d
 [<ffffffff8104e36e>] ? worker_thread+0x0/0xea
 [<ffffffff8104e44d>] worker_thread+0xdf/0xea
 [<ffffffff8105159b>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff81051477>] kthread+0x49/0x76
 [<ffffffff8100d048>] child_rip+0xa/0x12
 [<ffffffff8105142e>] ? kthread+0x0/0x76
 [<ffffffff8100d03e>] ? child_rip+0x0/0x12

kobject_add failed for 0:0:0:0 with -EEXIST, don't try to register things with the same name in the
same directory.
Pid: 713, comm: scsi_wq_0 Not tainted 2.6.24-ibmrt2.10-pf #1

Call Trace:
 [<ffffffff81136f3f>] kobject_add+0x185/0x1ba
 [<ffffffff811b0010>] device_add+0x9f/0x61d
 [<ffffffff8805e5f1>] :scsi_mod:scsi_sysfs_add_sdev+0x3c/0x1c9
 [<ffffffff8805c310>] :scsi_mod:scsi_probe_and_add_lun+0x9ed/0xb16
 [<ffffffff8805cdcb>] :scsi_mod:__scsi_scan_target+0x438/0x601
 [<ffffffff8805d486>] :scsi_mod:scsi_scan_target+0x9a/0xaf
 [<ffffffff88095b08>] :scsi_transport_fc:fc_scsi_scan_rport+0x7a/0x9d
 [<ffffffff8104d5d3>] run_workqueue+0x108/0x1d8
 [<ffffffff88095a8e>] ? :scsi_transport_fc:fc_scsi_scan_rport+0x0/0x9d
 [<ffffffff8104e36e>] ? worker_thread+0x0/0xea
 [<ffffffff8104e44d>] worker_thread+0xdf/0xea
 [<ffffffff8105159b>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff81051477>] kthread+0x49/0x76
 [<ffffffff8100d048>] child_rip+0xa/0x12
 [<ffffffff8105142e>] ? kthread+0x0/0x76
 [<ffffffff8100d03e>] ? child_rip+0x0/0x12

error 1
scsi 0:0:0:0: Unexpected response from lun 0 while scanning, scan aborted
device-mapper: multipath: Failing path 8:128.
Unable to handle kernel NULL pointer dereference at 0000000000000020 RIP: 
 [<ffffffff881c199e>] :dm_multipath:activate_path+0x4b/0x1b0
PGD 0 
Oops: 0000 [1] PREEMPT SMP 
CPU 0 
Modules linked in: autofs4 hidp rfcomm l2cap bluetooth sunrpc nf_conntrack_netbios_ns ipt_REJECT
nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables ip6t_REJECT xt_tcpudp
ip6table_filter ip6_tables x_tables video output sbs sbshc battery ac ipv6 parport_pc lp parport sg
mptsas bnx2 mptscsih mptbase netxen_nic scsi_transport_sas button serio_raw shpchp pcspkr
dm_round_robin dm_multipath dm_snapshot dm_zero dm_mirror dm_mod qla2xxx scsi_transport_fc scsi_tgt
scsi_dh_rdac scsi_dh sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd
Pid: 776, comm: kmpath_handlerd Not tainted 2.6.24-ibmrt2.10-pf #1
RIP: 0010:[<ffffffff881c199e>]  [<ffffffff881c199e>] :dm_multipath:activate_path+0x4b/0x1b0
RSP: 0018:ffff81014d5cde60  EFLAGS: 00010202
RAX: ffff81014d5c4140 RBX: ffff81014e00ce18 RCX: 0000000000000000
RDX: ffff81014d5c4140 RSI: ffff81014e00ce68 RDI: ffff81014e00ce18
RBP: ffff81014d5cde90 R08: ffff81014e00ce68 R09: ffff81000900b9e0
R10: 0000000000000000 R11: ffff81014fa9fe90 R12: ffff81014e00ce00
R13: 0000000000000020 R14: ffff81014e0a1838 R15: ffff81014e00ce58
FS:  000000004009c940(0000) GS:ffffffff813ef100(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000020 CR3: 0000000000201000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kmpath_handlerd (pid: 776, threadinfo ffff81014d5cc000, task ffff81014d5c4140)
Stack:  ffff81014d5c4488 ffff81014e0a1800 ffff81014e00ce78 ffff81014e0a1838
 ffff81014e0a1838 ffff81014e00ce58 ffff81014d5cded0 ffffffff8104d5d3
 ffffffff881c1953 ffff81014e0a1800 ffffffff8104e36e ffff81014d595c18
Call Trace:
 [<ffffffff8104d5d3>] run_workqueue+0x108/0x1d8
 [<ffffffff881c1953>] ? :dm_multipath:activate_path+0x0/0x1b0
 [<ffffffff8104e36e>] ? worker_thread+0x0/0xea
 [<ffffffff8104e44d>] worker_thread+0xdf/0xea
 [<ffffffff8105159b>] ? autoremove_wake_function+0x0/0x38
 [<ffffffff81051477>] kthread+0x49/0x76
 [<ffffffff8100d048>] child_rip+0xa/0x12
 [<ffffffff8105142e>] ? kthread+0x0/0x76
 [<ffffffff8100d03e>] ? child_rip+0x0/0x12


Code: 0c f9 4d 8b ac 24 90 00 00 00 48 89 df 49 c7 84 24 90 00 00 00 00 00 00 00 e8 8c 59 0c f9 49
83 c5 20 4d 85 ed 0f 84 58 01 00 00 <49> 8b 45 00 4d 8d 75 e0 48 8b 40 18 48 8b 80 d0 00 00 00 48 8b 
RIP  [<ffffffff881c199e>] :dm_multipath:activate_path+0x4b/0x1b0
 RSP <ffff81014d5cde60>
CR2: 0000000000000020
---[ end trace 3b17eae69e6aaa51 ]---

=Comment: #2=================================================
Venkateswarara Jujjuri <jvrao@us.ibm.com> - 
This is a upstream defect.
Problem is there in the mainline too.

=Comment: #3=================================================
Venkateswarara Jujjuri <jvrao@us.ibm.com> - 
Apparently this got fixed in the upstream under:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit
h=32aeef605aa01e1fee45e052eceffb00e72ba2b0

diff --git a/drivers/scsi/scsi.c b/drivers/scsi/scsi.c
index 42e72a2..cbcd3f6 100644 (file)
--- a/drivers/scsi/scsi.c
+++ b/drivers/scsi/scsi.c
@@ -1095,7 +1095,8 @@ EXPORT_SYMBOL(__starget_for_each_device);
  * Description: Looks up the scsi_device with the specified @lun for a given
  * @starget.  The returned scsi_device does not have an additional
  * reference.  You must hold the host's host_lock over this call and
- * any access to the returned scsi_device.
+ * any access to the returned scsi_device. A scsi_device in state
+ * SDEV_DEL is skipped.
  *
  * Note:  The only reason why drivers should use this is because
  * they need to access the device list in irq context.  Otherwise you
@@ -1107,6 +1108,8 @@ struct scsi_device *__scsi_device_lookup_by_target(struct scsi_target *starget,
        struct scsi_device *sdev;
 
        list_for_each_entry(sdev, &starget->devices, same_target_siblings) {
+               if (sdev->sdev_state == SDEV_DEL)
+                       continue;
                if (sdev->lun ==lun)
                        return sdev;
        }


I will back port this patch and test out our R2.
=Comment: #4=================================================
Venkateswarara Jujjuri <jvrao@us.ibm.com> - 

This patch is checked into R2-SR1 to fix the problem.
Comment 1 IBM Bug Proxy 2009-04-22 05:40:49 EDT
Created attachment 340703 [details]
This patch is checked into R2-SR1 to fix the problem.
Comment 2 IBM Bug Proxy 2009-06-29 12:21:06 EDT
------- Comment From sripathik@in.ibm.com 2009-06-29 12:12 EDT-------
I downloaded kernel-rt-2.6.29.4-23.el5rt.src.rpm and saw this patch in it. Since MRG1.2 kernel is going to be based on 2.6.29, we can safely assume the fix is in MRG. Closing this bug.

Note You need to log in before you can comment on or make changes to this bug.