Bug 514073
| Summary: | RHEL 5.4 cxgb3i (open-iscsi) hits skb_over_panic() on write | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 5 | Reporter: | kxie | ||||
| Component: | kernel | Assignee: | Mike Christie <mchristi> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Red Hat Kernel QE team <kernel-qe> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 5.4 | CC: | andriusb, aparanja, bugproxy, coughlan, cward, dzickus, hjia, jjarvis, mchristi, sekharan | ||||
| Target Milestone: | rc | Keywords: | OtherQA | ||||
| Target Release: | 5.4 | ||||||
| Hardware: | powerpc | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2009-09-02 08:31:32 UTC | Type: | --- | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 445210 | ||||||
| Attachments: |
|
||||||
This is a new driver for 5.4, so it cannot add a regression. Chelsio is doing the QE on this and it is their driver. They understand the risk for a change this late and think it is best to fix now. Adding devel ack. This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release. I found the problem originally and reported to Karen. Thanks for the fast fix Karen. I tested the same scenario with the attached patch, and I see no panic and the system is working as expected and staying healthy/ ------- Comment From sekharan.com 2009-07-24 10:46 EDT------- ---Problem Description--- Doing cp to a ext3 file system built on top of a iSCSI disk connected through the Chelsio cxgb3i driver panics the OS. ------------- [root@rulerlp8 ~]# ./lsscsi [0:0:1:0] disk AIX VDASD 0001 /dev/sda [0:0:2:0] disk AIX VDASD 0001 /dev/sdb [1:0:0:0] disk IBM 2107900 .181 /dev/sdc [1:0:0:1] disk IBM 2107900 .181 /dev/sdd [1:0:1:0] disk IBM 2107900 .181 /dev/sde [1:0:1:1] disk IBM 2107900 .181 /dev/sdf [1:0:2:0] disk IBM 1724-100 FAStT 0542 /dev/sdg [1:0:2:1] disk IBM 1724-100 FAStT 0542 /dev/sdh [1:0:3:0] disk IBM 2107900 .181 /dev/sdi [1:0:3:1] disk IBM 2107900 .181 /dev/sdj [1:0:4:0] disk IBM 2107900 .181 /dev/sdk [1:0:4:1] disk IBM 2107900 .181 /dev/sdl [2:0:0:0] disk NETAPP LUN 0.2 /dev/sdm [2:0:0:1] disk NETAPP LUN 0.2 /dev/sdn [2:0:0:2] disk NETAPP LUN 0.2 /dev/sdo [2:0:0:3] disk NETAPP LUN 0.2 /dev/sdp [root@rulerlp8 ~]# [root@rulerlp8 ~]# [root@rulerlp8 ~]# mount /dev/sdn1 /test1 [root@rulerlp8 ~]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup01-LogVol00 62308260 2404672 56687412 5% / /dev/sda2 93335 17950 70566 21% /boot tmpfs 2603328 0 2603328 0% /dev/shm /dev/sdn1 6879960 146568 6383904 3% /test1 [root@rulerlp8 ~]# ls -l /test1 total 16 drwx------ 2 root root 16384 Jul 24 00:20 lost+found [root@rulerlp8 ~]# lsmod | grep cxgb cxgb3i 81893 2 libiscsi_tcp 43869 2 iscsi_tcp,cxgb3i libiscsi2 73573 5 ib_iser,iscsi_tcp,bnx2i,cxgb3i,libiscsi_tcp scsi_transport_iscsi2 75041 7 ib_iser,iscsi_tcp,bnx2i,cxgb3i,libiscsi2 cxgb3 297409 1 cxgb3i 8021q 51801 1 cxgb3 scsi_mod 242249 12 ib_iser,iscsi_tcp,bnx2i,cxgb3i,libiscsi2,scsi_transport_iscsi2,scsi_dh,sg,lpfc,scsi_transport_fc,ibmvscsic,sd_mod [root@rulerlp8 ~]# cp -fr /etc/ /test1 ------------------------------------ skb_over_panic: text:d000000000a423b8 len:15920 put:15872 head:c000000003507500 data:c000000003507540 tail:c00000000350b370 end:c000000003507680 dev:<NULL> kernel BUG in skb_over_panic at net/core/skbuff.c:94! cpu 0x0: Vector: 700 (Program Check) at [c00000013fd13720] pc: c000000000338484: .skb_over_panic+0x50/0x68 lr: c000000000338480: .skb_over_panic+0x4c/0x68 sp: c00000013fd139a0 msr: 8000000000029032 current = 0xc00000000e113290 paca = 0xc00000000053c500 pid = 2035, comm = iscsi_q_2 kernel BUG in skb_over_panic at net/core/skbuff.c:94! enter ? for help 0:mon> t [c00000013fd13a30] d000000000a423cc .cxgb3i_conn_init_pdu+0x394/0x650 [cxgb3i] [c00000013fd13b00] d000000000980484 .iscsi_tcp_task_init+0x148/0x174 [libiscsi_tcp] [c00000013fd13b90] d0000000009e18d4 .iscsi_prep_scsi_cmd_pdu+0x498/0x5cc [libiscsi2] [c00000013fd13cb0] d0000000009e1ce4 .iscsi_xmitworker+0x16c/0x2dc [libiscsi2] [c00000013fd13d50] c000000000082240 .run_workqueue+0xdc/0x168 [c00000013fd13df0] c000000000082fac .worker_thread+0x12c/0x19c [c00000013fd13ee0] c000000000087ac8 .kthread+0x128/0x178 [c00000013fd13f90] c0000000000275c4 .kernel_thread+0x4c/0x68 0:mon> Contact Information = chandra.seetharaman.com ---Additional Hardware Info--- Ethernet controller: Chelsio Communications Inc T310 10GbE Single Port Adapter ---uname output--- Linux rulerlp8.upt.austin.ibm.com 2.6.18-158.el5 #1 SMP Mon Jul 13 16:27:25 EDT 2009 ppc64 ppc64 ppc64 GNU/Linux Machine Type = power lpar ---Debugger--- A debugger is not configured ---Steps to Reproduce--- - configure a iSCSI disk thru the Chelsio network adapter. - create a ext3 file system on the disk - mount a iSCSI disk partition onto /test - do cp -fr /etc/ /test ---Kernel - Drivers Component Data--- Stack trace output: skb_over_panic: text:d000000000a423b8 len:15920 put:15872 head:c000000003507500 data:c000000003507540 tail:c00000000350b370 end:c000000003507680 dev:<NULL> kernel BUG in skb_over_panic at net/core/skbuff.c:94! cpu 0x0: Vector: 700 (Program Check) at [c00000013fd13720] pc: c000000000338484: .skb_over_panic+0x50/0x68 lr: c000000000338480: .skb_over_panic+0x4c/0x68 sp: c00000013fd139a0 msr: 8000000000029032 current = 0xc00000000e113290 paca = 0xc00000000053c500 pid = 2035, comm = iscsi_q_2 kernel BUG in skb_over_panic at net/core/skbuff.c:94! enter ? for help 0:mon> t [c00000013fd13a30] d000000000a423cc .cxgb3i_conn_init_pdu+0x394/0x650 [cxgb3i] [c00000013fd13b00] d000000000980484 .iscsi_tcp_task_init+0x148/0x174 [libiscsi_tcp] [c00000013fd13b90] d0000000009e18d4 .iscsi_prep_scsi_cmd_pdu+0x498/0x5cc [libiscsi2] [c00000013fd13cb0] d0000000009e1ce4 .iscsi_xmitworker+0x16c/0x2dc [libiscsi2] [c00000013fd13d50] c000000000082240 .run_workqueue+0xdc/0x168 [c00000013fd13df0] c000000000082fac .worker_thread+0x12c/0x19c [c00000013fd13ee0] c000000000087ac8 .kthread+0x128/0x178 [c00000013fd13f90] c0000000000275c4 .kernel_thread+0x4c/0x68 0:mon> Oops output: no System Dump Info: The system is not configured to capture a system dump. *Additional Instructions for chandra.seetharaman.com: -Post a private note with access information to the machine that the bug is occuring on. -Attach sysctl -a output output to the bug. ------- Comment From sekharan.com 2009-07-28 18:05 EDT------- Karen Xie (our Chelsio partner) has opened a bug in Redhat Bugzilla for the same issue. https://bugzilla.redhat.com/show_bug.cgi?id=514073 Can this bug be mirrored/duped to that one ? ------- Comment From sekharan.com 2009-07-28 21:14 EDT------- Here is the description from the bugzilla in RedHat: Description of problem: On systems where the page size is large (64K) and skb's MAX_SKB_FRAGS is relatively small (say PPC64), for write command the cxgb3i could wrongly allocate a much smaller skb than what is needed, thus trigger the skb_over_panic() when trying to write data to the skb's head. How reproducible: Steps to Reproduce: 1. On a system with large page size (64K), start cxgb3i and connect it to target 2. do some write operations on the initiator disk Actual results: skb_over_panic() could be triggered during write operation. Expected results: There should be no error in completing the write operation. ------- Comment From sekharan.com 2009-07-28 21:16 EDT------- The problem is very easily reproducible, Creating a file system and doing a simple cp to that file system would cause this problem. It can be hit by any users that the iscsi storage connected thru the cxgb3 adapter. in kernel-2.6.18-162.el5 You can download this test kernel from http://people.redhat.com/dzickus/el5 Please do NOT transition this bugzilla state to VERIFIED until our QE team has sent specific instructions indicating when to do so. However feel free to provide a comment indicating that this fix has been verified. ------- Comment From sekharan.com 2009-08-05 19:14 EDT------- Downloaded 161.el5, verified that the problem is fixed. Will test again when next snapshot is released and close the bug. ------- Comment From sekharan.com 2009-08-05 19:15 EDT------- (In reply to comment #18) > Downloaded 161.el5, verified that the problem is fixed. > > Will test again when next snapshot is released and close the bug. > read as 162.el5 IBM, we're looking forward to received your test feedback on the -162 kernel. Thanks Dear RedHat, Please see comment #9 above your comment :) -162 kernel has been tested and the problem is verified to be fixed. ------- Comment From sekharan.com 2009-08-21 02:03 EDT------- Installed RHEL 5.4 RC2 and verified that this problem is fixed. Also ran the standard FVT tests to check for regression. Tests were successful. An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2009-1243.html |
Created attachment 355320 [details] kernel patch generated against dzickus 158.el5 Description of problem: On systems where the page size is large (64K) and skb's MAX_SKB_FRAGS is relatively small (say PPC64), for write command the cxgb3i could wrongly allocate a much smaller skb than what is needed, thus trigger the skb_over_panic() when trying to write data to the skb's head. How reproducible: Steps to Reproduce: 1. On a system with large page size (64K), start cxgb3i and connect it to target 2. do some write operations on the initiator disk Actual results: skb_over_panic() could be triggered during write operation. Expected results: There should be no error in completing the write operation.