Red Hat Bugzilla – Bug 105912
QLA12160 instability (Kernel may crash)
Last modified: 2007-11-30 17:06:58 EST
Description of problem:
When a large sequential I/O to multiple disks through single HBA,
a lot of SCSI time-out message appears for several hard disks
and kernel eventually panics.
The cause of this is low level driver accepts too many requests to handle.
The solution is setting driver parameter to proper value.
Patch will be attached.
Moreover, the version of QLA12160 driver is 3.00 Beta which is very old
and not stable. Even AS 2.1 has 3.23.19 Beta driver.
Version-Release number of selected component (if applicable):
Everytime after typical I/O load.
Steps to Reproduce:
Do following sequence.
1. Connect 4 hard disks to 1 HBA
(They are called as /dev/sda, /dev/sdb, /dev/sdc, /dev/sdd in following
2. Create partitions larger than 5GB on each disks
# parted /dev/sda mkpartfs primary ext2 0.000 5000.000
# parted /dev/sdb mkpartfs primary ext2 0.000 5000.000
# parted /dev/sdc mkpartfs primary ext2 0.000 5000.000
# parted /dev/sdd mkpartfs primary ext2 0.000 5000.000
3. Mount the partitions
# mkdir /mnt/0; mkdir /mnt/1; mkdir /mnt/2; mkdir /mnt/3
# mount /dev/sda1 /mnt/0
# mount /dev/sdb1 /mnt/1
# mount /dev/sdc1 /mnt/2
# mount /dev/sdd1 /mnt/3
4. Execute the shell script below
while true; do
i=`expr $i + 1`
echo -n "Loop = $i, "
dd if=/dev/zero of=/mnt/0/zero0 bs=16M count=120 &
dd if=/dev/zero of=/mnt/1/zero1 bs=16M count=120 &
dd if=/dev/zero of=/mnt/2/zero2 bs=16M count=120 &
dd if=/dev/zero of=/mnt/3/zero3 bs=16M count=120
Kernel will panic or SCSI timeout will occur about an hour after the step 4.
Panic and/or timeout should not occur.
Mid level driver (e.g. scsi.c) puts request into per-LUN queues of low
level drivers and invokes timer for request time-out handling.
The maximum number of queueable requests is determined by 'can_queue'
member of Scsi_Host_Template. qla1280.c sets can_queue as 0xfffff.
Low level driver (e.g. qla1280.c) issues requests using HBA's ring buffer.
The size of ring buffer is hardcoded as 256 (=REQUEST_ENTRY_CNT).
Thus, mid level can puts, at maximum, 0xfffff * (number of LUNs) requests
simultaneously. On the other hand, low level can handle only 256 at the
same time. This causes the situation that many requests remain in
low level waiting for room in ring buffer. ....(1)
The maximum number of issue-able requests per LUN is determined by
hiwat member of bus_param_t.
The hiwat value is set as max_queue_depth (NVRAM parameter) minus 1.
The max_queue_depth seems to be set as 256 and not being able to change
from QLA12160 BIOS utility.
low level driver puts requests from queue to ring buffer until the one
of following conditions is fulfilled:
- the queue becomes empty
- the requests of the LUN reaches the hiwat value
- the ring buffer becomes full
When the ring buffer becomes full, remaining requests will be put back
to the queue. Low level driver will re-try putting into ring buffer
when outstanding I/O will complete.
As the size of the ring buffer and the hiwat value are very close
and the low level driver can put as many request as possible for single
LUN, the ring buffer can be occupied by the requests for the LUN.
Once the occupation occurs, the LUN again gets the next chance to issue
requests after I/O completion.
Thus, the inequality of issueing requests continues. ...(2)
The problems (1) and (2) brings up the situation that many requests for some
LUNs causes request time-out without actually issued to HBAs.
The many time-outs cause kernel panic or disability of normal operation
Created attachment 94814 [details]
patch to limit max queue depth
This patch does following things:
- Change can_queue value of Scsi_Host from 0xfffff to REQUEST_ENTRY_CNT.
- Add static variable qla1280_max_queue_depth_limit.
Set qla1280_max_queue_depth_limit to REQUEST_ENTRY_CNT/16.
- If max_queue_depth parameter obtained from NVRAM exceeds the
set max_queue_depth as qla1280_max_queue_depth_limit.
The qla1280 is unmaintained in the 2.4 kernel. The RHEL 3 release notes indicate
this, and indicate that Red Hat only supports this driver on x86 platforms. As a
result, changes to this driver are risky and of limited benefit. We are not
planning to apply the requested patch to RHEL 3 for these reasons.