Red Hat Bugzilla – Bug 78312
Kernel hang (possibly from disk or network load)
Last modified: 2005-10-31 17:00:50 EST
Description of Problem:
System hangs unexpectedly, apparently after significant disk activity. Network
access fails, alt-sysreq works (but alt-sysreq-b completely jams system).
This has happened twice with two separate kernels, separated by about a week.
Version-Release number of selected component (if applicable):
Unclear. Loading the machine seems to help.
First hang happened while lightly loading Oracle (including DB changes) and
Second happened while importing large files (Oracle backup files) over network
4 CPU P4 Xeon system (Dell PowerEdge 6650)
Dell PERC 3/QC
All disks are SCSI, on the PERC RAID.
See bug 77508. This hang occurred while trying to diagnose another problem. I
now think this one is unrelated. This is all the same machine.
This machine did do similar tasks using an earlier kernal (2.4.18-3 or
2.4.18-10---I forget which) without the problems described here.
I have two sets of sysreq-T/Ps to attach from the second hang. They are
separated by about 20 minutes and a sysreq-S/U. Actually, they are so similar I
will attach the first and a diff.
Created attachment 85824 [details]
Syslog output containing sysreq-T/P from 1st hang (duplicated from bug 77508)
Created attachment 85825 [details]
Serial console log of sysreq-T/P from 2nd hang
Created attachment 85826 [details]
Diff between sysreq-T/P attempts for 2nd hang
In the second case, there is nothing suspicious at all in the logs. There are
just a few processes waiting for disk IO. There is absolutely no deadlocked
So, all we know is that a disk IO went missing --- we scheduled it to disk, but
it never completed, and one by one a bunch of kernel processes started waiting
uninterruptibly for that lost IO.
In conjunction with the first hang's trace, which indicates being locked in the
MegaServ process, this looks even more like a driver-level problem --- either a
driver bug, a firmware problem or a hardware problem causing lost IOs. Are
there any other IO messages in the logs?
There are absolutely no other messages that seem connected to the crash---the
only thing close in time are a few securelog messages recording my use of ssh
and su. I see no evidence that a single sector was written to the disk once it
hung (for the 2nd hang), and the last thing logged to the console before the
sysreqs are boot messages.
I've upped sys.printk to 15. Is there anything else I can do to get more
debugging info out of this system?
Upgrading to 2.4.18-19.7.xbigmem fixed this.
We've gotten over 45 days of uptime and no freezes since.