Bug 78312

Summary:

Kernel hang (possibly from disk or network load)

Product:

[Retired] Red Hat Linux

Reporter:

Need Real Name <mjeffery>

Component:

kernel

Assignee:

Arjan van de Ven <arjanv>

Status:

CLOSED ERRATA

QA Contact:

Brian Brock <bbrock>

Severity:

high

Docs Contact:

Priority:

high

Version:

7.3

CC:

sct

Target Milestone:

---

Target Release:

---

Hardware:

i686

OS:

Linux

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2003-03-01 03:05:40 UTC

Type:

---

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
Syslog output containing sysreq-T/P from 1st hang (duplicated from bug 77508)	none
Serial console log of sysreq-T/P from 2nd hang	none
Diff between sysreq-T/P attempts for 2nd hang	none

Description Need Real Name 2002-11-21 03:33:33 UTC

Description of Problem:

System hangs unexpectedly, apparently after significant disk activity.  Network
access fails, alt-sysreq works (but alt-sysreq-b completely jams system).

This has happened twice with two separate kernels, separated by about a week.


Version-Release number of selected component (if applicable):

kernel-bigmem-2.4.18-17.7.x.i686    (1st)
kernel-bigmem-2.4.18-18.7.x.i686    (2nd)


How Reproducible:

Unclear.  Loading the machine seems to help.

First hang happened while lightly loading Oracle (including DB changes) and
filesystem.

Second happened while importing large files (Oracle backup files) over network
using ssh.


Additional Information:

4 CPU P4 Xeon system (Dell PowerEdge 6650)
Dell PERC 3/QC
All disks are SCSI, on the PERC RAID.

See bug 77508.  This hang occurred while trying to diagnose another problem.  I
now think this one is unrelated.  This is all the same machine.

This machine did do similar tasks using an earlier kernal (2.4.18-3 or
2.4.18-10---I forget which) without the problems described here.

I have two sets of sysreq-T/Ps to attach from the second hang.  They are
separated by about 20 minutes and a sysreq-S/U.  Actually, they are so similar I
will attach the first and a diff.

Comment 1 Need Real Name 2002-11-21 03:35:50 UTC

Created attachment 85824 [details]
Syslog output containing sysreq-T/P from 1st hang (duplicated from bug 77508)

Comment 2 Need Real Name 2002-11-21 03:37:08 UTC

Created attachment 85825 [details]
Serial console log of sysreq-T/P from 2nd hang

Comment 3 Need Real Name 2002-11-21 03:38:19 UTC

Created attachment 85826 [details]
Diff between sysreq-T/P attempts for 2nd hang

Comment 4 Stephen Tweedie 2002-11-21 12:14:14 UTC

In the second case, there is nothing suspicious at all in the logs.  There are
just a few processes waiting for disk IO.  There is absolutely no deadlocked
processes.

So, all we know is that a disk IO went missing --- we scheduled it to disk, but
it never completed, and one by one a bunch of kernel processes started waiting
uninterruptibly for that lost IO.

In conjunction with the first hang's trace, which indicates being locked in the
MegaServ process, this looks even more like a driver-level problem --- either a
driver bug, a firmware problem or a hardware problem causing lost IOs.  Are
there any other IO messages in the logs?

Comment 5 Need Real Name 2002-11-21 21:23:49 UTC

There are absolutely no other messages that seem connected to the crash---the
only thing close in time are a few securelog messages recording my use of ssh
and su.  I see no evidence that a single sector was written to the disk once it
hung (for the 2nd hang), and the last thing logged to the console before the
sysreqs are boot messages.

I've upped sys.printk to 15.  Is there anything else I can do to get more
debugging info out of this system?

Comment 6 Need Real Name 2003-03-01 03:05:40 UTC

Upgrading to 2.4.18-19.7.xbigmem fixed this.
We've gotten over 45 days of uptime and no freezes since.