Bug 198892

Description of problem:

Customer reported recurring hangs on LVS director (Red Hat Cluster Suite). At
one hang he was able to produce a crash dump by pressing Alt-SysRq-c.

Computer has 4 physical CPU's (2 dual core CPU's):

  SYSTEM MAP: kernel/boot/System.map-2.6.9-34.0.1.ELsmp
DEBUG KERNEL: /usr/lib/debug/lib/modules/2.6.9-34.0.1.ELsmp/vmlinux
(2.6.9-34.0.1.ELsmp)
    DUMPFILE: cluster2/vmcore
        CPUS: 4
        DATE: Sun Jul  9 18:48:38 2006
      UPTIME: 9 days, 09:00:04
LOAD AVERAGE: 43.99, 43.97, 43.91
       TASKS: 116
    NODENAME: cluster2
     RELEASE: 2.6.9-34.0.1.ELsmp
     VERSION: #1 SMP Wed May 17 17:05:24 EDT 2006
     MACHINE: i686  (2794 Mhz)
      MEMORY: 2 GB
       PANIC: "Oops: 0002 [#1]" (check log for details)
         PID: 20928
     COMMAND: "ftp_chk"
        TASK: f35d87b0  [THREAD_INFO: f377b000]
         CPU: 0
       STATE: TASK_RUNNING (SYSRQ)

Crash dump analysis revealed that the last four processes of the process list
were in a deadlock situation (see attachment "ps").

These four processes were "Service Monitoring Scripts" written using the
following sample script from the documentation of Red Hat Cluster Suite
(see
http://www.redhat.com/docs/manuals/csgfs/browse/rh-cs-en/s1-piranha-virtservs.html
)

---------------
#!/bin/sh

TEST=`dig -t soa example.com @$1 | grep -c dns.example.com

if [ $TEST != "1" ]; then
	echo "OK
else
	echo "FAIL"
fi
--------------

I am going to attach a copy of the actual scripts running at the time of the
deadlock. These scripts should have finished within a second, but they were
running for about 3 hours already when the user pressed Alt-SysRq-c (see
attachment "ps-t").

Each of the four processes were running on one of the CPU's, and each of them
was waiting on a spinlock in the read system call on the file /proc/meminfo (see
attachment "bt-a"). Thereby they managed to hang the machine, because no CPU was
left free to work.

This looks like a kernel deadlock.

Version-Release number of selected component (if applicable):

2.6.9-34.0.1.ELsmp

How reproducible:

system hang occurred regularly at a 1-2 weeks interval

Steps to Reproduce:
1. install RH Cluster suite with 2 LVS nodes and 3 real servers
2. set it up to run the supplied scripts to monitor the named services on all
three real servers
3. wait for a week or two
  
Actual results:

system hung

Expected results:

system goes on working normally

Additional info:

a possible workaround is to use monitoring scripts that avoid reading /proc/meminfo