Created attachment 93335 [details]
tar file includes meinfo, slabinfo and alt sysre output during freeze (files are bzipped)

What is the 'oast' test tool ?

I'd like to try reproducing the bug here, so we can see exactly what's going on.

Martin Jenner has this tool. "Oracle automated stress 
tool". But you may not able to reproduce this bug as 
you may not have database which I have in size.

Hete are steps to reproduce this bug.

1) up kernel
2) 10i databse ( I do not know martin has that or 
not? I can provide tarball to martin but I do not 
know how?)
3) 4 GB of memory
4) create 200 ware house database using oast tool ( 
I am using 5 data disks of 18 GB)
5) use directIO option in init.ora file
6) run 1000 user test with 1.8 GB of sga

May be it is difficult to reproduce this test at Red 
Hat. I will help my best so that you can reproduce it 
at Red Hat.

Please let me know if you need any other information. 
I can reproduce it very easily at Oracle.

I have verified this bug in 411 kernel. I have colloected meminfo, slabinfo top 
at every minutes. You can see in meminfo there is no collection during 20-30 
minutes of interval. Also profile during that time is colloected at every 
minute. Also this test is done on up and ODIRECT is enabled.

Created attachment 93970 [details]
meminfo, slabinfo, top and profile collected during test.

I have verified this bug on 2.EL kernel as per Larry's request. I have 
colloected meminfo, slabinfo at every minutes and top at every 10 
minutes. You can see in meminfo there is no collection during 20-30 
minutes of interval. Also this test is done on up and ODIRECT is enabled. 
 I was running tpcc (oast) test on 10g database with 1000 user. Test 
should be done in about 45 to 50 minutes, but took 1 hour and 40 
minutes. Log for meminfo, slabinfo, top is attached. Please let me know 
if you need anything

Created attachment 94613 [details]
meminfo, slabinfo, top collected during test.

The attached didnt show us what was going on unfortunately.  Can you
get us AltSysrq-M, AltSysrq-T and AltSysrq-W outputs when the system
is in the frozen state.

Thanks, Larry Woodman

Requested out for alt-sysrq,m,p,t and w attached.

Created attachment 94646 [details]
alt sysrq output - bzipped

OK, we are making progress here(albeit slowly!).  Please get the system into
this state and get several(hundreds) "AltSysrq P" outputs so we can see if
one process is stuck in update_queue()/try_atomic_semop() or there is a bunch
of context switching between several processes in this state.

Larry

alt-sysrq-p (hundreds) output attached. During hang 
two or three different time output was taken.

Created attachment 94657 [details]
alt sysrq p output - bzipped

I can reproduce this bug in smp and 
hugemem kernel both. I will update bug 
with alt-sysrq-p, m and w as soon as I 
can.

Please use this smp kernel to collect the AltSysrq data.

http://people.redhat.com/~lwoodman/.for_oracle/kernel-smp-2.4.21-3.EL.debug.i686.rpm


Thanks, Larry

With 3.EL kernel I reproduced bug using 
hugemem kernel. System hangs. BUt it did 
not came out like it used to. When I tried 
to take alt-sysrq-m it hung. SO I have to 
reboot machine.

I will try debug kernel given by you. I 
will also attach output of 
meminfo,slabinfo and top. Also output of 
alt-sysrq-m (after this machine hangs and 
I have to switch off machine)

Created attachment 94835 [details]
memory and top output when hugemem kernel hangs

Created attachment 94836 [details]
alt-sysrq-m output when hugemem hangs

Please retry this rest with the kernel in:
http://people.redhat.com:/~lwoodman/.for_oracle

We think we hit the same problem in an other case and this
kernel fixed that hang.  Please let us know how it worked ASAP.

Larry Woodman

I tried with Kernel 
2.4.21-3.EL.sock.kswapd.io.debug.20smp. I still got 
similar behavior in test. Machine got freeze (paused) 
for long time while running same tests. I could not 
take alt-sysrq output while machine was paused. 

Suhua, please try this new kernel.  It has more debug output and
a panic if some bad things happen.

http://people.redhat.com/~lwoodman/.for_oracle/

Larry

Attached is a low-budget simulation of the problem
seen in this case.  Hundreds of processes are doing
sys_semtimedop() calls with a timeout of 10 
milliseconds (HZ resolution).  There is therefore
a scheduling storm from schedule_timeout() with each 
clock tick.

The attached program replaces the sys_semtimedop()
calls with empty select() calls with a timeout of 
10000 microseconds.  When run with an argument of 
say, 2000 processes, the same system "freeze" occurs.

When run on AS2.1, no such problem occurs with 
either the same Oracle test or running 2000 
processes from this "select.c" program.

Dave Anderson (entered while at Oracle)

Created attachment 95624 [details]
select.c file

A correction to my posting re: the select.c program, where I stated
that "When run on AS2.1, no such problem occurs".

If I'm not mistaken, the quick test of 2000 processes on an AS2.1
machine we did at Oracle was most likely done on an SMP machine.
When I run it here on a UP AS2.1 box, the same problem occurs as with
RHEL3 UP.  If I run 2000 users on a UP using a 2.4.23-based kernel.org
kernel (which doesn't have the O(1) scheduler used in AS2.1 and RHEL3)
the performance is even worse.  My guess is that when we ran the 2000
user test on that "other" OS while at Oracle, it was also an SMP
machine.

In any case, unless proven otherwise, it does not appear to be
a regression from AS2.1.

I verified that this select program given by you only hangs on up 
kernel. Both on AS 2.1 and RHEL 3. 

Program do not hang on smp kernel on AS 2.1 or RHEL 3.

But my original test hangs/freeze machine for some time (10 minutes 
to 3 hours ) on up, smp and hugemem. On hugemem and smp is little 
difficult to reproduce.

Right -- if you increase the number of users to the select program
(from 2000 upward) it will eventually hang on an SMP as well.

It's due to the the in-kernel schedule_timeout() that gets done
by both your oracle stress test and the select.c program, which in
both programs is a timeout of 1 clock tick.  In both cases, the
individual tasks timeout, return to user space, and then immediately
go back into the kernel for another 1-clock-tick timeout.  The oracle
tasks do it via the new sys_semtimedop() system call, while the
"select" programs do it via the select() system call.  The oracle
tasks do quite a bit more work in the kernel, and perhaps in user
space upon return from the sys_semtimedop with an EAGAIN errno,
before coming right back into the kernel and calling sys_semtimedop()
again.  Eventually there are hundreds of these backed-up, constantly
timing-out, oracle tasks on the runqueue, each timing out for a clock
tick, getting scheduled to run, running and returning to user space,
calling sys_semtimedop again, etc.  This floods the runqueues with
ready-to-run processes that make no progress, but end up flooding the
highest priority runqueue on each CPU, keeping other processes from
running for a long time.  Eventually the job gets done, but the manner
in which the oracle stress tests use the new sys_semtimedop() (new in
RHEL3) ends up impeding their own progress.

Not really.  But unless #43793 shows several hundred
processes blocked in sys_semtimedop(), it's doubtful
that it's the same issue.  

Actually, make that "several thousand" on an SMP machine...

Suhua, does this bugzilla need to remain private?  If not, please uncheck
the "Oracle Confidential Group" box below.  Thanks.

Anyone know whether this issue was resolved?  And if so, by what patch?

Making this bug private as a whole, but marking initial comment private
(in lieu of Suhua's response to comment #30).

Is this problem still reproducable with the latest kernel???

Larry Woodman

I don't have system to verify the latest kernel.

Larry, this bug may be closed.

Closing based on last comment.