Created attachment 90767 [details]
oom_patch1.patch

Created attachment 90768 [details]
oom_patch2.patch

Created attachment 90769 [details]
oom_patch3.patch

Did more analysis into this one. At the point of hang, oom_kill has infact 
sent a kill signal to one of the processes, but the corresponding process 
never gets to handle the signal and do an exit. As that process is waiting on 
memory, and never gets woken up as we have a memory shortage condition.

Following are the issues that when fixed will resolve this hang:
1) wakeup_memwaiters() in kswapd should wake up all processes waiting on 
memory everytime. Not only at (!VM_SHOULD_SLEEP) condition.
This will prevent the situation where oom_kill sends a kill signal to a 
processs, and that process never gets to handle the signal as it is waiting on 
memory. 
2.4.18+ kernels use this sort of mechanism to wakeup all processes waiting, 
ireespective of available memory, in kswapd.
Attached oom_patch1.patch does this.

2) Second issue is what should happen if the process that got the first kill 
signal is sleeping. The current code keeps on sending the kill signal to the 
same process, thus ending up not doing any forward progress.
A better approach, as suggested by Rik in lkml, is to mark the processes as 
oom_kill is sent to them, and try killing some other process when we reach the 
low memory condition next time. Attached patch oom_patch2.patch originally 
posted by Rik, rebased to AS 2.1.

3) Looks like a real bug in page_alloc.c. In _wrapped_alloc_pages(), in a 
check to retry alloc_pages, there is a condition where we should check 
(!free_shortage()), in place of (free_shortage()). Only when there is no 
shortage, we need to keep on retrying. If there is a shortage and we are 
looking for order > 0, then we should return failure rather than waiting 
forever.
At the hang we also noticed that if the process getting the kill signal 
happens to be in this state, wherein it keeps on retrying indefinitely to get 
memory, doing non-interruptible sleeps inbetween. This again can result in 
hang as no process gets actually killed, even after oom_kill sends a kill 
signal.
Another change that can help low memory condition is to retry indefinitely 
only on zero order pages. And not for page order <= 3.
Attached oom_patch3.patch does the above changes.

Per Larry Woodman:

Problem 1)above:  fixed and in latest kernel build for AS 2.1 x86 (sent to Intel)
Problem 2) above:  under review, but patch would require a great deal of testing.
Problem 3) above:  fixed and in latest kernel build for AS 2.1 x86 (sent to Intel)

Please test latest AS 2.1 x86 kernel build (pointer sent to
Paul.Gutierrez@intel.com in email)and report back.

Adding in more detail, also based on input from Larry

Overall, we do not consider this problem to be of highest criticality.  Reason
being that it refers to an edge condition whereby all of memory and swap have
been consumed.  We can not compromise the integrity of the normal operating
condition for an edge condition.

Problem 1)above:  fixed and in latest kernel build for AS 2.1 x86 (sent to Intel)
Problem 2) above:  we do not agree with the proposed patch.  
Problem 3) above:  Improved in the prior Q1 errata. If we blindly accepted this
patch, the end result would be substantially more process killing than
necessary.  Definitely too heavy handed, as our primary goal under these
circumstances is to keep the system alive.

We feel that the the majority of the problems highlighted here have been
addressed under the best tradeoff policies.  There may still be cases in which
processes don't get killed during complete depletion, but that balance is deemed
appropriate.

This was a part of the changes in patch 3 above. Somehow I still believe that 
this is a bug in the code. mm/page_alloc.c has a condition
if (!order || free_shortage()) {
and I feel it should be
if (!order || !free_shortage()) {

As per the existing check, we do a retry on the page request when we are low 
on memory, and we _do_not_ do a retry on a page request (with order > 1) when 
we have no shortage of free memory. 

Am I missing something here?
 

Status update with kernel-2.4.9-e.16.3:
We ran the tests on two systems that used to display this problem before.
The issue still persists with the new kernel..
Dell 870 based platform:
            3/28/03 16:35 - gpgstress test started
            3/31/03 03:30 - system hung
            swap space left: 2007840K
        
Another 4-way system:
            3/28/03 22:23 - gpgstress test started        
            3/28/03 23:13 - system hung
            swap space left: 4K

I will try to do some analysis on these hanging system and will update the 
details later.

While we tend to agree that this test case is an edge condition wherein all of 
memory and swap have been consumed. 
But, we dont feel comfortable with the system going into hang state. I mean, 
even going into a panic() state, with this kind of workload is a much better 
way. With that atleast system can gracefully report the error and restart. If 
the system goes into the hang state, there is very little option for the 
system administrator, in terms of identifying the failure, especially with a 
remote system.
And another point of concern is, this can be reproduced, on some systems, 
within couple of hours of gpgstress test run.

Another fact observed from our tests. RH 8.0 survives this test on both 
platforms above, for more than 72 hours.

First of all, I dont think that the "if (!order || free_shortage())"
is wrong.  By the time you get to this test, previously in the __alloc_pages()
routine, you have woken up kswapd and yield()'d and you have drained any 
pages on the inactive clean list onto the free list without satisfying the 
order>0 allocation.  Looping back up and around again wont do anything for 
order>0 allocation unless there is a free shortage. nt forget that order>0
allocations must come from the buddy allocator free lists, the inactive free
pages cant be used for order>0 allocations. 

I already applied the equivalent of your patch 1 in the 2.4.9-e.14.1 kernel
although I dont instantly wakeup every process waiting inside wakeup_kswapd,
I dont let any process sleep any longer than 30 itterations of kswapd.  This
is less aggressive but provides the same finctionality.

I have include the equilivent of patch 3 in the latest kernel with the 
exception of the free_shortage() logic described above.  The current logic
is:

    if(order ==0) goto try_again
    if(order <= 3 && process is not being oom killed) goto try_again
    if(order > 3 && havent tried 3 times yet) goto try_agian

I have not included patch 2 because of the potential undesirable side
effects.  With this patch, it is possible to oom kill many processes
that are not currently in a killable state.  Once one process is killable
and it gets killed, the system will kill all other processes that were
previously marked to be killed.  This is very undesirable especially for
an edge condition that you really have to forcefully try yo get the system
into.  

Larry Woodman

Thanks for the detailed explanation.

The kernel that were used for over the weekend tests (e-16.3), does have a 
variant of patch 1. But, I didn't see the "order <= 3 && process is not being 
oom killed" check that you have mentioned above. Can we get the latest kernel, 
so that we can rerun the tests and see the progress.

As you say, patch 3 is kind of really aggressive. But, good thing about that 
is it wont hang with this workload, as eventually it would empty all killable 
processes and call panic(). But as you have mentioned, that may not help much 
in a normal workload.

Another thing that we found during our analysis here is:
The problem does not necessarily come from one single process that is waiting 
forever in alloc_pages. This process that is waiting forever, unfortunately is 
also holding some file system lock (inode, page) and there are a bunch of 
other process doing uninterruptible sleep on these locks. This way none of the 
process in this bunch can ever get killed too.

Thanks again.

Did the test with the latest update kernel. 

And we successfully completed 72 hours test run on Dell 870 based platform. No 
failures were seen on other platforms too.

We can go ahead and close this bug now.
Thanks for all the support.

PER ABOVE COMMENT BY  Venkatesh Pallipadi of Intel on 2003-04-04 17:52, CLOSING
BUGZILLA AS FIXED.