Description of problem:

Under some circumstances the current RHEL5 implementation of the OOM
killer does not work out very well, e.g. because it needs to scan the
whole task list before it can make its decision to kill a process, based
on the defined heuristics.

In upstream kernel 2.6.24, some additional functionality was
implemented, including

/proc/sys/vm/oom_kill_allocating_task

See also http://lwn.net/Articles/254559/

This enables or disables killing the OOM-triggering task in
out-of-memory situations.

If this is set to zero, the OOM killer will scan through the entire
tasklist and select a task based on heuristics to kill.  This normally
selects a rogue memory-hogging task that frees up a large amount of
memory when killed.

If this is set to non-zero, the OOM killer simply kills the task that
triggered the out-of-memory condition.  This avoids the expensive
tasklist scan.

If panic_on_oom is selected, it takes precedence over whatever value
is used in oom_kill_allocating_task.

The default value is 0.

The following test program might be helpful:

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

#define N       100000

int main (void) {
        int i, n = 0;
        char *pp[N];

        for (n = 0; n < N; n++) {
                pp[n] = malloc(1<<20);
                if (pp[n] == NULL)
                        break;
        }

        printf("malloc failure after %d MiB\n", n);

        for (i = 0; i < n; i++) {
                memset (pp[i], 0, (1<<20));
                printf("got %d MiB\n", i+1);
        }

        return 0;
}