While investigating a locking order bug that resulted in a hang (bz 439548) it was discovered that when the locking bug was fixed and the testcase was able to continue that it would cause an OOM kill. This problem is reproduceable upstream. The testcase is simple, untar it, edit the VAR file to point at a blank partition/volume that it will use, and then do sh run.sh and it will start, at some point you will see things getting killed. I'm also attaching a txt file with a sample of the OOM messages I got (on an upstream kernel).
Created attachment 313105 [details] reproducer
Created attachment 313106 [details] oom messages
I did look at it. I fixed it for RHEL 6 with the split LRU code. I'm not entirely convinced it is fixable for RHEL 5, though if I come up with an idea one day I'd like to :)
Too old for RHEL5.8. Closing.