Hi To reproduce the bug, create a XEN virtual machine with 512MB RAM and try to install RHEL5.5-beta into it (I tried also 5.3 and the bug is also present there). Don't create swap during installation. During installation, the kernel kills Anaconda with out of memory condition, although Anaconda uses only 176MB memory, total memory used by all processes is 277MB and there is 177MB of cached pages. The kernel shouldn't kill the process if there are so many cached pages, it should try free the cache instead. Creating the swap or allocating 1GB memory for the virtual machine avoids the problem --- but the memory traces suggest that there is something broken in the RHEL kernel OOM killer and it could potentially kill tasks spuriously not only during installation, but also during normal operation. I'm attaching several screenshots taken when the OOM kill happened to show that the kernel kills processes while there is plenty of cached memory.
Created attachment 394641 [details] oom killer report oom killer report, notice the "43589 pagecache pages" line.
Created attachment 394642 [details] top output during the crash "top" command run during the crash (the kernel dump for this crash is in the previous screenshot)
Created attachment 394643 [details] another out-of-memory crash Another installation try.
Created attachment 394645 [details] /proc/meminfo output Output of /proc/meminfo during the out-of-memory crash from the previous screenshot. I ran the command: while true; do echo `cat /proc/meminfo`; sleep 1; done on the available console to capture memory state in 1-second intervals. Notice the "Cached: 177892kB" entry, it corresponds with "43589 pagecache pages" in the previous screenshot. The kernel definitely must not kill processes when there is so much cached data.
Created attachment 394646 [details] RHEL 5.3 out-of-memory crash The bug exists even in RHEL 5.3. There are 37705 cached pages and the OOM killer triggers.
I've opened Fedora bug 553193 for the locale issue, but without any progress so far... Also it would be probably nice to see memory layout of processes just before oom starts to make some action? Maybe some 'while : ; do ps aux >> /tmp/log ; sleep 1; done' during installation could be added to ananconda ? Or passing 'sysrq memory' dump at the right moment if the time could be determined.
Actually, after further analysis, I realized that this is caused by the root tmpfs filesystem --- it has pages in pagecache and they are not discardable. So, it is not kernel misbehaviour. Therefore, I am closing this as NOTABUG.