Created attachment 397246 [details]
Test program that mallocs a lot of memory

Description of problem:
Several times recently, I've hit bugs where an application enters an infinite loop allocating memory and brings the system to a halt.  I wanted to mitigate the problem by setting an RLIMIT_DATA (ulimit -d) in my login scripts.  But RLIMIT_DATA has no effect because malloc falls back to mmap when brk fails:

http://sourceware.org/git/?p=glibc.git;a=blob;f=malloc/malloc.c;h=763852ea3b78cba613b63d9e92e00ecdd6fcaab6;hb=HEAD#l3206

Supposedly the fallback is needed on systems with address space holes.  If I'm right in thinking that Linux is not such a system, we could disable the fallback.

(Note that RLIMIT_AS does not solve my problem because OOo needs to map 2.5 GB, almost all of which is read-only, but if I set the limit that high, a process that allocated that much writable memory would bog down the system.)

Version-Release number of selected component (if applicable):
glibc-2.11.1-1.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Download and compile the attached memory-hog.c .
2. (ulimit -S -d 500000 && ./memory-hog)
  
Actual results:
"Successfully malloc-ed 1 GB"

Expected results:
"malloc failed (i = SOMETHING), exiting"

Additional info:
malloc also uses mmap for large allocations, so they don't count toward RLIMIT_DATA.  This has been noted:

https://bugs.launchpad.net/ubuntu/+source/pam/+bug/71024/comments/10
http://www.csamuel.org/2007/11/30/linux-ulimit-memory-enforcement-oddity

I'm guessing this won't affect me because infinite loops tend to involve small allocations.  But if it does, I can install an LD_PRELOAD library that calls mallopt(M_MMAP_MAX, 0) to disable the feature.