We have seen this VM accounting bug on kernel 2.6.9-42.0.2.ELsmp for x86_64 (but
it does not seem to be fixed in the latest revision either).  In particular,
vm.overcommit was set to 2, and the machines (which run a lot of short i386
binaries at high frequencies) refused to fork any more processes -- malloc() was
failing.  After killing all user processes, Committed_AS was still well over 8
GB, clearly an accounting leak.

Please make this a relatively high priority, as overcommit features on x86_64 is
essentially broken without it (we have set it to 0 for now).

Also reported as affecting RHEL3

Note that this problem (i.e., negative "vm_committed_space" value) is only
cosmetic (bogus /proc/meminfo "Committed_AS" value) when the system is booted
with the default setting for /proc/sys/vm/overcommit_memory.  It is only when
the "overcommit_memory" sysctl value has been set to 2 that the kernel will
alter its behavior based on the "vm_committed_space" value.

If a customer is experiencing ENOMEM errors in user-space programs on a system
with "overcommit_memory" set to 2, a work-around would be to use the default
setting instead.

In the case of RHEL4 (unlike RHEL3), running a 32-bit executable on x86_64
results in incrementing the "Committed_AS" value by the initial stack size.

This is because a call to security_vm_enough_memory() was made in the version
of setup_arg_pages() in  arch/x86_64/ia32/ia32_binfmt.c, but VM_ACCOUNT is not
set in __VM_STACK_FLAGS, and thus unmap_vmas() doesn't count the pages in its
*nr_accounted arg, which leads to exit_mmap() not counting the pages in its
call to vm_unacct_memory().

Since this problem doesn't occur on x86, I'm changing the arch to x86_64 (but
it might occur on other 64-bit arches, too ... not yet verified).

That isn't a workaround. The only case you set overcommit_memory to 2 is because
you can't tolerate out of memory kills. So turning it off isn't an answer. 
Doesn't look hard to fix fortunately, although the changes upstream are more
extensive than neccessary and fix stuff like mips that we don't care about.

Digging more if we want a quick hack solution (eg someone wants a hotfix) then
we can just knock the enough memory check out of setup_arg_pages(). This will
make the checks a little out but we have a fair bit of slack in the default 50%
memory + swap assumption so only a deliberately hostile environment would
mistrigger out of memory situations.  Not a good fix for the final release but
probably adequate if someone needs a hotfix.

Note that RHEL3 bug 224600 is a different problem.

I'll develop and test a fix for the RHEL4 bug tomorrow (probably
just adding VM_ACCOUNT to __VM_STACK_FLAGS on x86_64), which I'd
guess will be a minimal enough risk for a RHEL4 Update (and even
for a potential U5 respin, if it's deemed necessary).

I'm changing the subject line to reflect what the real problem is in RHEL4,
which exists solely on the x86_64 arch (in 32-bit exec support).

Note that the upstream changes indicated in the "GIT commit" link in the
initial comment of this bug report do not address the RHEL4 problem, which
is causing the kernel's "vm_committed_space" global variable to increase
inappropriately (eventually leading to user-space memory allocation failures
if a system were booted in the non-default overcommit_memory mode 2).

Note that this problem does not exist in RHEL3.  (But RHEL3 has a different
problem in maintaining the "vm_committed_space" global variable.)

Patch posted for internal review on 1-Feb-2007.

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

*** Bug 229587 has been marked as a duplicate of this bug. ***

This request was evaluated by Red Hat Kernel Team for inclusion in a Red
Hat Enterprise Linux maintenance release, and has moved to bugzilla 
status POST.

committed in stream U6 build 55.2. A test kernel with this patch is available
from http://people.redhat.com/~jbaron/rhel4/

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2007-0791.html