Found root cause. This issue is caused by ia32 program. As often as 
ia32 binary is executed, the vm_committed_space is getting rower. 
And then finally, vm_committed_space become negative value.

Could you please look into this issue? 
You can easily reproduce by following steps.


Steps to Reproduce:

1. Create ia32 binary by gcc with -m32 on RHEL3 x86_64 system.

# vi test.c
int main(int argc, char **argv)
{
  /* Do nothing */
}

# gcc -o ia32test -m32 ia32test.c

2. Run the program so many times.

# i=0; while [ "$i" -lt 100000 ]; do ./ia32test; i=`expr $i + 1`; done

3. Make sure Committed_AS shows negative value.

# cat /proc/meminfo 
        total:    used:    free:  shared: buffers:  cached:
Mem:  16034414592 237867008 15796547584        0 14868480 66973696
Swap: 17182601216        0 17182601216
MemTotal:     15658608 kB
MemFree:      15426316 kB
MemShared:           0 kB
Buffers:         14520 kB
Cached:          65404 kB
SwapCached:          0 kB
Active:         112508 kB
ActiveAnon:      53440 kB
ActiveCache:     59068 kB
Inact_dirty:     15192 kB
Inact_laundry:    5472 kB
Inact_clean:         0 kB
Inact_target:    26632 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     15658608 kB
LowFree:      15426316 kB
SwapTotal:    16779884 kB
SwapFree:     16779884 kB
CommitLimit:  24609188 kB
Committed_AS: 18446744073709534888 kB  <---------
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB


Evaluation:

The problem is ia32_setup_arg_pages() in arch/x86_64/ia32/ia32_binfmt.c.
After allocating vm_area_struct for stack segment, vm_enough_memory()
should be called in order to reduce vm_committed_space. But the following
code does not call it. I think this is the root cause.

int ia32_setup_arg_pages(struct linux_binprm *bprm, int executable_stack)
{
        unsigned long stack_base;
        struct vm_area_struct *mpnt;
        int i, ret;

        stack_base = IA32_STACK_TOP - MAX_ARG_PAGES*PAGE_SIZE;

        bprm->p += stack_base;
        if (bprm->loader)
                bprm->loader += stack_base;
        bprm->exec += stack_base;

        mpnt = kmem_cache_alloc(vm_area_cachep, SLAB_KERNEL);
        if (!mpnt)
                return -ENOMEM;
       
        down_write(&current->mm->mmap_sem);
        {
                mpnt->vm_mm = current->mm;
                mpnt->vm_start = PAGE_MASK & (unsigned long) bprm->p;
                mpnt->vm_end = IA32_STACK_TOP;
                if (executable_stack)
                        mpnt->vm_flags = vm_stack_flags32;
                else
                        mpnt->vm_flags = vm_stack_flags32 & ~VM_EXEC;
                mpnt->vm_page_prot = (mpnt->vm_flags & VM_EXEC) ? 
                        PAGE_COPY_EXEC : PAGE_COPY;
                mpnt->vm_ops = NULL;
                mpnt->vm_pgoff = 0;
                mpnt->vm_file = NULL;
                mpnt->vm_private_data = (void *) 0;
                if ((ret = insert_vm_struct(current->mm, mpnt))) {
                        up_write(&current->mm->mmap_sem);
                        kmem_cache_free(vm_area_cachep, mpnt);
                        return ret;
                }
                current->mm->total_vm = (mpnt->vm_end - mpnt->vm_start) >>
PAGE_SHIFT;
        }


Regards,
Ryutaro

>> should be called in order to reduce vm_committed_space. But the following
>> code does not call it. I think this is the root cause.

Sorry, "in order to add the size of vm_area_struct to vm_committed_space" is
correct.

This bug is filed against RHEL 3, which is in maintenance phase.
During the maintenance phase, only security errata and select mission
critical bug fixes will be released for enterprise products. Since
this bug does not meet that criteria, it is now being closed.
 
For more information of the RHEL errata support policy, please visit:
http://www.redhat.com/security/updates/errata/
 
If you feel this bug is indeed mission critical, please contact your
support representative. You may be asked to provide detailed
information on how this bug is affecting you.