Bug 703704

Here is some more information about this bug.  The root problem is the message "Could not allocate physical memory".

Running qemu-kvm directly (under strace) shows that qemu-kvm calls mmap() fails. Writing a simple C program that calls mmap() with the same args produces the same failure.

Here is the C program.

[mclosson@delint07 ~]$ cat mem.c
#include <stdlib.h>
#include <sys/mman.h>
#include <string.h>
#include <assert.h>

int
main()
{
    size_t s = 2168475648;
    char * ptr;
    assert(sizeof(size_t) == 8);
    ptr = mmap(NULL, s, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
    memset(ptr, 0xff, s);
    munmap(ptr, s);

    return 0;
}

Running the program on a machine that doesn't exhibit the bug shows the following:

[mclosson@delint07 ~]$ gcc mem.c
[mclosson@delint07 ~]$ strace -tt ./a.out
10:37:36.760124 execve("./a.out", ["./a.out"], [/* 51 vars */]) = 0
10:37:36.761018 brk(0)                  = 0xffd6000
10:37:36.761075 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aead8ca5000
10:37:36.761136 uname({sys="Linux", node="delint07", ...}) = 0
10:37:36.761234 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) 10:37:36.761300 open("/etc/ld.so.cache", O_RDONLY) = 3
10:37:36.761355 fstat(3, {st_mode=S_IFREG|0644, st_size=106135, ...}) = 0
10:37:36.761426 mmap(NULL, 106135, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2aead8ca6000
10:37:36.761466 close(3)                = 0
10:37:36.761516 open("/lib64/libc.so.6", O_RDONLY) = 3
10:37:36.761562 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\332\241\0033\0\0\0"..., 832) = 832
10:37:36.761614 fstat(3, {st_mode=S_IFREG|0755, st_size=1717800, ...}) = 0
10:37:36.761678 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aead8cc0000
10:37:36.761729 mmap(0x3303a00000, 3498328, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3303a00000
10:37:36.761771 mprotect(0x3303b4e000, 2093056, PROT_NONE) = 0
10:37:36.761814 mmap(0x3303d4d000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14d000) = 0x3303d4d000
10:37:36.761868 mmap(0x3303d52000, 16728, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x3303d52000
10:37:36.761913 close(3)                = 0
10:37:36.761963 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aead8cc1000
10:37:36.762008 arch_prctl(ARCH_SET_FS, 0x2aead8cc1210) = 0
10:37:36.762108 mprotect(0x3303d4d000, 16384, PROT_READ) = 0
10:37:36.762155 mprotect(0x330381b000, 4096, PROT_READ) = 0
10:37:36.762195 munmap(0x2aead8ca6000, 106135) = 0
10:37:36.762269 mmap(NULL, 2168475648, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aead8cc2000
10:37:38.680874 munmap(0x2aead8cc2000, 2168475648) = 0
10:37:38.840559 exit_group(0)           = ?


Running the same program on a machine that does reveal the bug shows the following:

[root@lxbst0501 ~]# strace -tt ./mem
10:35:32.122432 execve("./mem", ["./mem"], [/* 22 vars */]) = 0
10:35:32.125134 brk(0)                  = 0x17b53000
10:35:32.125294 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b53ab36d000
10:35:32.125386 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b53ab36e000
10:35:32.125455 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
10:35:32.125542 open("/etc/ld.so.cache", O_RDONLY) = 3
10:35:32.125607 fstat(3, {st_mode=S_IFREG|0644, st_size=29122, ...}) = 0
10:35:32.125707 mmap(NULL, 29122, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2b53ab36f000
10:35:32.125770 close(3)                = 0
10:35:32.125837 open("/lib64/libc.so.6", O_RDONLY) = 3
10:35:32.125901 read(3,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\332\301\0174\0\0\0"...,
832) = 832
10:35:32.125971 fstat(3, {st_mode=S_IFREG|0755, st_size=1722304, ...}) = 0
10:35:32.126131 mmap(0x340fc00000, 3502424, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x340fc00000
10:35:32.126197 mprotect(0x340fd4e000, 2097152, PROT_NONE) = 0 10:35:32.126260 mmap(0x340ff4e000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14e000) = 0x340ff4e000
10:35:32.126328 mmap(0x340ff53000, 16728, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x340ff53000
10:35:32.126386 close(3)                = 0
10:35:32.126454 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b53ab377000
10:35:32.126517 arch_prctl(ARCH_SET_FS, 0x2b53ab3776e0) = 0
10:35:32.126644 mprotect(0x340ff4e000, 16384, PROT_READ) = 0
10:35:32.126709 mprotect(0x340fa1b000, 4096, PROT_READ) = 0
10:35:32.126763 munmap(0x2b53ab36f000, 29122) = 0
10:35:32.126848 mmap(NULL, 2168475648, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
10:35:32.126906 --- SIGSEGV (Segmentation fault) @ 0 (0) ---
10:35:32.127056 +++ killed by SIGSEGV +++



It is unlikely that the machine is really out of memory:

[root@lxbst0501 ~]# cat /proc/meminfo
MemTotal:     24676304 kB
MemFree:      15206136 kB
Buffers:        329892 kB
Cached:        8340008 kB
SwapCached:          0 kB
Active:        5178052 kB
Inactive:      3828624 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     24676304 kB
LowFree:      15206136 kB
SwapTotal:     4192956 kB
SwapFree:      4192956 kB
Dirty:             708 kB
Writeback:           0 kB
AnonPages:      336768 kB
Mapped:          24760 kB
Slab:           374172 kB
PageTables:       7644 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  16531108 kB
Committed_AS:   737512 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    302652 kB
VmallocChunk: 34359399243 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Closing, as the bug is reproduced using a simple host program
https://bugzilla.redhat.com/show_bug.cgi?id=703704#c2
And it looks like a limit on the virtual memory
https://bugzilla.redhat.com/show_bug.cgi?id=703704#c4

Ronen, can you explain why there is a limit on virtual memory?  From comment #4 there is 15GB of free memory and 4GB of swap.  But yet 2GB of mem cannot be mmap'd.

In comment 4 it says
[virtual memory (kbytes, -v) 1536000]
This line is one of the outputs of "ulimit -a".
My guess is that the local sys-admin, puts a limit on the resources of the users. Somebody did.

Thanks!