Created attachment 862057 [details] Test program to reproduce the problem. Description of problem: Many of our custom applications that need a lot of (virtual) memory crashes with kernel 3.12.* with out of memory messages when there is enought memory available on that host. This is true for applications in perl, python, java, C, C++, etc. But the same programs run witout out-of-memory errors with kernel 3.11.* . The problem occures with Fedora 19 and Fedora 20 if they use the same kernel version. I have created a small test program to reproduce the problem of our complex applications of our users. I append this as an attachment. I have compiled it with "g++ -g" and called the program with gdb to get some values after the crash. The maximum memory before crash with "std::bad_alloc" depends on the size of the created element (array). VmPeak is the value listed by the output of the gdb command "info proc status". count is the last value printed before the crash, e.g. the number of loop iterations. new int [100]: VmPeak: 8658212 kB (8 GB) count=21281810 new int [1000]: VmPeak: 8658212 kB (8 GB) count=2204489 new int [10000]: VmPeak: 10250644 kB (9 GB) count=261991 new int [100000]: VmPeak: 25688080 kB (24 GB) count=65497 new int [1000000]: VmPeak: 255982564 kB (244 GB) count=65497 new int [10000000]: VmPeak: 2558665408 kB (2440 GB) count=65497 new int [100000000]: VmPeak: 25585365220 kB (24400 GB) count=65497 This shows thats it is not a fixed limit of virtual memory and the out of memory error does not correspond to the real memory or real+swap memory. The host I have used for testing has much free memory at the time of the tests (real memory: 256 GB). I don't know what the reason of this out of memory problem is. Version-Release number of selected component (if applicable): These are the kernel version that I used for the test and the test program has crashed: kernel-3.12.10-200.fc19.x86_64 kernel-3.12.9-201.fc19.x86_64 kernel-3.12.5-200.fc19.x86_64 kernel-3.12.9-301.fc20.x86_64 These are the kernel version that I used for the test and the test program has NOT crashed: kernel-3.11.9-200.fc19.x86_64 kernel-3.11.6-200.fc19.x86_64 How reproducible: Always. Steps to Reproduce: 1. Boot a system with one of the kernels listed above. 2. Compile the testcase program. 3. Run the testcase program. You may use gdb to examine the memory used when it has crashed. Actual results: Kernel 3.12.*: Program crashes with memory allocation error while memory is available (see attachment). Kernel 3.11.*: No crash while memory is available. Expected results: No crash while memory is available.
Created attachment 862058 [details] Output of gdb when running the test program I have compiled the test program with g++ -g testcase-out-of-memory-kernel-3.12.cpp -o testcase-out-of-memory-kernel-3.12 I have called gbd with gbd testcase-out-of-memory-kernel-3.12
Created attachment 862059 [details] Output of "cat /proc/meminfo"
I may be seeing this bug also. It also might be related to bug #1050990. So far, in 3.12.9-201 I get oom-killer activating once or twice a day. In 3.11.10, exact same userland, I have not ever seen oom-killer activate. I am currently switching back and forth for testing every few days. When the oom-killer hits my GUI becomes extremely slow (like a click takes 1 minute) and only by switching to a console can I do anything. The console remains quite responsive. In all cases I was unable to fix things up by killing things, etc, and the oom's came more frequently (several a minute) until my only recourse was to issue a reboot. The reboots proceeded normally. The bug has hit 4 times in 4 days, so I've been able to run a few tests, and I plan more. Running top during the problem shows tons of free (when you include used buffer/cache space) RAM (8GB total) and 99% unused swap (8GB total). In many cases I've barely started much of anything compared to my normal workload when it happens. I noticed that %wa was quite high at 60%+. So I ran iotop, and this last time it showed a weird thing: xfce4-panel and sawfish were using 99% IO though their read/write was negligible! Strange. I'm not sure if the other instances were high on sawfish/xfce but I will check next time. I have the oom-killer logs from the times these hit, if they are of any use to anyone. I also noticed that twice when the bug hit some high-IO process was running in the bg. Once it was a find of the whole fs, and once I was doing another find-type crawl of a large fs. I don't know if the other 2 hits also were doing high IO. I will watch for next time. I tried to get an strace on sawfish/xfce and it attached but the strace froze and no output was ever generated. Others seeing this bug, please run iotop (make sure it's installed beforehand) and see if anything is chewing up high IO and note if R/W of the same ps is strangely low. Check top to see what your wa% looks like. See who the top CPU users are. If anyone has any other ideas of what to check let me know and I can get it for you pretty quickly.
*Possibly* related, though might be a stretch: https://lkml.org/lkml/2013/11/21/158 There are symptoms he mentions in the long thread that are very similar to this bug, though his perhaps is Xen specific. He says they occur in 3.11 and 3.12, though and we here seem pretty sure they only occur in 3.12. I'm having a real hard time finding anything else of relevance, especially from the kernel guys. You'd think someone else would be seeing something like this somewhere. Usually at least one Ubuntu hit shows up. Weird.
(In reply to Trevor Cordes from comment #4) Thanks for your report. I have seen bug #1050990 too, and thought it may be related, or may be not. I found no oom-killer messages in /var/log/messages or journalctl (with "grep -i oom") on these servers where our processes got the out of memory messages. We got oom-killer messages on Fedora 16 (for example), I haven't found them on Fedora 19. Maybe that they are suppressed by (default) configuration? Do you have oom-killer messages in /var/log/messages or journalctl?
(In reply to Edgar Hoch from comment #5) > I found no oom-killer messages in /var/log/messages or journalctl (with > "grep -i oom") on these servers where our processes got the out of memory > messages. > We got oom-killer messages on Fedora 16 (for example), I haven't found them > on Fedora 19. Maybe that they are suppressed by (default) configuration? I must correct myself: I found some oom killer message on the Fedora 19 servers, but they are from the time when an older kernel was running (3.10.* oder 3.11.*). We have had running kernel 3.12.* only a few day and switched back to 3.11.9 because of the problems described.
Haven't yet read this in detail, but I'll note that 3.12 merged the "Reliably out of-memory handling" https://lwn.net/Articles/562211/#oom """ it causes the kernel to avoid calling the OOM killer altogether for most memory allocation failures. In particular, if the allocation is being made in response to a system call, the kernel will just cause the system call to fail with an ENOMEM error rather than trying to find a process to kill. That may cause system call failures to happen more often and in different contexts than they used to. But, naturally, that will not be a problem since all user-space code diligently checks the return status of every system call and responds with well-tested error-handling code when things go wrong. """
Perhaps #1064076 and #1050990 are similar/same bug then, but my oom-killer bug is something separate, if neither Edgar nor Amata are seeing oom-killer in the logs. For me, my apps aren't crashing with ENOMEM (that I can tell), it's the kernel and apps going crazy until (and after) oom-killer triggers. Edgar, your symptom is merely app is crashing? Not kernel or other programs going crazy/hanging? Michele's tip in Comment #7 may indeed explain the app problems you guys are seeing. However, that still leaves the question of why your apps run in 3.11 without triggering ENOMEM nor oom-killer. You'd think if you you really were "running out of memory" that one of the two must then occur. In that sense your bug is similar to mine: oom issues in 3.12 that don't exist in 3.11 when there is tons of mem available.
(In reply to Trevor Cordes from comment #8) > However, that still leaves the question of why your apps run in 3.11 without > triggering ENOMEM nor oom-killer. You'd think if you you really were > "running out of memory" that one of the two must then occur. In that sense > your bug is similar to mine: oom issues in 3.12 that don't exist in 3.11 > when there is tons of mem available. My colleagues told me that their applications (linguistic application) had crashed when they had allocated about 8 GB virtual memory, and they told me that there was enought free real memory available. These servers have 144 or 256 GB real memory. In my own tests with the test program to reproduce the problem I have found situations when the program have crashed when it has allocated about 8 GB virtual memory (see comment #0). And I am sure that I was the only (real) user on this machine at testing time, and the machine was booted short before my tests. The test maschine has 256 GB real memory, most of them was really free. The process has received the out of memory message even through there was much enought free memory available. I have observed this behavior only on 3.12.* kernels, not on earlier versions. My test program is in c++ and doesn't call system calls directly. It just creates new arrays. The g++ compiler translates this into "brk" system calls (tested with "strace"). I am not sure if this are the kind of system calls that are described in comment #7. I guess that there may be a (internal?) limit on some metadata of the virtual memory management that I have not found until now and that is not listed with "sysctl -a" (I have compared the output of this command with kernel 3.12.9 and 3.11.9), or the 3.12.* kernels calculate some "process behaviour" values that marks my test program and that of my colleagues as "bad" and deny them addtional virtual memory. Or there may be a bug in the internal data structures of the kernel - I don't know.
Is possible that this is related with upstream bug: https://bugzilla.kernel.org/show_bug.cgi?id=67651 Fix for that bug was merged into 3.12.11 & 3.13.3 . Please update the kernel and check if problem still happens there (If yum does not include latest kernel, it can be downloaded from koji http://koji.fedoraproject.org/koji/packageinfo?packageID=8)
Yes, the problem is fixed in kernel-3.12.11-201.fc19.x86_64. I have tested it on a machine with 32 GB ram. With kernel 3.12.9 my test program has crashed when it had allocated about 8 GB. With kernel 3.11.9 and 3.12.11 my test programm has allocated more than the available real memory (32 GB) and the system has started swapping. Then I have killed the program. Many thanks for searching and finding a solution!
As an aside, it turns out my bug was unrelated to this bug. Please see new bug 1075185 if you are seeing my oom-killer symptoms.