Created attachment 362820 [details] PID range checker Description of problem: After the fix for BZ 479182, a program (attached) that checking PID range will trigger the OOM pretty quickly. This has only been observed on a particular machine installed kernel-smp so far. UP kernel and kernel-hugemem is not affected. The problem exists for both RHEL4.7.z and RHEL4.8. Red Hat Enterprise Linux AS release 4 (Nahant Update 7) Kernel 2.6.9-78.0.14.ELsmp on an i686 gs-bl460cg1-01.rhts.bos.redhat.com login: oom-killer: gfp_mask=0xd0 Mem-info: DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 cpu 0 cold: low 0, high 2, batch 1 cpu 1 hot: low 2, high 6, batch 1 cpu 1 cold: low 0, high 2, batch 1 cpu 2 hot: low 2, high 6, batch 1 cpu 2 cold: low 0, high 2, batch 1 cpu 3 hot: low 2, high 6, batch 1 cpu 3 cold: low 0, high 2, batch 1 cpu 4 hot: low 2, high 6, batch 1 cpu 4 cold: low 0, high 2, batch 1 cpu 5 hot: low 2, high 6, batch 1 cpu 5 cold: low 0, high 2, batch 1 cpu 6 hot: low 2, high 6, batch 1 cpu 6 cold: low 0, high 2, batch 1 cpu 7 hot: low 2, high 6, batch 1 cpu 7 cold: low 0, high 2, batch 1 Normal per-cpu: cpu 0 hot: low 32, high 96, batch 16 cpu 0 cold: low 0, high 32, batch 16 cpu 1 hot: low 32, high 96, batch 16 cpu 1 cold: low 0, high 32, batch 16 cpu 2 hot: low 32, high 96, batch 16 cpu 2 cold: low 0, high 32, batch 16 cpu 3 hot: low 32, high 96, batch 16 cpu 3 cold: low 0, high 32, batch 16 cpu 4 hot: low 32, high 96, batch 16 cpu 4 cold: low 0, high 32, batch 16 cpu 5 hot: low 32, high 96, batch 16 cpu 5 cold: low 0, high 32, batch 16 cpu 6 hot: low 32, high 96, batch 16 cpu 6 cold: low 0, high 32, batch 16 cpu 7 hot: low 32, high 96, batch 16 cpu 7 cold: low 0, high 32, batch 16 HighMem per-cpu: cpu 0 hot: low 32, high 96, batch 16 cpu 0 cold: low 0, high 32, batch 16 cpu 1 hot: low 32, high 96, batch 16 cpu 1 cold: low 0, high 32, batch 16 cpu 2 hot: low 32, high 96, batch 16 cpu 2 cold: low 0, high 32, batch 16 cpu 3 hot: low 32, high 96, batch 16 cpu 3 cold: low 0, high 32, batch 16 cpu 4 hot: low 32, high 96, batch 16 cpu 4 cold: low 0, high 32, batch 16 cpu 5 hot: low 32, high 96, batch 16 cpu 5 cold: low 0, high 32, batch 16 cpu 6 hot: low 32, high 96, batch 16 cpu 6 cold: low 0, high 32, batch 16 cpu 7 hot: low 32, high 96, batch 16 cpu 7 cold: low 0, high 32, batch 16 Free pages: 65117304kB (65101120kB HighMem) Active:5996 inactive:2021 dirty:0 writeback:0 unstable:0 free:16279326 slab:88868 mapped:4435 pagetables:294 DMA free:12456kB min:64kB low:128kB high:192kB active:0kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? yes protections[]: 0 0 0 Normal free:3728kB min:3728kB low:7456kB high:11184kB active:56kB inactive:316kB present:901120kB pages_scanned:1155 all_unreclaimable? yes protections[]: 0 0 0 HighMem free:65101120kB min:512kB low:1024kB high:1536kB active:23928kB inactive:7768kB present:65929212kB pages_scanned:0 all_unreclaimable? no protections[]: 0 0 0 DMA: 4*4kB 3*8kB 2*16kB 3*32kB 4*64kB 2*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12456kB Normal: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3728kB HighMem: 1034*4kB 549*8kB 261*16kB 191*32kB 149*64kB 69*128kB 32*256kB 12*512kB 3*1024kB 1*2048kB 15880*4096kB = 65101120kB 5330 pagecache pages Swap cache: add 0, delete 0, find 0/0, race 0+0 0 bounce buffer pages Free swap: 5406712kB 16711679 pages of RAM 16285268 pages of HIGHMEM 329568 reserved pages 6191 pages shared 0 pages swap cached Kernel panic - not syncing: out of memory. panic_on_oom is selected Version-Release number of selected component (if applicable): kernel-2.6.9-78.0.14.EL kernel-2.6.9-89.11.EL How reproducible: always Steps to Reproduce: 1. reserve gs-bl460cg1-01.rhts.bos.redhat.com installed RHEL4.7 i386 for RHTS. 2. echo 99999 >/proc/sys/kernel/pid_max 3. gcc pidseqchk.c -o pidseqchk 4. ./pidseqchk Actual results: OOM. Expected results: No OOM.
After pulled out this patch for 2.6.9-78.0.14.EL kernel, OOM is gone. linux-2.6.9-pidhashing-fix-alloc_pidmap.patch
Does this problem also contribute to this bug? Bug 510371 - task_struct (and related slabcache) grow continously in RHEL 4 https://bugzilla.redhat.com/show_bug.cgi?id=510371
I was trying to reproduce this on intel-s5000phb-01.rhts.eng.bos.redhat.com x64_64 with no luck. However on gs-bl460cg1-01.rhts.bos.redhat.com i686 and 2.6.9-78.0.14.ELsmp kernel this is triggered in a moment. I tried to take 2.6.9-78 and only add patch linux-2.6.9-pidhashing-fix-alloc_pidmap.patch. This kernel works just fine. This kernel is on gs-bl460cg1-01.rhts.bos.redhat.com atm named 2.6.9-78.EL.testsmp. I also looked into a code and patched alloc_pidmap() is almost identical to the ones in RHEL5 and upstream. Therefore I think that the regression is brought by a different patch and my patch only uncovers the issue with pidseqchk app. Thoughts?
Very true, I also didn't find divergence with upstream in the patch.
Jiri, I have tried the 2.6.9-78.EL.testsmp kernel you mentioned, but it does not look like the patch is applied since BZ #479182 can still be reproducible there. # echo 99999 >/proc/sys/kernel/pid_max # ./pidseqchk ... sequence break: 13868 - 32767 (new 65536) sequence break: 65536 - 70188 (new 70190) sequence break: 70190 - 98303 (new 131072) sequence break: 131072 - 131072 (new 300) ...
It has rebuild a kernel and confirmed that, * 2.6.9-78.EL.smp + linux-2.6.9-pidhashing-fix-alloc_pidmap.patch = OK * 2.6.9-78.0.5.EL.smp + linux-2.6.9-pidhashing-fix-alloc_pidmap.patch = OOM I am continuing bisecting.
update: Indeed, my 2.6.9-78.EL.testsmp didn't include linux-2.6.9-pidhashing-fix-alloc_pidmap.patch. I put it there as linux-kernel-test.patch but it looks like it is ignored - Cai did it the same way and therefore he has negative result with 2.6.9-78.EL.smp in comment #8. I did another build of 2.6.9-78.EL.smp which includes my patch (2.6.9-78.EL.test2smp) and I have the oom killer too. So it looks like there is really something wrong with this patch. Investigation continues...
According to slabtop, size-4096 goes off the roof... Therefore the suspect is patch line: + void *page = kzalloc(PAGE_SIZE, GFP_KERNEL);
During the pidseqchk run slab-4096 goes up to ~300MB. This is allocated in the lowmem. Because gs-bl460cg1-01 has 63GB of the memory, the lowmem with 2.6.9-78.EL.test2smp is small and pidseqchk run causes to fill it all and eventually triggers the OOM killer. This doesn't happen on systems with e.g. 1GB of RAM (intel-s5000phb-01) when the lowmem is bigger. When you use hugemem kernel, the lowmem is significantly bigger and OOM doesn't appear. Anyway booting 2.6.9-78.EL.test2smp kernel on gs-bl460cg1-01 gives you warning you are using >16GB of RAM. Here are relevant parts of /proc/meminfo for hosts/kernels: gs-bl460cg1-01.rhts.bos.redhat.com 2.6.9-78.EL.test2hugemem MemTotal: 65526128 kB MemFree: 65400908 kB LowTotal: 2873716 kB LowFree: 2829260 kB ------------------------------------------------------------- gs-bl460cg1-01.rhts.bos.redhat.com 2.6.9-78.EL.test2smp MemTotal: 65528440 kB MemFree: 65406300 kB LowTotal: 387368 kB LowFree: 343580 kB ------------------------------------------------------------- intel-s5000phb-01.rhts.eng.bos.redhat.com 2.6.9-78.EL.test2smp MemTotal: 1028768 kB MemFree: 909168 kB LowTotal: 903516 kB LowFree: 866468 kB Hope this clears it up. Closing this as NOTABUG. Feel free to reopen.
Created attachment 363184 [details] Free Low memory watcher This application shows the low memory level and how it fluctuates.
Done some research for RHEL5, and found that kernel-PAE has also had around 300M low memory in this machine. However, the OOM is unable to be reproduced there due to it does not allow pid_max to be set to more than 32768 for 32-bit. # echo -n 32769 >/proc/sys/kernel/pid_max -bash: echo: write error: Invalid argument Therefore, how about bring a similar behaviour into RHEL4 as well -- unable to set pid_max to more than 32768 for 32-bit kernels other than hugemem?
Created attachment 363384 [details] untested patch
a.d. c#14: Okay Cai, that seems reasonable. Following two commits solves the problem: http://linux.bkbits.net:8080/linux-2.6/?PAGE=gnupatch&REV=1.1938.166.68 http://linux.bkbits.net:8080/linux-2.6/?PAGE=gnupatch&REV=1.1938.166.69 I've already backported these to RHEL4 and I'm going to test it.
Jiri, one question though. So, it is not allowed to set the max_pid bigger than 0x8000 even with hugemem kernel anymore? Since it has 4G/4G split there, so I am not sure if there is any existing RHEL4 customers expect larger max_pid in that setup.
(In reply to comment #18) > Jiri, one question though. So, it is not allowed to set the max_pid bigger than > 0x8000 even with hugemem kernel anymore? Since it has 4G/4G split there, so I > am not sure if there is any existing RHEL4 customers expect larger max_pid in > that setup. Correct. This is the way it's done in upstream kernel and also in RHEL5 kernel.
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
Committed in 89.43.EL . RPMS are available at http://people.redhat.com/vgoyal/rhel4/
Reproduced in 89.42.ELsmp and verified in 89.43.ELsmp.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0263.html