Bug 426822 - oom-killer is activated and kills processes but mem exists
oom-killer is activated and kills processes but mem exists
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel (Show other bugs)
4.5
i686 Linux
low Severity high
: rc
: ---
Assigned To: Larry Woodman
Martin Jenner
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2007-12-26 17:56 EST by Sev Binello
Modified: 2012-06-20 11:56 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-06-20 11:56:19 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Sev Binello 2007-12-26 17:56:46 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.13pre) Gecko/20071018 Red Hat/1.0.9-6.el4 SeaMonkey/1.0.9

Description of problem:
oom-killer starts killing processes
even though it looks like we have plenty of memory
verified with output of sar.

Version-Release number of selected component (if applicable):
 2.6.9-55.ELsmp

How reproducible:
Couldn't Reproduce


Steps to Reproduce:
1.
2.
3.

Actual Results:


Expected Results:


Additional info:
Dec 26 16:21:05 acnlinb6 kernel: oom-killer: gfp_mask=0x8d0
Dec 26 16:21:05 acnlinb6 kernel: Mem-info:
Dec 26 16:21:05 acnlinb6 kernel: DMA per-cpu:
Dec 26 16:21:05 acnlinb6 kernel: cpu 0 hot: low 2, high 6, batch 1
Dec 26 16:21:05 acnlinb6 kernel: cpu 0 cold: low 0, high 2, batch 1
Dec 26 16:21:05 acnlinb6 kernel: cpu 1 hot: low 2, high 6, batch 1
Dec 26 16:21:05 acnlinb6 kernel: cpu 1 cold: low 0, high 2, batch 1
Dec 26 16:21:05 acnlinb6 kernel: cpu 2 hot: low 2, high 6, batch 1
Dec 26 16:21:05 acnlinb6 kernel: cpu 2 cold: low 0, high 2, batch 1
Dec 26 16:21:05 acnlinb6 kernel: cpu 3 hot: low 2, high 6, batch 1
Dec 26 16:21:05 acnlinb6 kernel: cpu 3 cold: low 0, high 2, batch 1
Dec 26 16:21:05 acnlinb6 kernel: Normal per-cpu:
Dec 26 16:21:05 acnlinb6 kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 26 16:24:40 acnlinb6 rsh(pam_unix)[14498]: session closed for user mcr
Dec 26 16:24:40 acnlinb6 rsh(pam_unix)[27810]: session closed for user mcr
Dec 26 16:24:40 acnlinb6 rsh(pam_unix)[15128]: session closed for user mcr
Dec 26 16:24:40 acnlinb6 rsh(pam_unix)[2206]: session closed for user mcr
Dec 26 16:24:40 acnlinb6 rsh(pam_unix)[22838]: session closed for user mcr
Dec 26 16:24:40 acnlinb6 rsh(pam_unix)[18897]: session closed for user mcr
Dec 26 16:24:40 acnlinb6 rsh(pam_unix)[21144]: session closed for user mcr
Dec 26 16:24:40 acnlinb6 rsh(pam_unix)[5670]: session closed for user mcr
Dec 26 16:24:40 acnlinb6 rsh(pam_unix)[19436]: session closed for user mcr
Dec 26 16:24:40 acnlinb6 rsh(pam_unix)[25600]: session closed for user mcr
Dec 26 16:24:42 acnlinb6 kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 26 16:24:43 acnlinb6 crond[1679]: (ordo) CMD (/home/here/ordo/client/ordo fast > /dev/null 2>&1)
Dec 26 16:24:44 acnlinb6 sshd[1668]: Accepted publickey for root from ::ffff:130.199.105.237 port 37318 ssh2
Dec 26 16:24:45 acnlinb6 kernel: cpu 1 hot: low 32, high 96, batch 16
Dec 26 16:24:45 acnlinb6 rshd[1676]: connect second port: Connection refused
Dec 26 16:24:47 acnlinb6 pam_rhosts_auth[1664]: allowed to mcr@acnsun68.pbn.bnl.gov as mcr
Dec 26 16:24:47 acnlinb6 pam_rhosts_auth[1680]: allowed to mcr@acnsun68.pbn.bnl.gov as mcr
Dec 26 16:24:49 acnlinb6 kernel: cpu 1 cold: low 0, high 32, batch 16
Dec 26 16:24:50 acnlinb6 pam_rhosts_auth[1688]: allowed to mcr@acnlin85.pbn.bnl.gov as mcr
Dec 26 16:24:51 acnlinb6 rsh(pam_unix)[1664]: session opened for user mcr by (uid=0)
Dec 26 16:24:51 acnlinb6 rsh(pam_unix)[1680]: session opened for user mcr by (uid=0)
Dec 26 16:24:52 acnlinb6 pam_rhosts_auth[1698]: allowed to mcr@acnsun68.pbn.bnl.gov as mcr
Dec 26 16:24:52 acnlinb6 kernel: cpu 2 hot: low 32, high 96, batch 16
Dec 26 16:24:53 acnlinb6 rsh(pam_unix)[1688]: session opened for user mcr by (uid=0)
Dec 26 16:24:54 acnlinb6 kernel: cpu 2 cold: low 0, high 32, batch 16
Dec 26 16:24:54 acnlinb6 rsh(pam_unix)[1698]: session opened for user mcr by (uid=0)
Dec 26 16:24:55 acnlinb6 kernel: cpu 3 hot: low 32, high 96, batch 16
Dec 26 16:24:56 acnlinb6 kernel: cpu 3 cold: low 0, high 32, batch 16
Dec 26 16:24:57 acnlinb6 kernel: HighMem per-cpu:
Dec 26 16:24:58 acnlinb6 kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 26 16:24:59 acnlinb6 kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 26 16:24:59 acnlinb6 kernel: cpu 1 hot: low 32, high 96, batch 16
Dec 26 16:24:59 acnlinb6 kernel: cpu 1 cold: low 0, high 32, batch 16
Dec 26 16:25:00 acnlinb6 kernel: cpu 2 hot: low 32, high 96, batch 16
Dec 26 16:25:00 acnlinb6 kernel: cpu 2 cold: low 0, high 32, batch 16
Dec 26 16:25:01 acnlinb6 kernel: cpu 3 hot: low 32, high 96, batch 16
Dec 26 16:25:01 acnlinb6 kernel: cpu 3 cold: low 0, high 32, batch 16
Dec 26 16:25:02 acnlinb6 kernel: 
Dec 26 16:25:02 acnlinb6 kernel: Free pages:     2672776kB (2659328kB HighMem)
Dec 26 16:25:02 acnlinb6 kernel: Active:137201 inactive:14185 dirty:0 writeback:0 unstable:0 free:668194 slab:212310 mapped:137821 pagetables:1363
Dec 26 16:25:02 acnlinb6 kernel: DMA free:12536kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:2421 all_unreclaimable? yes
Dec 26 16:25:02 acnlinb6 kernel: protections[]: 0 0 0
Dec 26 16:25:03 acnlinb6 kernel: Normal free:912kB min:928kB low:1856kB high:2784kB active:140kB inactive:72kB present:901120kB pages_scanned:396 all_unreclaimable? yes
Dec 26 16:25:03 acnlinb6 kernel: protections[]: 0 0 0
Dec 26 16:25:03 acnlinb6 kernel: HighMem free:2659328kB min:512kB low:1024kB high:1536kB active:548664kB inactive:56668kB present:3801088kB pages_scanned:0 all_unreclaimable? no
Dec 26 16:25:03 acnlinb6 rsh(pam_unix)[1688]: session closed for user mcr
Dec 26 16:25:03 acnlinb6 rsh(pam_unix)[1680]: session closed for user mcr
Dec 26 16:25:03 acnlinb6 rsh(pam_unix)[1664]: session closed for user mcr
Dec 26 16:25:03 acnlinb6 kernel: protections[]: 0 0 0
Dec 26 16:25:04 acnlinb6 kernel: DMA: 2*4kB 4*8kB 3*16kB 3*32kB 3*64kB 3*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12536kB
Dec 26 16:25:05 acnlinb6 kernel: Normal: 0*4kB 0*8kB 1*16kB 28*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 912kB
Dec 26 16:25:05 acnlinb6 kernel: HighMem: 6804*4kB 14474*8kB 6368*16kB 5351*32kB 7402*64kB 5220*128kB 2734*256kB 668*512kB 56*1024kB 1*2048kB 0*4096kB = 2659328kB
Dec 26 16:25:05 acnlinb6 kernel: Swap cache: add 2010991, delete 2009629, find 118104530/118136562, race 0+0
Dec 26 16:25:05 acnlinb6 kernel: 0 bounce buffer pages
Dec 26 16:25:05 acnlinb6 kernel: Free swap:       2050476kB
Dec 26 16:25:06 acnlinb6 kernel: 1179648 pages of RAM
Dec 26 16:25:06 acnlinb6 kernel: 818936 pages of HIGHMEM
Dec 26 16:25:07 acnlinb6 kernel: 142030 reserved pages
Dec 26 16:25:07 acnlinb6 kernel: 36959 pages shared
Dec 26 16:25:08 acnlinb6 kernel: 1364 pages swap cached
Dec 26 16:25:08 acnlinb6 kernel: Out of Memory: Killed process 8911 (wfgman).
Dec 26 16:25:08 acnlinb6 kernel: oom-killer: gfp_mask=0xd0
Dec 26 16:25:08 acnlinb6 kernel: Mem-info:
Dec 26 16:25:09 acnlinb6 kernel: DMA per-cpu:
Dec 26 16:25:09 acnlinb6 kernel: cpu 0 hot: low 2, high 6, batch 1
Dec 26 16:25:09 acnlinb6 kernel: cpu 0 cold: low 0, high 2, batch 1
Dec 26 16:25:09 acnlinb6 kernel: cpu 1 hot: low 2, high 6, batch 1
Dec 26 16:25:09 acnlinb6 kernel: cpu 1 cold: low 0, high 2, batch 1
Dec 26 16:25:10 acnlinb6 kernel: cpu 2 hot: low 2, high 6, batch 1
Dec 26 16:25:10 acnlinb6 kernel: cpu 2 cold: low 0, high 2, batch 1
Dec 26 16:25:10 acnlinb6 kernel: cpu 3 hot: low 2, high 6, batch 1
Dec 26 16:25:10 acnlinb6 kernel: cpu 3 cold: low 0, high 2, batch 1
Dec 26 16:25:10 acnlinb6 kernel: Normal per-cpu:
Dec 26 16:25:11 acnlinb6 kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 26 16:25:11 acnlinb6 kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 26 16:25:12 acnlinb6 kernel: cpu 1 hot: low 32, high 96, batch 16
Dec 26 16:25:12 acnlinb6 kernel: cpu 1 cold: low 0, high 32, batch 16
Dec 26 16:25:12 acnlinb6 kernel: cpu 2 hot: low 32, high 96, batch 16
Dec 26 16:25:12 acnlinb6 kernel: cpu 2 cold: low 0, high 32, batch 16
Dec 26 16:25:12 acnlinb6 kernel: cpu 3 hot: low 32, high 96, batch 16
Dec 26 16:25:12 acnlinb6 kernel: cpu 3 cold: low 0, high 32, batch 16
Dec 26 16:25:13 acnlinb6 kernel: HighMem per-cpu:
Dec 26 16:25:13 acnlinb6 kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 26 16:25:13 acnlinb6 kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 26 16:25:13 acnlinb6 kernel: cpu 1 hot: low 32, high 96, batch 16
Dec 26 16:25:14 acnlinb6 kernel: cpu 1 cold: low 0, high 32, batch 16
Dec 26 16:25:14 acnlinb6 kernel: cpu 2 hot: low 32, high 96, batch 16
Dec 26 16:25:14 acnlinb6 kernel: cpu 2 cold: low 0, high 32, batch 16
Dec 26 16:25:14 acnlinb6 kernel: cpu 3 hot: low 32, high 96, batch 16
Dec 26 16:25:14 acnlinb6 kernel: cpu 3 cold: low 0, high 32, batch 16
Dec 26 16:25:14 acnlinb6 kernel: 
Dec 26 16:25:15 acnlinb6 kernel: Free pages:     2733064kB (2720192kB HighMem)
Dec 26 16:25:15 acnlinb6 kernel: Active:122669 inactive:13501 dirty:0 writeback:0 unstable:0 free:683266 slab:212465 mapped:121925 pagetables:1279
Dec 26 16:25:15 acnlinb6 kernel: DMA free:12536kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:2462 all_unreclaimable? yes
Dec 26 16:25:15 acnlinb6 kernel: protections[]: 0 0 0
Dec 26 16:25:15 acnlinb6 kernel: Normal free:336kB min:928kB low:1856kB high:2784kB active:128kB inactive:56kB present:901120kB pages_scanned:9867 all_unreclaimable? yes
Dec 26 16:25:16 acnlinb6 kernel: protections[]: 0 0 0
Dec 26 16:25:16 acnlinb6 kernel: HighMem free:2720192kB min:512kB low:1024kB high:1536kB active:490548kB inactive:53948kB present:3801088kB pages_scanned:0 all_unreclaimable? no
Dec 26 16:25:16 acnlinb6 kernel: protections[]: 0 0 0
Dec 26 16:25:17 acnlinb6 kernel: DMA: 2*4kB 4*8kB 3*16kB 3*32kB 3*64kB 3*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12536kB
Dec 26 16:25:17 acnlinb6 kernel: Normal: 0*4kB 0*8kB 1*16kB 10*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 336kB
Dec 26 16:25:17 acnlinb6 kernel: HighMem: 16200*4kB 17236*8kB 6418*16kB 5359*32kB 7402*64kB 5219*128kB 2733*256kB 669*512kB 56*1024kB 1*2048kB 0*4096kB = 2720192kB
Dec 26 16:25:17 acnlinb6 kernel: Swap cache: add 2010991, delete 2009629, find 118104530/118136562, race 0+0
Dec 26 16:25:17 acnlinb6 kernel: 0 bounce buffer pages
Dec 26 16:25:17 acnlinb6 kernel: Free swap:       2050476kB
Dec 26 16:25:18 acnlinb6 kernel: 1179648 pages of RAM
Dec 26 16:25:18 acnlinb6 kernel: 818936 pages of HIGHMEM
Dec 26 16:25:18 acnlinb6 kernel: 142030 reserved pages
Dec 26 16:25:18 acnlinb6 kernel: 35667 pages shared
Dec 26 16:25:18 acnlinb6 kernel: 1364 pages swap cached
Dec 26 16:25:18 acnlinb6 kernel: Out of Memory: Killed process 20024 (magman).
Dec 26 16:25:19 acnlinb6 kernel: oom-killer: gfp_mask=0xd0
Dec 26 16:25:19 acnlinb6 kernel: Mem-info:
Dec 26 16:25:19 acnlinb6 kernel: DMA per-cpu:
Dec 26 16:25:19 acnlinb6 kernel: cpu 0 hot: low 2, high 6, batch 1
Dec 26 16:25:19 acnlinb6 kernel: cpu 0 cold: low 0, high 2, batch 1
Dec 26 16:25:19 acnlinb6 kernel: cpu 1 hot: low 2, high 6, batch 1
Dec 26 16:25:19 acnlinb6 kernel: cpu 1 cold: low 0, high 2, batch 1
Dec 26 16:25:20 acnlinb6 kernel: cpu 2 hot: low 2, high 6, batch 1
Dec 26 16:25:20 acnlinb6 kernel: cpu 2 cold: low 0, high 2, batch 1
Dec 26 16:25:20 acnlinb6 kernel: cpu 3 hot: low 2, high 6, batch 1
Dec 26 16:25:20 acnlinb6 kernel: cpu 3 cold: low 0, high 2, batch 1
Dec 26 16:25:21 acnlinb6 kernel: Normal per-cpu:
Dec 26 16:25:21 acnlinb6 kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 26 16:25:21 acnlinb6 kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 26 16:25:21 acnlinb6 kernel: cpu 1 hot: low 32, high 96, batch 16
Dec 26 16:25:22 acnlinb6 kernel: cpu 1 cold: low 0, high 32, batch 16
Dec 26 16:25:22 acnlinb6 kernel: cpu 2 hot: low 32, high 96, batch 16
Dec 26 16:25:22 acnlinb6 kernel: cpu 2 cold: low 0, high 32, batch 16
Dec 26 16:25:22 acnlinb6 kernel: cpu 3 hot: low 32, high 96, batch 16
Dec 26 16:25:22 acnlinb6 kernel: cpu 3 cold: low 0, high 32, batch 16
Dec 26 16:25:23 acnlinb6 kernel: HighMem per-cpu:
Dec 26 16:25:23 acnlinb6 kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 26 16:25:23 acnlinb6 kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 26 16:25:23 acnlinb6 kernel: cpu 1 hot: low 32, high 96, batch 16
Dec 26 16:25:23 acnlinb6 kernel: cpu 1 cold: low 0, high 32, batch 16
Comment 1 Sev Binello 2007-12-27 12:28:21 EST
Had a similar incidents yesterday on a different server.
Here's some additional data from that machine....

All messages in /var/log/messages stopped 
It never recovered and was rebooted.

Dec 26 17:09:41 acnlinb8 kernel: Mem-info:
Dec 26 17:09:41 acnlinb8 kernel: DMA per-cpu:
Dec 26 17:09:41 acnlinb8 kernel: cpu 0 hot: low 2, high 6, batch 1
Dec 26 17:09:41 acnlinb8 kernel: cpu 0 cold: low 0, high 2, batch 1
Dec 26 17:09:41 acnlinb8 kernel: cpu 1 hot: low 2, high 6, batch 1
Dec 26 17:09:41 acnlinb8 kernel: cpu 1 cold: low 0, high 2, batch 1
Dec 26 17:09:41 acnlinb8 kernel: cpu 2 hot: low 2, high 6, batch 1
Dec 26 17:09:41 acnlinb8 kernel: cpu 2 cold: low 0, high 2, batch 1
Dec 26 17:09:41 acnlinb8 kernel: cpu 3 hot: low 2, high 6, batch 1
Dec 26 17:09:41 acnlinb8 kernel: cpu 3 cold: low 0, high 2, batch 1
Dec 26 17:09:41 acnlinb8 kernel: Normal per-cpu:
Dec 26 17:09:41 acnlinb8 kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 26 17:09:41 acnlinb8 kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 26 17:09:41 acnlinb8 rsh(pam_unix)[23309]: session closed for user mcr
Dec 26 17:09:42 acnlinb8 kernel: cpu 1 hot: low 32, high 96, batch 16
Dec 26 17:09:42 acnlinb8 kernel: cpu 1 cold: low 0, high 32, batch 16
Dec 26 17:09:42 acnlinb8 kernel: cpu 2 hot: low 32, high 96, batch 16
Dec 26 17:09:42 acnlinb8 kernel: cpu 2 cold: low 0, high 32, batch 16
Dec 26 17:09:42 acnlinb8 kernel: cpu 3 hot: low 32, high 96, batch 16
Dec 26 17:09:43 acnlinb8 kernel: cpu 3 cold: low 0, high 32, batch 16
Dec 26 17:09:43 acnlinb8 kernel: HighMem per-cpu:
Dec 26 17:09:43 acnlinb8 kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 26 17:09:43 acnlinb8 kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 26 17:09:43 acnlinb8 kernel: cpu 1 hot: low 32, high 96, batch 16
Dec 26 17:09:43 acnlinb8 kernel: cpu 1 cold: low 0, high 32, batch 16
Dec 26 17:09:43 acnlinb8 kernel: cpu 2 hot: low 32, high 96, batch 16
Dec 26 17:09:43 acnlinb8 kernel: cpu 2 cold: low 0, high 32, batch 16
Dec 26 17:09:43 acnlinb8 kernel: cpu 3 hot: low 32, high 96, batch 16
Dec 26 17:09:44 acnlinb8 kernel: cpu 3 cold: low 0, high 32, batch 16
Dec 26 17:09:44 acnlinb8 kernel: 
Dec 26 17:09:44 acnlinb8 kernel: Free pages:       16540kB (3072kB HighMem)
Dec 26 17:09:44 acnlinb8 kernel: Active:784468 inactive:27703 dirty:1688
writeback:567 unstable:62 free:4135 slab:211469 mapped:807085 pagetables:5350
Dec 26 17:09:44 acnlinb8 kernel: DMA free:12540kB min:16kB low:32kB high:48kB
active:0kB inactive:0kB present:16384kB pages_scanned:2645 all_unreclaimable? yes
Dec 26 17:09:44 acnlinb8 kernel: protections[]: 0 0 0
Dec 26 17:09:44 acnlinb8 kernel: Normal free:928kB min:928kB low:1856kB
high:2784kB active:404kB inactive:1072kB present:901120kB pages_scanned:7260
all_unreclaimable? yes
Dec 26 17:09:44 acnlinb8 kernel: protections[]: 0 0 0
Dec 26 17:09:44 acnlinb8 kernel: HighMem free:3072kB min:512kB low:1024kB
high:1536kB active:3137568kB inactive:109756kB present:3801088kB pages_scanned:0
all_unreclaimable? no
Dec 26 17:09:45 acnlinb8 kernel: protections[]: 0 0 0
Dec 26 17:09:45 acnlinb8 kernel: DMA: 1*4kB 5*8kB 3*16kB 3*32kB 3*64kB 3*128kB
2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12540kB
Dec 26 17:09:45 acnlinb8 kernel: Normal: 16*4kB 0*8kB 0*16kB 3*32kB 0*64kB
0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 928kB
Dec 26 17:09:45 acnlinb8 kernel: HighMem: 0*4kB 0*8kB 68*16kB 46*32kB 0*64kB
0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 3072kB
Dec 26 17:09:46 acnlinb8 kernel: Swap cache: add 130565, delete 74885, find
50220/57583, race 0+0
Dec 26 17:09:46 acnlinb8 kernel: 0 bounce buffer pages
Dec 26 17:09:46 acnlinb8 kernel: Free swap:       1810736kB
Dec 26 17:09:46 acnlinb8 kernel: 1179648 pages of RAM
Dec 26 17:09:46 acnlinb8 kernel: 818936 pages of HIGHMEM
Dec 26 17:09:46 acnlinb8 kernel: 142030 reserved pages
Dec 26 17:09:46 acnlinb8 kernel: 310807 pages shared
Dec 26 17:09:46 acnlinb8 kernel: 55680 pages swap cached
Dec 26 17:09:46 acnlinb8 kernel: Out of Memory: Killed process 8147 (GpmLogger).

Dec 26 18:24:34 acnlinb8 kernel: SELinux: initialized (dev 0:17, type nfs), uses
genfs_contexts
Dec 26 18:24:42 acnlinb8 pam_rhosts_auth[29384]: allowed to
mcr@acnsun68.pbn.bnl.gov as mcr
Dec 26 18:24:42 acnlinb8 rsh(pam_unix)[29384]: session opened for user mcr by
(uid=0)
Dec 26 18:24:43 acnlinb8 kernel: oom-killer: gfp_mask=0xd0
Dec 26 18:24:43 acnlinb8 kernel: Mem-info:
Dec 26 18:24:43 acnlinb8 kernel: DMA per-cpu:
Dec 26 18:24:43 acnlinb8 kernel: cpu 0 hot: low 2, high 6, batch 1
Dec 26 18:24:43 acnlinb8 kernel: cpu 0 cold: low 0, high 2, batch 1
Dec 26 18:24:43 acnlinb8 kernel: cpu 1 hot: low 2, high 6, batch 1
Dec 26 18:24:43 acnlinb8 kernel: cpu 1 cold: low 0, high 2, batch 1
Dec 26 18:24:43 acnlinb8 kernel: cpu 2 hot: low 2, high 6, batch 1
Dec 26 18:24:43 acnlinb8 kernel: cpu 2 cold: low 0, high 2, batch 1
Dec 26 18:24:43 acnlinb8 kernel: cpu 3 hot: low 2, high 6, batch 1
Dec 26 18:24:43 acnlinb8 kernel: cpu 3 cold: low 0, high 2, batch 1
Dec 26 18:24:43 acnlinb8 kernel: Normal per-cpu:
Dec 26 18:24:43 acnlinb8 kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 26 18:24:43 acnlinb8 kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 26 18:24:44 acnlinb8 kernel: cpu 1 hot: low 32, high 96, batch 16
Dec 26 18:24:44 acnlinb8 kernel: cpu 1 cold: low 0, high 32, batch 16
Dec 26 18:24:44 acnlinb8 kernel: cpu 2 hot: low 32, high 96, batch 16
Dec 26 18:24:44 acnlinb8 kernel: cpu 2 cold: low 0, high 32, batch 16
Dec 26 18:24:44 acnlinb8 kernel: cpu 3 hot: low 32, high 96, batch 16
Dec 26 18:24:45 acnlinb8 kernel: cpu 3 cold: low 0, high 32, batch 16
Dec 26 18:24:45 acnlinb8 kernel: HighMem per-cpu:
Dec 26 18:24:45 acnlinb8 kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 26 18:24:46 acnlinb8 kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 26 18:24:46 acnlinb8 kernel: cpu 1 hot: low 32, high 96, batch 16
Dec 26 18:24:47 acnlinb8 kernel: cpu 1 cold: low 0, high 32, batch 16
Dec 26 18:24:47 acnlinb8 kernel: cpu 2 hot: low 32, high 96, batch 16
Dec 26 18:24:47 acnlinb8 kernel: cpu 2 cold: low 0, high 32, batch 16
Dec 26 18:24:47 acnlinb8 kernel: cpu 3 hot: low 32, high 96, batch 16
Dec 26 18:24:47 acnlinb8 kernel: cpu 3 cold: low 0, high 32, batch 16
Dec 26 18:24:47 acnlinb8 rsh(pam_unix)[29384]: session closed for user mcr
Dec 26 18:24:47 acnlinb8 kernel: 
Dec 26 18:24:48 acnlinb8 kernel: Free pages:      309756kB (296320kB HighMem)
Dec 26 18:24:48 acnlinb8 kernel: Active:711985 inactive:26756 dirty:19
writeback:456 unstable:35 free:77439 slab:211739 mapped:735684 pagetables:5219
Dec 26 18:24:48 acnlinb8 kernel: DMA free:12540kB min:16kB low:32kB high:48kB
active:0kB inactive:0kB present:16384kB pages_scanned:20576 all_unreclaimable? yes
Dec 26 18:24:48 acnlinb8 kernel: protections[]: 0 0 0
Dec 26 18:24:48 acnlinb8 kernel: Normal free:896kB min:928kB low:1856kB
high:2784kB active:340kB inactive:164kB present:901120kB pages_scanned:957
all_unreclaimable? yes
Dec 26 18:24:48 acnlinb8 kernel: protections[]: 0 0 0
Dec 26 18:24:48 acnlinb8 kernel: HighMem free:296320kB min:512kB low:1024kB
high:1536kB active:2847600kB inactive:106860kB present:3801088kB pages_scanned:0
all_unreclaimable? no
Dec 26 18:24:48 acnlinb8 kernel: protections[]: 0 0 0
Dec 26 18:24:49 acnlinb8 kernel: DMA: 1*4kB 5*8kB 3*16kB 3*32kB 3*64kB 3*128kB
2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12540kB
Dec 26 18:24:49 acnlinb8 kernel: Normal: 0*4kB 0*8kB 2*16kB 3*32kB 0*64kB
0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 896kB
Dec 26 18:24:49 acnlinb8 kernel: HighMem: 10322*4kB 6495*8kB 2582*16kB 1279*32kB
764*64kB 328*128kB 103*256kB 7*512kB 0*1024kB 0*2048kB 0*4096kB = 296320kB
Dec 26 18:24:49 acnlinb8 kernel: Swap cache: add 130619, delete 74885, find
50261/57637, race 0+0
Dec 26 18:24:49 acnlinb8 kernel: 0 bounce buffer pages
Dec 26 18:24:49 acnlinb8 kernel: Free swap:       1810736kB
Dec 26 18:24:49 acnlinb8 kernel: 1179648 pages of RAM
Dec 26 18:24:49 acnlinb8 kernel: 818936 pages of HIGHMEM
Dec 26 18:24:49 acnlinb8 kernel: 142030 reserved pages
Dec 26 18:24:49 acnlinb8 kernel: 309176 pages shared
Dec 26 18:24:49 acnlinb8 kernel: 55734 pages swap cached
Dec 26 18:24:50 acnlinb8 kernel: Out of Memory: Killed process 23938 (GpmLogger).
Dec 26 18:24:50 acnlinb8 kernel: SELinux: initialized (dev 0:10, type nfs), uses
genfs_contexts
Dec 26 18:24:51 acnlinb8 pam_rhosts_auth[29442]: allowed to
mcr@acnlin88.pbn.bnl.gov as mcr
Dec 26 18:24:52 acnlinb8 rsh(pam_unix)[29442]: session opened for user mcr by
(uid=0)
Dec 26 18:24:53 acnlinb8 kernel: SELinux: initialized (dev 0:1a, type nfs), uses
genfs_contexts
Dec 26 18:24:54 acnlinb8 rsh(pam_unix)[29442]: session closed for user mcr
Dec 26 18:24:54 acnlinb8 kernel: SELinux: initialized (dev 0:1c, type nfs), uses
genfs_contexts
Dec 26 18:24:54 acnlinb8 kernel: SELinux: initialized (dev 0:1d, type nfs), uses
genfs_contexts
Dec 26 18:24:54 acnlinb8 kernel: oom-killer: gfp_mask=0xd0
Dec 26 18:24:54 acnlinb8 kernel: Mem-info:
Dec 26 18:24:54 acnlinb8 kernel: DMA per-cpu:
Dec 26 18:24:54 acnlinb8 kernel: cpu 0 hot: low 2, high 6, batch 1
Dec 26 18:24:54 acnlinb8 kernel: cpu 0 cold: low 0, high 2, batch 1
Dec 26 18:24:54 acnlinb8 kernel: cpu 1 hot: low 2, high 6, batch 1
Dec 26 18:24:54 acnlinb8 kernel: cpu 1 cold: low 0, high 2, batch 1
Dec 26 18:24:54 acnlinb8 kernel: cpu 2 hot: low 2, high 6, batch 1
Dec 26 18:24:54 acnlinb8 kernel: cpu 2 cold: low 0, high 2, batch 1
Dec 26 18:24:54 acnlinb8 kernel: cpu 3 hot: low 2, high 6, batch 1
Dec 26 18:24:54 acnlinb8 kernel: cpu 3 cold: low 0, high 2, batch 1
Dec 26 18:24:54 acnlinb8 kernel: Normal per-cpu:
Dec 26 18:24:54 acnlinb8 kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 26 18:24:55 acnlinb8 kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 26 18:24:55 acnlinb8 kernel: cpu 1 hot: low 32, high 96, batch 16
Dec 26 18:24:55 acnlinb8 kernel: cpu 1 cold: low 0, high 32, batch 16
Dec 26 18:24:55 acnlinb8 kernel: cpu 2 hot: low 32, high 96, batch 16
Dec 26 18:24:56 acnlinb8 kernel: cpu 2 cold: low 0, high 32, batch 16
Dec 26 18:24:56 acnlinb8 kernel: cpu 3 hot: low 32, high 96, batch 16
Dec 26 18:24:56 acnlinb8 kernel: cpu 3 cold: low 0, high 32, batch 16
Dec 26 18:24:56 acnlinb8 kernel: HighMem per-cpu:
Dec 26 18:24:56 acnlinb8 kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 26 18:24:57 acnlinb8 kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 26 18:24:57 acnlinb8 kernel: cpu 1 hot: low 32, high 96, batch 16
Dec 26 18:24:57 acnlinb8 kernel: cpu 1 cold: low 0, high 32, batch 16
Dec 26 18:24:57 acnlinb8 kernel: cpu 2 hot: low 32, high 96, batch 16
Dec 26 18:24:57 acnlinb8 kernel: cpu 2 cold: low 0, high 32, batch 16
Dec 26 18:24:57 acnlinb8 kernel: cpu 3 hot: low 32, high 96, batch 16
Dec 26 18:24:58 acnlinb8 kernel: cpu 3 cold: low 0, high 32, batch 16
Dec 26 18:24:58 acnlinb8 kernel: 
Dec 26 18:24:58 acnlinb8 kernel: Free pages:      319676kB (306240kB HighMem)
Dec 26 18:24:58 acnlinb8 kernel: Active:709819 inactive:26506 dirty:0
writeback:252 unstable:39 free:79919 slab:211788 mapped:733449 pagetables:5166
Dec 26 18:24:58 acnlinb8 kernel: DMA free:12540kB min:16kB low:32kB high:48kB
active:0kB inactive:0kB present:16384kB pages_scanned:21051 all_unreclaimable? yes
Dec 26 18:24:58 acnlinb8 kernel: protections[]: 0 0 0
Dec 26 18:24:58 acnlinb8 kernel: Normal free:896kB min:928kB low:1856kB
high:2784kB active:276kB inactive:132kB present:901120kB pages_scanned:1089
all_unreclaimable? yes
Dec 26 18:24:58 acnlinb8 kernel: protections[]: 0 0 0
Dec 26 18:24:58 acnlinb8 kernel: HighMem free:306240kB min:512kB low:1024kB
high:1536kB active:2839000kB inactive:105892kB present:3801088kB pages_scanned:0
all_unreclaimable? no
Dec 26 18:24:59 acnlinb8 kernel: protections[]: 0 0 0
Dec 26 18:24:59 acnlinb8 kernel: DMA: 1*4kB 5*8kB 3*16kB 3*32kB 3*64kB 3*128kB
2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12540kB
Dec 26 18:24:59 acnlinb8 kernel: Normal: 0*4kB 0*8kB 2*16kB 3*32kB 0*64kB
0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 896kB
Dec 26 18:24:59 acnlinb8 kernel: HighMem: 12432*4kB 6548*8kB 2638*16kB 1282*32kB
765*64kB 328*128kB 103*256kB 7*512kB 0*1024kB 0*2048kB 0*4096kB = 306240kB
Dec 26 18:24:59 acnlinb8 kernel: Swap cache: add 130619, delete 74885, find
50261/57637, race 0+0
Dec 26 18:24:59 acnlinb8 kernel: 0 bounce buffer pages
Dec 26 18:24:59 acnlinb8 kernel: Free swap:       1810736kB
Dec 26 18:25:00 acnlinb8 kernel: 1179648 pages of RAM
Dec 26 18:25:00 acnlinb8 kernel: 818936 pages of HIGHMEM
Dec 26 18:25:00 acnlinb8 kernel: 142030 reserved pages
Dec 26 18:25:00 acnlinb8 kernel: 306374 pages shared
Dec 26 18:25:00 acnlinb8 kernel: 55734 pages swap cached
Dec 26 18:25:00 acnlinb8 kernel: Out of Memory: Killed process 30828 (GpmLogger).
Dec 26 18:25:00 acnlinb8 kernel: Fixed up OOM kill of mm-less task
Dec 26 18:25:00 acnlinb8 kernel: SELinux: initialized (dev 0:1e, type nfs), uses
genfs_contexts
Dec 26 18:25:00 acnlinb8 kernel: SELinux: initialized (dev 0:1f, type nfs), uses
genfs_contexts
Dec 26 18:25:00 acnlinb8 kernel: SELinux: initialized (dev 0:20, type nfs), uses
genfs_contexts
Dec 26 18:25:00 acnlinb8 kernel: SELinux: initialized (dev 0:21, type nfs), uses
genfs_contexts
Dec 26 18:25:00 acnlinb8 kernel: SELinux: initialized (dev 0:22, type nfs), uses
genfs_contexts
Dec 26 18:25:01 acnlinb8 kernel: oom-killer: gfp_mask=0xd0
Dec 26 18:25:01 acnlinb8 kernel: Mem-info:
Dec 26 18:25:01 acnlinb8 kernel: DMA per-cpu:
Dec 26 18:25:01 acnlinb8 kernel: cpu 0 hot: low 2, high 6, batch 1
Dec 26 18:25:02 acnlinb8 kernel: cpu 0 cold: low 0, high 2, batch 1
Dec 26 18:25:02 acnlinb8 kernel: cpu 1 hot: low 2, high 6, batch 1
Dec 26 18:25:02 acnlinb8 kernel: cpu 1 cold: low 0, high 2, batch 1
Dec 26 18:25:02 acnlinb8 kernel: cpu 2 hot: low 2, high 6, batch 1
Dec 26 18:25:02 acnlinb8 kernel: cpu 2 cold: low 0, high 2, batch 1
Dec 26 18:25:02 acnlinb8 kernel: cpu 3 hot: low 2, high 6, batch 1
Dec 26 18:25:02 acnlinb8 kernel: cpu 3 cold: low 0, high 2, batch 1
Dec 26 18:25:02 acnlinb8 kernel: Normal per-cpu:
Dec 26 18:25:03 acnlinb8 kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 26 18:25:03 acnlinb8 kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 26 18:25:03 acnlinb8 kernel: cpu 1 hot: low 32, high 96, batch 16
Dec 26 18:25:03 acnlinb8 kernel: cpu 1 cold: low 0, high 32, batch 16
Dec 26 18:25:03 acnlinb8 kernel: cpu 2 hot: low 32, high 96, batch 16
Dec 26 18:25:03 acnlinb8 kernel: cpu 2 cold: low 0, high 32, batch 16
Dec 26 18:25:03 acnlinb8 kernel: cpu 3 hot: low 32, high 96, batch 16
Dec 26 18:25:03 acnlinb8 kernel: cpu 3 cold: low 0, high 32, batch 16
Dec 26 18:25:03 acnlinb8 kernel: HighMem per-cpu:
Dec 26 18:25:03 acnlinb8 kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 26 18:25:03 acnlinb8 kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 26 18:25:03 acnlinb8 kernel: cpu 1 hot: low 32, high 96, batch 16
Dec 26 18:25:03 acnlinb8 kernel: cpu 1 cold: low 0, high 32, batch 16
Dec 26 18:25:03 acnlinb8 kernel: cpu 2 hot: low 32, high 96, batch 16
Dec 26 18:25:04 acnlinb8 kernel: cpu 2 cold: low 0, high 32, batch 16
Dec 26 18:25:04 acnlinb8 kernel: cpu 3 hot: low 32, high 96, batch 16
Dec 26 18:25:04 acnlinb8 kernel: cpu 3 cold: low 0, high 32, batch 16
Dec 26 18:25:04 acnlinb8 kernel: 
Dec 26 18:25:04 acnlinb8 kernel: Free pages:      326684kB (313216kB HighMem)
Dec 26 18:25:04 acnlinb8 kernel: Active:707934 inactive:26628 dirty:67
writeback:17 unstable:0 free:81671 slab:211791 mapped:731707 pagetables:5160
Dec 26 18:25:04 acnlinb8 kernel: DMA free:12540kB min:16kB low:32kB high:48kB
active:0kB inactive:0kB present:16384kB pages_scanned:21267 all_unreclaimable? yes
Dec 26 18:25:04 acnlinb8 kernel: protections[]: 0 0 0
Dec 26 18:25:04 acnlinb8 kernel: Normal free:928kB min:928kB low:1856kB
high:2784kB active:240kB inactive:48kB present:901120kB pages_scanned:651
all_unreclaimable? no
Dec 26 18:25:04 acnlinb8 kernel: protections[]: 0 0 0
Dec 26 18:25:04 acnlinb8 kernel: HighMem free:313216kB min:512kB low:1024kB
high:1536kB active:2831616kB inactive:106456kB present:3801088kB pages_scanned:0
all_unreclaimable? no
Dec 26 18:25:04 acnlinb8 kernel: protections[]: 0 0 0
Dec 26 18:25:04 acnlinb8 kernel: DMA: 1*4kB 5*8kB 3*16kB 3*32kB 3*64kB 3*128kB
2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12540kB
Dec 26 18:25:04 acnlinb8 kernel: Normal: 0*4kB 8*8kB 0*16kB 3*32kB 0*64kB
0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 928kB
Dec 26 18:25:04 acnlinb8 kernel: HighMem: 12648*4kB 7144*8kB 2694*16kB 1284*32kB
769*64kB 329*128kB 103*256kB 7*512kB 0*1024kB 0*2048kB 0*4096kB = 313216kB
Dec 26 18:25:04 acnlinb8 kernel: Swap cache: add 130619, delete 74885, find
50261/57637, race 0+0
Dec 26 18:25:04 acnlinb8 kernel: 0 bounce buffer pages
Dec 26 18:25:04 acnlinb8 kernel: Free swap:       1810736kB
Dec 26 18:25:04 acnlinb8 kernel: 1179648 pages of RAM
Dec 26 18:25:04 acnlinb8 kernel: 818936 pages of HIGHMEM
Dec 26 18:25:04 acnlinb8 kernel: 142030 reserved pages
Dec 26 18:25:04 acnlinb8 kernel: 304966 pages shared
Dec 26 18:25:04 acnlinb8 kernel: 55734 pages swap cached
Dec 26 18:25:05 acnlinb8 kernel: Out of Memory: Killed process 8156 (GpmLogger).
Dec 26 18:25:05 acnlinb8 kernel: Fixed up OOM kill of mm-less task
Dec 26 18:25:05 acnlinb8 kernel: SELinux: initialized (dev 0:23, type nfs), uses
genfs_contexts
Dec 26 18:25:05 acnlinb8 kernel: SELinux: initialized (dev 0:24, type nfs), uses
genfs_contexts
Dec 26 18:25:05 acnlinb8 kernel: SELinux: initialized (dev 0:25, type nfs), uses
genfs_contexts
Dec 26 18:25:05 acnlinb8 kernel: SELinux: initialized (dev 0:26, type nfs), uses
genfs_contexts
Dec 26 18:25:05 acnlinb8 kernel: SELinux: initialized (dev 0:27, type nfs), uses
genfs_contexts
Dec 26 18:26:24 acnlinb8 pam_rhosts_auth[29601]: allowed to
mcr@acnlin88.pbn.bnl.gov as mcr
Dec 26 18:26:24 acnlinb8 rsh(pam_unix)[29601]: session opened for user mcr by
(uid=0)
Dec 26 18:26:26 acnlinb8 kernel: SELinux: initialized (dev 0:10, type nfs), uses
genfs_contexts
Dec 26 18:26:26 acnlinb8 rsh(pam_unix)[29601]: session closed for user mcr
Dec 26 18:26:31 acnlinb8 pam_rhosts_auth[29620]: allowed to
mcr@acnlin85.pbn.bnl.gov as mcr
Dec 26 18:26:31 acnlinb8 rsh(pam_unix)[29620]: session opened for user mcr by
(uid=0)
Dec 26 18:26:32 acnlinb8 rsh(pam_unix)[29620]: session closed for user mcr
Dec 26 18:26:40 acnlinb8 pam_rhosts_auth[29639]: allowed to
mcr@acnsun68.pbn.bnl.gov as mcr
Dec 26 18:26:40 acnlinb8 rsh(pam_unix)[29639]: session opened for user mcr by
(uid=0)
Dec 26 18:26:41 acnlinb8 rsh(pam_unix)[29639]: session closed for user mcr
Dec 26 18:27:05 acnlinb8 kernel: SELinux: initialized (dev 0:17, type nfs), uses
genfs_contexts
Dec 26 18:27:55 acnlinb8 pam_rhosts_auth[29680]: allowed to
mcr@acnlin88.pbn.bnl.gov as mcr
Dec 26 18:27:55 acnlinb8 rsh(pam_unix)[29680]: session opened for user mcr by
(uid=0)
Dec 26 18:27:57 acnlinb8 kernel: SELinux: initialized (dev 0:10, type nfs), uses
genfs_contexts
Dec 26 18:27:57 acnlinb8 rsh(pam_unix)[29680]: session closed for user mcr
Dec 26 18:28:33 acnlinb8 pam_rhosts_auth[29728]: allowed to
mcr@acnsun68.pbn.bnl.gov as mcr
Dec 26 18:28:34 acnlinb8 rsh(pam_unix)[29728]: session opened for user mcr by
(uid=0)
Dec 26 18:28:34 acnlinb8 rsh(pam_unix)[29728]: session closed for user mcr
Dec 26 18:28:47 acnlinb8 kernel: SELinux: initialized (dev 0:17, type nfs), uses
genfs_contexts
Dec 26 18:29:23 acnlinb8 pam_rhosts_auth[29758]: allowed to
mcr@acnlin85.pbn.bnl.gov as mcr
Dec 26 18:29:23 acnlinb8 rsh(pam_unix)[29758]: session opened for user mcr by
(uid=0)
Dec 26 18:29:24 acnlinb8 kernel: SELinux: initialized (dev 0:10, type nfs), uses
genfs_contexts
Dec 26 18:29:24 acnlinb8 rsh(pam_unix)[29758]: session closed for user mcr
Dec 26 18:29:26 acnlinb8 pam_rhosts_auth[29785]: allowed to
mcr@acnlin88.pbn.bnl.gov as mcr
Dec 26 18:29:26 acnlinb8 rsh(pam_unix)[29785]: session opened for user mcr by
(uid=0)
Dec 26 18:29:26 acnlinb8 rsh(pam_unix)[29785]: session closed for user mcr
Dec 26 18:29:42 acnlinb8 pam_rhosts_auth[29828]: allowed to
mcr@acnlin85.pbn.bnl.gov as mcr
Dec 26 18:29:42 acnlinb8 kernel: SELinux: initialized (dev 0:1a, type nfs), uses
genfs_contexts
Dec 26 18:29:43 acnlinb8 rsh(pam_unix)[29828]: session opened for user mcr by
(uid=0)
Dec 26 18:29:43 acnlinb8 rsh(pam_unix)[29828]: session closed for user mcr
Dec 26 18:29:43 acnlinb8 kernel: SELinux: initialized (dev 0:1c, type nfs), uses
genfs_contexts
Dec 26 18:29:45 acnlinb8 kernel: SELinux: initialized (dev 0:1d, type nfs), uses
genfs_contexts
Dec 26 18:29:45 acnlinb8 kernel: SELinux: initialized (dev 0:1e, type nfs), uses
genfs_contexts
Dec 26 18:29:45 acnlinb8 kernel: SELinux: initialized (dev 0:1f, type nfs), uses
genfs_contexts
Dec 26 18:29:45 acnlinb8 kernel: SELinux: initialized (dev 0:20, type nfs), uses
genfs_contexts
Dec 26 18:29:46 acnlinb8 kernel: SELinux: initialized (dev 0:21, type nfs), uses
genfs_contexts
Dec 26 18:29:46 acnlinb8 kernel: SELinux: initialized (dev 0:22, type nfs), uses
genfs_contexts
Dec 26 18:29:46 acnlinb8 kernel: SELinux: initialized (dev 0:23, type nfs), uses
genfs_contexts
Dec 26 18:29:46 acnlinb8 kernel: SELinux: initialized (dev 0:24, type nfs), uses
genfs_contexts
Dec 26 18:29:46 acnlinb8 kernel: SELinux: initialized (dev 0:25, type nfs), uses
genfs_contexts
Dec 26 18:29:46 acnlinb8 kernel: SELinux: initialized (dev 0:26, type nfs), uses
genfs_contexts
Dec 26 18:29:47 acnlinb8 kernel: SELinux: initialized (dev 0:27, type nfs), uses
genfs_contexts
Dec 26 18:30:01 acnlinb8 crond[29972]: (root) CMD (/usr/lib/sa/sa1 1 1)
Dec 26 18:30:26 acnlinb8 pam_rhosts_auth[29978]: allowed to
mcr@acnsun68.pbn.bnl.gov as mcr
Dec 26 18:30:26 acnlinb8 rsh(pam_unix)[29978]: session opened for user mcr by
(uid=0)
Dec 26 18:30:27 acnlinb8 rsh(pam_unix)[29978]: session closed for user mcr
Dec 26 18:30:47 acnlinb8 kernel: SELinux: initialized (dev 0:10, type nfs), uses
genfs_contexts
Dec 26 18:30:56 acnlinb8 pam_rhosts_auth[30011]: allowed to
mcr@acnlin88.pbn.bnl.gov as mcr
Dec 26 18:30:57 acnlinb8 rsh(pam_unix)[30011]: session opened for user mcr by
(uid=0)
Dec 26 18:30:58 acnlinb8 kernel: SELinux: initialized (dev 0:17, type nfs), uses
genfs_contexts
Dec 26 18:30:59 acnlinb8 rsh(pam_unix)[30011]: session closed for user mcr
Dec 26 18:31:01 acnlinb8 crond[30038]: (ordo) CMD (/home/here/ordo/client/ordo
fast > /dev/null 2>&1)
Dec 26 18:32:08 acnlinb8 kernel: SELinux: initialized (dev 0:10, type nfs), uses
genfs_contexts
Dec 26 18:32:22 acnlinb8 pam_rhosts_auth[30224]: allowed to
mcr@acnsun68.pbn.bnl.gov as mcr
Dec 26 18:32:22 acnlinb8 rsh(pam_unix)[30224]: session opened for user mcr by
(uid=0)
Dec 26 18:32:23 acnlinb8 kernel: SELinux: initialized (dev 0:17, type nfs), uses
genfs_contexts
Dec 26 18:32:24 acnlinb8 rsh(pam_unix)[30224]: session closed for user mcr
Dec 26 18:32:27 acnlinb8 pam_rhosts_auth[30246]: allowed to
mcr@acnlin88.pbn.bnl.gov as mcr
Dec 26 18:32:28 acnlinb8 rsh(pam_unix)[30246]: session opened for user mcr by
(uid=0)
Dec 26 18:32:29 acnlinb8 rsh(pam_unix)[30246]: session closed for user mcr
Dec 26 18:32:52 acnlinb8 pam_rhosts_auth[30266]: allowed to
mcr@acnlin85.pbn.bnl.gov as mcr
Dec 26 18:32:52 acnlinb8 rsh(pam_unix)[30266]: session opened for user mcr by
(uid=0)
Dec 26 18:32:53 acnlinb8 rsh(pam_unix)[30266]: session closed for user mcr
Dec 26 18:33:57 acnlinb8 pam_rhosts_auth[30298]: allowed to
mcr@acnlin88.pbn.bnl.gov as mcr
Dec 26 18:33:57 acnlinb8 rsh(pam_unix)[30298]: session opened for user mcr by
(uid=0)
Dec 26 18:33:59 acnlinb8 kernel: SELinux: initialized (dev 0:10, type nfs), uses
genfs_contexts
Dec 26 18:33:59 acnlinb8 rsh(pam_unix)[30298]: session closed for user mcr
Dec 26 18:34:17 acnlinb8 pam_rhosts_auth[30319]: allowed to
mcr@acnsun68.pbn.bnl.gov as mcr
Dec 26 18:34:18 acnlinb8 rsh(pam_unix)[30319]: session opened for user mcr by
(uid=0)
Dec 26 18:34:19 acnlinb8 rsh(pam_unix)[30319]: session closed for user mcr
Dec 26 18:34:23 acnlinb8 pam_rhosts_auth[30340]: allowed to
mcr@acnlin85.pbn.bnl.gov as mcr
Dec 26 18:34:24 acnlinb8 rsh(pam_unix)[30340]: session opened for user mcr by
(uid=0)
Dec 26 18:34:24 acnlinb8 rsh(pam_unix)[30340]: session closed for user mcr
Dec 26 18:34:35 acnlinb8 kernel: SELinux: initialized (dev 0:17, type nfs), uses
genfs_contexts
Dec 26 18:34:43 acnlinb8 kernel: SELinux: initialized (dev 0:1a, type nfs), uses
genfs_contexts
Dec 26 18:34:44 acnlinb8 kernel: SELinux: initialized (dev 0:1c, type nfs), uses
genfs_contexts
Dec 26 18:34:46 acnlinb8 kernel: SELinux: initialized (dev 0:1d, type nfs), uses
genfs_contexts
Dec 26 18:34:46 acnlinb8 kernel: SELinux: initialized (dev 0:1e, type nfs), uses
genfs_contexts
Dec 26 18:34:46 acnlinb8 kernel: SELinux: initialized (dev 0:1f, type nfs), uses
genfs_contexts
Dec 26 18:34:46 acnlinb8 kernel: SELinux: initialized (dev 0:20, type nfs), uses
genfs_contexts
Dec 26 18:34:46 acnlinb8 kernel: SELinux: initialized (dev 0:21, type nfs), uses
genfs_contexts
Dec 26 18:34:46 acnlinb8 kernel: SELinux: initialized (dev 0:22, type nfs), uses
genfs_contexts
Dec 26 18:34:46 acnlinb8 kernel: SELinux: initialized (dev 0:23, type nfs), uses
genfs_contexts
Dec 26 18:34:46 acnlinb8 kernel: SELinux: initialized (dev 0:24, type nfs), uses
genfs_contexts
Dec 26 18:34:48 acnlinb8 kernel: SELinux: initialized (dev 0:25, type nfs), uses
genfs_contexts
Dec 26 18:34:49 acnlinb8 kernel: SELinux: initialized (dev 0:26, type nfs), uses
genfs_contexts
Dec 26 18:34:50 acnlinb8 kernel: SELinux: initialized (dev 0:27, type nfs), uses
genfs_contexts
Dec 26 18:35:29 acnlinb8 pam_rhosts_auth[30521]: allowed to
mcr@acnlin88.pbn.bnl.gov as mcr
Dec 26 18:35:29 acnlinb8 rsh(pam_unix)[30521]: session opened for user mcr by
(uid=0)
Dec 26 18:35:30 acnlinb8 kernel: SELinux: initialized (dev 0:10, type nfs), uses
genfs_contexts
Dec 26 18:35:31 acnlinb8 rsh(pam_unix)[30521]: session closed for user mcr
Dec 26 18:36:03 acnlinb8 pam_rhosts_auth[30572]: allowed to
mcr@acnlin85.pbn.bnl.gov as mcr
Dec 26 18:36:03 acnlinb8 rsh(pam_unix)[30572]: session opened for user mcr by
(uid=0)
Dec 26 18:36:04 acnlinb8 rsh(pam_unix)[30572]: session closed for user mcr
Dec 26 18:36:11 acnlinb8 pam_rhosts_auth[30591]: allowed to
mcr@acnsun68.pbn.bnl.gov as mcr
Dec 26 18:36:12 acnlinb8 rsh(pam_unix)[30591]: session opened for user mcr by
(uid=0)
Dec 26 18:36:13 acnlinb8 rsh(pam_unix)[30591]: session closed for user mcr
Dec 26 18:37:00 acnlinb8 pam_rhosts_auth[30618]: allowed to
mcr@acnlin88.pbn.bnl.gov as mcr
Dec 26 18:37:01 acnlinb8 rsh(pam_unix)[30618]: session opened for user mcr by
(uid=0)
Dec 26 18:37:02 acnlinb8 kernel: SELinux: initialized (dev 0:10, type nfs), uses
genfs_contexts
Dec 26 18:37:02 acnlinb8 rsh(pam_unix)[30618]: session closed for user mcr
Dec 26 18:37:43 acnlinb8 kernel: SELinux: initialized (dev 0:17, type nfs), uses
genfs_contexts
Dec 26 18:38:05 acnlinb8 pam_rhosts_auth[30647]: allowed to
mcr@acnsun68.pbn.bnl.gov as mcr
Dec 26 18:38:06 acnlinb8 rsh(pam_unix)[30647]: session opened for user mcr by
(uid=0)
Dec 26 18:38:07 acnlinb8 rsh(pam_unix)[30647]: session closed for user mcr
Dec 26 18:38:32 acnlinb8 pam_rhosts_auth[30672]: allowed to
mcr@acnlin88.pbn.bnl.gov as mcr
Dec 26 18:38:32 acnlinb8 rsh(pam_unix)[30672]: session opened for user mcr by
(uid=0)
Dec 26 18:38:33 acnlinb8 kernel: SELinux: initialized (dev 0:10, type nfs), uses
genfs_contexts
Dec 26 18:38:33 acnlinb8 rsh(pam_unix)[30672]: session closed for user mcr
Dec 26 18:39:13 acnlinb8 pam_rhosts_auth[30699]: allowed to
mcr@acnlin85.pbn.bnl.gov as mcr
Dec 26 18:39:14 acnlinb8 rsh(pam_unix)[30699]: session opened for user mcr by
(uid=0)
Dec 26 18:39:15 acnlinb8 rsh(pam_unix)[30699]: session closed for user mcr
Dec 26 18:39:23 acnlinb8 pam_rhosts_auth[30717]: allowed to
mcr@acnlin85.pbn.bnl.gov as mcr
Dec 26 18:39:23 acnlinb8 rsh(pam_unix)[30717]: session opened for user mcr by
(uid=0)
Dec 26 18:39:24 acnlinb8 rsh(pam_unix)[30717]: session closed for user mcr
Dec 26 18:39:35 acnlinb8 kernel: SELinux: initialized (dev 0:17, type nfs), uses
genfs_contexts
Dec 26 18:39:42 acnlinb8 kernel: SELinux: initialized (dev 0:10, type nfs), uses
genfs_contexts
Dec 26 18:39:43 acnlinb8 kernel: SELinux: initialized (dev 0:1a, type nfs), uses
genfs_contexts
Dec 26 18:39:43 acnlinb8 kernel: oom-killer: gfp_mask=0xd0
Dec 26 18:39:43 acnlinb8 kernel: Mem-info:
Dec 26 18:39:43 acnlinb8 kernel: DMA per-cpu:
Dec 26 18:39:43 acnlinb8 kernel: cpu 0 hot: low 2, high 6, batch 1
Dec 26 18:39:43 acnlinb8 kernel: cpu 0 cold: low 0, high 2, batch 1
Dec 26 18:39:43 acnlinb8 kernel: cpu 1 hot: low 2, high 6, batch 1
Dec 26 18:39:43 acnlinb8 kernel: cpu 1 cold: low 0, high 2, batch 1
Dec 26 18:39:43 acnlinb8 kernel: cpu 2 hot: low 2, high 6, batch 1
Dec 26 18:39:43 acnlinb8 kernel: cpu 2 cold: low 0, high 2, batch 1
Dec 26 18:39:43 acnlinb8 kernel: cpu 3 hot: low 2, high 6, batch 1
Dec 26 18:39:43 acnlinb8 kernel: cpu 3 cold: low 0, high 2, batch 1
Dec 26 18:39:43 acnlinb8 kernel: Normal per-cpu:
Dec 26 18:39:43 acnlinb8 kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 26 18:39:43 acnlinb8 kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 26 18:39:43 acnlinb8 kernel: cpu 1 hot: low 32, high 96, batch 16
Dec 26 18:39:43 acnlinb8 kernel: cpu 1 cold: low 0, high 32, batch 16
Dec 26 18:39:43 acnlinb8 kernel: cpu 2 hot: low 32, high 96, batch 16
Dec 26 18:39:44 acnlinb8 kernel: cpu 2 cold: low 0, high 32, batch 16
Dec 26 18:39:44 acnlinb8 kernel: cpu 3 hot: low 32, high 96, batch 16
Dec 26 18:39:44 acnlinb8 kernel: cpu 3 cold: low 0, high 32, batch 16
Dec 26 18:39:44 acnlinb8 kernel: HighMem per-cpu:
Dec 26 18:39:45 acnlinb8 kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 26 18:39:45 acnlinb8 kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 26 18:39:45 acnlinb8 kernel: cpu 1 hot: low 32, high 96, batch 16
Dec 26 18:39:45 acnlinb8 kernel: cpu 1 cold: low 0, high 32, batch 16
Dec 26 18:39:45 acnlinb8 kernel: cpu 2 hot: low 32, high 96, batch 16
Dec 26 18:39:45 acnlinb8 kernel: cpu 2 cold: low 0, high 32, batch 16
Dec 26 18:39:46 acnlinb8 kernel: cpu 3 hot: low 32, high 96, batch 16
Dec 26 18:39:46 acnlinb8 kernel: cpu 3 cold: low 0, high 32, batch 16
Dec 26 18:39:46 acnlinb8 kernel: 
Dec 26 18:39:46 acnlinb8 kernel: Free pages:      543204kB (529792kB HighMem)
Dec 26 18:39:46 acnlinb8 kernel: Active:654057 inactive:26497 dirty:56
writeback:0 unstable:2 free:135801 slab:211793 mapped:677887 pagetables:5028
Dec 26 18:39:46 acnlinb8 kernel: DMA free:12540kB min:16kB low:32kB high:48kB
active:0kB inactive:0kB present:16384kB pages_scanned:38242 all_unreclaimable? yes
Dec 26 18:39:47 acnlinb8 kernel: protections[]: 0 0 0
Dec 26 18:39:47 acnlinb8 kernel: Normal free:872kB min:928kB low:1856kB
high:2784kB active:128kB inactive:304kB present:901120kB pages_scanned:2758
all_unreclaimable? yes
Dec 26 18:39:47 acnlinb8 kernel: protections[]: 0 0 0
Dec 26 18:39:47 acnlinb8 kernel: HighMem free:529792kB min:512kB low:1024kB
high:1536kB active:2616100kB inactive:105684kB present:3801088kB pages_scanned:0
all_unreclaimable? no
Dec 26 18:39:47 acnlinb8 kernel: protections[]: 0 0 0
Dec 26 18:39:47 acnlinb8 kernel: DMA: 1*4kB 5*8kB 3*16kB 3*32kB 3*64kB 3*128kB
2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12540kB
Dec 26 18:39:47 acnlinb8 kernel: Normal: 0*4kB 1*8kB 0*16kB 3*32kB 0*64kB
0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 872kB
Dec 26 18:39:47 acnlinb8 kernel: HighMem: 40068*4kB 16162*8kB 3934*16kB
1420*32kB 798*64kB 337*128kB 109*256kB 11*512kB 2*1024kB 1*2048kB 0*4096kB =
529792kB
Dec 26 18:39:47 acnlinb8 kernel: Swap cache: add 130619, delete 74933, find
50261/57637, race 0+0
Dec 26 18:39:48 acnlinb8 kernel: 0 bounce buffer pages
Dec 26 18:39:48 acnlinb8 kernel: Free swap:       1822508kB
Dec 26 18:39:48 acnlinb8 kernel: 1179648 pages of RAM
Dec 26 18:39:48 acnlinb8 kernel: 818936 pages of HIGHMEM
Dec 26 18:39:48 acnlinb8 kernel: 142030 reserved pages
Dec 26 18:39:49 acnlinb8 kernel: 307427 pages shared
Dec 26 18:39:49 acnlinb8 kernel: 55686 pages swap cached
Dec 26 18:39:49 acnlinb8 kernel: Out of Memory: Killed process 30016 (GpmLogger).
Dec 26 18:39:49 acnlinb8 kernel: Fixed up OOM kill of mm-less task
Dec 26 18:39:49 acnlinb8 kernel: SELinux: initialized (dev 0:1c, type nfs), uses
genfs_contexts
Dec 26 18:39:49 acnlinb8 kernel: oom-killer: gfp_mask=0xd0
Dec 26 18:39:50 acnlinb8 kernel: Mem-info:
Dec 26 18:39:50 acnlinb8 kernel: DMA per-cpu:
Dec 26 18:39:50 acnlinb8 kernel: cpu 0 hot: low 2, high 6, batch 1
Dec 26 18:39:50 acnlinb8 kernel: cpu 0 cold: low 0, high 2, batch 1
Dec 26 18:39:50 acnlinb8 kernel: cpu 1 hot: low 2, high 6, batch 1
Dec 26 18:39:50 acnlinb8 kernel: cpu 1 cold: low 0, high 2, batch 1
Dec 26 18:39:50 acnlinb8 kernel: cpu 2 hot: low 2, high 6, batch 1
Dec 26 18:39:50 acnlinb8 kernel: cpu 2 cold: low 0, high 2, batch 1
Dec 26 18:39:51 acnlinb8 kernel: cpu 3 hot: low 2, high 6, batch 1
Dec 26 18:39:51 acnlinb8 kernel: cpu 3 cold: low 0, high 2, batch 1
Dec 26 18:39:51 acnlinb8 kernel: Normal per-cpu:
Dec 26 18:39:51 acnlinb8 kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 26 18:39:51 acnlinb8 kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 26 18:39:51 acnlinb8 kernel: cpu 1 hot: low 32, high 96, batch 16
Dec 26 18:39:52 acnlinb8 kernel: cpu 1 cold: low 0, high 32, batch 16
Dec 26 18:39:52 acnlinb8 kernel: cpu 2 hot: low 32, high 96, batch 16
Dec 26 18:39:52 acnlinb8 kernel: cpu 2 cold: low 0, high 32, batch 16
Dec 26 18:39:52 acnlinb8 kernel: cpu 3 hot: low 32, high 96, batch 16
Dec 26 18:39:52 acnlinb8 kernel: cpu 3 cold: low 0, high 32, batch 16
Dec 26 18:39:52 acnlinb8 kernel: HighMem per-cpu:
Dec 26 18:39:52 acnlinb8 kernel: cpu 0 hot: low 32, high 96, batch 16
Dec 26 18:39:52 acnlinb8 kernel: cpu 0 cold: low 0, high 32, batch 16
Dec 26 18:39:52 acnlinb8 kernel: cpu 1 hot: low 32, high 96, batch 16
Dec 26 18:39:53 acnlinb8 kernel: cpu 1 cold: low 0, high 32, batch 16
Dec 26 18:39:53 acnlinb8 kernel: cpu 2 hot: low 32, high 96, batch 16
Dec 26 18:39:53 acnlinb8 kernel: cpu 2 cold: low 0, high 32, batch 16
Dec 26 18:39:53 acnlinb8 kernel: cpu 3 hot: low 32, high 96, batch 16
Dec 26 18:39:53 acnlinb8 kernel: cpu 3 cold: low 0, high 32, batch 16
Dec 26 18:39:53 acnlinb8 kernel: 
Dec 26 18:39:53 acnlinb8 kernel: Free pages:      550492kB (537024kB HighMem)
Dec 26 18:39:53 acnlinb8 kernel: Active:652274 inactive:26542 dirty:12
writeback:1 unstable:8 free:137623 slab:211794 mapped:675998 pagetables:5014
Dec 26 18:39:53 acnlinb8 kernel: DMA free:12540kB min:16kB low:32kB high:48kB
active:0kB inactive:0kB present:16384kB pages_scanned:38433 all_unreclaimable? yes
Dec 26 18:39:54 acnlinb8 kernel: protections[]: 0 0 0
Dec 26 18:39:54 acnlinb8 kernel: Normal free:928kB min:928kB low:1856kB
high:2784kB active:480kB inactive:0kB present:901120kB pages_scanned:3828
all_unreclaimable? yes
Dec 26 18:39:54 acnlinb8 kernel: protections[]: 0 0 0
Dec 26 18:39:54 acnlinb8 kernel: HighMem free:537024kB min:512kB low:1024kB
high:1536kB active:2608616kB inactive:106168kB present:3801088kB pages_scanned:0
all_unreclaimable? no
Dec 26 18:39:54 acnlinb8 kernel: protections[]: 0 0 0
Dec 26 18:39:54 acnlinb8 kernel: DMA: 1*4kB 5*8kB 3*16kB 3*32kB 3*64kB 3*128kB
2*256kB 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12540kB
Dec 26 18:39:55 acnlinb8 kernel: Normal: 0*4kB 2*8kB 3*16kB 3*32kB 0*64kB
0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 928kB
Dec 26 18:39:55 acnlinb8 kernel: HighMem: 41854*4kB 16167*8kB 3935*16kB
1419*32kB 797*64kB 338*128kB 109*256kB 11*512kB 2*1024kB 1*2048kB 0*4096kB =
537024kB
Dec 26 18:39:55 acnlinb8 kernel: Swap cache: add 130619, delete 74933, find
50261/57637, race 0+0
Dec 26 18:39:55 acnlinb8 kernel: 0 bounce buffer pages
Dec 26 18:39:55 acnlinb8 kernel: Free swap:       1822508kB
Dec 26 18:39:55 acnlinb8 kernel: 1179648 pages of RAM
Dec 26 18:39:55 acnlinb8 kernel: 818936 pages of HIGHMEM
Dec 26 18:39:55 acnlinb8 kernel: 142030 reserved pages
Dec 26 18:39:55 acnlinb8 kernel: 306111 pages shared
Dec 26 18:39:55 acnlinb8 kernel: 55686 pages swap cached
Dec 26 18:39:55 acnlinb8 kernel: Out of Memory: Killed process 30205 (GpmLogger).
Comment 2 Larry Woodman 2008-04-11 18:07:14 EDT
The problem is this is a 32-bit system and lowmem is consumed by the slabache.n

>>>slab:212310 

>>>Normal free:912kB  active:140kB inactive:72kB present:901120kB
pages_scanned:396 all_unreclaimable? yes


Can you get a /proc/slabinfo output when the OOM kill occurs to see who is using
al of the slab memory???

Larry Woodman
Comment 3 Sev Binello 2008-06-16 11:03:07 EDT
Problem just reoccured, here is the slabinfo data 
Do you need anything else ?


acnlinb6 101:cat /proc/slabinfo
slabinfo - version: 2.0
# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab>
: tunables <batchcount> <limit> <sharedfactor> : slabdata <ac tive_slabs>
<num_slabs> <sharedavail>
nfs_direct_cache       0      0     68   58    1 : tunables  120   60    8 :
slabdata      0      0      0
nfs_write_data        36     36    448    9    1 : tunables   54   27    8 :
slabdata      4      4      0
nfs_read_data         32     36    448    9    1 : tunables   54   27    8 :
slabdata      4      4      0
nfs_inode_cache      166    426    672    6    1 : tunables   54   27    8 :
slabdata     71     71    128
nfs_page               0     61     64   61    1 : tunables  120   60    8 :
slabdata      0      1      0
rpc_buffers            8      8   2048    2    1 : tunables   24   12    8 :
slabdata      4      4      0
rpc_tasks              8     20    192   20    1 : tunables  120   60    8 :
slabdata      1      1      0
rpc_inode_cache       36     42    512    7    1 : tunables   54   27    8 :
slabdata      6      6      0
dm_tio                 0      0     16  226    1 : tunables  120   60    8 :
slabdata      0      0      0
dm_io                  0      0     20  185    1 : tunables  120   60    8 :
slabdata      0      0      0
dm-bvec-(256)          0      0   3072    2    2 : tunables   24   12    8 :
slabdata      0      0      0
dm-bvec-128            0      0   1536    5    2 : tunables   24   12    8 :
slabdata      0      0      0
dm-bvec-64             0      0    768    5    1 : tunables   54   27    8 :
slabdata      0      0      0
dm-bvec-16             0      0    192   20    1 : tunables  120   60    8 :
slabdata      0      0      0
dm-bvec-4              0      0     64   61    1 : tunables  120   60    8 :
slabdata      0      0      0
dm-bvec-1              0      0     16  226    1 : tunables  120   60    8 :
slabdata      0      0      0
dm-bio                 0      0    128   31    1 : tunables  120   60    8 :
slabdata      0      0      0
fib6_nodes             5    119     32  119    1 : tunables  120   60    8 :
slabdata      1      1      0
ip6_dst_cache          4     15    256   15    1 : tunables  120   60    8 :
slabdata      1      1      0
ndisc_cache            1     20    192   20    1 : tunables  120   60    8 :
slabdata      1      1      0
rawv6_sock             6     11    704   11    2 : tunables   54   27    8 :
slabdata      1      1      0
udpv6_sock             0      0    704   11    2 : tunables   54   27    8 :
slabdata      0      0      0
tcpv6_sock             6     12   1216    3    1 : tunables   24   12    8 :
slabdata      4      4      0
ip_fib_alias          10    226     16  226    1 : tunables  120   60    8 :
slabdata      1      1      0
ip_fib_hash           10    119     32  119    1 : tunables  120   60    8 :
slabdata      1      1      0
avtab_node         25497  25585     32  119    1 : tunables  120   60    8 :
slabdata    215    215      0
ext3_inode_cache     375    763    552    7    1 : tunables   54   27    8 :
slabdata    109    109      0
ext3_xattr             8     81     48   81    1 : tunables  120   60    8 :
slabdata      1      1      0
journal_handle        63    135     28  135    1 : tunables  120   60    8 :
slabdata      1      1      0
journal_head          70    243     48   81    1 : tunables  120   60    8 :
slabdata      3      3     15
revoke_table          12    290     12  290    1 : tunables  120   60    8 :
slabdata      1      1      0
revoke_record          0      0     16  226    1 : tunables  120   60    8 :
slabdata      0      0      0
scsi_cmd_cache        32     90    384   10    1 : tunables   54   27    8 :
slabdata      9      9      6
sgpool-128            32     33   2560    3    2 : tunables   24   12    8 :
slabdata     11     11      0
sgpool-64             32     33   1280    3    1 : tunables   24   12    8 :
slabdata     11     11      0
sgpool-32             32     36    640    6    1 : tunables   54   27    8 :
slabdata      6      6      0
sgpool-16             32     36    320   12    1 : tunables   54   27    8 :
slabdata      3      3      0
sgpool-8             113    220    192   20    1 : tunables  120   60    8 :
slabdata     11     11      0
unix_sock             51     91    512    7    1 : tunables   54   27    8 :
slabdata     13     13      0
ip_mrt_cache           0      0    128   31    1 : tunables  120   60    8 :
slabdata      0      0      0
tcp_tw_bucket         33     62    128   31    1 : tunables  120   60    8 :
slabdata      2      2      0
tcp_bind_bucket       50    226     16  226    1 : tunables  120   60    8 :
slabdata      1      1      0
tcp_open_request       0      0    128   31    1 : tunables  120   60    8 :
slabdata      0      0      0
inet_peer_cache        4     61     64   61    1 : tunables  120   60    8 :
slabdata      1      1      0
secpath_cache          0      0    128   31    1 : tunables  120   60    8 :
slabdata      0      0      0
xfrm_dst_cache         0      0    256   15    1 : tunables  120   60    8 :
slabdata      0      0      0
ip_dst_cache         158    180    256   15    1 : tunables  120   60    8 :
slabdata     12     12      0
arp_cache             29     80    192   20    1 : tunables  120   60    8 :
slabdata      4      4      0
raw_sock               5      7    576    7    1 : tunables   54   27    8 :
slabdata      1      1      0
udp_sock              66     70    576    7    1 : tunables   54   27    8 :
slabdata     10     10      0
tcp_sock              16     42   1152    7    2 : tunables   24   12    8 :
slabdata      6      6      0
flow_cache             0      0    128   31    1 : tunables  120   60    8 :
slabdata      0      0      0
mqueue_inode_cache      1      7    576    7    1 : tunables   54   27    8 :
slabdata      1      1      0
relayfs_inode_cache      0      0    348   11    1 : tunables   54   27    8 :
slabdata      0      0      0
isofs_inode_cache      0      0    372   10    1 : tunables   54   27    8 :
slabdata      0      0      0
hugetlbfs_inode_cache      1     11    344   11    1 : tunables   54   27    8 :
slabdata      1      1      0
ext2_inode_cache       0      0    488    8    1 : tunables   54   27    8 :
slabdata      0      0      0
ext2_xattr             0      0     48   81    1 : tunables  120   60    8 :
slabdata      0      0      0
dquot                  0      0    144   27    1 : tunables  120   60    8 :
slabdata      0      0      0
eventpoll_pwq          1    107     36  107    1 : tunables  120   60    8 :
slabdata      1      1      0
eventpoll_epi          1     31    128   31    1 : tunables  120   60    8 :
slabdata      1      1      0
kioctx                 0      0    192   20    1 : tunables  120   60    8 :
slabdata      0      0      0
kiocb                  0      0    128   31    1 : tunables  120   60    8 :
slabdata      0      0      0
dnotify_cache         17    185     20  185    1 : tunables  120   60    8 :
slabdata      1      1      0
fasync_cache           0      0     16  226    1 : tunables  120   60    8 :
slabdata      0      0      0
shmem_inode_cache    312    333    444    9    1 : tunables   54   27    8 :
slabdata     37     37      0
posix_timers_cache      0      0    112   35    1 : tunables  120   60    8 :
slabdata      0      0      0
uid_cache              3     61     64   61    1 : tunables  120   60    8 :
slabdata      1      1      0
cfq_pool             138    357     32  119    1 : tunables  120   60    8 :
slabdata      3      3      0
crq_pool             106    288     40   96    1 : tunables  120   60    8 :
slabdata      3      3      0
deadline_drq           0      0     52   75    1 : tunables  120   60    8 :
slabdata      0      0      0
as_arq                 0      0     64   61    1 : tunables  120   60    8 :
slabdata      0      0      0
blkdev_ioc            60    185     20  185    1 : tunables  120   60    8 :
slabdata      1      1      0
blkdev_queue          19     24    488    8    1 : tunables   54   27    8 :
slabdata      3      3      0
blkdev_requests       99    200    160   25    1 : tunables  120   60    8 :
slabdata      7      8      0
biovec-(256)         256    256   3072    2    2 : tunables   24   12    8 :
slabdata    128    128      0
biovec-128           256    260   1536    5    2 : tunables   24   12    8 :
slabdata     52     52      0
biovec-64            256    265    768    5    1 : tunables   54   27    8 :
slabdata     52     53      0
biovec-16            257    280    192   20    1 : tunables  120   60    8 :
slabdata     14     14      0
biovec-4             257    305     64   61    1 : tunables  120   60    8 :
slabdata      5      5      0
biovec-1             351    678     16  226    1 : tunables  120   60    8 :
slabdata      3      3      0
bio                  359    496    128   31    1 : tunables  120   60    8 :
slabdata     16     16      0
file_lock_cache        4     41     96   41    1 : tunables  120   60    8 :
slabdata      1      1      0
sock_inode_cache     124    153    448    9    1 : tunables   54   27    8 :
slabdata     17     17      0
skbuff_head_cache    361    480    192   20    1 : tunables  120   60    8 :
slabdata     24     24     15
sock                   6     20    384   10    1 : tunables   54   27    8 :
slabdata      2      2      0
proc_inode_cache      32     88    360   11    1 : tunables   54   27    8 :
slabdata      7      8      0
sigqueue               2     54    148   27    1 : tunables  120   60    8 :
slabdata      2      2      0
radix_tree_node      771   2170    276   14    1 : tunables   54   27    8 :
slabdata    155    155     16
bdev_cache            26     35    512    7    1 : tunables   54   27    8 :
slabdata      5      5      0
mnt_cache             50     93    128   31    1 : tunables  120   60    8 :
slabdata      3      3      0
audit_watch_cache      0      0     48   81    1 : tunables  120   60    8 :
slabdata      0      0      0
inode_cache          790   1078    344   11    1 : tunables   54   27    8 :
slabdata     98     98      0
dentry_cache        1936   4966    152   26    1 : tunables  120   60    8 :
slabdata    191    191    288
filp                 704   1400    192   20    1 : tunables  120   60    8 :
slabdata     70     70      0
names_cache           20     20   4096    1    1 : tunables   24   12    8 :
slabdata     20     20      0
avc_node             504   2100     52   75    1 : tunables  120   60    8 :
slabdata     28     28      0
key_jar                6     31    128   31    1 : tunables  120   60    8 :
slabdata      1      1      0
idr_layer_cache       86    116    136   29    1 : tunables  120   60    8 :
slabdata      4      4      0
buffer_head        14954  67575     52   75    1 : tunables  120   60    8 :
slabdata    901    901     30
mm_struct             58    154    704   11    2 : tunables   54   27    8 :
slabdata     14     14      0
vm_area_struct      1535   3465     88   45    1 : tunables  120   60    8 :
slabdata     77     77     15
fs_cache              64    183     64   61    1 : tunables  120   60    8 :
slabdata      3      3      0
files_cache           61    108    448    9    1 : tunables   54   27    8 :
slabdata     12     12      0
signal_cache          95    240    192   20    1 : tunables  120   60    8 :
slabdata     12     12      0
sighand_cache         99    114   1344    3    1 : tunables   24   12    8 :
slabdata     38     38      0
task_struct          104    145   1408    5    2 : tunables   24   12    8 :
slabdata     29     29      0
anon_vma             756   1808     16  226    1 : tunables  120   60    8 :
slabdata      8      8      0
pgd                   51    357     32  119    1 : tunables  120   60    8 :
slabdata      3      3      0
pmd                  150    152   4096    1    1 : tunables   24   12    8 :
slabdata    150    152      0
size-131072(DMA)       0      0 131072    1   32 : tunables    8    4    0 :
slabdata      0      0      0
size-131072            0      0 131072    1   32 : tunables    8    4    0 :
slabdata      0      0      0
size-65536(DMA)        0      0  65536    1   16 : tunables    8    4    0 :
slabdata      0      0      0
size-65536             2      2  65536    1   16 : tunables    8    4    0 :
slabdata      2      2      0
size-32768(DMA)        0      0  32768    1    8 : tunables    8    4    0 :
slabdata      0      0      0
size-32768             7      7  32768    1    8 : tunables    8    4    0 :
slabdata      7      7      0
size-16384(DMA)        0      0  16384    1    4 : tunables    8    4    0 :
slabdata      0      0      0
size-16384             4      4  16384    1    4 : tunables    8    4    0 :
slabdata      4      4      0
size-8192(DMA)         0      0   8192    1    2 : tunables    8    4    0 :
slabdata      0      0      0
size-8192             12     12   8192    1    2 : tunables    8    4    0 :
slabdata     12     12      0
size-4096(DMA)         0      0   4096    1    1 : tunables   24   12    8 :
slabdata      0      0      0
size-4096            170    176   4096    1    1 : tunables   24   12    8 :
slabdata    170    176      0
size-2048(DMA)         0      0   2048    2    1 : tunables   24   12    8 :
slabdata      0      0      0
size-2048            347    354   2048    2    1 : tunables   24   12    8 :
slabdata    177    177      0
size-1620(DMA)         0      0   1664    4    2 : tunables   24   12    8 :
slabdata      0      0      0
size-1620             19     28   1664    4    2 : tunables   24   12    8 :
slabdata      7      7      0
size-1024(DMA)         0      0   1024    4    1 : tunables   54   27    8 :
slabdata      0      0      0
size-1024            201    252   1024    4    1 : tunables   54   27    8 :
slabdata     63     63      0
size-512(DMA)          0      0    512    8    1 : tunables   54   27    8 :
slabdata      0      0      0
size-512             564   2504    512    8    1 : tunables   54   27    8 :
slabdata    313    313      6
size-256(DMA)          0      0    256   15    1 : tunables  120   60    8 :
slabdata      0      0      0
size-256          274119 3088950    256   15    1 : tunables  120   60    8 :
slabdata 205930 205930      0
size-128(DMA)          0      0    128   31    1 : tunables  120   60    8 :
slabdata      0      0      0
size-128            1673   4588    128   31    1 : tunables  120   60    8 :
slabdata    148    148      0
size-64(DMA)           0      0     64   61    1 : tunables  120   60    8 :
slabdata      0      0      0
size-64             6338  12810     64   61    1 : tunables  120   60    8 :
slabdata    210    210    288
size-32(DMA)           0      0     32  119    1 : tunables  120   60    8 :
slabdata      0      0      0
size-32             9306  15827     32  119    1 : tunables  120   60    8 :
slabdata    133    133      0
kmem_cache           150    150    256   15    1 : tunables  120   60    8 :
slabdata     10     10      0
acnlinb6 102:
Comment 4 Larry Woodman 2008-06-18 09:10:42 EDT
The problem is caused by fragmentation in the slabaches.  Some driver is
allocating 256 byte chunks of slabcache which caused enough fragmentation to
prevent ~200000 pages from being freed.
-----------------------------------------------------------------------------
size-256          274119 3088950    256   15    1 : tunables  120   60    8 :
slabdata 205930 205930      0
-----------------------------------------------------------------------------

Please include an lsmod output so we can see if there is a likely cause in some
driver or other loadable module.

Larry Woodman
Comment 5 Larry Woodman 2008-06-18 09:12:59 EDT
BTW, running the hugemem kernel might prevent this problem from happening since
the kernel space incerases be a factor of 4.

Larry Woodman
Comment 6 Sev Binello 2008-06-18 10:41:58 EDT
Here is a listing of the kernel modules

acnlinb6 102:lsmod
Module                  Size  Used by
nfs                   219561  5
lockd                  65897  2 nfs
nfs_acl                 7745  1 nfs
parport_pc             28033  0
lp                     15661  0
parport                38025  2 parport_pc,lp
netconsole             10845  0
netdump                16225  0
autofs4                26053  2
i2c_dev                14529  0
i2c_core               26049  1 i2c_dev
sunrpc                144037  9 nfs,lockd,nfs_acl
dm_mirror              31557  0
dm_mod                 67177  1 dm_mirror
button                 10705  0
battery                12997  0
ac                      8901  0
joydev                 14465  0
md5                     8129  1
ipv6                  243553  32
ohci_hcd               24273  0
ehci_hcd               32325  0
k8_edac                19173  0
edac_mc                28297  1 k8_edac
tg3                   106437  0
ext3                  119497  6
jbd                    59865  1 ext3
sata_svw               11973  0
libata                105757  1 sata_svw
mptscsih                5441  0
mptsas                 24405  8 mptscsih
mptspi                 13777  1 mptscsih
mptscsi                42193  2 mptsas,mptspi
mptbase                67105  3 mptsas,mptspi,mptscsi
sd_mod                 20545  8
scsi_mod              120269  5 libata,mptsas,mptspi,mptscsi,sd_mod
Comment 7 xiaojun.wu 2009-09-24 02:17:35 EDT
I found this problem at RHEL5.2 32bit .and This is no hugemem kernel on RHEL5.2  ,How can I do that to get it out?

Thanks
Comment 10 Jiri Pallich 2012-06-20 11:56:19 EDT
Thank you for submitting this issue for consideration in Red Hat Enterprise Linux. The release for which you requested us to review is now End of Life. 
Please See https://access.redhat.com/support/policy/updates/errata/

If you would like Red Hat to re-consider your feature request for an active release, please re-open the request via appropriate support channels and provide additional supporting details about the importance of this issue.

Note You need to log in before you can comment on or make changes to this bug.