From Bugzilla Helper: User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1 Description of problem: RH4ES, kernel 2.6.9-42.0.3.ELsmp. Two node cluster with common storage. System was left running with heavy load over night. gfs lock count 100000...300000. Load avg 2..40. Morning check revealed that gfs_tool failed to report counters on one node. gfs_tool worked on second node. gfs mount working. Load was dropped to nill and after several hours (4) gfs_tool started reporting counters again without reboot or remount. gfs_tool counters reported [root@p20-is03 etc]# gfs_tool counters /usr/db/mail/ gfs_tool: can't get counters: Cannot allocate memory See additional information for gfs_tool error message and dmesg printout. Version-Release number of selected component (if applicable): GFS-6.1.5-0 How reproducible: Didn't try Steps to Reproduce: 1. Two node setup 2. Generate heavy load with lock changing between 100000..300000 3. Leave running Actual Results: Expected Results: Additional info: [root@p20-is03 etc]# gfs_tool counters /usr/db/mail/ gfs_tool: can't get counters: Cannot allocate memory [root@p20-is03 etc]# uptime 06:14:20 up 15:52, 1 user, load average: 37.53, 52.79, 58.09 [root@p20-is03 etc]# gfs_tool counters /usr/db/mail/ gfs_tool: can't get counters: Cannot allocate memory [root@p20-is03 etc]# ps -ef | grep [root@p20-is03 etc]# date Wed Feb 21 06:14:29 UTC 2007 [root@p20-is03 etc]# free total used free shared buffers cached Mem: 4025548 1892064 2133484 0 2264 1316976 -/+ buffers/cache: 572824 3452724 Swap: 2040212 0 2040212 [root@p20-is03 etc]# date Wed Feb 21 06:14:39 UTC 2007 [root@p20-is03 etc]# dmesg showed: mptctl: Registered with Fusion MPT base driver mptctl: /dev/mptctl @ (major,minor=10,220) gfs_tool: page allocation failure. order:4, mode:0xd0 [<c0144257>] __alloc_pages+0x28b/0x29d [<c0144281>] __get_free_pages+0x18/0x24 [<c0146d5c>] kmem_getpages+0x1c/0xbb [<c01478aa>] cache_grow+0xab/0x138 [<c0147a9c>] cache_alloc_refill+0x165/0x19d [<c0147e70>] __kmalloc+0x76/0x88 [<f8bf734c>] gi_skeleton+0x4c/0xd3 [gfs] [<f8bf7dbb>] gi_get_counters+0x0/0xb72 [gfs] [<f8bfb15d>] gfs_ioctl_i+0x1b4/0x507 [gfs] [<c015a300>] sys_chown+0x10/0x3c [<f8c06e34>] gfs_ioctl+0x75/0x7f [gfs] [<c016add6>] sys_ioctl+0x227/0x269 [<c02d47cb>] syscall_call+0x7/0xb Mem-info: DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 cpu 0 cold: low 0, high 2, batch 1 cpu 1 hot: low 2, high 6, batch 1 cpu 1 cold: low 0, high 2, batch 1 Normal per-cpu: cpu 0 hot: low 32, high 96, batch 16 cpu 0 cold: low 0, high 32, batch 16 cpu 1 hot: low 32, high 96, batch 16 cpu 1 cold: low 0, high 32, batch 16 HighMem per-cpu: cpu 0 hot: low 32, high 96, batch 16 cpu 0 cold: low 0, high 32, batch 16 cpu 1 hot: low 32, high 96, batch 16 cpu 1 cold: low 0, high 32, batch 16 Free pages: 2079780kB (2064832kB HighMem) Active:212965 inactive:156261 dirty:11699 writeback:0 unstable:0 free:519945 slab:112464 mapped:29893 pagetables:905 DMA free:12548kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:12315 all_unreclaimable? yes protections[]: 0 0 0 Normal free:2400kB min:928kB low:1856kB high:2784kB active:16kB inactive:404408kB present:901120kB pages_scanned:0 all_unreclaimable? no protections[]: 0 0 0 HighMem free:2064832kB min:512kB low:1024kB high:1536kB active:851844kB inactive:220636kB present:3145664kB pages_scanned:0 all_unreclaimable? no protections[]: 0 0 0 DMA: 3*4kB 3*8kB 4*16kB 3*32kB 3*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12548kB Normal: 144*4kB 104*8kB 62*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2400kB HighMem: 11738*4kB 14885*8kB 16677*16kB 3915*32kB 776*64kB 179*128kB 40*256kB 15*512kB 3*1024kB 0*2048kB 345*4096kB = 2064832kB Swap cache: add 0, delete 0, find 0/0, race 0+0 0 bounce buffer pages Free swap: 2040212kB 1015792 pages of RAM 786416 pages of HIGHMEM 9405 reserved pages 357352 pages shared 0 pages swap cached gfs_tool: page allocation failure. order:4, mode:0xd0 [<c0144257>] __alloc_pages+0x28b/0x29d [<c0144281>] __get_free_pages+0x18/0x24 [<c0146d5c>] kmem_getpages+0x1c/0xbb [<c01478aa>] cache_grow+0xab/0x138 [<c0147a9c>] cache_alloc_refill+0x165/0x19d [<c0147e70>] __kmalloc+0x76/0x88 [<f8bf734c>] gi_skeleton+0x4c/0xd3 [gfs] [<f8bf7dbb>] gi_get_counters+0x0/0xb72 [gfs] [<f8bfb15d>] gfs_ioctl_i+0x1b4/0x507 [gfs] [<c015a300>] sys_chown+0x10/0x3c [<f8c06e34>] gfs_ioctl+0x75/0x7f [gfs] [<c016add6>] sys_ioctl+0x227/0x269 [<c02d47cb>] syscall_call+0x7/0xb Mem-info: DMA per-cpu: cpu 0 hot: low 2, high 6, batch 1 cpu 0 cold: low 0, high 2, batch 1 cpu 1 hot: low 2, high 6, batch 1 cpu 1 cold: low 0, high 2, batch 1 Normal per-cpu: cpu 0 hot: low 32, high 96, batch 16 cpu 0 cold: low 0, high 32, batch 16 cpu 1 hot: low 32, high 96, batch 16 cpu 1 cold: low 0, high 32, batch 16 HighMem per-cpu: cpu 0 hot: low 32, high 96, batch 16 cpu 0 cold: low 0, high 32, batch 16 cpu 1 hot: low 32, high 96, batch 16 cpu 1 cold: low 0, high 32, batch 16 Free pages: 2118108kB (2098880kB HighMem) Active:208441 inactive:151319 dirty:11702 writeback:0 unstable:0 free:529527 slab:112390 mapped:29893 pagetables:905 DMA free:12548kB min:16kB low:32kB high:48kB active:0kB inactive:0kB present:16384kB pages_scanned:12321 all_unreclaimable? yes protections[]: 0 0 0 Normal free:6680kB min:928kB low:1856kB high:2784kB active:28kB inactive:400532kB present:901120kB pages_scanned:0 all_unreclaimable? no protections[]: 0 0 0 HighMem free:2098880kB min:512kB low:1024kB high:1536kB active:833736kB inactive:204744kB present:3145664kB pages_scanned:0 all_unreclaimable? no protections[]: 0 0 0 DMA: 3*4kB 3*8kB 4*16kB 3*32kB 3*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 2*4096kB = 12548kB Normal: 1200*4kB 111*8kB 62*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6680kB HighMem: 18992*4kB 15514*8kB 16677*16kB 3915*32kB 776*64kB 179*128kB 40*256kB 15*512kB 3*1024kB 0*2048kB 345*4096kB = 2098880kB Swap cache: add 0, delete 0, find 0/0, race 0+0 0 bounce buffer pages Free swap: 2040212kB 1015792 pages of RAM 786416 pages of HIGHMEM 9405 reserved pages 347901 pages shared 0 pages swap cached
I'll work on this one.
Created attachment 148501 [details] RHEL4 patch to fix the problem Well, gfs_tool was requesting a 64KB chunk of memory for parsing the counters. That got sent up to the kernel level, which requested it in contiguous kernel memory. When the system is under heavy stress, the kernel memory can be tight, and so in some circumstances, there wasn't a 64K chunk of contiguous memory to spare. In reality, the whole thing only needs a little over 1K to do this work, so 64K was about a 50X overkill. This patch backs the memory requirement down to 4K, which is one page on most systems. It should therefore always work unless the system is completely out of memory.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2007-0139.html