Bug 691697

Summary: udevd invoked oom-killer: gfp_mask=0x800d0, order=0, oom_adj=-17
Product: Red Hat Enterprise Linux 6 Reporter: Chao Ye <cye>
Component: kexec-toolsAssignee: Cong Wang <amwang>
Status: CLOSED NOTABUG QA Contact: Kernel Dump QE <kernel-dump-qe>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.1CC: czhang, phan, rkhan
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-03-30 02:42:19 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Chao Ye 2011-03-29 09:48:49 UTC 
Description of problem:
Kdump failed on dell-pe2800-01.rhts.eng.bos.redhat.com due to oom:
====================================================================
readmem: type_addr: 0, addr:ffff880000040d40, size:4 
_exclude_free_page: Can't get nr_zones. 
create_2nd_bitmap: Can't exclude unnecessary pages. 
Attempting to enter user-space to capture vmcore 
Resetting kernel time value to BIOS time and timezone value to UTC. 
Free memory/Total memory (free %): 35476 / 115040 ( 30.838 ) 
Creating root device. 
Free memory/Total memory (free %): 35452 / 115040 ( 30.8171 ) 
Checking root filesystem. 
fsck (busybox 1.15.1, 2010-11-30 08:10:31 EST) 
e2fsck 1.41.12 (17-May-2010) 
Superblock last mount time is in the future. 
	(by less than a day, probably due to the hardware clock being incorrectly set)  Fix? yes 
 
Superblock last write time is in the future. 
	(by less than a day, probably due to the hardware clock being incorrectly set).  Fix? yes 
 
/dev/mapper/vg_dellpe280001-lv_root: clean, 99569/3276800 files, 1074789/13107200 blocks 
Mounting root filesystem: mount -t ext4 /dev/mapper/vg_dellpe280001-lv_root /sysroot 
EXT4-fs (dm-0): mounted filesystem with ordered data mode 
grep: /proc/cmdline: No such file or directory 
Loading SELINUX policy 
/sbin/load_policy:  Policy is already loaded and initial load requested 
Policy load failed, a relabel will be required on reboot 
Free memory/Total memory (free %): 31300 / 115040 ( 27.2079 ) 
Switching to new root and running init. 
		Welcome to Red Hat Enterprise Linux Server 
Starting udev: udevd invoked oom-killer: gfp_mask=0x800d0, order=0, oom_adj=-17 
udevd cpuset=/ mems_allowed=0 
Pid: 1586, comm: udevd Not tainted 2.6.32-125.el6.x86_64 #1 
Call Trace: 
 [<ffffffff810c0021>] ? cpuset_print_task_mems_allowed+0x91/0xb0 
 [<ffffffff811100db>] ? oom_kill_process+0xcb/0x2e0 
 [<ffffffff811106a0>] ? select_bad_process+0xd0/0x110 
 [<ffffffff81110738>] ? __out_of_memory+0x58/0xc0 
 [<ffffffff81110939>] ? out_of_memory+0x199/0x210 
 [<ffffffff811200fd>] ? __alloc_pages_nodemask+0x80d/0x8b0 
 [<ffffffff81159592>] ? kmem_getpages+0x62/0x170 
 [<ffffffff8115a1aa>] ? fallback_alloc+0x1ba/0x270 
 [<ffffffff81159bff>] ? cache_grow+0x2cf/0x320 
 [<ffffffff81159f29>] ? ____cache_alloc_node+0x99/0x160 
 [<ffffffff8115a88b>] ? kmem_cache_alloc+0x11b/0x190 
 [<ffffffff81266a66>] ? radix_tree_preload+0x36/0xa0 
 [<ffffffff8110d5bb>] ? add_to_page_cache_locked+0x7b/0x140 
 [<ffffffff8110d6ac>] ? add_to_page_cache_lru+0x2c/0x80 
 [<ffffffff811abad6>] ? mpage_readpages+0xb6/0x130 
 [<ffffffffa0026060>] ? ext4_get_block+0x0/0x120 [ext4] 
 [<ffffffff811542aa>] ? alloc_pages_current+0xaa/0x110 
 [<ffffffffa002185d>] ? ext4_readpages+0x1d/0x20 [ext4] 
 [<ffffffff81122b15>] ? __do_page_cache_readahead+0x185/0x210 
 [<ffffffff81122bc1>] ? ra_submit+0x21/0x30 
 [<ffffffff8110e803>] ? filemap_fault+0x4c3/0x500 
 [<ffffffff81136e34>] ? __do_fault+0x54/0x510 
 [<ffffffff811373e7>] ? handle_pte_fault+0xf7/0xb50 
 [<ffffffff8120f3c9>] ? selinux_inode_alloc_security+0x69/0xb0 
 [<ffffffff8104fc08>] ? update_curr+0xf8/0x1e0 
 [<ffffffff8105055a>] ? enqueue_entity+0x13a/0x340 
 [<ffffffff81138018>] ? handle_mm_fault+0x1d8/0x2a0 
 [<ffffffff810414e9>] ? __do_page_fault+0x139/0x480 
 [<ffffffff8118fd20>] ? mntput_no_expire+0x30/0x110 
 [<ffffffff8117dc25>] ? putname+0x35/0x50 
 [<ffffffff81180d5e>] ? sys_symlinkat+0x5e/0x110 
 [<ffffffff814df28e>] ? do_page_fault+0x3e/0xa0 
 [<ffffffff814dc635>] ? page_fault+0x25/0x30 
Mem-Info: 
Node 0 DMA per-cpu: 
CPU    0: hi:    0, btch:   1 usd:   0 
Node 0 DMA32 per-cpu: 
CPU    0: hi:   42, btch:   7 usd:  37 
active_anon:8911 inactive_anon:974 isolated_anon:0 
 active_file:21 inactive_file:6 isolated_file:0 
 unevictable:1 dirty:0 writeback:0 unstable:0 
 free:451 slab_reclaimable:1617 slab_unreclaimable:12020 
 mapped:3 shmem:21 pagetables:734 bounce:0 
Node 0 DMA free:396kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes 
lowmem_reserve[]: 0 126 126 126 
Node 0 DMA32 free:1408kB min:1432kB low:1788kB high:2148kB active_anon:35644kB inactive_anon:3896kB active_file:84kB inactive_file:24kB unevictable:4kB isolated(anon):0kB isolated(file):0kB present:129408kB mlocked:0kB dirty:0kB writeback:0kB mapped:12kB shmem:84kB slab_reclaimable:6468kB slab_unreclaimable:48080kB kernel_stack:1240kB pagetables:2936kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:284 all_unreclaimable? yes 
lowmem_reserve[]: 0 0 0 0 
Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 396kB 
Node 0 DMA32: 4*4kB 12*8kB 1*16kB 0*32kB 0*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1408kB 
55 total pagecache pages 
0 pages in swap cache 
Swap cache stats: add 0, delete 0, find 0/0 
Free swap  = 0kB 
Total swap = 0kB 
41210 pages RAM 
12450 pages reserved 
12222 pages shared 
23187 pages non-shared 
Out of memory: kill process 1490 (rc.sysinit) score 2541 or a child 
......

Version-Release number of selected component (if applicable):
kernel-2.6.32-125
kexec-tools-2.0.0-174

How reproducible:
found on dell-pe2800-01.rhts.eng.bos.redhat.com

Steps to Reproduce:
1.Install RHEL6.1-20110323.1
2.Setup kdump
3.Trigger crash
  
Actual results:
oom

Expected results:
vmcore saved

Additional info:
I checked /proc/cmdline, got crashkernel=129M@0M
https://beaker.engineering.redhat.com/recipes/138480
Comment 2 Cong Wang 2011-03-30 02:42:19 UTC 
It is not a bug, it enters user-space to capture the core, this is why udevd gets running, it is always a cause of OOM. Please increase your reserved memory size.