Hide Forgot
Description of problem: Kdump failed on dell-pe2800-01.rhts.eng.bos.redhat.com due to oom: ==================================================================== readmem: type_addr: 0, addr:ffff880000040d40, size:4 _exclude_free_page: Can't get nr_zones. create_2nd_bitmap: Can't exclude unnecessary pages. Attempting to enter user-space to capture vmcore Resetting kernel time value to BIOS time and timezone value to UTC. Free memory/Total memory (free %): 35476 / 115040 ( 30.838 ) Creating root device. Free memory/Total memory (free %): 35452 / 115040 ( 30.8171 ) Checking root filesystem. fsck (busybox 1.15.1, 2010-11-30 08:10:31 EST) e2fsck 1.41.12 (17-May-2010) Superblock last mount time is in the future. (by less than a day, probably due to the hardware clock being incorrectly set) Fix? yes Superblock last write time is in the future. (by less than a day, probably due to the hardware clock being incorrectly set). Fix? yes /dev/mapper/vg_dellpe280001-lv_root: clean, 99569/3276800 files, 1074789/13107200 blocks Mounting root filesystem: mount -t ext4 /dev/mapper/vg_dellpe280001-lv_root /sysroot EXT4-fs (dm-0): mounted filesystem with ordered data mode grep: /proc/cmdline: No such file or directory Loading SELINUX policy /sbin/load_policy: Policy is already loaded and initial load requested Policy load failed, a relabel will be required on reboot Free memory/Total memory (free %): 31300 / 115040 ( 27.2079 ) Switching to new root and running init. Welcome to Red Hat Enterprise Linux Server Starting udev: udevd invoked oom-killer: gfp_mask=0x800d0, order=0, oom_adj=-17 udevd cpuset=/ mems_allowed=0 Pid: 1586, comm: udevd Not tainted 2.6.32-125.el6.x86_64 #1 Call Trace: [<ffffffff810c0021>] ? cpuset_print_task_mems_allowed+0x91/0xb0 [<ffffffff811100db>] ? oom_kill_process+0xcb/0x2e0 [<ffffffff811106a0>] ? select_bad_process+0xd0/0x110 [<ffffffff81110738>] ? __out_of_memory+0x58/0xc0 [<ffffffff81110939>] ? out_of_memory+0x199/0x210 [<ffffffff811200fd>] ? __alloc_pages_nodemask+0x80d/0x8b0 [<ffffffff81159592>] ? kmem_getpages+0x62/0x170 [<ffffffff8115a1aa>] ? fallback_alloc+0x1ba/0x270 [<ffffffff81159bff>] ? cache_grow+0x2cf/0x320 [<ffffffff81159f29>] ? ____cache_alloc_node+0x99/0x160 [<ffffffff8115a88b>] ? kmem_cache_alloc+0x11b/0x190 [<ffffffff81266a66>] ? radix_tree_preload+0x36/0xa0 [<ffffffff8110d5bb>] ? add_to_page_cache_locked+0x7b/0x140 [<ffffffff8110d6ac>] ? add_to_page_cache_lru+0x2c/0x80 [<ffffffff811abad6>] ? mpage_readpages+0xb6/0x130 [<ffffffffa0026060>] ? ext4_get_block+0x0/0x120 [ext4] [<ffffffff811542aa>] ? alloc_pages_current+0xaa/0x110 [<ffffffffa002185d>] ? ext4_readpages+0x1d/0x20 [ext4] [<ffffffff81122b15>] ? __do_page_cache_readahead+0x185/0x210 [<ffffffff81122bc1>] ? ra_submit+0x21/0x30 [<ffffffff8110e803>] ? filemap_fault+0x4c3/0x500 [<ffffffff81136e34>] ? __do_fault+0x54/0x510 [<ffffffff811373e7>] ? handle_pte_fault+0xf7/0xb50 [<ffffffff8120f3c9>] ? selinux_inode_alloc_security+0x69/0xb0 [<ffffffff8104fc08>] ? update_curr+0xf8/0x1e0 [<ffffffff8105055a>] ? enqueue_entity+0x13a/0x340 [<ffffffff81138018>] ? handle_mm_fault+0x1d8/0x2a0 [<ffffffff810414e9>] ? __do_page_fault+0x139/0x480 [<ffffffff8118fd20>] ? mntput_no_expire+0x30/0x110 [<ffffffff8117dc25>] ? putname+0x35/0x50 [<ffffffff81180d5e>] ? sys_symlinkat+0x5e/0x110 [<ffffffff814df28e>] ? do_page_fault+0x3e/0xa0 [<ffffffff814dc635>] ? page_fault+0x25/0x30 Mem-Info: Node 0 DMA per-cpu: CPU 0: hi: 0, btch: 1 usd: 0 Node 0 DMA32 per-cpu: CPU 0: hi: 42, btch: 7 usd: 37 active_anon:8911 inactive_anon:974 isolated_anon:0 active_file:21 inactive_file:6 isolated_file:0 unevictable:1 dirty:0 writeback:0 unstable:0 free:451 slab_reclaimable:1617 slab_unreclaimable:12020 mapped:3 shmem:21 pagetables:734 bounce:0 Node 0 DMA free:396kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes lowmem_reserve[]: 0 126 126 126 Node 0 DMA32 free:1408kB min:1432kB low:1788kB high:2148kB active_anon:35644kB inactive_anon:3896kB active_file:84kB inactive_file:24kB unevictable:4kB isolated(anon):0kB isolated(file):0kB present:129408kB mlocked:0kB dirty:0kB writeback:0kB mapped:12kB shmem:84kB slab_reclaimable:6468kB slab_unreclaimable:48080kB kernel_stack:1240kB pagetables:2936kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:284 all_unreclaimable? yes lowmem_reserve[]: 0 0 0 0 Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 396kB Node 0 DMA32: 4*4kB 12*8kB 1*16kB 0*32kB 0*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1408kB 55 total pagecache pages 0 pages in swap cache Swap cache stats: add 0, delete 0, find 0/0 Free swap = 0kB Total swap = 0kB 41210 pages RAM 12450 pages reserved 12222 pages shared 23187 pages non-shared Out of memory: kill process 1490 (rc.sysinit) score 2541 or a child ...... Version-Release number of selected component (if applicable): kernel-2.6.32-125 kexec-tools-2.0.0-174 How reproducible: found on dell-pe2800-01.rhts.eng.bos.redhat.com Steps to Reproduce: 1.Install RHEL6.1-20110323.1 2.Setup kdump 3.Trigger crash Actual results: oom Expected results: vmcore saved Additional info: I checked /proc/cmdline, got crashkernel=129M@0M https://beaker.engineering.redhat.com/recipes/138480
It is not a bug, it enters user-space to capture the core, this is why udevd gets running, it is always a cause of OOM. Please increase your reserved memory size.