Hide Forgot
Description of problem: Target system failed kdump testing, as kdump kernel failed to boot due to: load_policy: page allocation failure. order:0, mode:0xd0 Version-Release number of selected component (if applicable): distro: RHEL-6.7 Server x86_64 kernel: 2.6.32-573.26.1.el6 How reproducible: unknown Steps to Reproduce: 1. Install target system system listed in comment#1 with RHEL-6.7 Server x86_64 2. Install kernel-2.6.32-573.26.1.el6 3. Trigger a crash echo c > /proc/sysrq-trigger Actual results: https://beaker.engineering.redhat.com/recipes/2645087 http://beaker-archive.app.eng.bos.redhat.com/beaker-logs/2016/04/12999/1299965/2645087/console.log ---<-snip->--- Out of memory: Kill process 2300 (load_selinux_po) score 1 or sacrifice child Killed process 2302, UID 0, (load_policy) total-vm:115612kB, anon-rss:132kB, file-rss:4kB rport-6:0-13: blocked FC remote port time out: removing rport load_policy: page allocation failure. order:0, mode:0xd0 Pid: 2302, comm: load_policy Not tainted 2.6.32-573.26.1.el6.x86_64 #1 Call Trace: [<ffffffff81137a8c>] ? __alloc_pages_nodemask+0x7dc/0x950 [<ffffffff811373d9>] ? __alloc_pages _nodemask+0x129/66666 vvV&��� � �k+k�;+�� ;+�[ ��&� ��v � [<ffffffff811781ca>] ? fallback_alloc+0x1ba/0x270 [<ffffffff81177c1f>] ? cache_grow+0x2cf/0x320 [<ffffffff81177f49>] ? ____cache_alloc_node+0x99/0x160 [<ffffffff81178d53>] ? kmem_cache_alloc+0x123/0x190 [<ffffffff8124873d>] ? avtab_insert_nonunique+0xcd/0x140 [<ffffffff8100bc0e>] ? apic_timer_interrupt+0xe/0x20 [<ffffffff81252726>] ? cond_insertf+0x116/0x200 [<ffffffff81248a5e>] ? avtab_read_item+0x15e/0x390 [<ffffffff8100bc0e>] ? apic_timer_interrupt+0xe/0x20 [<ffffffff81252610>] ? cond_insertf+0x0/0x200 [<ffffffff81252610>] ? cond_insertf+0x0/0x200 [<ffffffff81248901>] ? avtab_read_item+0x1/0x390 [<ffffffff81252bd1>] ? cond_read_av_list+0xb1/0xd0 [<ffffffff81252e82>] ? cond_read_list+0x162/0x2a0 [<ffffffff8124ca66>] ? policydb_read+0x576/0x1080 [<ffffffff8117b686>] ? kmem_cache_create+0x456/0x5a0 [<ffffffff812507dc>] ? security_load_policy+0x5c/0x410 [<ffffffff81184b47>] ? mem_cgroup_update_file_mapped+0x17/0x90 [<ffffffff81127a37>] ? unlock_page+0x27/0x30 [<ffffffff811526d9>] ? __do_fault+0x469/0x530 [<ffffffff81152897>] ? handle_pte_fault+0xf7/0xb20 [<ffffffff8153927e>] ? thread_return+0x4e/0x7d0 [<ffffffff8117097a>] ? alloc_pages_current+0xaa/0x110 [<ffffffff8100bc0e>] ? apic_timer_interrupt+0xe/0x20 [<ffffffff81153559>] ? handle_mm_fault+0x299/0x3d0 [<ffffffff8104f204>] ? __do_page_fault+0x1f4/0x500 [<ffffffff81134f4e>] ? __get_free_pages+0xe/0x50 [<ffffffff8115fa69>] ? vmap_page_range_noflush+0x279/0x370 [<ffffffff8153f90e>] ? do_page_fault+0x3e/0xa0 [<ffffffff8153cc55>] ? page_fault+0x25/0x30 [<ffffffff812443eb>] ? sel_write_load+0xdb/0x710 [<ffffffff8123f2bb>] ? selinux_file_permission+0xfb/0x150 [<ffffffff81232026>] ? security_file_permission+0x16/0x20 [<ffffffff81192208>] ? vfs_write+0xb8/0x1a0 [<ffffffff811936f6>] ? fget_light_pos+0x16/0x50 [<ffffffff81192d41>] ? sys_write+0x51/0xb0 [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b ---<-snip->--- Expected results: successful boot of kdump kernel and capture of vmcore Additional info:
I don't think this is something for selinux-policy component. Paul, Can you look on this? Thank you.
It appears that the kernel is having a difficult time allocating kernel memory for the SELinux policy load, I'm guessing this is due to memory pressure caused by the fact that kernel is a kdump recovery kernel and a good portion of the physical memory is reserved for dumping to disk. Is there any reason why we are loading the SELinux policy in the kdump kernel? This kernel boot instance is single user and exists only to write the memory image to disk, yes? If so, it seems like we could avoid loading the SELinux policy in this case and avoiding this problem completely.
(In reply to Paul Moore from comment #4) > It appears that the kernel is having a difficult time allocating kernel > memory for the SELinux policy load, I'm guessing this is due to memory > pressure caused by the fact that kernel is a kdump recovery kernel and a > good portion of the physical memory is reserved for dumping to disk. > > Is there any reason why we are loading the SELinux policy in the kdump > kernel? This kernel boot instance is single user and exists only to write > the memory image to disk, yes? If so, it seems like we could avoid loading > the SELinux policy in this case and avoiding this problem completely. All, Adding kdump team for response... ruyang? Best, -pbunyan
We load selinux policy when dumping to rootfs, the purpose is to labeling the dump files correctly. For the oom, Paul, can you retest with kdump.conf option "debug_mem_level 3" so that we can get who is using most of the memory? Thanks Dave
Could you look on comment#6 and retest it with changed kdump.conf? Thanks.
*** Bug 1301089 has been marked as a duplicate of this bug. ***
*** Bug 1323086 has been marked as a duplicate of this bug. ***
This looks more kernel bug then selinux-policy issue. Closing as CANTFIX and this should be fixed in #1379260.
In reply to PaulB from comment #12) > ----------------------------------------------------------- > However, looking at the console log I do see the following: > ----------------------------------------------------------- > ---<-snip->--- > Loading SELINUX policy > load_policy invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0, > oom_score_adj=0 Hi Paul, You should not see these messages with "Release 2.0.0-304". Following patch removes selinux policy load from kdump kernel. 98f374eeff89 mkdumprd: Remove load_selinux_policy script ~Pratyush
(In reply to Pratyush Anand from comment #13) > In reply to PaulB from comment #12) > > ----------------------------------------------------------- > > However, looking at the console log I do see the following: > > ----------------------------------------------------------- > > ---<-snip->--- > > Loading SELINUX policy > > load_policy invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0, > > oom_score_adj=0 > > Hi Paul, > > You should not see these messages with "Release 2.0.0-304". > > Following patch removes selinux policy load from kdump kernel. > > 98f374eeff89 mkdumprd: Remove load_selinux_policy script > > ~Pratyush Pratyush, You are correct. The Call Trace in comment #12 is no longer seen: host: dell-per320-02.khw.lab.eng.bos.redhat.com distro: RHEL-6.8 kernel: 2.6.32-642.el6 kexec-tools: 2.0.0-304.el6 See here: https://beaker.engineering.redhat.com/jobs/1558254 Best, -pbunyan