Bug 1046695
| Summary: | libvirtd: page allocation failure in vhost_net_open | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | R P Herrold <herrold> | ||||
| Component: | libvirt | Assignee: | Ján Tomko <jtomko> | ||||
| Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | Virtualization Bugs <virt-bugs> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 7.0 | CC: | acathrow, dyuan, herrold, mzhan, ydu, zhwang | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2014-01-10 16:39:49 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
I don't see any segfaults in the log, just an 'out of memory' error in the kernel. Could you install the debug infos and provide a backtrace of the hanging libvirtd? ( as described on http://libvirt.org/bugs.html#quality ) I spoke perhaps imprecisely --- something that produces a kernel process trace in the dmesg's is to me, something that needs to be fixed, as the owning process should have caught the error, into syslog / stderror space, rather than kernel 'dmesg' space --- so, not a 'segfault' in the sense of a null pointer ref or such, but still needing an error catch and handle [94429.505462] libvirtd: page allocation failure: order:4, mode:0x1040d0 [94429.505474] CPU: 0 PID: 10985 Comm: libvirtd Not tainted 3.10.0-54.0.1.el7.x86_64 #1 [94429.505478] Hardware name: Dell Inc. Precision WorkStation 380 /0G9322, BIOS A09 01/08/2007 [94429.505481] 0000000000000000 ffff88006af139e0 ffffffff815b4dde ffff88006af13a68 [94429.505488] ffffffff81139910 ffff88020bfd4b38 000000000000000e ffffffff8113c1a6 [94429.505493] ffff880000000040 ffffffff810bb243 0000000000000001 0000000000000010 [94429.505498] Call Trace: [94429.505510] [<ffffffff815b4dde>] dump_stack+0x19/0x1b [94429.505516] [<ffffffff81139910>] warn_alloc_failed+0xf0/0x160 [94429.505521] [<ffffffff8113c1a6>] ? drain_local_pages+0x16/0x20 [94429.505527] [<ffffffff810bb243>] ? on_each_cpu_mask+0x43/0x60 [94429.505532] [<ffffffff8113d8df>] __alloc_pages_nodemask+0x7ff/0xa00 [94429.505538] [<ffffffff81178659>] alloc_pages_current+0xa9/0x170 [94429.505544] [<ffffffff8113889e>] __get_free_pages+0xe/0x50 [94429.505552] [<ffffffff81182bae>] kmalloc_order_trace+0x2e/0xa0 [94429.505560] [<ffffffffa08b8d79>] vhost_net_open+0x29/0x1b0 [vhost_net] [94429.505566] [<ffffffff8137487f>] misc_open+0xaf/0x1c0 [94429.505571] [<ffffffff811a2a12>] chrdev_open+0x92/0x1d0 [94429.505576] [<ffffffff8119bdef>] do_dentry_open+0x1ef/0x2a0 [94429.505581] [<ffffffff811a97e2>] ? __inode_permission+0x52/0xc0 [94429.505585] [<ffffffff811a2980>] ? cdev_put+0x30/0x30 [94429.505589] [<ffffffff8119bed1>] finish_open+0x31/0x40 [94429.505593] [<ffffffff811ad32c>] do_last+0x56c/0x11f0 [94429.505599] [<ffffffff8123dbfc>] ? selinux_file_alloc_security+0x3c/0x60 [94429.505603] [<ffffffff811ae060>] path_openat+0xb0/0x480 [94429.505608] [<ffffffff8115d4b5>] ? handle_mm_fault+0x285/0x3a0 [94429.505612] [<ffffffff811aeb98>] do_filp_open+0x38/0x80 [94429.505617] [<ffffffff811bacc7>] ? __alloc_fd+0xa7/0x130 [94429.505621] [<ffffffff8119d1a9>] do_sys_open+0xe9/0x1c0 [94429.505625] [<ffffffff8119d29e>] SyS_open+0x1e/0x20 [94429.505630] [<ffffffff815c4c99>] system_call_fastpath+0x16/0x1b [94429.505633] Mem-Info: [94429.505635] Node 0 DMA per-cpu: [94429.505639] CPU 0: hi: 0, btch: 1 usd: 0 [94429.505642] CPU 1: hi: 0, btch: 1 usd: 0 I cannot directly reproduce that load environment, as the machine in question has been torn down and re-provisioned I don't think libvirtd possibly can (or wants to) prevent kernel from logging that stack trace. If open returned ENOMEM, libvirt should've caught it and reported an error as well instead of hanging. But I'm afraid there's not much to be done without knowing what exactly was libvirtd doing at the time. Please reopen the bug if you have more information: some steps to reproduce this, a backtrace of the hanging libvirtd or debug logs from libvirtd. |
Created attachment 841954 [details] dmesg dump ... system startup at top, segfault at bottom Description of problem: segfault of unknown genesis Version-Release number of selected component (if applicable): 3.10.0-54.0.1.el7 How reproducible: not clear why it happened -- it is on an active buildbox, and may have a failure to be able to allocate ram down in a SELinux process from the look of it Steps to Reproduce: we restarted the libvirtd when it appeared to be hung, thus the different PIDs cannot provoke at will, but we are getting these with some regularity [root@centos7-64-herc bin]# dmesg | grep libvirtd [94429.505462] libvirtd: page allocation failure: order:4, mode:0x1040d0 [94429.505474] CPU: 0 PID: 10985 Comm: libvirtd Not tainted 3.10.0-54.0.1.el7.x86_64 #1 [94431.926885] libvirtd: page allocation failure: order:4, mode:0x1040d0 [94431.926893] CPU: 1 PID: 10985 Comm: libvirtd Not tainted 3.10.0-54.0.1.el7.x86_64 #1 [95268.783462] libvirtd: page allocation failure: order:4, mode:0x1040d0 [95268.783470] CPU: 1 PID: 10982 Comm: libvirtd Not tainted 3.10.0-54.0.1.el7.x86_64 #1 [95271.606514] libvirtd: page allocation failure: order:4, mode:0x1040d0 [95271.606523] CPU: 1 PID: 10982 Comm: libvirtd Not tainted 3.10.0-54.0.1.el7.x86_64 #1 [95536.611678] libvirtd: page allocation failure: order:4, mode:0x1040d0 [95536.611686] CPU: 1 PID: 10985 Comm: libvirtd Not tainted 3.10.0-54.0.1.el7.x86_64 #1 [96028.740930] libvirtd: page allocation failure: order:4, mode:0x1040d0 [96028.740939] CPU: 0 PID: 10981 Comm: libvirtd Not tainted 3.10.0-54.0.1.el7.x86_64 #1 [96029.440471] libvirtd: page allocation failure: order:4, mode:0x1040d0 [96029.440478] CPU: 1 PID: 10981 Comm: libvirtd Not tainted 3.10.0-54.0.1.el7.x86_64 #1 [root@centos7-64-herc bin]# Actual results: virtualization hangs and segfaults Expected results: no hangs nor segfaults Additional info: