Bug 1220860

Summary: no core files when ABRT signal is sent to qemu-kvm process
Product: Red Hat Enterprise Virtualization Manager Reporter: Jiri Belka <jbelka>
Component: vdsmAssignee: Nobody <nobody>
Status: CLOSED DUPLICATE QA Contact: Ilanit Stein <istein>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.5.3CC: bazulay, ecohen, gklein, huding, juzhang, lpeer, lsurette, michal.skrivanek, virt-maint, xfu, yeylon
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: virt
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-06-02 08:48:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1131460    

Description Jiri Belka 2015-05-12 15:53:07 UTC 
Description of problem:

Red Hat Enterprise Linux Server release 7.1 (Maipo)

I sent ABRT signal to a qemu-kvm process and got no core file.

May 12 17:43:21 dell-r210ii-04 kvm: 1 guest now active
May 12 17:43:21 dell-r210ii-04 journal: metadata not found: Requested metadata element is not present
iMay 12 17:43:34 dell-r210ii-04 journal: internal error: End of file from monitor
May 12 17:43:34 dell-r210ii-04 kernel: ovirtmgmt: port 2(vnet0) entered disabled state
May 12 17:43:34 dell-r210ii-04 journal: Unable to get index for interface vnet0: No such device
May 12 17:43:34 dell-r210ii-04 kernel: device vnet0 left promiscuous mode
May 12 17:43:34 dell-r210ii-04 kernel: ovirtmgmt: port 2(vnet0) entered disabled state
May 12 17:43:34 dell-r210ii-04 kvm: 0 guests now active
May 12 17:43:34 dell-r210ii-04 systemd-machined: Machine qemu-server terminated.
May 12 17:43:34 dell-r210ii-04 journal: vdsm root WARNING File: /var/lib/libvirt/qemu/channels/16e0e48c-a1ee-4bad-929e-e1fa24638a28.com.redhat.rhevm.vdsm already removed
May 12 17:43:34 dell-r210ii-04 journal: vdsm root WARNING File: /var/lib/libvirt/qemu/channels/16e0e48c-a1ee-4bad-929e-e1fa24638a28.org.qemu.guest_agent.0 already removed

[root@dell-r210ii-04 yum.repos.d]# sysctl -a | grep core_
kernel.core_pattern = /var/log/core/core.%p.%t.dump
kernel.core_pipe_limit = 0
kernel.core_uses_pid = 1

[root@dell-r210ii-04 yum.repos.d]# ls -ldZ /var/log/core
drwxrwxrwt. root root system_u:object_r:virt_cache_t:s0 /var/log/core

[root@dell-r210ii-04 yum.repos.d]# find /var/log/core/ -ls
403483988    0 drwxrwxrwt   2 root     root            6 Apr 11 08:15 /var/log/core/

kernel-3.10.0-229.1.2.el7.x86_64
systemd-208-20.el7_1.2.x86_64
qemu-kvm-rhev-2.1.2-23.el7_1.2.x86_64

Version-Release number of selected component (if applicable):
kernel-3.10.0-229.1.2.el7.x86_64
systemd-208-20.el7_1.2.x86_64
qemu-kvm-rhev-2.1.2-23.el7_1.2.x86_64

How reproducible:
100%

Steps to Reproduce:
1. run a VM under rhel7 / ovirt 3.6 env
2. pkill -ABRT qemu-kvm
3. find /var/log/core

Actual results:
no core file

Expected results:
a core file for aborted qemu-kvm process

Additional info:
Comment 1 Jiri Belka 2015-05-26 15:30:36 UTC 
so libvirt doesn't send any resource limits. doing detective work, it is (super)vdsmd which sets resource limits:

# grep LimitCORE /usr/lib/systemd/system/{super,}vdsmd.service 
/usr/lib/systemd/system/supervdsmd.service:LimitCORE=infinity
/usr/lib/systemd/system/vdsmd.service:LimitCORE=infinity

but using this wrapper i see `ulimit -c' is '0':

[root@dell-r210ii-04 libexec]# cat qemu-kvm
#!/bin/sh

ulimit -c > /tmp/ulimit
exec /usr/bin/strace -o /tmp/strace/strace.out -f /usr/libexec/qemu-kvm.orig "$@"

i tried with and without 'core_dump_enable = true' in vdsm.conf

so imo settings resource limit is lost somewhere in translation :)
Comment 2 Jiri Belka 2015-05-26 15:33:03 UTC 
FYI it works ok on RHEL6:

- RHEL6:

supervdsm - DAEMON_COREFILE_LIMIT=unlimited... daemon function reads that
vdsm - DAEMON_COREFILE_LIMIT=unlimite.. daemon function reads that

- RHEL7:

supervdsmd.service - LimitCORE=infinity
vdsmd.service - LimitCORE=infitity
Comment 3 Jiri Belka 2015-05-26 15:34:57 UTC 
i also tried tracing it but only (semi-)relevant line is:

6846  getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
Comment 4 Michal Skrivanek 2015-06-02 08:48:48 UTC 
we don't do core dumps for qemu process

see bug 917062 for ABRT integration which would allow proper troubleshooting info

*** This bug has been marked as a duplicate of bug 917062 ***