Bug 1220993

Summary: No VM's core dumps after kill vm EL7
Product: Red Hat Enterprise Virtualization Manager Reporter: Douglas Schilling Landgraf <dougsland>
Component: vdsmAssignee: Yaniv Bronhaim <ybronhei>
Status: CLOSED ERRATA QA Contact: Jiri Belka <jbelka>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.5.1CC: asegundo, bazulay, danken, eedri, fdeutsch, gklein, jdenemar, lpeer, lsurette, oourfali, yeylon, ykaul, ylavi
Target Milestone: ovirt-3.6.0-rcKeywords: ZStream
Target Release: 3.6.0   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: ovirt-3.6.0-2.1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1229312 (view as bug list) Environment:
Last Closed: 2016-03-09 19:39:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1215554, 1229312    

Description Douglas Schilling Landgraf 2015-05-13 04:18:43 UTC
Description of problem:

No VM's core dumps after kill vm.

Red Hat Enterprise Virtualization Hypervisor 7.1 (20150420.0.el7ev)
ovirt-node-3.2.2-3.el7.noarch
vdsm-4.16.13.1-1.el7ev.x86_64
ovirt-node-selinux-3.2.2-3.el7.noarch
selinux-policy-3.13.1-23.el7.noarch
selinux-policy-targeted-3.13.1-23.el7.noarch

# rpm -qf /var/log/core/
vdsm-4.16.13.1-1.el7ev.x86_64

Red Hat Enterprise Virtualization Manager Version: 3.5.1-0.4.el6ev

Steps to Reproduce:
1. Install RHEVH-7.1-20150420.0
2. Register to RHEVM3.5.1.
3. Create VM.
4. Run command "pkill -ABRT -f 'qemu-kvm'" on host.
5. Run  ll /var/log/core/ on host.

Additional info:

There is a vdsm patch that starts the coredump configuration [1] with systemd, however it's required one additional step, which is add into libvirtd.service LimitCORE=infinity as the libvirt will trigger qemu-kvm.

i.e: /usr/lib/systemd/system/libvirtd.service
# cat ./usr/lib/systemd/system/libvirtd.service
[Unit]
Description=Virtualization daemon
Before=libvirt-guests.service
After=network.target
After=dbus.service
After=iscsid.service
After=apparmor.service
Documentation=man:libvirtd(8)
Documentation=http://libvirt.org

[Service]
Type=notify
LimitCORE=infinity <-------------------  Added
EnvironmentFile=-/etc/sysconfig/libvirtd
ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
# Override the maximum number of opened files
#LimitNOFILE=2048

[Install]
WantedBy=multi-user.target


[1]supervdsm: enable coredumping
https://gerrit.ovirt.org/#/c/35745/

Comment 2 Yaniv Bronhaim 2015-05-17 11:56:13 UTC
I don't think vdsm shouldn't touch libvirtd.service. I prefer to hear more views - ccing Danken to hear his opinion and Jiri to understand why the default is with coredump limitation and if its more reasonable to change that default.

Comment 3 Yaniv Bronhaim 2015-05-17 15:47:19 UTC
s/shouldn't/should ^

Comment 4 Jiri Denemark 2015-05-18 08:42:47 UTC
Normal systems (RHEL, for example) run abrt daemon which will handle the crash and process or store the core dump depending on its configuration regardless on LimitCORE settings. On hosts without abrt daemon, setting LimitCore to unlimited by default is not a very good idea since core files may be really big, especially those of qemu-kvm processes running big guests. And I believe even vdsm should not enable them by default. You should rather start using abrt which should be even able to process the core file without storing it on a disk first (although this functionality is not available in RHEL 7.1, AFAIK).

However, if you still want to set LimitCORE=unlimited (either by default or on demand), you should not mess with /usr/lib/systemd/system/libvirtd.service. You should do this by creating a new file in /etc/systemd/system/libvirtd.service.d/. For example,

$ cat /etc/systemd/system/libvirtd.service.d/unlimited-core.conf
[Service]
LimitCORE=unlimited

Comment 5 Dan Kenigsberg 2015-05-18 11:55:22 UTC
Yaniv, as Jiri hints, the best solution is to implement bug 917062. But until this happens, we should solve our el7 regression, and add the suggested libvirt.conf to our libvirt configurator.

Comment 6 Yaniv Bronhaim 2015-05-20 09:20:48 UTC
Thanks for the replies! Moving to 3.6 and we'll consider to add it soon. I prefer not to introduce this change in 3.5

Comment 7 Jiri Belka 2015-06-05 08:35:05 UTC
FYI I reported this as well and got closed by mskrivan@, he told me it has never been intended to have core files. See https://bugzilla.redhat.com/show_bug.cgi?id=1220860

Comment 9 Jiri Belka 2015-06-25 08:51:29 UTC
ok, vdsm-4.17.0-1028.git7f5e1f2.el7.noarch

# rpm -qf /etc/systemd/system/libvirtd.service.d/unlimited-core.conf 
vdsm-4.17.0-1028.git7f5e1f2.el7.noarch

# cat /etc/systemd/system/libvirtd.service.d/unlimited-core.conf 
## beginning of configuration section by vdsm
[Service]
LimitCORE=infinity
## end of configuration section by vdsm

# file /var/log/core/core.519.1435222212.dump 
/var/log/core/core.519.1435222212.dump: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/libexec/qemu-kvm -name test -S -machine rhel6.5.0,accel=kvm,usb=off -cpu S'

Comment 11 errata-xmlrpc 2016-03-09 19:39:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0362.html