Description of problem: No VMs core dumps because of missing SELinux local policies. type=AVC msg=audit(1401786061.559:42800): avc: denied { getattr } for pid=10087 comm="logrotate" path="/var/log/core/core.9784.1401785994.dump" dev=dm-9 ino=3670018 scont ext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:virt_cache_t:s0:c462,c953 tclass=file # cat /etc/redhat-release Red Hat Enterprise Virtualization Hypervisor release 6.5 (20140527.0.el6ev) [root@dell-r210ii-04 ~]# semanage fcontext -l | grep /var/log/core [root@dell-r210ii-04 ~]# # cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.5 (Santiago) [root@srh-03 ~]# semanage fcontext -l |grep /var/log/core /var/log/core(/.*)? all files system_u:object_r:virt_cache_t:s0 [root@srh-03 ~]# On normal RHEL qemu is able to write into 'virt_cache_t' dirs. On RHEVH it cannot. 'semanage fcontext -a -t virt_cache_t "/var/log/core(./*)?"' solves the problem. Without core dumps one is not able to debug VM crashes, like recent VMs' craches because of spice - BZ1092380. Version-Release number of selected component (if applicable): selinux-policy-3.7.19-231.el6_5.3.noarch selinux-policy-targeted-3.7.19-231.el6_5.3.noarch vdsm-python-4.14.7-3.el6ev.x86_64 vdsm-python-zombiereaper-4.14.7-3.el6ev.noarch vdsm-xmlrpc-4.14.7-3.el6ev.noarch vdsm-4.14.7-3.el6ev.x86_64 vdsm-reg-4.14.7-3.el6ev.noarch vdsm-hook-vhostmd-4.14.7-3.el6ev.noarch vdsm-cli-4.14.7-3.el6ev.noarch How reproducible: 100% Steps to Reproduce: 1. pkill -ABRT -f 'qemu-kvm' 2. 3. Actual results: no core dump Expected results: VMs core dumps available Additional info:
Ah, relevant bug about core dumping qemu processes (because of spice) is in fact BZ1072101, not the BZ written in #0 (cut&paste issue).
(In reply to Jiri Belka from comment #0) > Description of problem: > No VMs core dumps because of missing SELinux local policies. > > type=AVC msg=audit(1401786061.559:42800): avc: denied { getattr } for > pid=10087 comm="logrotate" path="/var/log/core/core.9784.1401785994.dump" > dev=dm-9 ino=3670018 scont > ext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 > tcontext=system_u:object_r:virt_cache_t:s0:c462,c953 tclass=file > > # cat /etc/redhat-release > Red Hat Enterprise Virtualization Hypervisor release 6.5 (20140527.0.el6ev) > [root@dell-r210ii-04 ~]# semanage fcontext -l | grep /var/log/core > [root@dell-r210ii-04 ~]# > > # cat /etc/redhat-release > Red Hat Enterprise Linux Server release 6.5 (Santiago) > [root@srh-03 ~]# semanage fcontext -l |grep /var/log/core > /var/log/core(/.*)? all files > system_u:object_r:virt_cache_t:s0 > [root@srh-03 ~]# Hey Jiri, the paste above looks like you were using RHEL. Were you using RHEV-H and if - please tell me which RHEV-H and which ovirt-node version you were using.
#0 contains outputs from RHEVH and RHEL for comparison. # cat /etc/redhat-release Red Hat Enterprise Virtualization Hypervisor release 6.5 (20140527.0.el6ev) # rpm -qa | egrep "ovirt|redhat-release" redhat-release-server-6Server-6.5.0.2.el6_5.x86_64 ovirt-host-deploy-offline-1.2.1-1.el6ev.x86_64 ovirt-node-plugin-cim-3.0.1-18.el6_5.10.noarch ovirt-node-3.0.1-18.el6_5.10.noarch ovirt-node-plugin-rhn-3.0.1-18.el6_5.10.noarch ovirt-node-plugin-snmp-3.0.1-18.el6_5.10.noarch ovirt-node-selinux-3.0.1-18.el6_5.10.noarch ovirt-node-plugin-vdsm-0.1.1-22.el6ev.noarch
Hey Jiri, thanks.
RHEV-H QE can reproduce this bug. Test version: rhev-hypervisor6-6.5-20140624.0.el6ev ovirt-node-3.0.1-18.el6_5.11.noarch vdsm-4.14.7-3.el6ev.x86_64 RHEVM av10 selinux-policy-3.7.19-231.el6_5.3.noarch selinux-policy-targeted-3.7.19-231.el6_5.3.noarch vdsm-python-4.14.7-3.el6ev.x86_64 vdsm-python-zombiereaper-4.14.7-3.el6ev.noarch vdsm-xmlrpc-4.14.7-3.el6ev.noarch vdsm-4.14.7-3.el6ev.x86_64 vdsm-reg-4.14.7-3.el6ev.noarch vdsm-hook-vhostmd-4.14.7-3.el6ev.noarch vdsm-cli-4.14.7-3.el6ev.noarch # cat /etc/redhat-release Red Hat Enterprise Virtualization Hypervisor release 6.5 (20140624.0.el6ev) (Edited) [root@localhost admin]# semanage fcontext -l | grep /var/log/core # cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.5 (Santiago) [root@dell-op740-03 tmp]# semanage fcontext -l |grep /var/log/core /var/log/core(/.*)? all files system_u:object_r:virt_cache_t:s0 Test steps: 1. Install RHEV-H and register to RHEVM. 2. Create VM. 3. Run command "pkill -ABRT -f 'qemu-kvm'" on host. Test result: no core dump
Hi Jbelka, I append 'semanage fcontext -a -t virt_cache_t "/var/log/core(./*)?"' to RHEV-H host, and then run command "pkill -ABRT -f 'qemu-kvm'", but there is still no core dump in vm. Can you point me which step was wrong? Test steps: 1. Install RHEV-H and register to RHEVM. 2. Create VM and config kdump to local(/data/core) 3. Run workaround: 'semanage fcontext -a -t virt_cache_t "/var/log/core(./*)?"' to RHEV-H host, 4. Run command "pkill -ABRT -f 'qemu-kvm'" on host. 5. Start the VM again, check the core dump file under(/data/core) Test result: step4, all VMs were killed. step5, no core dump. Thanks!
/data/core is probably not correctly labeled. See audit.log for denials.
Finally the core dump file can generate successful. Details workaround is here: #'semanage fcontext -a -t virt_cache_t "/var/log/core(./*)?" #restorecon -RFv /var/log/core #ls -ldZ /data/core /var/log/core drwxrwxrwt. root root system_u:object_r:virt_cache_t:s0 /data/core drwxrwxrwt. root root system_u:object_r:virt_cache_t:s0 /var/log/core
Test version: rhev-hypervisor6-6.6-20150112.0 ovirt-node-3.1.0-0.42.20150109gitd06b7c5.el6.noarch selinux-policy-3.7.19-260.el6_6.1.noarch selinux-policy-targeted-3.7.19-260.el6_6.1.noarch Test steps: 1. Install RHEV-H and register to RHEVM. 2. Create VM. 3. Run command "pkill -ABRT -f 'qemu-kvm'" on host. 4. Run ll /var/log/core/ on host. Test result: VMs core dumps available # ll /var/log/core/ total 959676 -rw-------. 1 qemu qemu 1302220800 2015-01-14 07:23 core.21219.1421220217.dump So the bug was fixed with above rhevh6.0 build. =================================================== RHEVH7.0 rhev-hypervisor7-7.0-20150112.0 ovirt-node-3.1.0-0.42.20150109gitd06b7c5.el7.noarch selinux-policy-3.12.1-153.el7_0.13.noarch selinux-policy-targeted-3.12.1-153.el7_0.13.noarch Still no VM's core dumps in RHEV-H 7.0, I noticed that the bug fixed in version: rhev-hypervisor6-6.6-20141218.0.iso rhev-hypervisor7-7.0-20141218.0.iso. So can I verify this bug directly according the test result of rhevh6.6? Or assigned this bug until RHEVH7.0 fix this issue?
Created attachment 980419 [details] SElinux.tar.gz
Chen, does the core dump work in permissive mode? So booting with enforcing=0
(In reply to Fabian Deutsch from comment #15) > Chen, does the core dump work in permissive mode? So booting with enforcing=0 Test version: Red Hat Enterprise Virtualization Hypervisor release 7.0 (20150114.0.2.el7ev) ovirt-node-3.2.1-1.el7.noarch selinux-policy-targeted-3.12.1-153.el7_0.13.noarch selinux-policy-3.12.1-153.el7_0.13.noarch # cat /proc/cmdline BOOT_IMAGE=/vmlinuz0 root=live:LABEL=Root ro rootfstype=auto rootflags=ro rd.live.image rd.live.check crashkernel=256M elevator=deadline quiet max_loop=256 rhgb rd.luks=0 rd.md=0 rd.dm=0 nomodeset enforcing=0 Test steps: 1. Install RHEV-H add booting with enforcing=0 2. Register to RHEVM. 3. Create VM. 4. Run command "pkill -ABRT -f 'qemu-kvm'" on host. 5. Run ll /var/log/core/ on host. Test result: Still no VM's core dumps in RHEV-H 7.0,
Okay, then it is not an selinux issue - and thus it can not be said if this bug is fixed or not, thus I am setting it back to ON_QA Miroslav, can you tell how we can debug this problem?
I check the issue with Chen and it looks like ulimit problem. Waiting for test results.
Do we need ABRT for dump? I can't see it in the system.
No, ABRT is not needed. But the change to security.conf had no immediate effect in comment 19, that's why the dump did not work. Setting ulimit -S -c unlimited enables core dumps. Steps to reproduce: 1. Login as root 2. ulimit -S -c unlimited 3. /usr/libexec/qemu-kvm & 4. pkill -ABRT qemu-kvm 5. ls -shal -var/log/core/ 5 Will show a core dump But note that ulimits for services are set elsewhere. Does this bug also happen on RHEL?
On RHEL both, soft and hard limits are set to unlimited. On RHEV-H the soft limit is set to 0.
(In reply to Fabian Deutsch from comment #22) > On RHEL both, soft and hard limits are set to unlimited. On RHEV-H the soft > limit is set to 0. That was not correct. By default it is the same as on RHEV-H. Chen, can you reproduce this bug on RHEL 7 as well?
Verify it with RHEL 6.6 and 7, on RHEL 6.6 I got core dump but on RHEL 7 the process killed and the core dump did not created. outputs: RHEL 6.6: [root@dingo-vdsc ~]# ps -ef | grep qemu qemu 14748 1 34 14:07 ? 00:05:06 /usr/libexec/qemu-kvm -name VM-6.6_test -S -M rhel6.5.0 -cpu Penryn -enable-kvm -m 1024 -realtime mlock=off -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 -uuid 45a46df9-166d-42cc- ... ... [root@dingo-vdsc ~]# pkill -ABRT -f 'qemu' [root@dingo-vdsc ~]# ps -ef | grep qemu qemu 16336 14896 0 Jan20 ? 00:00:00 [supervdsmServer] <defunct> root 16839 16588 0 14:22 pts/0 00:00:00 grep qemu [root@dingo-vdsc ~]# ll /var/log/core/ total 1380304 -rw-------. 1 qemu qemu 1618182144 Jan 22 14:22 core.14748.1421929347.dump -rw-------. 1 vdsm kvm 189968384 Dec 5 14:49 core.9864.1417783798.dump [root@dingo-vdsc ~]# date Thu Jan 22 14:23:23 IST 2015 [root@dingo-vdsc ~]# RHEL 7: [root@puma18 ~]# ps -ef | grep qemu qemu 18122 1 15 14:31 ? 00:00:19 /usr/libexec/qemu-kvm -name VM-7_test -S -machine rhel6.5.0,accel=kvm,usb=off -cpu Penryn -m 1024 -realtime mlock=off -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 -uuid ... ... [root@puma18 ~]# pkill -ABRT -f 'qemu' [root@puma18 ~]# ll /var/log/core/ total 0 [root@puma18 ~]# ps -ef | grep qemu qemu 15096 14849 0 14:07 ? 00:00:00 [supervdsmServer] <defunct> root 18531 17658 0 14:34 pts/0 00:00:00 grep --color=auto qemu
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2015-0160.html