Red Hat Bugzilla – Bug 1265547
[vdsm] logrotate for /var/log/core again not working
Last modified: 2016-04-12 02:56:10 EDT
Description of problem: logrotate for /var/log/core again not working. explanation: /var/log/core contant is DAC 600. logrotate script is run as vdsm:kvm, vdsm has additional group 'qemu' so it can enter the path but the script got egid only in this context and thus cannot read files as group permission are missing. # logrotate -v -f /etc/vdsm/logrotate/vdsm 2>&1 | sed -n '/^rotating pattern: \/var\/log\/core/,$p' rotating pattern: /var/log/core/*.dump forced from command line (1 rotations) empty log files are rotated, old logs are removed switching euid to 36 and egid to 36 considering log /var/log/core/core.24839.1442294276.dump log needs rotating considering log /var/log/core/core.60957.1442272065.dump log needs rotating rotating log /var/log/core/core.24839.1442294276.dump, log->rotateCount is 1 dateext suffix '-20150923' glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' renaming /var/log/core/core.24839.1442294276.dump.1.xz to /var/log/core/core.24839.1442294276.dump.2.xz (rotatecount 1, logstart 1, i 1), old log /var/log/core/core.24839.1442294276.dump.1.xz does not exist renaming /var/log/core/core.24839.1442294276.dump.0.xz to /var/log/core/core.24839.1442294276.dump.1.xz (rotatecount 1, logstart 1, i 0), old log /var/log/core/core.24839.1442294276.dump.0.xz does not exist log /var/log/core/core.24839.1442294276.dump.2.xz doesn't exist -- won't try to dispose of it renaming /var/log/core/core.24839.1442294276.dump to /var/log/core/core.24839.1442294276.dump.1 error: failed to rename /var/log/core/core.24839.1442294276.dump to /var/log/core/core.24839.1442294276.dump.1: Operation not permitted rotating log /var/log/core/core.60957.1442272065.dump, log->rotateCount is 1 dateext suffix '-20150923' glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' renaming /var/log/core/core.60957.1442272065.dump.1.xz to /var/log/core/core.60957.1442272065.dump.2.xz (rotatecount 1, logstart 1, i 1), old log /var/log/core/core.60957.1442272065.dump.1.xz does not exist renaming /var/log/core/core.60957.1442272065.dump.0.xz to /var/log/core/core.60957.1442272065.dump.1.xz (rotatecount 1, logstart 1, i 0), old log /var/log/core/core.60957.1442272065.dump.0.xz does not exist log /var/log/core/core.60957.1442272065.dump.2.xz doesn't exist -- won't try to dispose of it renaming /var/log/core/core.60957.1442272065.dump to /var/log/core/core.60957.1442272065.dump.1 error: failed to rename /var/log/core/core.60957.1442272065.dump to /var/log/core/core.60957.1442272065.dump.1: Operation not permitted switching euid to 0 and egid to 0 # mail Message 45: From root@ibm-p8-rhevm-04.rhts.eng.bos.example.com Wed Sep 16 14:01:01 2015 Return-Path: <root@ibm-p8-rhevm-04.rhts.eng.bos.example.com> X-Original-To: root Delivered-To: root@ibm-p8-rhevm-04.rhts.eng.bos.example.com From: "(Cron Daemon)" <root@ibm-p8-rhevm-04.rhts.eng.bos.example.com> To: root@ibm-p8-rhevm-04.rhts.eng.bos.example.com Subject: Cron <root@ibm-p8-rhevm-04> run-parts /etc/cron.hourly Content-Type: text/plain; charset=UTF-8 Auto-Submitted: auto-generated Precedence: bulk X-Cron-Env: <XDG_SESSION_ID=19041> X-Cron-Env: <XDG_RUNTIME_DIR=/run/user/0> X-Cron-Env: <LANG=en_US.UTF-8> X-Cron-Env: <SHELL=/bin/bash> X-Cron-Env: <PATH=/sbin:/bin:/usr/sbin:/usr/bin> X-Cron-Env: <MAILTO=root> X-Cron-Env: <HOME=/root> X-Cron-Env: <LOGNAME=root> X-Cron-Env: <USER=root> Date: Wed, 16 Sep 2015 14:01:01 -0500 (CDT) Status: RO /etc/cron.hourly/vdsm-logrotate: error: failed to rename /var/log/core/core.24839.1442294276.dump to /var/log/core/core.24839.1442294276.dump.1: Operation not permitted error: failed to rename /var/log/core/core.60957.1442272065.dump to /var/log/core/core.60957.1442272065.dump.1: Operation not permitted # ls -ld /var/log/core drwxrwxrwt 2 qemu qemu 83 Sep 23 03:09 /var/log/core [root@ibm-p8-rhevm-04 ~]# getenforce Disabled [root@ibm-p8-rhevm-04 ~]# ls -l /var/log/core/ total 2186272 -rw------- 1 qemu qemu 1209073664 Sep 15 00:17 core.24839.1442294276.dump -rw------- 1 qemu qemu 1209073664 Sep 14 18:07 core.60957.1442272065.dump # su -s /bin/bash - vdsm Last login: Wed Sep 23 03:10:29 CDT 2015 on pts/1 -bash-4.2$ id uid=36(vdsm) gid=36(kvm) groups=36(kvm),107(qemu),179(sanlock) -bash-4.2$ cd /var/log/core -bash-4.2$ touch vdsmtest -bash-4.2$ mv core.24839.1442294276.dump core.24839.1442294276.dump.1 mv: cannot move 'core.24839.1442294276.dump' to 'core.24839.1442294276.dump.1': Operation not permitted Version-Release number of selected component (if applicable): vdsm-4.17.7-1.el7ev.noarch How reproducible: 100% Steps to Reproduce: 1. logrotate -v -f /etc/vdsm/logrotate/vdsm 2>&1 | sed -n '/^rotating pattern: \/var\/log\/core/,$p' 2. (yum -y install mailx ; mail) 3. Actual results: huge files are not rotated and could fill filesystem (and as linux people are usually ignoring good partitioning it would fill root fs) Expected results: logrotate should work Additional info: issue found on ppc64le but same exists on x86 platform as well
This bug report has Keywords: Regression or TestBlocker. Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.
I tried that and it worked fine: please see below and tell me what I miss.. [root@vm-17-62 core]# kill -6 5185 [root@vm-17-62 core]# ls core.5185.1444820851.dump [root@vm-17-62 core]# [root@vm-17-62 core]# [root@vm-17-62 core]# logrotate -v -f /etc/vdsm/logrotate/vdsm 2>&1 | sed -n '/^rotating pattern: \/var\/log\/core/,$p' rotating pattern: /var/log/core/*.dump forced from command line (1 rotations) empty log files are rotated, old logs are removed switching euid to 36 and egid to 36 considering log /var/log/core/core.5185.1444820851.dump log needs rotating rotating log /var/log/core/core.5185.1444820851.dump, log->rotateCount is 1 dateext suffix '-20151014' glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' renaming /var/log/core/core.5185.1444820851.dump.1.xz to /var/log/core/core.5185.1444820851.dump.2.xz (rotatecount 1, logstart 1, i 1), old log /var/log/core/core.5185.1444820851.dump.1.xz does not exist renaming /var/log/core/core.5185.1444820851.dump.0.xz to /var/log/core/core.5185.1444820851.dump.1.xz (rotatecount 1, logstart 1, i 0), old log /var/log/core/core.5185.1444820851.dump.0.xz does not exist log /var/log/core/core.5185.1444820851.dump.2.xz doesn't exist -- won't try to dispose of it fscreate context set to system_u:object_r:virt_cache_t:s0 renaming /var/log/core/core.5185.1444820851.dump to /var/log/core/core.5185.1444820851.dump.1 compressing log with: /usr/bin/xz switching uid to 36 and gid to 36 switching euid to 0 and egid to 0 set default create context [root@vm-17-62 core]# geten getenforce getent [root@vm-17-62 core]# geten getenforce getent [root@vm-17-62 core]# getenforce Enforcing [root@vm-17-62 core]#
(In reply to Yaniv Bronhaim from comment #2) > I tried that and it worked fine: please see below and tell me what I miss.. > > [root@vm-17-62 core]# kill -6 5185 > [root@vm-17-62 core]# ls > core.5185.1444820851.dump this info is not complete, what about `ls -l' on this file? who's owner, what group is set? > [root@vm-17-62 core]# > [root@vm-17-62 core]# > [root@vm-17-62 core]# logrotate -v -f /etc/vdsm/logrotate/vdsm 2>&1 | sed -n > '/^rotating pattern: \/var\/log\/core/,$p' > rotating pattern: /var/log/core/*.dump forced from command line (1 > rotations) > empty log files are rotated, old logs are removed > switching euid to 36 and egid to 36 > considering log /var/log/core/core.5185.1444820851.dump > log needs rotating > rotating log /var/log/core/core.5185.1444820851.dump, log->rotateCount is 1 > dateext suffix '-20151014' > glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' > renaming /var/log/core/core.5185.1444820851.dump.1.xz to > /var/log/core/core.5185.1444820851.dump.2.xz (rotatecount 1, logstart 1, i > 1), > old log /var/log/core/core.5185.1444820851.dump.1.xz does not exist > renaming /var/log/core/core.5185.1444820851.dump.0.xz to > /var/log/core/core.5185.1444820851.dump.1.xz (rotatecount 1, logstart 1, i > 0), > old log /var/log/core/core.5185.1444820851.dump.0.xz does not exist > log /var/log/core/core.5185.1444820851.dump.2.xz doesn't exist -- won't try > to dispose of it > fscreate context set to system_u:object_r:virt_cache_t:s0 > renaming /var/log/core/core.5185.1444820851.dump to > /var/log/core/core.5185.1444820851.dump.1 > compressing log with: /usr/bin/xz > switching uid to 36 and gid to 36 > switching euid to 0 and egid to 0 > set default create context no info about OS at all. fully reproducible on: [root@dell-r210ii-04 core]# rpm -qa kernel systemd redhat-release-server selinux-policy-targeted selinux-policy-targeted-3.13.1-60.el7.noarch redhat-release-server-7.2-7.el7.x86_64 systemd-219-19.el7.x86_64 kernel-3.10.0-324.el7.x86_64 [root@dell-r210ii-04 core]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.2 Beta (Maipo) [root@dell-r210ii-04 core]# rpm -qa kernel systemd redhat-release-server selinux-policy-targeted selinux-policy-targeted-3.13.1-60.el7.noarch redhat-release-server-7.2-7.el7.x86_64 systemd-219-19.el7.x86_64 kernel-3.10.0-324.el7.x86_64 installed from: http://download.englab.brq.redhat.com/pub/rhel/rel-eng/RHEL-7.2-20151015.0/compose/Server/x86_64/os/ [root@dell-r210ii-04 core]# ls -l total 52960 -rw-------. 1 qemu qemu 1269874688 Oct 23 17:39 core.15104.1445614743.dump [root@dell-r210ii-04 core]# logrotate -v -f /etc/vdsm/logrotate/vdsm 2>&1 | sed -n '/^rotating pattern: \/var\/log\/core/,$p' rotating pattern: /var/log/core/*.dump forced from command line (1 rotations) empty log files are rotated, old logs are removed switching euid to 36 and egid to 36 considering log /var/log/core/core.15104.1445614743.dump log needs rotating rotating log /var/log/core/core.15104.1445614743.dump, log->rotateCount is 1 dateext suffix '-20151023' glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' renaming /var/log/core/core.15104.1445614743.dump.1.xz to /var/log/core/core.15104.1445614743.dump.2.xz (rotatecount 1, logstart 1, i 1), old log /var/log/core/core.15104.1445614743.dump.1.xz does not exist renaming /var/log/core/core.15104.1445614743.dump.0.xz to /var/log/core/core.15104.1445614743.dump.1.xz (rotatecount 1, logstart 1, i 0), old log /var/log/core/core.15104.1445614743.dump.0.xz does not exist log /var/log/core/core.15104.1445614743.dump.2.xz doesn't exist -- won't try to dispose of it error: error opening /var/log/core/core.15104.1445614743.dump: Permission denied switching euid to 0 and egid to 0 set default create context
Of course it doesn't work.. this is qemu core dump. vdsm configuration rotates the file as vdsm user - therefore it can rotate only vdsm core dump files which are created by vdsm user - do you except vdsm configuration to rotate any core file in this directory? If yes we need to remove the "su vdsm kvm" in /etc/vdsm/logrotate/vdsm. Dan, this was added quite long ago - http://gerrit.ovirt.org/971 we configure to throw all coredumps to /var/log/core , also libvirt's which are created by root .. What do you say? su root root?
I'm curious, what is then that core file in /var/log/core ? :) I just send ABRT signal to qemu-kvm.
Yaniv, I'm afraid that I don't recall the motivation for adding "su" to logrotate. Until we integrate properly with ABRT, and as long we mess with /var/log/core, we should be able to log-rotate everything that is put there.
so be it - I can't just remove the su to rotate files. it will give us: error: skipping "/var/log/core/core.5626.1445764996.dump" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation. Therefore, we will need "su root root" there ..
*** Bug 1221464 has been marked as a duplicate of this bug. ***
ok, vdsm-4.17.15-0.el7ev.noarch # grep su /etc/vdsm/logrotate/vdsm ; rpm -qf /etc/vdsm/logrotate/vdsm su root root vdsm-4.17.15-0.el7ev.noarch # ls -l /var/log/core/core.16118.1452686885.dump -rw-------. 1 qemu qemu 1839423488 Jan 13 13:08 /var/log/core/core.16118.1452686885.dump # logrotate -v -f /etc/vdsm/logrotate/vdsm 2>&1 | sed -n '/^rotating pattern: \/var\/log\/core/,$p' rotating pattern: /var/log/core/*.dump forced from command line (1 rotations) empty log files are rotated, old logs are removed considering log /var/log/core/core.16118.1452686885.dump log needs rotating rotating log /var/log/core/core.16118.1452686885.dump, log->rotateCount is 1 dateext suffix '-20160113' glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' renaming /var/log/core/core.16118.1452686885.dump.1.xz to /var/log/core/core.16118.1452686885.dump.2.xz (rotatecount 1, logstart 1, i 1), old log /var/log/core/core.16118.1452686885.dump.1.xz does not exist renaming /var/log/core/core.16118.1452686885.dump.0.xz to /var/log/core/core.16118.1452686885.dump.1.xz (rotatecount 1, logstart 1, i 0), old log /var/log/core/core.16118.1452686885.dump.0.xz does not exist log /var/log/core/core.16118.1452686885.dump.2.xz doesn't exist -- won't try to dispose of it fscreate context set to system_u:object_r:virt_cache_t:s0 renaming /var/log/core/core.16118.1452686885.dump to /var/log/core/core.16118.1452686885.dump.1 compressing log with: /usr/bin/xz set default create context [root@dell-r210ii-13 ~]# ls -l /var/log/core/ total 150896 -rw-------. 1 qemu qemu 154513628 Jan 13 13:08 core.16118.1452686885.dump.1.xz
oVirt 3.6.1 has been released, closing current release
*** Bug 1311053 has been marked as a duplicate of this bug. ***