Bug 1381719

Summary: [RFE] move default state file to /var/lib/logrotate/logrotate.status
Product: Red Hat Enterprise Linux 7 Reporter: Douglas Schilling Landgraf <dougsland>
Component: logrotateAssignee: Kamil Dudka <kdudka>
Status: CLOSED ERRATA QA Contact: Robin Hack <rhack>
Severity: high Docs Contact: Lenka Kimlickova <lkimlick>
Priority: medium    
Version: 7.3CC: bbreard, cshao, dougsland, fdeutsch, kdudka, otsuka, rhack, vdanek, ycui
Target Milestone: rcKeywords: FutureFeature, Patch
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: logrotate-3.8.6-14.el7 Doc Type: Release Note
Doc Text:
`logrotate` now uses `/var/lib/logrotate/logrotate.status` as the default state file Previously, the `logrotate cron job` used a modified path to the `logrotate` state file. Consequently, the path used by the cron job did not match the default state file path used by `logrotate` itself. To prevent confusion, the default state file path used by `logrotate` has been changed to match the state file path used by `logrotate cron job`. As a result, `logrotate` now uses `/var/lib/logrotate/logrotate.status` as the default state file path in both scenarios.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-01 16:19:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1380596, 1393867, 1400961    

Description Douglas Schilling Landgraf 2016-10-04 19:53:51 UTC
Description of problem:
logrotate using /var/lib/logrotate.status instead of /var/lib/logrotate/logrotate.status

Version-Release number of selected component (if applicable):
# rpm -qf /usr/sbin/logrotate 
logrotate-3.8.6-12.el7.x86_64

How reproducible:

On RHEV-H 7.3 (based on RHEL 7.3)
====================================
# cat /etc/redhat-release 
Red Hat Enterprise Virtualization Hypervisor release 7.3 (20161004.0.el7ev)

# touch /tmp/test
# logrotate /tmp/test
error: error creating state file /var/lib/logrotate.status: Read-only file system


On RHEL 7.2, installed logrotate-3.8.6-12
==============================================

# touch /tmp/test
# strace -o out -f -s255 logrotate /tmp/test
# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.2 (Maipo)

5580  stat("/var/lib/logrotate.status", {st_mode=S_IFREG|0644, st_size=29, ...}) = 0
5580  open("/var/lib/logrotate.status.tmp", O_RDWR|O_CREAT|O_EXCL|O_TRUNC|O_NOFOLLOW, 0600) = 3
5580  rename("/var/lib/logrotate.status.tmp", "/var/lib/logrotate.status") = 0


Actual results:
logrotate still uses /var/lib/logrotate.status

Expected results:
logrotate should use /var/lib/logrotate/logrotate.status

Comment 1 Douglas Schilling Landgraf 2016-10-04 19:58:08 UTC
My understand is that such behaviour should be resolved by bz#1272236 but I still see it.

Comment 3 Kamil Dudka 2016-10-05 07:22:59 UTC
(In reply to Douglas Schilling Landgraf from comment #1)
> My understand is that such behaviour should be resolved by bz#1272236 but I
> still see it.

The fix for bug #1272236 only changes the cron script to pass -s /var/lib/logrotate/logrotate.status to logrotate:

+++ b/examples/logrotate.cron
@@ -1,6 +1,6 @@
 #!/bin/sh

-/usr/sbin/logrotate /etc/logrotate.conf
+/usr/sbin/logrotate -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf
 EXITVALUE=$?
 if [ $EXITVALUE != 0 ]; then
     /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"

According to logrotate(8) man page, /var/lib/logrotate.status is still the default state file.

I am not the author of this downstream-only change, so it is difficult for me to explain the reasoning behind, but it works the same way in Fedora for quite some time now.

Comment 4 Douglas Schilling Landgraf 2016-10-11 20:49:12 UTC
Hi Kamil,

(In reply to Kamil Dudka from comment #3)
> (In reply to Douglas Schilling Landgraf from comment #1)
> > My understand is that such behaviour should be resolved by bz#1272236 but I
> > still see it.
> 
> The fix for bug #1272236 only changes the cron script to pass -s
> /var/lib/logrotate/logrotate.status to logrotate:
> 
> +++ b/examples/logrotate.cron
> @@ -1,6 +1,6 @@
>  #!/bin/sh
> 
> -/usr/sbin/logrotate /etc/logrotate.conf
> +/usr/sbin/logrotate -s /var/lib/logrotate/logrotate.status
> /etc/logrotate.conf
>  EXITVALUE=$?
>  if [ $EXITVALUE != 0 ]; then
>      /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
> 
> According to logrotate(8) man page, /var/lib/logrotate.status is still the
> default state file.

Sure, I have a proposal patch for logrotate that will change it [1].

> 
> I am not the author of this downstream-only change, so it is difficult for
> me to explain the reasoning behind, but it works the same way in Fedora for
> quite some time now.

Instead of downstream-only proposal, I have sent a proposal upstream that should solve these RO scenarios. Please let me know if you have any thoughts/suggestions about it.

[1] https://github.com/logrotate/logrotate/commit/c76dcf905dfdc129ebab578caa5beee7ab2e8fa3

Thanks!

Comment 5 Douglas Schilling Landgraf 2016-10-18 02:57:40 UTC
More data:

# cat /etc/redhat-release 
Red Hat Enterprise Virtualization Hypervisor release 7.3 (20161004.0.el7ev)

# rpm -qa | grep -i logrotate
logrotate-3.8.6-12.el7.x86_64

# cat /var/log/cron 
Oct 18 02:47:40 localhost crond[1871]: (CRON) INFO (Syslog will be used instead of sendmail.)
Oct 18 02:47:40 localhost crond[1871]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 54% if used.)
Oct 18 02:47:41 localhost crond[1871]: (CRON) INFO (running with inotify support)
Oct 18 02:48:01 localhost CROND[2391]: (root) CMD (python < /usr/share/rhn/virtualization/poller.py)
Oct 18 02:50:01 localhost CROND[11511]: (root) CMD (python < /usr/share/rhn/virtualization/poller.py)
Oct 18 02:50:01 localhost CROND[11512]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Oct 18 02:50:01 localhost CROND[11513]: (root) CMD (/usr/sbin/logrotate /etc/logrotate.conf)
Oct 18 02:50:01 localhost CROND[11505]: (root) CMDOUT (error: error creating state file /var/lib/logrotate.status: Read-only file system)
Oct 18 02:51:20 localhost crond[1871]: (CRON) INFO (Shutting down)
Oct 18 02:51:57 localhost crond[2021]: (CRON) INFO (Syslog will be used instead of sendmail.)
Oct 18 02:51:57 localhost crond[2021]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 26% if used.)
Oct 18 02:51:58 localhost crond[2021]: (CRON) INFO (running with inotify support)
Oct 18 02:52:01 localhost CROND[4969]: (root) CMD (python < /usr/share/rhn/virtualization/poller.py)
Oct 18 02:54:01 localhost CROND[17518]: (root) CMD (python < /usr/share/rhn/virtualization/poller.py)

Comment 6 Kamil Dudka 2016-10-18 10:03:12 UTC
Looks weird.  Are you using the cron script installed by logrotate?

Please try the following:
# rpm -V logrotate
# grep status /etc/cron.daily/logrotate

Comment 7 Douglas Schilling Landgraf 2016-10-18 15:55:58 UTC
From a fresh install:

# cat /etc/redhat-release 
Red Hat Enterprise Virtualization Hypervisor release 7.3 (20161004.0.el7ev)

# rpm -qa | grep -i logrotate
logrotate-3.8.6-12.el7.x86_64

# rpm -V logrotate
missing   c /etc/cron.daily/logrotate
missing   c /var/lib/logrotate/logrotate.status

You might wondering why the /etc/cron.daily is not there, I have found in our legacy code the below statement from 2000-2001. However, keep in mind that we have text user interface that allow users to change logrotate settings to their need.

"""
recipe/common-post.ks
<snip>
# Remove the default logrotate daily cron job
# since we run it every 10 minutes instead.
rm -f /etc/cron.daily/logrotate
</snip>
"""

We already discussed it by email but let's make the record here as well:
In our case, there are services like vdsm and ovirt-node that adds in cron a logrotate call:

#etc> grep -rni logrotate *
<snip>
cron.d/ovirt-logrotate:2:*/10 * * * * root /usr/sbin/logrotate
/etc/logrotate.conf

Specially this one:
=========================
cron.hourly/vdsm-logrotate:3:/usr/sbin/logrotate /etc/vdsm/logrotate/vdsm
</snip>


# cat /etc/vdsm/logrotate/vdsm 
/var/log/vdsm/*.log {
    rotate 100
    missingok
    size 15M
    compress
    compresscmd /usr/bin/xz
    uncompresscmd /usr/bin/unxz
    compressext .xz
}
/var/log/core/*.dump {
    rotate 1
    missingok
    size 0
    compress
    compresscmd /usr/bin/xz
    uncompresscmd /usr/bin/unxz
    compressext .xz
    su root root
}

This case is very bad, because the logging rotate of vdsm is not working and might create a full disk scenario as I can demonstrate here:

1) Create 65M log file:
# cd /var/log/vdsm
# dd if=/dev/zero of=vdsm.log bs=64M count=1 
1+0 records in
1+0 records out

2) Let's mimic the vdsm logrotate call from cron:

# logrotate /etc/vdsm/logrotate/vdsm 
error: error creating state file /var/lib/logrotate.status: Read-only file system

3) Still 65M, which is wrong as vdsm should rotate when log is 15M:
# du -sh *
4.0K	backup
4.0K	connectivity.log
4.0K	mom.log
12K	supervdsm.log
4.0K	upgrade.log
65M	vdsm.log

4) Remounting the FS as RW, the logrotate works and reduce the size:
# mount -o remount,rw /
# logrotate /etc/vdsm/logrotate/vdsm 
# du -sh *
4.0K	backup
4.0K	connectivity.log
4.0K	mom.log
12K	supervdsm.log
4.0K	upgrade.log
4.0K	vdsm.log
12K	vdsm.log.1.xz

Comment 8 Kamil Dudka 2016-10-18 16:21:33 UTC
Thanks for the clarification!  This is definitely not a regression in logrotate.  The fix for bug #1272236 is just less ambitious than you would like it to be.  I am fine with changing the default state file in RHEL-7 if the change gets merged in upstream and Fedora.

Comment 9 Douglas Schilling Landgraf 2016-10-18 16:26:07 UTC
(In reply to Kamil Dudka from comment #8)
> Thanks for the clarification!  This is definitely not a regression in
> logrotate.  The fix for bug #1272236 is just less ambitious than you would
> like it to be.  I am fine with changing the default state file in RHEL-7 if
> the change gets merged in upstream and Fedora.

Sure, I have noticed new comments in my pull request. I am working on an updated patch/pull request.

Thanks!

Comment 11 Kamil Dudka 2016-11-24 15:22:13 UTC
fixed in logrotate-3.10.0-4.fc26 for now:
http://pkgs.fedoraproject.org/cgit/rpms/logrotate.git/commit/?id=3799546e

Comment 18 errata-xmlrpc 2017-08-01 16:19:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1889