Bug 1127415 - error renaming temp state file /var/lib/logrotate.status.tmp
Summary: error renaming temp state file /var/lib/logrotate.status.tmp
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: logrotate
Version: 23
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Jan Kaluža
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On: 1228531
Blocks: 1272236
TreeView+ depends on / blocked
 
Reported: 2014-08-06 20:25 UTC by John Florian
Modified: 2017-05-03 13:08 UTC (History)
5 users (show)

Fixed In Version: logrotate-3.9.1-2.fc23
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-06-29 21:57:23 UTC


Attachments (Terms of Use)

Description John Florian 2014-08-06 20:25:13 UTC
Description of problem:
logrotate fails with the above message when attempting to run from within a live image (e.g., one created with livecd-creator).

Version-Release number of selected component (if applicable):
logrotate-3.8.7-1.fc20.i686

How reproducible:
always

Steps to Reproduce:
1. Create a Live spin that includes logrotate.
2. Boot.
3. /usr/sbin/logrotate /etc/logrotate.conf

Actual results:
# /usr/sbin/logrotate /etc/logrotate.conf; echo $?
error: error renaming temp state file /var/lib/logrotate.status.tmp
1

Expected results:
Same as with a regular install.

Additional info:
# ls -l /var/lib/logrotate.status*
-rw-r--r--. 1 root root 29 Aug  6 15:35 /var/lib/logrotate.status

# rm /var/lib/logrotate.status
rm: cannot remove ‘/var/lib/logrotate.status’: Device or resource busy

# lsof /var/lib/logrotate.status
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
systemd-j  418 root  mem    REG   0,34  8388608 15408 /var/log/journal/2150106cce5d48d883d7253aa337cacd/system.journal
systemd-j  418 root   26u   REG   0,34  8388608 15408 /var/log/journal/2150106cce5d48d883d7253aa337cacd/system.journal
dhclient   824 root    5w   REG   0,34      519 15931 /var/lib/dhclient/dhclient--eth0.lease
journalct 1448 root  mem    REG   0,34  8388608 15408 /var/log/journal/2150106cce5d48d883d7253aa337cacd/system.journal
journalct 1448 root    5r   REG   0,34  8388608 15408 /var/log/journal/2150106cce5d48d883d7253aa337cacd/system.journal
anacron   5640 root  cwd    DIR   0,34      100 11927 /var/spool/anacron
anacron   5640 root    3uW  REG   0,34        0 11928 /var/spool/anacron/cron.daily
anacron   5640 root    5uW  REG   0,34        0 11930 /var/spool/anacron/cron.weekly
anacron   5640 root    6uW  REG   0,34        0 11929 /var/spool/anacron/cron.monthly

Comment 1 John Florian 2014-08-06 20:48:26 UTC
If I run with an alternate state file, it exits cleanly:

# /usr/sbin/logrotate --state /tmp/foo /etc/logrotate.conf; echo $?
0

Unfortunately I can't use this as a workaround, short of patching the cron job, because there doesn't appear to be any equivalent directive in the conf file.  Something like the following would help in that regard:

--- /etc/cron.daily/logrotate.orig      2012-08-01 06:56:47.000000000 -0400
+++ /etc/cron.daily/logrotate   2014-08-06 16:47:36.845630093 -0400
@@ -1,6 +1,10 @@
 #!/bin/sh
 
-/usr/sbin/logrotate /etc/logrotate.conf
+if [ -r /etc/sysconfig/logrotate ]; then
+    . /etc/sysconfig/logrotate
+fi
+
+/usr/sbin/logrotate $OPTIONS /etc/logrotate.conf
 EXITVALUE=$?
 if [ $EXITVALUE != 0 ]; then
     /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"

Comment 2 John Florian 2014-08-07 14:33:03 UTC
I had no such problems with my last live spins that were based on Fedora 18 with logrotate-3.8.3-1.fc18.i686.  This prompted me into a little testing on a machine running my Fedora 18 spin with various releases of logrotate that I can find in our mirrors.  Here are the results:

logrotate-3.8.3-1.fc18.i686      WORKS
logrotate-3.8.4-1.fc19.i686      WORKS
logrotate-3.8.4-3.fc19.i686      WORKS
logrotate-3.8.7-1.fc20.i686      FAILS

So it got broke between 3.8.4 and 3.8.7.  Don't have anything between those in our mirror, but http://koji.fedoraproject.org/koji/packageinfo?packageID=146 looks helpful here.

logrotate-3.8.5-1.fc20.i686      FAILS

Comment 3 John Florian 2014-08-07 17:49:24 UTC
The following may be worth noting as well.  This was run on a Fedora 20 Live spin with logrotate-3.8.7-1.fc20.i686.

# /usr/sbin/logrotate /etc/logrotate.conf
error: error renaming temp state file /var/lib/logrotate.status.tmp

The above fails just like the cron.daily job.  This should effectively be the same:

# /usr/sbin/logrotate --state /var/lib/logrotate.status /etc/logrotate.conf
error: error renaming temp state file /var/lib/logrotate.status.tmp

And appears to be so.  However, an alternate state file in the same directory works as expected:

#/usr/sbin/logrotate --state /var/lib/logrotate.status-alt /etc/logrotate.conf
# echo $?
0


I think the lsof output in #c1 (comment 1) is quite relevant here.  I have to admit that I'm at a complete loss as to why systemd-journald, journalctl and especially dhclient have this file open.  I don't see this behavior with a regular Fedora install.

Comment 4 John Florian 2014-08-08 15:46:40 UTC
I believe I've identified the true problem here.  My live spins are very dependent on the stateless supported provided by /usr/lib/systemd/fedora-readonly.  (I have TEMPORARY_STATE=yes in /etc/sysconfig/readonly-root.)

Within /etc/rwtab there exists "files   /var/lib/logrotate.status" which effectively causes the readonly-root script to create a tmpfs, copy the state file into it, and use a bind mount to make the original location operate on the tmpfs location instead.   So when logrotate runs and attempts to remove the old state file and rename its replacement, it fails because the remove is essentially operating on a busy mount point.

The easiest fix for this bug probably simply entails changing the state file from /var/lib/logrotate.status to /var/lib/logrotate/status and changing the entry in rwtab to "dirs   /var/lib/logrotate".

Comment 5 Jan Kaluža 2014-08-11 05:08:26 UTC
Thanks for the information. Sorry for late response, I've been on Flock 2014 Fedora conference. I'm thinking about moving state file to some write-able location, but as a workaround for now you have to use the way you described in previous comment.

Comment 6 jjj092353 2015-05-20 14:37:07 UTC
(In reply to John Florian from comment #4)
> I believe I've identified the true problem here.  My live spins are very
> dependent on the stateless supported provided by
> /usr/lib/systemd/fedora-readonly.  (I have TEMPORARY_STATE=yes in
> /etc/sysconfig/readonly-root.)
> 
> Within /etc/rwtab there exists "files   /var/lib/logrotate.status" which
> effectively causes the readonly-root script to create a tmpfs, copy the
> state file into it, and use a bind mount to make the original location
> operate on the tmpfs location instead.   So when logrotate runs and attempts
> to remove the old state file and rename its replacement, it fails because
> the remove is essentially operating on a busy mount point.
> 
> The easiest fix for this bug probably simply entails changing the state file
> from /var/lib/logrotate.status to /var/lib/logrotate/status and changing the
> entry in rwtab to "dirs   /var/lib/logrotate".

ok - I am habing this problem on my system and I'm a newby so can someone tell me how to : change the state file from /var/lib/logrotate.status to .var/lib/logrotate/status and changing the entry in rwtab to "dirs   /var/lib/logrotate".

???

thanks in advance.

Comment 7 Fedora End Of Life 2015-05-29 12:34:21 UTC
This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 8 Andrew J. Schorr 2015-06-03 17:26:30 UTC
This problem is still present in Fedora 21.  Can somebody please update the version?  I don't seem to be able to do so.

The simple fix is to apply this patch:

--- etc/cron.daily/logrotate.orig       2012-08-01 06:56:47.000000000 -0400
+++ etc/cron.daily/logrotate    2015-06-03 13:15:41.826644853 -0400
@@ -1,6 +1,6 @@
 #!/bin/sh
 
-/usr/sbin/logrotate /etc/logrotate.conf
+/usr/sbin/logrotate -s /var/spool/logrotate.status /etc/logrotate.conf
 EXITVALUE=$?
 if [ $EXITVALUE != 0 ]; then
     /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"

This should work since the entire /var/spool tree is in /etc/rwtab.

Regards,
Andy

Comment 9 Andrew J. Schorr 2015-06-03 17:29:20 UTC
Note: this change occurred in version 3.8.5 as documented in the CHANGES file:
https://fedorahosted.org/logrotate/browser/tags/r3-9-1/CHANGES

3.8.4 -> 3.8.5
...
        - Make logrotate.status creation atomic.

I guess somebody was worried about a race condition.  It's not clear to me why that was a concern.

Regards,
Andy

Comment 10 Jan Kaluža 2015-06-04 15:14:27 UTC
I'm going to move logrotate.status in rawhide into some rwtab directory soon (Probably tomorrow.). The "problem" is that in logrotate's RPM %post script, there has to be script to copy old logrotate.status into new destination, so it will need some testing.

Comment 11 Jan Kaluža 2015-06-05 06:52:30 UTC
SELinux change is needed to fix this bug. Added as Bug 1228531.

Comment 12 Fedora End Of Life 2015-06-29 21:57:23 UTC
Fedora 20 changed to end-of-life (EOL) status on 2015-06-23. Fedora 20 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.