Bug 1374550

Summary: After a failure, logrotate doesn't clean up its incomplete work, leaving it in an unrunnable state
Product: Red Hat Enterprise Linux 7 Reporter: Daphne Shaw <dshaw>
Component: logrotateAssignee: Kamil Dudka <kdudka>
Status: CLOSED ERRATA QA Contact: Frantisek Sumsal <fsumsal>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.2CC: fsumsal, gregoire.menuel, kdudka, phil, toracat, vdanek
Target Milestone: rcKeywords: Patch
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: logrotate-3.8.6-17.el7 Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-30 11:10:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1546552, 1549617, 1549689    

Description Daphne Shaw 2016-09-09 02:20:11 UTC
Description of problem:

If logrotate is using copytruncate, and the copy fails (for example, due to disk full), logrotate can leave behind partially written files.  Unfortunately, these partially written files prevent logrotate from rotating the affected log file again, requiring manual assistance.

Version-Release number of selected component (if applicable):

logrotate-3.8.6-7.el7_2.x86_64

How reproducible:

Every time (albeit requiring a disk full or close to disk full situation)

Steps to Reproduce:
1. echo foobar > /var/log/foobar.log

2. dd if=/dev/zero of=/var/log/fill-this-disk
   (wait for it to fill the disk and eventually fail)

3. logrotate -f /etc/logrotate.conf

   Fails for foobar.log with:
   error: error writing to /var/log/foobar.log.1: No space left on device

Actual results:

After following the steps above, logrotate cannot rotate foobar.log any longer.  Even once the disk-full situation is addressed (for this example just rm /var/log/fill-this-disk), it will always fail to rotate that log file with:

  error: error creating output file /var/log/foobar.log.1: File exists

Filling the whole disk via dd is just an example to force the problem to happen.  It can happen naturally if the log file in question is large enough.

Expected results:

If logrotate created foobar.log.1 and could not successfully write to it, it should clean up after itself and delete it.

Additional info:

This is the config I used for foobar.log:

# cat /etc/logrotate.d/foobar
/var/log/foobar.log {
    rotate 10
    size 10M
    create
    copytruncate
}

Note that essentially the same thing happens using "delaycompress".  Using the above config, remove "copytruncate" and add "compress" and "delaycompress" and follow these steps:

1. echo "foobar" > /var/log/foobar.log
2. logrotate -f /etc/logrotate.conf
3. dd if=/dev/zero of=/var/log/fill-this-disk
2. logrotate -f /etc/logrotate.conf
4. rm /var/log/fill-this-disk

# logrotate -f /etc/logrotate.conf
error: error creating output file /var/log/foobar.log.1.gz: File exists

Comment 2 Kamil Dudka 2016-09-12 16:55:42 UTC
This seems to be a known issue, for which we have no solution yet:

https://github.com/logrotate/logrotate/issues/45

Comment 3 Daphne Shaw 2016-09-12 17:21:52 UTC
The bug reported in https://github.com/logrotate/logrotate/issues/45 is in regards to a corrupt /var/log/logrotate.status statefile.  This is somewhat different - even if the statefile is fine, logrotate won't rotate if one of its target files already exists (or clean up a target file when writing to it fails).

Comment 4 Kamil Dudka 2016-09-16 09:44:02 UTC
(In reply to David Shaw from comment #3)
> The bug reported in https://github.com/logrotate/logrotate/issues/45 is in
> regards to a corrupt /var/log/logrotate.status statefile.  This is somewhat
> different - even if the statefile is fine, logrotate won't rotate if one of
> its target files already exists (or clean up a target file when writing to
> it fails).

True.  This needs to be fixed, too.

Comment 6 gregoire.menuel 2018-02-23 09:01:40 UTC
Hi,
The bug seems to be corrected upstream: https://github.com/logrotate/logrotate/commit/e349752ccbbfa9ae8480ca504ded3af87e718298

Comment 7 Kamil Dudka 2018-02-26 09:28:32 UTC
(In reply to gregoire.menuel from comment #6)
> The bug seems to be corrected upstream:
> https://github.com/logrotate/logrotate/commit/
> e349752ccbbfa9ae8480ca504ded3af87e718298

Thanks for the reference!  This ^^^ seems safe to be applied on el7 logrotate.  It is not going to fix all the troubles one encounters while running logrotate without having any free space on the file system, but it should improve the current behavior without breaking compatibility.  I am fine with backporting the above patch.

Comment 12 errata-xmlrpc 2018-10-30 11:10:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3202