Bug 1024759 - logrotate rotates atop.log but process created atop_YYYYMMDD files
logrotate rotates atop.log but process created atop_YYYYMMDD files
Status: CLOSED ERRATA
Product: Fedora EPEL
Classification: Fedora
Component: atop (Show other bugs)
el6
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Gwyn Ciesla
Fedora Extras Quality Assurance
:
: 1052404 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-30 07:31 EDT by Edward Moore
Modified: 2014-02-03 15:51 EST (History)
3 users (show)

See Also:
Fixed In Version: atop-1.27-2.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-02-03 15:50:35 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Edward Moore 2013-10-30 07:31:50 EDT
Description of problem:
/var/log/atop.log is always size 0 because atop logs to /var/log/atop_YYYYMMDD. These logs never rotate and are never archived by logrotate because it's looking at atop.log. Logs need managing manually.


Version-Release number of selected component (if applicable):
atop-1.27-1.el6.x86_64


How reproducible:


Steps to Reproduce:
1. Install atop-1.27-1.el6.x86_64
2. Start atop (service atop start or /etc/rc.d/init.d/atop start)
3. ps -ef |grep atop, note logging to /var/log/atop/atop_YYYYMMDD
4. logrotate -v /etc/logrotate.conf, observe rotation

Actual results:


Expected results:


Additional info:
Comment 1 Edward Moore 2013-10-30 07:34:33 EDT
Fedora EPEL package added to RHEL 6.2
Comment 2 Edward Moore 2013-11-05 04:09:17 EST
Any update please?
Comment 3 Gwyn Ciesla 2013-11-05 11:43:02 EST
Sorry for the delay, I've been swamped, I hope to look at this tomorrow.
Comment 4 Edward Moore 2013-11-12 04:20:54 EST
Any news?
Comment 5 Dominique Brazziel 2013-11-14 20:02:50 EST
This is happening on V2.0.2 as well.  I just removed ~141M of log files going back to January of this year trying to fix this problem.  Using pattern '/var/log/atop/atop_*' fails because logrotate tries to rotate 4 versions of each '_YYMMDD' log and ends up failing, so no rotation is done:

renaming /var/log/atop/atop_20131108.4 to /var/log/atop/atop_20131108.5 (rotatecount 4, logstart 1, i 4), 
renaming /var/log/atop/atop_20131108.3 to /var/log/atop/atop_20131108.4 (rotatecount 4, logstart 1, i 3), 
renaming /var/log/atop/atop_20131108.2 to /var/log/atop/atop_20131108.3 (rotatecount 4, logstart 1, i 2), 
renaming /var/log/atop/atop_20131108.1 to /var/log/atop/atop_20131108.2 (rotatecount 4, logstart 1, i 1), 
renaming /var/log/atop/atop_20131108.0 to /var/log/atop/atop_20131108.1 (rotatecount 4, logstart 1, i 0), 
fscreate context set to system_u:object_r:var_log_t:s0
renaming /var/log/atop/atop_20131108 to /var/log/atop/atop_20131108.1
creating new /var/log/atop/atop_20131108 mode = 0600 uid = 0 gid = 0
removing old log /var/log/atop/atop_20131108.5
error: error opening /var/log/atop/atop_20131108.5: No such file or directory

It seems that rotation can't work if the raw log file already has the date extension on it. 

Workaround I used to get some sort of log rotation was to change the atop startup script (/usr/bin/atopd) as follows:

#$BINPATH/atop -a -w $LOGPATH/atop_$CURDAY $INTERVAL > $LOGPATH/atop.log 2>&1 &
$BINPATH/atop -a -w $LOGPATH/atop.log $INTERVAL > $LOGPATH/atop_error.log 2>&1 &

It looks like the current 'atop.log' is only there to report errors (and was always empty), so I renamed it to 'atop_error.log' to claim back 'atop.log' which makes log rotation work again (I added the 'dateext' parameter to '/etc/logrotate.d/atop'):

logrotate -d -v -f ./atop 2>&1 | tee ~/logs/lr.log
reading config file ./atop

Handling 1 logs

rotating pattern: /var/log/atop/atop.log  forced from command line (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/atop/atop.log
  log needs rotating
rotating log /var/log/atop/atop.log, log->rotateCount is 4
dateext suffix '-20131114'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
glob finding old rotated logs failed
fscreate context set to unconfined_u:object_r:var_log_t:s0
renaming /var/log/atop/atop.log to /var/log/atop/atop.log-20131114
creating new /var/log/atop/atop.log mode = 0600 uid = 0 gid = 0
Comment 6 Dominique Brazziel 2013-11-15 10:17:52 EST
ACK - that rotated logs nicely but thoroughly botched up atop service restart.
The code and infrastructure is geared to the log files having a date extension.

I think the root cause is missing '/etc/atop/atop.daily' file, which actually
trims daily log files older than 4 weeks old.  Reference 'man atop' section 'RAW   DATA STORAGE'.  The logrotate file ('/etc/logrotate.d/atop') is not included in the source distribution, and because 'atop.daily' actually maintains the logs, a separate logrotate file is not needed.
Comment 7 Dominique Brazziel 2013-11-15 10:47:31 EST
Turns out '/usr/bin/atopd' is almost the same as '/etc/atop/atop.daily' except
it misses the logfile maintenance part at the end.  Added these lines at the end:

# delete logfiles older than four weeks
#
(sleep 3; find $LOGPATH -name 'atop_*' -mtime +28 -exec rm {} \; )&
Comment 8 Dominique Brazziel 2013-11-15 15:26:09 EST
Still a problem with the doc ('man atop') not matching what actually happens vis-a-vis restarting atop:  the systemd service executes '/usr/bin/atopd', the man page references '/etc/atop/atop.daily'.
Comment 9 Edward Moore 2013-11-19 04:02:05 EST
Hi Jon. Any news please?

Daniel's comment that the man page differs from actual behaviour is also true in v1.27. Compared to his issue, I'm not seeing logrotate create 4x versions of 'atop_YYMMDD' logs: it performs no rotation at all for us because atop.log is always empty. However his observations still largely apply, including atop.log being used for reporting errors and not for the atop data.
Comment 10 Edward Moore 2013-11-19 04:02:58 EST
(Sorry, Dominique's comments. My apologies Dominique.)
Comment 11 Gwyn Ciesla 2013-11-19 09:50:58 EST
Sorry, still buried under other things.  I'd be happy to commit and build something if you've managed to come up with a workable solution.
Comment 12 Edward Moore 2013-12-05 10:53:23 EST
Hi Jon

I've been looking more at this. Here's the current setup:

/var/log/atop/atop.log {
    missingok
    weekly
    rotate 4
    notifempty
    create 0600 root root
}

And below is my proposed solution:

/var/log/atop/atop_201[0-9][0-9][0-9][0-9][0-9] {
    missingok
    daily
    rotate 1
    ifempty
    nocreate
    postrotate
      /usr/bin/find /var/log/atop/ -maxdepth 1 -mount -name atop_201\[0-9\]\[0-9\]\[0-9\]\[0-9\]\[0-9\] -mtime -40 -exec /bin/rm {} \;
    endscript
}

REASONING:
- We need to manage atop_<date> and not atop.log, so pattern change
- If I use a * in the pattern it rotates the rotations, hence [0-9] x5
- nocreate added to stop it replacing the rotated files with empty ones
- ifempty added to tidy empty files away (otherwise they'll build up)
- Finally "rotate 1" as explained below.


The logrotate pattern match seems to only rotate older .N (.1, .2, .3 etc) files if the original file name exists. Once you add nocreate it never rotates .N again. Therefore my work-around is a "find" command. (The -40 day mtime is personal preference.) Admittedly, the whole solution could be a find command in a crontab, but as people are expecting logrotate to manage the atop logs, at least they'll find the archiving mechanism. The postrotate section in only ran after a rotation, so "rotate 1" is needed to create some kind of rotation.

Please see if you like this solution to take forward or have a better one. Personally, I believe atop lacks the options for a neater solution.

Kind Regards,
Edd Moore
Comment 13 Edward Moore 2013-12-06 04:57:24 EST
Sorry, please read that find command as +40. I briefly changed it to a - for testing.
Comment 14 Edward Moore 2013-12-23 07:27:56 EST
Hi John

Here's my solution, as below, for logrotate to tidy the atop logs:

# cat /etc/logrotate.d/atop
/var/log/atop/atop_201[0-9][0-9][0-9][0-9][0-9] {
    missingok
    daily
    nodateext
    rotate 1
    ifempty
    nocreate
    postrotate
      /usr/bin/find /var/log/atop/ -maxdepth 1 -mount -name atop_201\[0-9\]\[0-9\]\[0-9\]\[0-9\]\[0-9\]\* -mtime +40 -exec /bin/rm {} \;
    endscript
}
#

A cleaner fix would be to have additional logrotate functionality that can handle this issue without adding the "find" command above. Please see previous post 2013-12-05 10:53:23 EST. Also please note Dominique's note about the man page documentation being incorrect (2013-11-15 15:26:09 EST).

Any good? I suspect this work-around is not ideal to add to your build. That's me done, for it will keep me happy for now until something is released.

Kind Regards,
Edd
Comment 15 Gwyn Ciesla 2014-01-16 09:46:34 EST
*** Bug 1052404 has been marked as a duplicate of this bug. ***
Comment 16 Gwyn Ciesla 2014-01-16 09:48:42 EST
I'm building this for el6 and el5 with the latest version, sorry for the delay, I've been swamped.  Please test and give karma on the updates once they're issued.
Comment 17 Gwyn Ciesla 2014-01-16 09:48:57 EST
I'm building this for el6 and el5 with the latest version, sorry for the delay, I've been swamped.  Please test and give karma on the updates once they're issued.
Comment 18 Fedora Update System 2014-01-16 10:09:16 EST
atop-1.27-2.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/atop-1.27-2.el6
Comment 19 Fedora Update System 2014-01-16 10:09:24 EST
atop-1.27-2.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/atop-1.27-2.el5
Comment 20 Fedora Update System 2014-01-18 13:18:44 EST
Package atop-1.27-2.el5:
* should fix your issue,
* was pushed to the Fedora EPEL 5 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=epel-testing atop-1.27-2.el5'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-EPEL-2014-0211/atop-1.27-2.el5
then log in and leave karma (feedback).
Comment 21 Fedora Update System 2014-02-03 15:50:35 EST
atop-1.27-2.el5 has been pushed to the Fedora EPEL 5 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 22 Fedora Update System 2014-02-03 15:51:04 EST
atop-1.27-2.el6 has been pushed to the Fedora EPEL 6 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.