Description of problem: atop's logrotate is not compatible with the way that atop creates it's daily logfile. Together they create a multitude of rotated files. Version-Release number of selected component (if applicable): 1.23-8.el5 How reproducible: 100% Steps to Reproduce: 1. install atop 2. wait 4 days 3. ls -l /var/log/atop Actual results: [root@thisnode ~]# ls -l /var/log/atop/ total 1264 -rw------- 1 root root 0 Nov 30 04:02 atop_20091124 -rw------- 1 root root 0 Nov 30 04:02 atop_20091124.1 -rw------- 1 root root 0 Nov 30 04:02 atop_20091124.1.1 -rw------- 1 root root 0 Nov 29 04:02 atop_20091124.1.1.1 -rw------- 1 root root 0 Nov 30 04:02 atop_20091125 -rw------- 1 root root 0 Nov 30 04:02 atop_20091125.1 -rw------- 1 root root 0 Nov 30 04:02 atop_20091125.1.1 -rw------- 1 root root 0 Nov 29 04:02 atop_20091125.1.1.1 -rw------- 1 root root 0 Nov 30 04:02 atop_20091126 -rw------- 1 root root 0 Nov 30 04:02 atop_20091126.1 -rw------- 1 root root 0 Nov 29 04:02 atop_20091126.1.1 -rw------- 1 root root 0 Nov 30 04:02 atop_20091127 -rw------- 1 root root 0 Nov 30 04:02 atop_20091127.1 -rw------- 1 root root 0 Nov 29 04:02 atop_20091127.1.1 -rw------- 1 root root 0 Nov 30 04:02 atop_20091128 -rw------- 1 root root 0 Nov 29 04:02 atop_20091128.1 -rw------- 1 root root 0 Nov 30 04:02 atop_20091129 -rw-r--r-- 1 root root 838518 Nov 30 00:00 atop_20091129.1 -rw-r--r-- 1 root root 356578 Nov 30 11:02 atop_20091130 -rw------- 1 root root 0 Nov 30 04:02 atop.log -rw-r--r-- 1 root root 0 Nov 29 04:02 atop.log.1 Expected results: [root@betternode ~]# ls -l /var/log/atop/ total 1264 -rw------- 1 root root 0 Nov 24 04:02 atop_20091124 -rw------- 1 root root 0 Nov 25 04:02 atop_20091125 -rw------- 1 root root 0 Nov 26 04:02 atop_20091126 -rw------- 1 root root 0 Nov 27 04:02 atop_20091127 -rw------- 1 root root 0 Nov 28 04:02 atop_20091128 -rw-r--r-- 1 root root 838518 Nov 29 16:23 atop_20091129 -rw-r--r-- 1 root root 356578 Nov 30 11:02 atop_20091130 -rw------- 1 root root 0 Nov 30 04:02 atop.log -rw-r--r-- 1 root root 0 Nov 29 04:02 atop.log.1 Additional info: Since atop already starts a new logfile each night, it's logrotate should not rotate these at all, and should certainly not create empty new files.
*** Bug 608514 has been marked as a duplicate of this bug. ***
I've seen this exact same behavior. Its worth noting that not only is logrotate unnecessary, but it may somehow be causing the data to be discarded. In the example Matthias posted, the rawfiles for 20091124 through 20091128 probably shouldn't be empty.
i don`t think that atop needs this logrotate script at all. There is already cron script which restarts atop and change the name of the log. Its much better to add old log purging to it, then to do this with logrotate.
Also log files are binary and compression rate is very small (and files itself are not huge), so its just wasting of the CPU time.
So, is the fix to disable the section for /var/log/atop/atop_* in /etc/logrotate.d/atop and remove all files ending in multiple .1's? I guess the section for /var/log/atop/atop.log can stay, right?
This bug seems to be the cause of hundreds of thousands of files in /var/log/atop on our servers that we're cleaning up right now. If anyone has a workaround or a fix, that would be great.
Not sure why this wasn't assigned to me. . .
I'll just drop the logrotate bits since atop handles it, once we figure out the cron bit.
Shouldn't the logrotate script be kept for rotating /var/log/atop/atop.log? I believe only the first section in the logrotate script, the one which rotates /var/log/atop/atop_*, is causing problems. And I guess there needs to be some process which removes old /var/log/atop/atop_* files. I don't know if atop has built-in functionality for this and if the period to keep /var/log/atop/atop_* history needs to be configurable, but without something like that the number of /var/log/atop/atop_* files will increase daily.
Ah, I see. I'll put that back modulo the following. It looks like logrotate doesn't support date-stamped logs as input, only output. Changing the ifempty option to notifempty will help, but it looks like by default atop keeps all the dated logs. You could set up something like find /var/log/atop/atop_* -mtime +5 -exec rm {} \; which would remove anything older than 5 days, as an example. I'm hesitant to include that as a default, though, since it modifies atop's expected upstream behaviour.
I know I initially argued for just dropping the logrotate script, and that's what I did on my systems, but this leads to the problem of the binary logs (e.g. /var/log/atop/atop_20120508) never being purged. Someone could patch the cron script to purge them. Keep in mind that the cron script and init script are essentially duplicates, so all cron really needs to do is call the '/etc/init.d/atop condrestart'. Alternately, someone could look in more detail at what Debian does. They drop the cron job and use three logrotate scripts. http://packages.debian.org/squeeze/atop
I know that I had changed the logrotate when I encountered this issue, but by now I simply do not remember what exactly I did. Sorry for not being much help...
Oops, I meant to say that /etc/cron.d/atop and /usr/bin/atopd are essentially duplicates; all cron really needs to do is call '/etc/init.d/atop condrestart' I mentioned this before at https://bugzilla.redhat.com/show_bug.cgi?id=445174 As to the find command for removing old logs, that's a good idea. I modified /etc/sysconfig/atop to make this configurable and added the find command to /usr/bin/atopd. I used find's built-in delete action rather than execing rm. You can see how I have atop configured in my environment at https://github.com/sciurus/puppet-atop.
Ok, have a look at this scratch build. http://koji.fedoraproject.org/koji/taskinfo?taskID=4068062
Removing old /var/log/atop/atop_* data files is not handled in this build as far as I can see. Otherwise it looks fine to me.
The /var/log/atop/atop_* data files are a couple of megabytes each, so if they don't get removed after some time the /var/log/atop/atop_* files would start to take up quite a bit of space.
Ok. I understand, but as far as I can tell upstream makes to provision for handling that, so users would expect all their data to be there until they did something about it, like setting up a cron job with the above find command or something akin. I'll add a note with that find command to the docs.
atop-1.26-5.fc15 has been submitted as an update for Fedora 15. https://admin.fedoraproject.org/updates/atop-1.26-5.fc15
atop-1.26-2.el5.1 has been submitted as an update for Fedora EPEL 5. https://admin.fedoraproject.org/updates/atop-1.26-2.el5.1
atop-1.26-5.fc16 has been submitted as an update for Fedora 16. https://admin.fedoraproject.org/updates/atop-1.26-5.fc16
atop-1.26-2.el6 has been submitted as an update for Fedora EPEL 6. https://admin.fedoraproject.org/updates/atop-1.26-2.el6
atop-1.26-5.fc17 has been submitted as an update for Fedora 17. https://admin.fedoraproject.org/updates/atop-1.26-5.fc17
I think it's okay to leave the log rotation script for atop.log. However, as far as I can see atop will never write to atop.log. I took a look at upstream today and realized they are distributing their own RPMS. They have differences from the EPEL RPMS, not just in terms of file layout as I expected but also in file contents. For example, they include a find command to delete old logs after 28 days. Here are the files in http://www.atoptool.nl/download/atop-1.26-2.i586.rpm ./var ./var/log ./var/log/atop ./usr ./usr/bin ./usr/bin/atopsar ./usr/bin/atop ./usr/share ./usr/share/doc ./usr/share/doc/packages ./usr/share/doc/packages/atop ./usr/share/doc/packages/atop/AUTHOR ./usr/share/doc/packages/atop/COPYING ./usr/share/doc/packages/atop/README ./usr/share/doc/packages/atop/ChangeLog ./usr/share/man ./usr/share/man/man1 ./usr/share/man/man1/atop.1.gz ./usr/share/man/man1/atopsar.1.gz ./usr/share/man/man5 ./usr/share/man/man5/atoprc.5.gz ./etc ./etc/init.d ./etc/init.d/atop ./etc/logrotate.d ./etc/logrotate.d/psaccu_atop ./etc/logrotate.d/psaccs_atop ./etc/atop ./etc/atop/atop.daily ./etc/atop/45atoppm ./etc/cron.d ./etc/cron.d/atop
<sigh> If I were them, I'd just write everything to one log file, with dates in each line, and make the code that consumes the log data parse the dates and ignore the filenames, so you could do the whole thing with logrotate. Oh well.
Package atop-1.26-5.fc17: * should fix your issue, * was pushed to the Fedora 17 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing atop-1.26-5.fc17' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2012-7733/atop-1.26-5.fc17 then log in and leave karma (feedback).
atop-1.26-6.fc17 has been submitted as an update for Fedora 17. https://admin.fedoraproject.org/updates/atop-1.26-6.fc17
atop-1.26-3.el5 has been submitted as an update for Fedora EPEL 5. https://admin.fedoraproject.org/updates/atop-1.26-3.el5
atop-1.26-3.el6 has been submitted as an update for Fedora EPEL 6. https://admin.fedoraproject.org/updates/atop-1.26-3.el6
atop-1.26-6.fc15 has been submitted as an update for Fedora 15. https://admin.fedoraproject.org/updates/atop-1.26-6.fc15
atop-1.26-6.fc16 has been submitted as an update for Fedora 16. https://admin.fedoraproject.org/updates/atop-1.26-6.fc16
atop-1.26-6.fc16 has been pushed to the Fedora 16 stable repository. If problems still persist, please make note of it in this bug report.
atop-1.26-6.fc15 has been pushed to the Fedora 15 stable repository. If problems still persist, please make note of it in this bug report.
atop-1.26-6.fc17 has been pushed to the Fedora 17 stable repository. If problems still persist, please make note of it in this bug report.
atop-1.26-3.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report.
atop-1.26-3.el5 has been pushed to the Fedora EPEL 5 stable repository. If problems still persist, please make note of it in this bug report.