Bug 542598 - atop's logrotate creates avalanche of logfiles
Summary: atop's logrotate creates avalanche of logfiles
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora EPEL
Classification: Fedora
Component: atop
Version: el5
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
Assignee: Gwyn Ciesla
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 608514 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-11-30 10:24 UTC by Matthias Schroder
Modified: 2012-05-31 01:15 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-05-24 23:58:53 UTC


Attachments (Terms of Use)

Description Matthias Schroder 2009-11-30 10:24:10 UTC
Description of problem: atop's logrotate is not compatible with the way that atop creates it's daily logfile. Together they create a multitude of rotated files.


Version-Release number of selected component (if applicable): 1.23-8.el5


How reproducible: 100%


Steps to Reproduce:
1. install atop
2. wait 4 days
3. ls -l /var/log/atop
  
Actual results:

[root@thisnode ~]# ls -l /var/log/atop/
total 1264
-rw------- 1 root root      0 Nov 30 04:02 atop_20091124
-rw------- 1 root root      0 Nov 30 04:02 atop_20091124.1
-rw------- 1 root root      0 Nov 30 04:02 atop_20091124.1.1
-rw------- 1 root root      0 Nov 29 04:02 atop_20091124.1.1.1
-rw------- 1 root root      0 Nov 30 04:02 atop_20091125
-rw------- 1 root root      0 Nov 30 04:02 atop_20091125.1
-rw------- 1 root root      0 Nov 30 04:02 atop_20091125.1.1
-rw------- 1 root root      0 Nov 29 04:02 atop_20091125.1.1.1
-rw------- 1 root root      0 Nov 30 04:02 atop_20091126
-rw------- 1 root root      0 Nov 30 04:02 atop_20091126.1
-rw------- 1 root root      0 Nov 29 04:02 atop_20091126.1.1
-rw------- 1 root root      0 Nov 30 04:02 atop_20091127
-rw------- 1 root root      0 Nov 30 04:02 atop_20091127.1
-rw------- 1 root root      0 Nov 29 04:02 atop_20091127.1.1
-rw------- 1 root root      0 Nov 30 04:02 atop_20091128
-rw------- 1 root root      0 Nov 29 04:02 atop_20091128.1
-rw------- 1 root root      0 Nov 30 04:02 atop_20091129
-rw-r--r-- 1 root root 838518 Nov 30 00:00 atop_20091129.1
-rw-r--r-- 1 root root 356578 Nov 30 11:02 atop_20091130
-rw------- 1 root root      0 Nov 30 04:02 atop.log
-rw-r--r-- 1 root root      0 Nov 29 04:02 atop.log.1

Expected results:
[root@betternode ~]# ls -l /var/log/atop/
total 1264
-rw------- 1 root root      0 Nov 24 04:02 atop_20091124
-rw------- 1 root root      0 Nov 25 04:02 atop_20091125
-rw------- 1 root root      0 Nov 26 04:02 atop_20091126
-rw------- 1 root root      0 Nov 27 04:02 atop_20091127
-rw------- 1 root root      0 Nov 28 04:02 atop_20091128
-rw-r--r-- 1 root root 838518 Nov 29 16:23 atop_20091129
-rw-r--r-- 1 root root 356578 Nov 30 11:02 atop_20091130
-rw------- 1 root root      0 Nov 30 04:02 atop.log
-rw-r--r-- 1 root root      0 Nov 29 04:02 atop.log.1


Additional info: Since atop already starts a new logfile each night, it's logrotate should not rotate these at all, and should certainly not create empty new files.

Comment 1 Brian Pitts 2010-06-28 14:42:25 UTC
*** Bug 608514 has been marked as a duplicate of this bug. ***

Comment 2 Brian Pitts 2010-06-28 16:15:46 UTC
I've seen this exact same behavior. Its worth noting that not only is logrotate unnecessary, but it may somehow be causing the data to be discarded. In the example Matthias posted, the rawfiles for 20091124 through 20091128 probably shouldn't be empty.

Comment 3 Alex Samorukov 2010-08-09 08:40:20 UTC
i don`t think that atop needs this logrotate script at all. There is already cron script which restarts atop and change the name of the log. Its much better to add old log purging to it, then to do this with logrotate.

Comment 4 Alex Samorukov 2010-08-09 08:52:25 UTC
Also log files are binary and compression rate is very small (and files itself are not huge), so its just wasting of the CPU time.

Comment 5 Nils Breunese 2012-05-07 12:57:59 UTC
So, is the fix to disable the section for /var/log/atop/atop_* in /etc/logrotate.d/atop and remove all files ending in multiple .1's?

I guess the section for /var/log/atop/atop.log can stay, right?

Comment 6 Nils Breunese 2012-05-07 13:12:14 UTC
This bug seems to be the cause of hundreds of thousands of files in /var/log/atop on our servers that we're cleaning up right now. If anyone has a workaround or a fix, that would be great.

Comment 7 Gwyn Ciesla 2012-05-07 16:40:48 UTC
Not sure why this wasn't assigned to me. . .

Comment 8 Gwyn Ciesla 2012-05-07 18:52:11 UTC
I'll just drop the logrotate bits since atop handles it, once we figure out the cron bit.

Comment 9 Nils Breunese 2012-05-09 14:21:59 UTC
Shouldn't the logrotate script be kept for rotating /var/log/atop/atop.log? I believe only the first section in the logrotate script, the one which rotates /var/log/atop/atop_*, is causing problems.

And I guess there needs to be some process which removes old /var/log/atop/atop_* files. I don't know if atop has built-in functionality for this and if the period to keep /var/log/atop/atop_* history needs to be configurable, but without something like that the number of /var/log/atop/atop_* files will increase daily.

Comment 10 Gwyn Ciesla 2012-05-09 14:49:49 UTC
Ah, I see.  I'll put that back modulo the following.

It looks like logrotate doesn't support date-stamped logs as input, only output.  Changing the ifempty option to notifempty will help, but it looks like by default atop keeps all the dated logs.  You could set up something like

find /var/log/atop/atop_* -mtime +5 -exec rm {} \;

which would remove anything older than 5 days, as an example.  I'm hesitant to include that as a default, though, since it modifies atop's expected upstream behaviour.

Comment 11 Brian Pitts 2012-05-09 15:00:13 UTC
I know I initially argued for just dropping the logrotate script, and that's
what I did on my systems, but this leads to the problem of the binary logs
(e.g. /var/log/atop/atop_20120508) never being purged.

Someone could patch the cron script to purge them. Keep in mind that the cron script and init script are essentially duplicates, so all cron really needs to do is call the '/etc/init.d/atop condrestart'.

Alternately, someone could look in more detail at what Debian does. They drop
the cron job and use three logrotate scripts.

http://packages.debian.org/squeeze/atop

Comment 12 Matthias Schroder 2012-05-09 16:01:25 UTC
I know that I had changed the logrotate when I encountered this issue, but by now I simply do not remember what exactly I did. Sorry for not being much help...

Comment 13 Brian Pitts 2012-05-09 21:06:02 UTC
Oops, I meant to say that /etc/cron.d/atop and /usr/bin/atopd are essentially duplicates; all cron really needs to do is call '/etc/init.d/atop condrestart' I mentioned this before at https://bugzilla.redhat.com/show_bug.cgi?id=445174

As to the find command for removing old logs, that's a good idea. I modified /etc/sysconfig/atop to make this configurable and added the find command to /usr/bin/atopd. I used find's built-in delete action rather than execing rm.

You can see how I have atop configured in my environment at https://github.com/sciurus/puppet-atop.

Comment 14 Gwyn Ciesla 2012-05-10 14:17:45 UTC
Ok, have a look at this scratch build.

http://koji.fedoraproject.org/koji/taskinfo?taskID=4068062

Comment 15 Nils Breunese 2012-05-10 19:48:46 UTC
Removing old /var/log/atop/atop_* data files is not handled in this build as far as I can see. Otherwise it looks fine to me.

Comment 16 Nils Breunese 2012-05-10 19:52:56 UTC
The /var/log/atop/atop_* data files are a couple of megabytes each, so if they don't get removed after some time the /var/log/atop/atop_* files would start to take up quite a bit of space.

Comment 17 Gwyn Ciesla 2012-05-11 12:41:32 UTC
Ok.  I understand, but as far as I can tell upstream makes to provision for handling that, so users would expect all their data to be there until they did something about it, like setting up a cron job with the above find command or something akin.  I'll add a note with that find command to the docs.

Comment 18 Fedora Update System 2012-05-11 13:53:29 UTC
atop-1.26-5.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/atop-1.26-5.fc15

Comment 19 Fedora Update System 2012-05-11 13:53:48 UTC
atop-1.26-2.el5.1 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/atop-1.26-2.el5.1

Comment 20 Fedora Update System 2012-05-11 13:54:24 UTC
atop-1.26-5.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/atop-1.26-5.fc16

Comment 21 Fedora Update System 2012-05-11 13:54:40 UTC
atop-1.26-2.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/atop-1.26-2.el6

Comment 22 Fedora Update System 2012-05-11 13:54:56 UTC
atop-1.26-5.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/atop-1.26-5.fc17

Comment 23 Brian Pitts 2012-05-11 15:14:31 UTC
I think it's okay to leave the log rotation script for atop.log. However, as far as I can see atop will never write to atop.log.

I took a look at upstream today and realized they are distributing their own RPMS. They have differences from the EPEL RPMS, not just in terms of file layout as I expected but also in file contents. For example, they include a find command to delete old logs after 28 days.

Here are the files in http://www.atoptool.nl/download/atop-1.26-2.i586.rpm
./var
./var/log
./var/log/atop
./usr
./usr/bin
./usr/bin/atopsar
./usr/bin/atop
./usr/share
./usr/share/doc
./usr/share/doc/packages
./usr/share/doc/packages/atop
./usr/share/doc/packages/atop/AUTHOR
./usr/share/doc/packages/atop/COPYING
./usr/share/doc/packages/atop/README
./usr/share/doc/packages/atop/ChangeLog
./usr/share/man
./usr/share/man/man1
./usr/share/man/man1/atop.1.gz
./usr/share/man/man1/atopsar.1.gz
./usr/share/man/man5
./usr/share/man/man5/atoprc.5.gz
./etc
./etc/init.d
./etc/init.d/atop
./etc/logrotate.d
./etc/logrotate.d/psaccu_atop
./etc/logrotate.d/psaccs_atop
./etc/atop
./etc/atop/atop.daily
./etc/atop/45atoppm
./etc/cron.d
./etc/cron.d/atop

Comment 24 Gwyn Ciesla 2012-05-11 15:20:45 UTC
<sigh>

If I were them, I'd just write everything to one log file, with dates in each line, and make the code that consumes the log data parse the dates and ignore the filenames, so you could do the whole thing with logrotate.

Oh well.

Comment 25 Fedora Update System 2012-05-11 21:53:08 UTC
Package atop-1.26-5.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing atop-1.26-5.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-7733/atop-1.26-5.fc17
then log in and leave karma (feedback).

Comment 26 Fedora Update System 2012-05-14 14:22:42 UTC
atop-1.26-6.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/atop-1.26-6.fc17

Comment 27 Fedora Update System 2012-05-14 14:23:06 UTC
atop-1.26-3.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/atop-1.26-3.el5

Comment 28 Fedora Update System 2012-05-14 14:23:28 UTC
atop-1.26-3.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/atop-1.26-3.el6

Comment 29 Fedora Update System 2012-05-14 14:23:51 UTC
atop-1.26-6.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/atop-1.26-6.fc15

Comment 30 Fedora Update System 2012-05-14 14:24:34 UTC
atop-1.26-6.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/atop-1.26-6.fc16

Comment 31 Fedora Update System 2012-05-24 23:58:53 UTC
atop-1.26-6.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 32 Fedora Update System 2012-05-25 00:00:00 UTC
atop-1.26-6.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 33 Fedora Update System 2012-05-26 07:45:06 UTC
atop-1.26-6.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 34 Fedora Update System 2012-05-31 01:13:08 UTC
atop-1.26-3.el6 has been pushed to the Fedora EPEL 6 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 35 Fedora Update System 2012-05-31 01:15:41 UTC
atop-1.26-3.el5 has been pushed to the Fedora EPEL 5 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.