Description of problem: I have 10GB log files of mongodb /var/lib/mongodb directory. Please add housekeeping: - Compress log files - Delete ancient log files [crash] root@li-lc-1017:/var/lib/mongodb# df -h /var/lib/mongodb Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_li_lc_1017_app1-lv_mongodb 59G 53G 3.5G 94% /var/lib/mongodb [crash] root@li-lc-1017:/var/lib/mongodb# ls -l total 51848016 drwxr-xr-x. 2 mongodb mongodb 4096 Mar 14 10:48 journal drwxr-xr-x. 2 root root 4096 Mar 14 10:48 local -rw-------. 1 root root 67108864 Mar 14 10:48 local.0 -rw-------. 1 root root 16777216 Mar 14 10:48 local.ns drwx------. 2 root root 16384 Oct 14 17:15 lost+found -rw-r--r--. 1 mongodb mongodb 369421818 Mar 14 10:48 mongodb.log -rw-r--r--. 1 mongodb mongodb 106265322 Oct 25 03:47 mongodb.log.2015-10-25T03-47-07 -rw-r--r--. 1 mongodb mongodb 2341141066 Nov 24 03:23 mongodb.log.2015-11-24T03-23-28 -rw-r--r--. 1 mongodb mongodb 739217660 Dec 1 03:11 mongodb.log.2015-12-01T03-11-29 -rw-r--r--. 1 mongodb mongodb 641503463 Dec 6 03:20 mongodb.log.2015-12-06T03-20-32 -rw-r--r--. 1 mongodb mongodb 1452726916 Dec 17 03:20 mongodb.log.2015-12-17T03-20-28 -rw-r--r--. 1 mongodb mongodb 77058127 Dec 19 23:15 mongodb.log.2015-12-20T03-18-29 -rw-r--r--. 1 mongodb mongodb 856449942 Dec 30 03:47 mongodb.log.2015-12-30T03-47-15 -rw-r--r--. 1 mongodb mongodb 1711882544 Jan 26 03:48 mongodb.log.2016-01-26T03-49-14 -rw-r--r--. 1 mongodb mongodb 271079611 Jan 31 02:48 mongodb.log.2016-01-31T03-50-45 -rw-r--r--. 1 mongodb mongodb 1470183265 Mar 4 03:27 mongodb.log.2016-03-04T03-37-19 -rw-r--r--. 1 mongodb mongodb 91353295 Mar 6 02:30 mongodb.log.2016-03-06T03-18-34 -rwxr-xr-x. 1 mongodb mongodb 0 Mar 14 10:48 mongod.lock -rw-------. 1 mongodb mongodb 67108864 Mar 14 10:48 pulp_database.0 -rw-------. 1 mongodb mongodb 134217728 Mar 14 05:38 pulp_database.1 -rw-------. 1 mongodb mongodb 2146435072 Mar 14 08:41 pulp_database.10 -rw-------. 1 mongodb mongodb 2146435072 Mar 14 08:41 pulp_database.11 -rw-------. 1 mongodb mongodb 2146435072 Mar 14 08:41 pulp_database.12 -rw-------. 1 mongodb mongodb 2146435072 Mar 14 05:38 pulp_database.13 -rw-------. 1 mongodb mongodb 2146435072 Mar 14 05:38 pulp_database.14 -rw-------. 1 mongodb mongodb 2146435072 Mar 14 05:38 pulp_database.15 -rw-------. 1 mongodb mongodb 2146435072 Mar 14 05:38 pulp_database.16 -rw-------. 1 mongodb mongodb 2146435072 Mar 14 08:41 pulp_database.17 -rw-------. 1 mongodb mongodb 2146435072 Mar 14 05:38 pulp_database.18 -rw-------. 1 mongodb mongodb 2146435072 Mar 14 05:38 pulp_database.19 -rw-------. 1 mongodb mongodb 268435456 Mar 14 05:38 pulp_database.2 -rw-------. 1 mongodb mongodb 2146435072 Mar 14 05:38 pulp_database.20 -rw-------. 1 mongodb mongodb 2146435072 Mar 14 05:07 pulp_database.21 -rw-------. 1 mongodb mongodb 2146435072 Mar 14 05:38 pulp_database.22 -rw-------. 1 mongodb mongodb 2146435072 Feb 7 23:21 pulp_database.23 -rw-------. 1 mongodb mongodb 536870912 Mar 14 05:21 pulp_database.3 -rw-------. 1 mongodb mongodb 1073741824 Mar 14 08:41 pulp_database.4 -rw-------. 1 mongodb mongodb 2146435072 Mar 14 05:38 pulp_database.5 -rw-------. 1 mongodb mongodb 2146435072 Mar 14 08:41 pulp_database.6 -rw-------. 1 mongodb mongodb 2146435072 Mar 14 08:41 pulp_database.7 -rw-------. 1 mongodb mongodb 2146435072 Mar 14 08:41 pulp_database.8 -rw-------. 1 mongodb mongodb 2146435072 Mar 14 08:41 pulp_database.9 -rw-------. 1 mongodb mongodb 16777216 Mar 14 08:41 pulp_database.ns Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
The other BZ https://bugzilla.redhat.com/show_bug.cgi?id=1269227 is about the wrong location and selinux context. This BZ is about housekeeping
Moving 6.2 bugs out to sat-backlog.
Situation with Sat6.2.4 on RHEL7.3 improved. Partly it uses our systemwide logrotation configuration [crash] root@li-lc-1017:/etc# cat logrotate.d/mongodb /var/log/mongodb/*.log { weekly rotate 10 copytruncate delaycompress compress notifempty missingok postrotate /bin/kill -USR1 `cat /var/run/mongodb/mongod.pid 2>/dev/null` 2> /dev/null|| true endscript } But there are strange duplicate files with date suffixes like '2016-10-23T03-19-25' and on 20161021 there is even a duplicate between compressed and uncompressed. $ sudo ls -l /var/log/mongodb/ total 1272816 -rw-r--r--. 1 mongodb mongodb 882338206 Dec 1 10:43 mongodb.log -rw-r--r--. 1 mongodb mongodb 55039530 Oct 23 03:19 mongodb.log-20161023.gz -rw-r--r--. 1 mongodb mongodb 0 Oct 23 03:19 mongodb.log.2016-10-23T03-19-25 -rw-r--r--. 1 mongodb mongodb 19605531 Oct 30 03:21 mongodb.log-20161030.gz -rw-r--r--. 1 mongodb mongodb 0 Oct 30 03:21 mongodb.log.2016-10-30T03-21-51 -rw-r--r--. 1 mongodb mongodb 29261929 Nov 6 03:46 mongodb.log-20161106.gz -rw-r--r--. 1 mongodb mongodb 0 Nov 6 03:46 mongodb.log.2016-11-06T03-46-44 -rw-r--r--. 1 mongodb mongodb 63012359 Nov 13 03:35 mongodb.log-20161113.gz -rw-r--r--. 1 mongodb mongodb 0 Nov 13 03:35 mongodb.log.2016-11-13T03-35-56 -rw-r--r--. 1 mongodb mongodb 83346603 Nov 21 03:41 mongodb.log-20161121.gz -rw-r--r--. 1 mongodb mongodb 655373184 Nov 21 03:42 mongodb.log.2016-11-21T03-42-35 -rw-r--r--. 1 mongodb mongodb 170731102 Nov 27 03:37 mongodb.log-20161127 -rw-r--r--. 1 mongodb mongodb 0 Nov 27 03:37 mongodb.log.2016-11-27T03-37-46
Looking at https://docs.mongodb.com/v2.6/tutorial/rotate-log-files/ it looks like logrotation is also enabled in mongodb itself.
The original reported issue was fixed here: https://bugzilla.redhat.com/show_bug.cgi?id=1269227 As for the strange 0-byte files you're now seeing, I have a theory on that. Although I don't see that behavior being reproduced with Satellite 6.2.6 on either el6 or el7. The theory, inspired by comment #6: logrotate does its thing. We end up with a 0-byte "mongodb.log", and a new "mongodb.log-xxxxxxxx.gz". In the "postrotate", it sends SIGUSR1 to mongodb, which actually triggers mongodb to do *its own log rotation* per the docs [0]. But mongodb.log is empty, which is why the rotated file is also empty. The only problem with this theory is your rotate artifacts from Nov 21, where the uncompressed file is quite large. <shrug> Why don't we see these secondary-rotation artifacts appearing with 6.2.6? So far I have no idea. [0] https://docs.mongodb.com/v2.6/tutorial/rotate-log-files/
Correct, the empty log files is a general issue like. A bit of googling gives also other discussions like http://serverfault.com/questions/540423/mongodb-proper-way-to-rotate-logs
The huge uncompressed file is also explained: It is a spare file, where mongodb is still writing a filepostion N after the logrotate did the rotation and before the USR1 is received by mongod See the Size vs. Blocks in the stat and the hexdump of the first 20 0x00 characters: [crash] root@li-lc-1017:~# head -c 20 /var/log/mongodb/mongodb.log.2016-12-05T03-22-28 | xxd 0000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000010: 0000 0000 .... [crash] root@li-lc-1017:~# stat /var/log/mongodb/mongodb.log.2016-12-05T03-22-28 File: ‘/var/log/mongodb/mongodb.log.2016-12-05T03-22-28’ Size: 1342099293 Blocks: 16 IO Block: 4096 regular file Device: fd07h/64775d Inode: 11156 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 184/ mongodb) Gid: ( 130/ mongodb) Context: system_u:object_r:mongod_log_t:s0 Access: 2017-01-04 16:14:49.667227212 +0000 Modify: 2016-12-05 03:22:28.217426986 +0000 Change: 2016-12-05 03:22:28.388427487 +0000 Birth: -
Thank you for providing this helpful info. I filed a bug about the logrotate behavior against the upstream package: https://bugzilla.redhat.com/show_bug.cgi?id=1410238 That's the best place to track progress on fixing the logrotate issue for now. Once they fix it, that change will make its way into Satellite. Since the original problem reported via this issue was previously resolved, I am closing this as a duplicate. *** This bug has been marked as a duplicate of bug 1269227 ***