Bug 1317479 - Mongodb log files are huge, rotation does not do compression and deletion
Summary: Mongodb log files are huge, rotation does not do compression and deletion
Keywords:
Status: CLOSED DUPLICATE of bug 1269227
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.1.7
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Katello QA List
URL:
Whiteboard:
Depends On:
Blocks: 1122832
TreeView+ depends on / blocked
 
Reported: 2016-03-14 10:57 UTC by Peter Vreman
Modified: 2017-01-04 21:31 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-04 21:31:46 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1269227 0 high CLOSED MongoDB logs are directed to incorrect location / incorrect selinux context 2021-02-22 00:41:40 UTC

Description Peter Vreman 2016-03-14 10:57:24 UTC
Description of problem:
I have 10GB log files of mongodb /var/lib/mongodb directory.
Please add housekeeping:
- Compress log files
- Delete ancient log files


[crash] root@li-lc-1017:/var/lib/mongodb# df -h /var/lib/mongodb
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_li_lc_1017_app1-lv_mongodb
                       59G   53G  3.5G  94% /var/lib/mongodb

[crash] root@li-lc-1017:/var/lib/mongodb# ls -l
total 51848016
drwxr-xr-x. 2 mongodb mongodb       4096 Mar 14 10:48 journal
drwxr-xr-x. 2 root    root          4096 Mar 14 10:48 local
-rw-------. 1 root    root      67108864 Mar 14 10:48 local.0
-rw-------. 1 root    root      16777216 Mar 14 10:48 local.ns
drwx------. 2 root    root         16384 Oct 14 17:15 lost+found
-rw-r--r--. 1 mongodb mongodb  369421818 Mar 14 10:48 mongodb.log
-rw-r--r--. 1 mongodb mongodb  106265322 Oct 25 03:47 mongodb.log.2015-10-25T03-47-07
-rw-r--r--. 1 mongodb mongodb 2341141066 Nov 24 03:23 mongodb.log.2015-11-24T03-23-28
-rw-r--r--. 1 mongodb mongodb  739217660 Dec  1 03:11 mongodb.log.2015-12-01T03-11-29
-rw-r--r--. 1 mongodb mongodb  641503463 Dec  6 03:20 mongodb.log.2015-12-06T03-20-32
-rw-r--r--. 1 mongodb mongodb 1452726916 Dec 17 03:20 mongodb.log.2015-12-17T03-20-28
-rw-r--r--. 1 mongodb mongodb   77058127 Dec 19 23:15 mongodb.log.2015-12-20T03-18-29
-rw-r--r--. 1 mongodb mongodb  856449942 Dec 30 03:47 mongodb.log.2015-12-30T03-47-15
-rw-r--r--. 1 mongodb mongodb 1711882544 Jan 26 03:48 mongodb.log.2016-01-26T03-49-14
-rw-r--r--. 1 mongodb mongodb  271079611 Jan 31 02:48 mongodb.log.2016-01-31T03-50-45
-rw-r--r--. 1 mongodb mongodb 1470183265 Mar  4 03:27 mongodb.log.2016-03-04T03-37-19
-rw-r--r--. 1 mongodb mongodb   91353295 Mar  6 02:30 mongodb.log.2016-03-06T03-18-34
-rwxr-xr-x. 1 mongodb mongodb          0 Mar 14 10:48 mongod.lock
-rw-------. 1 mongodb mongodb   67108864 Mar 14 10:48 pulp_database.0
-rw-------. 1 mongodb mongodb  134217728 Mar 14 05:38 pulp_database.1
-rw-------. 1 mongodb mongodb 2146435072 Mar 14 08:41 pulp_database.10
-rw-------. 1 mongodb mongodb 2146435072 Mar 14 08:41 pulp_database.11
-rw-------. 1 mongodb mongodb 2146435072 Mar 14 08:41 pulp_database.12
-rw-------. 1 mongodb mongodb 2146435072 Mar 14 05:38 pulp_database.13
-rw-------. 1 mongodb mongodb 2146435072 Mar 14 05:38 pulp_database.14
-rw-------. 1 mongodb mongodb 2146435072 Mar 14 05:38 pulp_database.15
-rw-------. 1 mongodb mongodb 2146435072 Mar 14 05:38 pulp_database.16
-rw-------. 1 mongodb mongodb 2146435072 Mar 14 08:41 pulp_database.17
-rw-------. 1 mongodb mongodb 2146435072 Mar 14 05:38 pulp_database.18
-rw-------. 1 mongodb mongodb 2146435072 Mar 14 05:38 pulp_database.19
-rw-------. 1 mongodb mongodb  268435456 Mar 14 05:38 pulp_database.2
-rw-------. 1 mongodb mongodb 2146435072 Mar 14 05:38 pulp_database.20
-rw-------. 1 mongodb mongodb 2146435072 Mar 14 05:07 pulp_database.21
-rw-------. 1 mongodb mongodb 2146435072 Mar 14 05:38 pulp_database.22
-rw-------. 1 mongodb mongodb 2146435072 Feb  7 23:21 pulp_database.23
-rw-------. 1 mongodb mongodb  536870912 Mar 14 05:21 pulp_database.3
-rw-------. 1 mongodb mongodb 1073741824 Mar 14 08:41 pulp_database.4
-rw-------. 1 mongodb mongodb 2146435072 Mar 14 05:38 pulp_database.5
-rw-------. 1 mongodb mongodb 2146435072 Mar 14 08:41 pulp_database.6
-rw-------. 1 mongodb mongodb 2146435072 Mar 14 08:41 pulp_database.7
-rw-------. 1 mongodb mongodb 2146435072 Mar 14 08:41 pulp_database.8
-rw-------. 1 mongodb mongodb 2146435072 Mar 14 08:41 pulp_database.9
-rw-------. 1 mongodb mongodb   16777216 Mar 14 08:41 pulp_database.ns



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Peter Vreman 2016-03-14 13:04:38 UTC
The other BZ https://bugzilla.redhat.com/show_bug.cgi?id=1269227 is about the wrong location and selinux context.
This BZ is about housekeeping

Comment 2 Bryan Kearney 2016-07-26 19:01:56 UTC
Moving 6.2 bugs out to sat-backlog.

Comment 5 Peter Vreman 2016-12-01 12:17:01 UTC
Situation with Sat6.2.4 on RHEL7.3 improved.
Partly it uses our systemwide logrotation configuration

[crash] root@li-lc-1017:/etc# cat logrotate.d/mongodb
/var/log/mongodb/*.log {
       weekly
       rotate 10
       copytruncate
       delaycompress
       compress
       notifempty
       missingok
       postrotate
          /bin/kill -USR1 `cat /var/run/mongodb/mongod.pid 2>/dev/null` 2> /dev/null|| true
       endscript
}

But there are strange duplicate files with date suffixes like '2016-10-23T03-19-25' and on 20161021  there is even a duplicate between compressed and uncompressed.


$ sudo ls -l /var/log/mongodb/
total 1272816
-rw-r--r--. 1 mongodb mongodb 882338206 Dec  1 10:43 mongodb.log
-rw-r--r--. 1 mongodb mongodb  55039530 Oct 23 03:19 mongodb.log-20161023.gz
-rw-r--r--. 1 mongodb mongodb         0 Oct 23 03:19 mongodb.log.2016-10-23T03-19-25
-rw-r--r--. 1 mongodb mongodb  19605531 Oct 30 03:21 mongodb.log-20161030.gz
-rw-r--r--. 1 mongodb mongodb         0 Oct 30 03:21 mongodb.log.2016-10-30T03-21-51
-rw-r--r--. 1 mongodb mongodb  29261929 Nov  6 03:46 mongodb.log-20161106.gz
-rw-r--r--. 1 mongodb mongodb         0 Nov  6 03:46 mongodb.log.2016-11-06T03-46-44
-rw-r--r--. 1 mongodb mongodb  63012359 Nov 13 03:35 mongodb.log-20161113.gz
-rw-r--r--. 1 mongodb mongodb         0 Nov 13 03:35 mongodb.log.2016-11-13T03-35-56
-rw-r--r--. 1 mongodb mongodb  83346603 Nov 21 03:41 mongodb.log-20161121.gz
-rw-r--r--. 1 mongodb mongodb 655373184 Nov 21 03:42 mongodb.log.2016-11-21T03-42-35
-rw-r--r--. 1 mongodb mongodb 170731102 Nov 27 03:37 mongodb.log-20161127
-rw-r--r--. 1 mongodb mongodb         0 Nov 27 03:37 mongodb.log.2016-11-27T03-37-46

Comment 6 Peter Vreman 2016-12-01 12:19:52 UTC
Looking at https://docs.mongodb.com/v2.6/tutorial/rotate-log-files/ it looks like logrotation is also enabled in mongodb itself.

Comment 7 Michael Hrivnak 2017-01-04 15:55:15 UTC
The original reported issue was fixed here: https://bugzilla.redhat.com/show_bug.cgi?id=1269227

As for the strange 0-byte files you're now seeing, I have a theory on that. Although I don't see that behavior being reproduced with Satellite 6.2.6 on either el6 or el7.

The theory, inspired by comment #6: logrotate does its thing. We end up with a 0-byte "mongodb.log", and a new "mongodb.log-xxxxxxxx.gz". In the "postrotate", it sends SIGUSR1 to mongodb, which actually triggers mongodb to do *its own log rotation* per the docs [0]. But mongodb.log is empty, which is why the rotated file is also empty. The only problem with this theory is your rotate artifacts from Nov 21, where the uncompressed file is quite large. <shrug>

Why don't we see these secondary-rotation artifacts appearing with 6.2.6? So far I have no idea.

[0] https://docs.mongodb.com/v2.6/tutorial/rotate-log-files/

Comment 8 Peter Vreman 2017-01-04 16:11:36 UTC
Correct, the empty log files is a general issue like. A bit of googling gives also other discussions like http://serverfault.com/questions/540423/mongodb-proper-way-to-rotate-logs

Comment 9 Peter Vreman 2017-01-04 16:19:50 UTC
The huge uncompressed file is also explained: It is a spare file, where mongodb is still writing a filepostion N after the logrotate did the rotation and before the USR1 is received by mongod

See the Size vs. Blocks in the stat and the hexdump of the first 20 0x00 characters:

[crash] root@li-lc-1017:~# head -c 20 /var/log/mongodb/mongodb.log.2016-12-05T03-22-28 | xxd
0000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000010: 0000 0000                                ....

[crash] root@li-lc-1017:~# stat /var/log/mongodb/mongodb.log.2016-12-05T03-22-28
  File: ‘/var/log/mongodb/mongodb.log.2016-12-05T03-22-28’
  Size: 1342099293      Blocks: 16         IO Block: 4096   regular file
Device: fd07h/64775d    Inode: 11156       Links: 1
Access: (0644/-rw-r--r--)  Uid: (  184/ mongodb)   Gid: (  130/ mongodb)
Context: system_u:object_r:mongod_log_t:s0
Access: 2017-01-04 16:14:49.667227212 +0000
Modify: 2016-12-05 03:22:28.217426986 +0000
Change: 2016-12-05 03:22:28.388427487 +0000
 Birth: -

Comment 10 Michael Hrivnak 2017-01-04 21:31:46 UTC
Thank you for providing this helpful info. I filed a bug about the logrotate behavior against the upstream package:

https://bugzilla.redhat.com/show_bug.cgi?id=1410238

That's the best place to track progress on fixing the logrotate issue for now. Once they fix it, that change will make its way into Satellite.

Since the original problem reported via this issue was previously resolved, I am closing this as a duplicate.

*** This bug has been marked as a duplicate of bug 1269227 ***


Note You need to log in before you can comment on or make changes to this bug.