Bug 1317479

The other BZ https://bugzilla.redhat.com/show_bug.cgi?id=1269227 is about the wrong location and selinux context.
This BZ is about housekeeping

Moving 6.2 bugs out to sat-backlog.

Situation with Sat6.2.4 on RHEL7.3 improved.
Partly it uses our systemwide logrotation configuration

[crash] root@li-lc-1017:/etc# cat logrotate.d/mongodb
/var/log/mongodb/*.log {
       weekly
       rotate 10
       copytruncate
       delaycompress
       compress
       notifempty
       missingok
       postrotate
          /bin/kill -USR1 `cat /var/run/mongodb/mongod.pid 2>/dev/null` 2> /dev/null|| true
       endscript
}

But there are strange duplicate files with date suffixes like '2016-10-23T03-19-25' and on 20161021  there is even a duplicate between compressed and uncompressed.


$ sudo ls -l /var/log/mongodb/
total 1272816
-rw-r--r--. 1 mongodb mongodb 882338206 Dec  1 10:43 mongodb.log
-rw-r--r--. 1 mongodb mongodb  55039530 Oct 23 03:19 mongodb.log-20161023.gz
-rw-r--r--. 1 mongodb mongodb         0 Oct 23 03:19 mongodb.log.2016-10-23T03-19-25
-rw-r--r--. 1 mongodb mongodb  19605531 Oct 30 03:21 mongodb.log-20161030.gz
-rw-r--r--. 1 mongodb mongodb         0 Oct 30 03:21 mongodb.log.2016-10-30T03-21-51
-rw-r--r--. 1 mongodb mongodb  29261929 Nov  6 03:46 mongodb.log-20161106.gz
-rw-r--r--. 1 mongodb mongodb         0 Nov  6 03:46 mongodb.log.2016-11-06T03-46-44
-rw-r--r--. 1 mongodb mongodb  63012359 Nov 13 03:35 mongodb.log-20161113.gz
-rw-r--r--. 1 mongodb mongodb         0 Nov 13 03:35 mongodb.log.2016-11-13T03-35-56
-rw-r--r--. 1 mongodb mongodb  83346603 Nov 21 03:41 mongodb.log-20161121.gz
-rw-r--r--. 1 mongodb mongodb 655373184 Nov 21 03:42 mongodb.log.2016-11-21T03-42-35
-rw-r--r--. 1 mongodb mongodb 170731102 Nov 27 03:37 mongodb.log-20161127
-rw-r--r--. 1 mongodb mongodb         0 Nov 27 03:37 mongodb.log.2016-11-27T03-37-46

Looking at https://docs.mongodb.com/v2.6/tutorial/rotate-log-files/ it looks like logrotation is also enabled in mongodb itself.

The original reported issue was fixed here: https://bugzilla.redhat.com/show_bug.cgi?id=1269227

As for the strange 0-byte files you're now seeing, I have a theory on that. Although I don't see that behavior being reproduced with Satellite 6.2.6 on either el6 or el7.

The theory, inspired by comment #6: logrotate does its thing. We end up with a 0-byte "mongodb.log", and a new "mongodb.log-xxxxxxxx.gz". In the "postrotate", it sends SIGUSR1 to mongodb, which actually triggers mongodb to do *its own log rotation* per the docs [0]. But mongodb.log is empty, which is why the rotated file is also empty. The only problem with this theory is your rotate artifacts from Nov 21, where the uncompressed file is quite large. <shrug>

Why don't we see these secondary-rotation artifacts appearing with 6.2.6? So far I have no idea.

[0] https://docs.mongodb.com/v2.6/tutorial/rotate-log-files/

Correct, the empty log files is a general issue like. A bit of googling gives also other discussions like http://serverfault.com/questions/540423/mongodb-proper-way-to-rotate-logs

The huge uncompressed file is also explained: It is a spare file, where mongodb is still writing a filepostion N after the logrotate did the rotation and before the USR1 is received by mongod

See the Size vs. Blocks in the stat and the hexdump of the first 20 0x00 characters:

[crash] root@li-lc-1017:~# head -c 20 /var/log/mongodb/mongodb.log.2016-12-05T03-22-28 | xxd
0000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000010: 0000 0000                                ....

[crash] root@li-lc-1017:~# stat /var/log/mongodb/mongodb.log.2016-12-05T03-22-28
  File: ‘/var/log/mongodb/mongodb.log.2016-12-05T03-22-28’
  Size: 1342099293      Blocks: 16         IO Block: 4096   regular file
Device: fd07h/64775d    Inode: 11156       Links: 1
Access: (0644/-rw-r--r--)  Uid: (  184/ mongodb)   Gid: (  130/ mongodb)
Context: system_u:object_r:mongod_log_t:s0
Access: 2017-01-04 16:14:49.667227212 +0000
Modify: 2016-12-05 03:22:28.217426986 +0000
Change: 2016-12-05 03:22:28.388427487 +0000
 Birth: -

Thank you for providing this helpful info. I filed a bug about the logrotate behavior against the upstream package:

https://bugzilla.redhat.com/show_bug.cgi?id=1410238

That's the best place to track progress on fixing the logrotate issue for now. Once they fix it, that change will make its way into Satellite.

Since the original problem reported via this issue was previously resolved, I am closing this as a duplicate.

*** This bug has been marked as a duplicate of bug 1269227 ***