Hide Forgot
Description of problem: In OSP15, the logs files of services such as nova, neutron keep getting bigger up to a few 100MBs but they are not being compressed and rotated. Version-Release number of selected component (if applicable): 15 How reproducible: 100% Steps to Reproduce: 1. Deploy OSP 15 and leave it around for sometime after putting load 2. 3. Actual results: [root@overcloud-controller-0 containers]# cd neutron/ [root@overcloud-controller-0 neutron]# ls -tarlh total 490M drwxr-xr-x. 16 root root 221 Jun 12 14:04 .. drwxr-xr-x. 2 42435 42435 24 Jun 12 14:04 . -rw-r--r--. 1 42435 42435 234M Jun 12 14:09 server.log [root@overcloud-controller-0 nova]# ls -tarlh total 610M -rw-------. 1 42436 42436 90K May 31 18:08 nova-manage.log -rw-r--r--. 1 42436 42436 3.0K May 31 18:12 nova-conductor.log -rw-r--r--. 1 42436 42436 1.9K May 31 18:12 nova-consoleauth.log -rw-r--r--. 1 42436 42436 622 May 31 18:12 nova-novncproxy.log -rw-r--r--. 1 42436 42436 8.5K May 31 18:12 nova-metadata-api.log -rw-r--r--. 1 42436 42436 8.8K May 31 18:14 nova-scheduler.log drwxr-xr-x. 2 42436 42436 241 Jun 1 00:26 . -rw-r--r--. 1 42436 42436 0 Jun 1 00:26 nova-rowsflush.log drwxr-xr-x. 16 root root 221 Jun 12 14:04 .. -rw-r--r--. 1 42436 42436 8.5M Jun 12 14:09 nova-placement-api.log -rw-r--r--. 1 42436 42436 346M Jun 12 14:09 nova-api.log Expected results: Log files need to be rotated Additional info:
[heat-admin@overcloud-controller-0 ~]$ sudo podman ps | grep log b2748b1d7046 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-cron:20190607.1 dumb-init --singl... 4 days ago Up 4 days ago logrotate_crond [heat-admin@overcloud-controller-0 ~]$ cat /var/lib/config-data/puppet-generated/crond/etc/logrotate-crond.conf cat: /var/lib/config-data/puppet-generated/crond/etc/logrotate-crond.conf: Permission denied [heat-admin@overcloud-controller-0 ~]$ sudo cat /var/lib/config-data/puppet-generated/crond/etc/logrotate-crond.conf /var/log/containers/*/*log /var/log/containers/*/*/*log /var/log/containers/*/*err { daily rotate 14 maxage 14 # minsize 1 is required for GDPR compliance, all files in # /var/log/containers not managed with logrotate will be purged! minsize 1 # Do not use size as it's not compatible with time-based rotation rules # required for GDPR compliance. maxsize 10M missingok notifempty copytruncate delaycompress compress } However the log rotation is not working
Hello, Would love to get some SELinux info about the system: # getenforce # grep denied /var/log/audit/audit.log I'm pretty sure there are some witchcraft in there. Cheers, C.
[root@overcloud-controller-0 heat-admin]# getenforce Enforcing [root@overcloud-controller-0 heat-admin]# grep denied /var/log/audit/audit.log [root@overcloud-controller-0 heat-admin]#
Hello again, hmmm. Sad news then. Do you have a live env we can access? Would be best in order to investigate instead of playing Pong :). Cheers, C.
I'll email access details in a few hours.
Hello, I've just built an undercloud with osp-15 on rhel8, and am playing a bit with the logs. It appears for some reasons logrotate doesn't see the log, as this output shows it: ()[root@d739d17ec240 /]# /usr/sbin/logrotate -vv -s /var/lib/logrotate/logrotate-crond.status /etc/logrotate-crond.conf reading config file /etc/logrotate-crond.conf Reading state from file: /var/lib/logrotate/logrotate-crond.status Allocating hash table for state file, size 64 entries Handling 1 logs rotating pattern: /var/log/containers/*/*log /var/log/containers/*/*/*log /var/log/containers/*/*err after 1 days (14 rotations) empty log files are not rotated, only log files >= 1 bytes are rotated, log files >= 10485760 are rotated earlier, old logs are removed No logs found. Rotation not needed. Fun fact: I've tweaked a log that WILL trigger a rotation: -rw-r--r--. 1 42436 42436 274M Jun 20 09:42 nova-placement-api.log and that one appears in the patterns set in logrotate.conf: ()[root@d739d17ec240 /]# ls /var/log/containers/*/*log | grep nova-placement-api.log /var/log/containers/nova/nova-placement-api.log Last time I had this kind of issue, it was due to the status file - I'll dig a bit more around that one. "fun" fact: it was working when I first worked on logrotate, a couple of months ago :/. Stay tuned! Cheers, C.
Me again, Just found out: logrotate doesn't support multiple wildcards in paths anymore. This means the following doesn't match anything for logrotate: /var/log/containers/*/*log while it should, at least, match my nova-placement-api.log due to its size. I have to compare with centos-7/rhel7 logrotate version, but I'm pretty sure it did work back then. Stay tuned! C.
Created a bug usptream against logrotate in order to get some more information.
May be a blocker, need to find a proper workaround in order to make logrotate work in case it's not corrected in time.
Added backport link for upstream/Stein.
can we move it to MODIFIED? Is the workaround enough ?
Verification scenario: 1. New deployment, 3 controllers, 2 compute, left overnight. After the night nova-api.log got to 9.7 M. No logs bigger than 10M: sudo podman exec logrotate_crond ls -tarlh /var/log/containers/nova/ 2. sudo podman exec -ti cat /var/log/containers/nova/nova-api.log | tee -a /var/log/containers/nova/nova-api.log, wait for 2 sec See that the log becomes bigger that 10M 3. sudo podman exec logrotate_crond logrotate -s -d /var/spool/... /etc/logrotate-crond.conf See that the old file becomes nova-api.log.1 and the new one appears - it means that the rotation is working. 4. cat /var/lib/logrotate/logrotate-crond.status - see that status is not empty - it means that the files are rotated Verified
Quick note: logrotate has a patch on its own in order to not crash on broken symlink, referenced here: - https://bugzilla.redhat.com/show_bug.cgi?id=1723265 - https://github.com/logrotate/logrotate/issues/251 So we're all good now :).
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:2811