Bug 1719770

[heat-admin@overcloud-controller-0 ~]$ sudo podman ps | grep log
b2748b1d7046  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-cron:20190607.1                dumb-init --singl...  4 days ago  Up 4 days ago           logrotate_crond
[heat-admin@overcloud-controller-0 ~]$ cat /var/lib/config-data/puppet-generated/crond/etc/logrotate-crond.conf
cat: /var/lib/config-data/puppet-generated/crond/etc/logrotate-crond.conf: Permission denied
[heat-admin@overcloud-controller-0 ~]$ sudo cat /var/lib/config-data/puppet-generated/crond/etc/logrotate-crond.conf
/var/log/containers/*/*log /var/log/containers/*/*/*log /var/log/containers/*/*err {
  daily
  rotate 14
  maxage 14
  # minsize 1 is required for GDPR compliance, all files in
  # /var/log/containers not managed with logrotate will be purged!
  minsize 1
  # Do not use size as it's not compatible with time-based rotation rules
  # required for GDPR compliance.
  maxsize 10M
  missingok
  notifempty

  copytruncate


  delaycompress


  compress

}


However the log rotation is not working

Hello,

Would love to get some SELinux info about the system:
# getenforce
# grep denied /var/log/audit/audit.log

I'm pretty sure there are some witchcraft in there.

Cheers,

C.

[root@overcloud-controller-0 heat-admin]# getenforce
Enforcing
[root@overcloud-controller-0 heat-admin]#  grep denied /var/log/audit/audit.log
[root@overcloud-controller-0 heat-admin]#

Hello again,

hmmm. Sad news then.

Do you have a live env we can access? Would be best in order to investigate instead of playing Pong :).

Cheers,

C.

I'll email access details in a few hours.

Hello,

I've just built an undercloud with osp-15 on rhel8, and am playing a bit with the logs.
It appears for some reasons logrotate doesn't see the log, as this output shows it:
()[root@d739d17ec240 /]# /usr/sbin/logrotate -vv -s /var/lib/logrotate/logrotate-crond.status /etc/logrotate-crond.conf
reading config file /etc/logrotate-crond.conf
Reading state from file: /var/lib/logrotate/logrotate-crond.status
Allocating hash table for state file, size 64 entries

Handling 1 logs

rotating pattern: /var/log/containers/*/*log /var/log/containers/*/*/*log /var/log/containers/*/*err  after 1 days (14 rotations)
empty log files are not rotated, only log files >= 1 bytes are rotated, log files >= 10485760 are rotated earlier, old logs are removed
No logs found. Rotation not needed.

Fun fact:
I've tweaked a log that WILL trigger a rotation:
-rw-r--r--. 1 42436 42436 274M Jun 20 09:42 nova-placement-api.log

and that one appears in the patterns set in logrotate.conf:
()[root@d739d17ec240 /]# ls /var/log/containers/*/*log | grep nova-placement-api.log
/var/log/containers/nova/nova-placement-api.log


Last time I had this kind of issue, it was due to the status file - I'll dig a bit more around that one.

"fun" fact: it was working when I first worked on logrotate, a couple of months ago :/.

Stay tuned!

Cheers,

C.

Me again,

Just found out: logrotate doesn't support multiple wildcards in paths anymore. This means the following doesn't match anything for logrotate: /var/log/containers/*/*log while it should, at least, match my nova-placement-api.log due to its size.

I have to compare with centos-7/rhel7 logrotate version, but I'm pretty sure it did work back then.

Stay tuned!

C.

Created a bug usptream against logrotate in order to get some more information.

May be a blocker, need to find a proper workaround in order to make logrotate work in case it's not corrected in time.

Added backport link for upstream/Stein.

can we move it to MODIFIED? Is the workaround enough ?

Verification scenario:
1. New deployment, 3 controllers, 2 compute, left overnight. After the night nova-api.log got to 9.7 M. No logs bigger than 10M:
sudo podman exec logrotate_crond ls -tarlh /var/log/containers/nova/
2.  sudo podman exec -ti cat /var/log/containers/nova/nova-api.log | tee -a /var/log/containers/nova/nova-api.log, wait for 2 sec
See that the log becomes bigger that 10M
3. sudo podman exec logrotate_crond logrotate -s -d /var/spool/... /etc/logrotate-crond.conf
See that the old file becomes nova-api.log.1 and the new one appears - it means that the rotation is working.
4. cat /var/lib/logrotate/logrotate-crond.status - see that status is not empty - it means that the files are rotated

Verified

Quick note: logrotate has a patch on its own in order to not crash on broken symlink, referenced here:
- https://bugzilla.redhat.com/show_bug.cgi?id=1723265
- https://github.com/logrotate/logrotate/issues/251

So we're all good now :).

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811