Bug 1719770 - Log files not being rotated
Summary: Log files not being rotated
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: 15.0 (Stein)
Assignee: Rabi Mishra
QA Contact: Sasha Smolyak
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-06-12 14:10 UTC by Sai Sindhur Malleni
Modified: 2019-09-26 10:52 UTC (History)
10 users (show)

Fixed In Version: openstack-tripleo-heat-templates-10.5.1-0.20190627050406.22ee26f.el8ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-21 11:23:00 UTC
Target Upstream Version:
Embargoed:
cjeanner: needinfo-


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github logrotate logrotate issues 251 0 None None None 2019-06-20 11:58:49 UTC
OpenStack gerrit 666898 0 None None None 2019-06-24 11:08:58 UTC
OpenStack gerrit 667501 0 None None None 2019-06-26 04:17:19 UTC
Red Hat Product Errata RHEA-2019:2811 0 None None None 2019-09-21 11:23:23 UTC

Description Sai Sindhur Malleni 2019-06-12 14:10:03 UTC
Description of problem: In OSP15, the logs files of services such as nova, neutron keep getting bigger up to a few 100MBs but they are not being compressed and rotated.


Version-Release number of selected component (if applicable):
15

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP 15 and leave it around for sometime after putting load
2.
3.

Actual results:

[root@overcloud-controller-0 containers]# cd neutron/
[root@overcloud-controller-0 neutron]# ls -tarlh
total 490M
drwxr-xr-x. 16 root  root   221 Jun 12 14:04 ..
drwxr-xr-x.  2 42435 42435   24 Jun 12 14:04 .
-rw-r--r--.  1 42435 42435 234M Jun 12 14:09 server.log


[root@overcloud-controller-0 nova]# ls -tarlh
total 610M
-rw-------.  1 42436 42436  90K May 31 18:08 nova-manage.log
-rw-r--r--.  1 42436 42436 3.0K May 31 18:12 nova-conductor.log
-rw-r--r--.  1 42436 42436 1.9K May 31 18:12 nova-consoleauth.log
-rw-r--r--.  1 42436 42436  622 May 31 18:12 nova-novncproxy.log
-rw-r--r--.  1 42436 42436 8.5K May 31 18:12 nova-metadata-api.log
-rw-r--r--.  1 42436 42436 8.8K May 31 18:14 nova-scheduler.log
drwxr-xr-x.  2 42436 42436  241 Jun  1 00:26 .
-rw-r--r--.  1 42436 42436    0 Jun  1 00:26 nova-rowsflush.log
drwxr-xr-x. 16 root  root   221 Jun 12 14:04 ..
-rw-r--r--.  1 42436 42436 8.5M Jun 12 14:09 nova-placement-api.log
-rw-r--r--.  1 42436 42436 346M Jun 12 14:09 nova-api.log


Expected results:
Log files need to be rotated

Additional info:

Comment 2 Sai Sindhur Malleni 2019-06-18 17:14:43 UTC
[heat-admin@overcloud-controller-0 ~]$ sudo podman ps | grep log
b2748b1d7046  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-cron:20190607.1                dumb-init --singl...  4 days ago  Up 4 days ago           logrotate_crond
[heat-admin@overcloud-controller-0 ~]$ cat /var/lib/config-data/puppet-generated/crond/etc/logrotate-crond.conf
cat: /var/lib/config-data/puppet-generated/crond/etc/logrotate-crond.conf: Permission denied
[heat-admin@overcloud-controller-0 ~]$ sudo cat /var/lib/config-data/puppet-generated/crond/etc/logrotate-crond.conf
/var/log/containers/*/*log /var/log/containers/*/*/*log /var/log/containers/*/*err {
  daily
  rotate 14
  maxage 14
  # minsize 1 is required for GDPR compliance, all files in
  # /var/log/containers not managed with logrotate will be purged!
  minsize 1
  # Do not use size as it's not compatible with time-based rotation rules
  # required for GDPR compliance.
  maxsize 10M
  missingok
  notifempty

  copytruncate


  delaycompress


  compress

}


However the log rotation is not working

Comment 3 Cédric Jeanneret 2019-06-19 11:36:23 UTC
Hello,

Would love to get some SELinux info about the system:
# getenforce
# grep denied /var/log/audit/audit.log

I'm pretty sure there are some witchcraft in there.

Cheers,

C.

Comment 4 Sai Sindhur Malleni 2019-06-19 13:12:40 UTC
[root@overcloud-controller-0 heat-admin]# getenforce
Enforcing
[root@overcloud-controller-0 heat-admin]#  grep denied /var/log/audit/audit.log
[root@overcloud-controller-0 heat-admin]#

Comment 5 Cédric Jeanneret 2019-06-19 13:15:09 UTC
Hello again,

hmmm. Sad news then.

Do you have a live env we can access? Would be best in order to investigate instead of playing Pong :).

Cheers,

C.

Comment 6 Sai Sindhur Malleni 2019-06-19 13:37:31 UTC
I'll email access details in a few hours.

Comment 7 Cédric Jeanneret 2019-06-20 09:45:10 UTC
Hello,

I've just built an undercloud with osp-15 on rhel8, and am playing a bit with the logs.
It appears for some reasons logrotate doesn't see the log, as this output shows it:
()[root@d739d17ec240 /]# /usr/sbin/logrotate -vv -s /var/lib/logrotate/logrotate-crond.status /etc/logrotate-crond.conf
reading config file /etc/logrotate-crond.conf
Reading state from file: /var/lib/logrotate/logrotate-crond.status
Allocating hash table for state file, size 64 entries

Handling 1 logs

rotating pattern: /var/log/containers/*/*log /var/log/containers/*/*/*log /var/log/containers/*/*err  after 1 days (14 rotations)
empty log files are not rotated, only log files >= 1 bytes are rotated, log files >= 10485760 are rotated earlier, old logs are removed
No logs found. Rotation not needed.

Fun fact:
I've tweaked a log that WILL trigger a rotation:
-rw-r--r--. 1 42436 42436 274M Jun 20 09:42 nova-placement-api.log

and that one appears in the patterns set in logrotate.conf:
()[root@d739d17ec240 /]# ls /var/log/containers/*/*log | grep nova-placement-api.log
/var/log/containers/nova/nova-placement-api.log


Last time I had this kind of issue, it was due to the status file - I'll dig a bit more around that one.

"fun" fact: it was working when I first worked on logrotate, a couple of months ago :/.

Stay tuned!

Cheers,

C.

Comment 8 Cédric Jeanneret 2019-06-20 11:26:26 UTC
Me again,

Just found out: logrotate doesn't support multiple wildcards in paths anymore. This means the following doesn't match anything for logrotate: /var/log/containers/*/*log while it should, at least, match my nova-placement-api.log due to its size.

I have to compare with centos-7/rhel7 logrotate version, but I'm pretty sure it did work back then.

Stay tuned!

C.

Comment 9 Cédric Jeanneret 2019-06-20 11:58:49 UTC
Created a bug usptream against logrotate in order to get some more information.

Comment 10 Cédric Jeanneret 2019-06-20 13:14:17 UTC
May be a blocker, need to find a proper workaround in order to make logrotate work in case it's not corrected in time.

Comment 11 Cédric Jeanneret 2019-06-24 11:08:59 UTC
Added backport link for upstream/Stein.

Comment 12 Emilien Macchi 2019-06-25 18:54:19 UTC
can we move it to MODIFIED? Is the workaround enough ?

Comment 19 Sasha Smolyak 2019-07-01 07:58:21 UTC
Verification scenario:
1. New deployment, 3 controllers, 2 compute, left overnight. After the night nova-api.log got to 9.7 M. No logs bigger than 10M:
sudo podman exec logrotate_crond ls -tarlh /var/log/containers/nova/
2.  sudo podman exec -ti cat /var/log/containers/nova/nova-api.log | tee -a /var/log/containers/nova/nova-api.log, wait for 2 sec
See that the log becomes bigger that 10M
3. sudo podman exec logrotate_crond logrotate -s -d /var/spool/... /etc/logrotate-crond.conf
See that the old file becomes nova-api.log.1 and the new one appears - it means that the rotation is working.
4. cat /var/lib/logrotate/logrotate-crond.status - see that status is not empty - it means that the files are rotated

Verified

Comment 20 Cédric Jeanneret 2019-07-01 09:41:38 UTC
Quick note: logrotate has a patch on its own in order to not crash on broken symlink, referenced here:
- https://bugzilla.redhat.com/show_bug.cgi?id=1723265
- https://github.com/logrotate/logrotate/issues/251

So we're all good now :).

Comment 23 errata-xmlrpc 2019-09-21 11:23:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811


Note You need to log in before you can comment on or make changes to this bug.