Description of problem: There is no log rotation configured in Rhos-14 for Nova logs, not sure about other components, but most likely its not configured for any of the openstack components. Version-Release number of selected component (if applicable): puppet-nova-13.3.1-0.20180917153245.6fdb591.el7ost.noarch Actual results: Rhos-13: [root@controller-0 heat-admin]# less /etc/logrotate.d/ aide mariadb openstack-mistral rabbitmq-server barbican-api numad openstack-neutron redis bootlog openstack-aodh openstack-nova samba chrony openstack-ceilometer openstack-octavia sensu corosync openstack-cinder openstack-panko sssd gnocchi openstack-dashboard openstack-sahara subscription-manager haproxy openstack-glance openstack-swift syslog httpd openstack-heat openstack-zaqar up2date iscsiuiolog openstack-ironic openvswitch wpa_supplicant libvirtd openstack-ironic-inspector pacemaker yum libvirtd.lxc openstack-keystone pcsd libvirtd.qemu openstack-manila puppet Rhos-14: Build: 2018-10-10.1 root@controller-0 nova]# less /etc/logrotate.d/ aide haproxy libvirtd.lxc openvswitch redis syslog bootlog httpd libvirtd.qemu pacemaker samba up2date chrony iscsiuiolog mariadb pcsd sssd wpa_supplicant corosync libvirtd numad puppet subscription-manager yum Expected results: Log-rotation present Additional info: This was discovered when the nova-scheduler.log rotated to nova-scheduler.log.1 but didnt create a new nova-scheduler.log, build id: 2018-10-10.1
There is now a logrotate_crond container which rotates the logs: $ docker ps | grep log 60427742db80 192.168.24.1:8787/rhosp14/openstack-cron:2018-10-10.1 "kolla_start" 18 hours ago Up 18 hours logrotate_crond (undercloud) [stack@undercloud-0 ~]$ docker exec -it -u root logrotate_crond /bin/bash ()[root@60427742db80 /]# cat /etc/logrotate-crond.conf /var/log/containers/*/*log /var/log/containers/*/*/*log { daily rotate 14 maxage 14 # minsize 1 is required for GDPR compliance, all files in # /var/log/containers not managed with logrotate will be purged! minsize 1 # Do not use size as it's not compatible with time-based rotation rules # required for GDPR compliance. maxsize 10M missingok notifempty delaycompress compress sharedscripts postrotate /usr/bin/find /var/log/containers -type f \ \( -mtime +14 -or \ -atime +14 -or \ -ctime +14 \) -exec rm -f {} \;; /sbin/lsof -nPs +L1 +D /var/log/containers 2>/dev/null|\ grep -v /var/log/httpd/ |\ awk '/\S+\s+[0-9]+\s.*\/var\/log\/.*\(deleted\)/ {print $2}' |\ sort -u | /bin/xargs -n1 -r -t kill -HUP; /sbin/lsof -nPs +L1 +D /var/log/containers 2>/dev/null|\ awk '/\S+\s+[0-9]+\s.*\/var\/log\/httpd\/.*\(deleted\)/ {print $2}' |\ sort -u | /bin/xargs -n1 -r -t kill -USR1 endscript } [root@controller-0 ~]# ll /var/log/containers/nova/ total 45400 -rw-r--r--. 1 42436 42436 9916939 Oct 12 07:43 nova-api.log -rw-r--r--. 1 42436 42436 1260924 Oct 12 00:00 nova-api.log.1 -rw-r--r--. 1 42436 42436 402253 Oct 11 23:01 nova-api.log.2.gz -rw-r--r--. 1 42436 42436 150567 Oct 11 15:04 nova-conductor.log.1 -rw-r--r--. 1 42436 42436 146496 Oct 11 15:04 nova-consoleauth.log.1 -rw-r--r--. 1 42436 42436 0 Oct 12 00:01 nova-manage.log -rw-r--r--. 1 42436 42436 283125 Oct 11 14:58 nova-manage.log.1 -rw-r--r--. 1 42436 42436 758492 Oct 11 15:05 nova-metadata-api.log.1 -rw-r--r--. 1 42436 42436 146048 Oct 12 07:43 nova-novncproxy.log -rw-r--r--. 1 42436 42436 169327 Oct 11 23:59 nova-novncproxy.log.1 -rw-r--r--. 1 42436 42436 7780705 Oct 12 07:43 nova-placement-api.log -rw-r--r--. 1 42436 42436 9654455 Oct 12 00:00 nova-placement-api.log.1 -rw-r--r--. 1 42436 42436 0 Oct 12 00:01 nova-rowsflush.log -rw-r--r--. 1 42436 42436 3970530 Oct 12 07:43 nova-scheduler.log -rw-r--r--. 1 42436 42436 4800002 Oct 12 00:00 nova-scheduler.log.1 Your issue is a duplicate of [1] and gets addressed by [2], which is not merged yet for rocky. There are services which do not tolerate SIGHUP, or support them nicely. That's why it result sometimes to not rotate them correct. The fix config to fallback to copytruncate containerized logs. I am going to close this as a dupe of [1], but feel free to reopen if I have missed something. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1614810 [2] https://review.openstack.org/#/c/607491/ [3] https://review.openstack.org/#/c/608594/ *** This bug has been marked as a duplicate of bug 1614810 ***