Bug 1027722 - openstack: there's no log rotation on all components
openstack: there's no log rotation on all components
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: distribution (Show other bugs)
4.0
Unspecified Unspecified
high Severity high
: rc
: 4.0
Assigned To: Pádraig Brady
Tzach Shefi
: Improvement, Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-11-07 05:34 EST by Yogev Rabl
Modified: 2016-01-04 09:42 EST (History)
8 users (show)

See Also:
Fixed In Version: openstack-nova-2013.2-8.el6ost openstack-glance-2013.2-4.el6ost openstack-keystone-2013.2-3.el6ost openstack-cinder-2013.2-6.el6ost openstack-ceilometer-2013.2-3.el6ost openstack-heat-2013.2-3.el6ost openstack-neutron-2013.2-13.el6ost
Doc Type: Enhancement
Doc Text:
By default, the logs of OpenStack services are now checked at the system logrotate frequency; logs are then rotated when their size reaches 10MB. This is in addition to the default logrotate frequency setting each service uses. This update was implemented to reduce disk waste and ease log management. Previously, logs were rotated weekly with no size limit; in some cases, this resulted in the creation of excessively large log files.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-12-19 19:34:32 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Yogev Rabl 2013-11-07 05:34:16 EST
Description of problem:
Openstack keeps writing to the same log files and doesn't not archive them - thus creating huge files. 


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Pádraig Brady 2013-11-13 11:23:52 EST
I'm not sure this is valid, as all services should have logrotate config in place. For example, nova's config is:

/var/log/nova/*.log {
    weekly
    rotate 4
    missingok
    compress
    minsize 100k
}

Do you mean that the config should be changed to be sized based?
(noting the caveat that logrotate is usually a daily cronjob)

/var/log/nova/*.log {
    rotate 4
    missingok
    compress
    size 10M
}
Comment 2 Dafna Ron 2013-11-19 12:18:20 EST
Padraig, Yogev opened this bug for me when I was away so please contact me for any further need info's. 

I think we should change logrotate for all services for max size + daily. 

Currently logs can reach very large sizes which makes debugging in QE env. extremely difficult and uploading logs to bugzilla is also a problem. 
we would also see the same issue in customer env. which means gss will have a hard time as well. 

I looked in my setup and we do have logrotate for services but its weekly and has no max size. 
so, the ideal for us would be to xz the logs daily and keep for a month, unless the log is over 2GB and than we should xz as well (I thin the max upload size in bugzilla is 2GB).
Comment 5 Dafna Ron 2013-12-11 12:30:25 EST
bug is verified for

openstack-nova-compute-2013.2-9.el6ost.noarch

[root@cougar06 ~(keystone_admin)]# ls -l /var/log/nova/
total 940764
-rw-r--r--. 1 nova nova  17546671 Dec 11 19:29 api.log
-rw-r--r--. 1 nova nova   1685308 Dec 11 19:28 cert.log
-rw-r--r--. 1 nova nova 247456778 Dec 11 19:29 compute.log
-rw-r--r--. 1 nova nova   5900194 Dec 11 03:48 compute.log-20131211.gz
-rw-r--r--. 1 nova nova 641882356 Dec 11 19:29 conductor.log
-rw-r--r--. 1 nova nova  16069786 Dec 11 03:48 conductor.log-20131211.gz
-rw-r--r--. 1 nova nova   2003009 Dec 11 19:29 consoleauth.log
-rw-r--r--. 1 nova nova    295496 Dec 11 19:29 console.log
-rw-r--r--. 1 nova nova      3520 Dec 11 17:40 metadata-api.log
-rw-r--r--. 1 nova nova  23647377 Dec 11 19:29 network.log
-rw-r--r--. 1 nova nova    451029 Dec 11 03:48 network.log-20131211.gz
-rw-r--r--. 1 root root     82306 Dec 11 19:12 nova-dhcpbridge.log
-rw-r--r--. 1 root root     64945 Dec 10 18:50 nova-manage.log
-rw-r--r--. 1 nova nova   6115017 Dec 11 19:29 scheduler.log
-rw-r--r--. 1 nova nova     76887 Dec 11 17:40 xvpvncproxy.log
Comment 7 errata-xmlrpc 2013-12-19 19:34:32 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2013-1859.html

Note You need to log in before you can comment on or make changes to this bug.