Bug 1027722 - openstack: there's no log rotation on all components
Summary: openstack: there's no log rotation on all components
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: distribution
Version: 4.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 4.0
Assignee: Pádraig Brady
QA Contact: Tzach Shefi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-11-07 10:34 UTC by Yogev Rabl
Modified: 2016-01-04 14:42 UTC (History)
8 users (show)

Fixed In Version: openstack-nova-2013.2-8.el6ost openstack-glance-2013.2-4.el6ost openstack-keystone-2013.2-3.el6ost openstack-cinder-2013.2-6.el6ost openstack-ceilometer-2013.2-3.el6ost openstack-heat-2013.2-3.el6ost openstack-neutron-2013.2-13.el6ost
Doc Type: Enhancement
Doc Text:
By default, the logs of OpenStack services are now checked at the system logrotate frequency; logs are then rotated when their size reaches 10MB. This is in addition to the default logrotate frequency setting each service uses. This update was implemented to reduce disk waste and ease log management. Previously, logs were rotated weekly with no size limit; in some cases, this resulted in the creation of excessively large log files.
Clone Of:
Environment:
Last Closed: 2013-12-20 00:34:32 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2013:1859 0 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform Enhancement Advisory 2013-12-21 00:01:48 UTC

Description Yogev Rabl 2013-11-07 10:34:16 UTC
Description of problem:
Openstack keeps writing to the same log files and doesn't not archive them - thus creating huge files. 


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Pádraig Brady 2013-11-13 16:23:52 UTC
I'm not sure this is valid, as all services should have logrotate config in place. For example, nova's config is:

/var/log/nova/*.log {
    weekly
    rotate 4
    missingok
    compress
    minsize 100k
}

Do you mean that the config should be changed to be sized based?
(noting the caveat that logrotate is usually a daily cronjob)

/var/log/nova/*.log {
    rotate 4
    missingok
    compress
    size 10M
}

Comment 2 Dafna Ron 2013-11-19 17:18:20 UTC
Padraig, Yogev opened this bug for me when I was away so please contact me for any further need info's. 

I think we should change logrotate for all services for max size + daily. 

Currently logs can reach very large sizes which makes debugging in QE env. extremely difficult and uploading logs to bugzilla is also a problem. 
we would also see the same issue in customer env. which means gss will have a hard time as well. 

I looked in my setup and we do have logrotate for services but its weekly and has no max size. 
so, the ideal for us would be to xz the logs daily and keep for a month, unless the log is over 2GB and than we should xz as well (I thin the max upload size in bugzilla is 2GB).

Comment 5 Dafna Ron 2013-12-11 17:30:25 UTC
bug is verified for

openstack-nova-compute-2013.2-9.el6ost.noarch

[root@cougar06 ~(keystone_admin)]# ls -l /var/log/nova/
total 940764
-rw-r--r--. 1 nova nova  17546671 Dec 11 19:29 api.log
-rw-r--r--. 1 nova nova   1685308 Dec 11 19:28 cert.log
-rw-r--r--. 1 nova nova 247456778 Dec 11 19:29 compute.log
-rw-r--r--. 1 nova nova   5900194 Dec 11 03:48 compute.log-20131211.gz
-rw-r--r--. 1 nova nova 641882356 Dec 11 19:29 conductor.log
-rw-r--r--. 1 nova nova  16069786 Dec 11 03:48 conductor.log-20131211.gz
-rw-r--r--. 1 nova nova   2003009 Dec 11 19:29 consoleauth.log
-rw-r--r--. 1 nova nova    295496 Dec 11 19:29 console.log
-rw-r--r--. 1 nova nova      3520 Dec 11 17:40 metadata-api.log
-rw-r--r--. 1 nova nova  23647377 Dec 11 19:29 network.log
-rw-r--r--. 1 nova nova    451029 Dec 11 03:48 network.log-20131211.gz
-rw-r--r--. 1 root root     82306 Dec 11 19:12 nova-dhcpbridge.log
-rw-r--r--. 1 root root     64945 Dec 10 18:50 nova-manage.log
-rw-r--r--. 1 nova nova   6115017 Dec 11 19:29 scheduler.log
-rw-r--r--. 1 nova nova     76887 Dec 11 17:40 xvpvncproxy.log

Comment 7 errata-xmlrpc 2013-12-20 00:34:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2013-1859.html


Note You need to log in before you can comment on or make changes to this bug.