Bug 1308872 - Need protection from logs filling up the disk space
Summary: Need protection from logs filling up the disk space
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo
Version: 7.0 (Kilo)
Hardware: All
OS: All
unspecified
low
Target Milestone: ---
: 8.0 (Liberty)
Assignee: James Slagle
QA Contact: yeylon@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-02-16 10:33 UTC by Prateek Arora
Modified: 2016-04-18 07:00 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-02-23 10:02:56 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Prateek Arora 2016-02-16 10:33:30 UTC
Description of problem:

The environment had some problems and as such the amount of logs that were produced were in the range of 1300 Gigs in a very short span of time in an hour so. 
The logrotate utility is configured by cron to run daily and keep only 140M logs but like here in case the logrotate didnt have a chance to run and the customer was in a fix before that

/var/log/neutron/*.log {
    rotate 14
    size 10M
    missingok
    compress
    copytruncate
}


Version-Release number of selected component (if applicable):

RHOS 7.0

How reproducible:

Always

Steps to Reproduce:
1.By having runaway logs on a RHOS deployment
2.
3.

Actual results:
The disk space filled out completely

Expected results:
Some defence mechanism to handle such things

Additional info:

The logs get compressed become very small like a 250 GB log got compressed to 4.4G, see below

656G    openvswitch-agent.log
4.4G    openvswitch-agent.log-20160121.gz ->~250GB
12G     openvswitch-agent.log-20160122.gz ->~684GB
635G    openvswitch-agent.log-20160123
3.2M    openvswitch-agent.log-20160124

but if the disk does not have space for compression to succeed, the problem arises with huge logs and no disk to write to them or do anything/


The logrotate utility is working fime and the cron is configured to run it daily and it is doing as expected


Note You need to log in before you can comment on or make changes to this bug.