Bug 1308872 - Need protection from logs filling up the disk space
Need protection from logs filling up the disk space
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo (Show other bugs)
7.0 (Kilo)
All All
unspecified Severity low
: ---
: 8.0 (Liberty)
Assigned To: James Slagle
Depends On:
  Show dependency treegraph
Reported: 2016-02-16 05:33 EST by Prateek Arora
Modified: 2016-04-18 03:00 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2016-02-23 05:02:56 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Prateek Arora 2016-02-16 05:33:30 EST
Description of problem:

The environment had some problems and as such the amount of logs that were produced were in the range of 1300 Gigs in a very short span of time in an hour so. 
The logrotate utility is configured by cron to run daily and keep only 140M logs but like here in case the logrotate didnt have a chance to run and the customer was in a fix before that

/var/log/neutron/*.log {
    rotate 14
    size 10M

Version-Release number of selected component (if applicable):

RHOS 7.0

How reproducible:


Steps to Reproduce:
1.By having runaway logs on a RHOS deployment

Actual results:
The disk space filled out completely

Expected results:
Some defence mechanism to handle such things

Additional info:

The logs get compressed become very small like a 250 GB log got compressed to 4.4G, see below

656G    openvswitch-agent.log
4.4G    openvswitch-agent.log-20160121.gz ->~250GB
12G     openvswitch-agent.log-20160122.gz ->~684GB
635G    openvswitch-agent.log-20160123
3.2M    openvswitch-agent.log-20160124

but if the disk does not have space for compression to succeed, the problem arises with huge logs and no disk to write to them or do anything/

The logrotate utility is working fime and the cron is configured to run it daily and it is doing as expected

Note You need to log in before you can comment on or make changes to this bug.