Bug 1312490 - Fluentd holds open log files that have been deleted
Fluentd holds open log files that have been deleted
Product: OpenShift Origin
Classification: Red Hat
Component: Logging (Show other bugs)
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: ewolinet
Depends On:
Blocks: OSOPS_V3
  Show dependency treegraph
Reported: 2016-02-26 16:32 EST by Sten Turpin
Modified: 2017-12-12 21:29 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2016-05-12 13:15:56 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
lsof output (823 bytes, text/plain)
2016-02-26 16:32 EST, Sten Turpin
no flags Details

  None (edit)
Description Sten Turpin 2016-02-26 16:32:52 EST
Created attachment 1130909 [details]
lsof output

Description of problem: Fluentd is holding log files open that have been deleted. The inode locks stay in place for hours, possibly longer, if the fluentd container is not restarted. 

Version-Release number of selected component (if applicable):

How reproducible: always, when metrics are on

Steps to Reproduce:
1. enable openshift metrics
2. rotate system logs

Actual results:
logs removed by logrotate are still open and taking space

Expected results:
fluentd should eventually let the files go

Additional info:
See attached lsof output
Comment 3 Sten Turpin 2016-03-11 12:18:04 EST
We ended up changing the fluentd image to set it to ignore gzipped/archived files, which has prevented issues with us running out of disk space.

Note You need to log in before you can comment on or make changes to this bug.