Bug 1312490 - Fluentd holds open log files that have been deleted
Summary: Fluentd holds open log files that have been deleted
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OKD
Classification: Red Hat
Component: Logging
Version: 3.x
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: ewolinet
QA Contact: chunchen
URL:
Whiteboard:
Depends On:
Blocks: OSOPS_V3
TreeView+ depends on / blocked
 
Reported: 2016-02-26 21:32 UTC by Sten Turpin
Modified: 2017-12-13 02:29 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-05-12 17:15:56 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
lsof output (823 bytes, text/plain)
2016-02-26 21:32 UTC, Sten Turpin 		no flags Details
View All

Description Sten Turpin 2016-02-26 21:32:52 UTC 
Created attachment 1130909 [details]
lsof output

Description of problem: Fluentd is holding log files open that have been deleted. The inode locks stay in place for hours, possibly longer, if the fluentd container is not restarted. 


Version-Release number of selected component (if applicable): 3.1.1.6


How reproducible: always, when metrics are on


Steps to Reproduce:
1. enable openshift metrics
2. rotate system logs

Actual results:
logs removed by logrotate are still open and taking space

Expected results:
fluentd should eventually let the files go

Additional info:
See attached lsof output
Comment 3 Sten Turpin 2016-03-11 17:18:04 UTC 
We ended up changing the fluentd image to set it to ignore gzipped/archived files, which has prevented issues with us running out of disk space.
Note You need to log in before you can comment on or make changes to this bug.