Bug 1312490
| Summary: | Fluentd holds open log files that have been deleted | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OKD | Reporter: | Sten Turpin <sten> | ||||
| Component: | Logging | Assignee: | ewolinet | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | chunchen <chunchen> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 3.x | CC: | aos-bugs, ewolinet, pportant, sten, wsun | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2016-05-12 17:15:56 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1303130 | ||||||
| Attachments: |
|
||||||
We ended up changing the fluentd image to set it to ignore gzipped/archived files, which has prevented issues with us running out of disk space. |
Created attachment 1130909 [details] lsof output Description of problem: Fluentd is holding log files open that have been deleted. The inode locks stay in place for hours, possibly longer, if the fluentd container is not restarted. Version-Release number of selected component (if applicable): 3.1.1.6 How reproducible: always, when metrics are on Steps to Reproduce: 1. enable openshift metrics 2. rotate system logs Actual results: logs removed by logrotate are still open and taking space Expected results: fluentd should eventually let the files go Additional info: See attached lsof output