Bug 1796133
| Summary: | Write fluent logs to stdout and exclude them from collection | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Jeff Cantrill <jcantril> | |
| Component: | Logging | Assignee: | Jeff Cantrill <jcantril> | |
| Status: | CLOSED ERRATA | QA Contact: | Qiaoling Tang <qitang> | |
| Severity: | unspecified | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 4.4 | CC: | aos-bugs, ocasalsa | |
| Target Milestone: | --- | |||
| Target Release: | 4.4.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: |
Cause: Fluent logs were written to host disk so they would be available for analysis but were not being properly cleaned up.
Consequence: This could result in the node not having enough disk to function correctly
Fix: Revert container logging to stdout and exclude them from collection to avoid feedback loops
Result: Disk space is no longer consumed by collector logs
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1798073 (view as bug list) | Environment: | ||
| Last Closed: | 2020-05-04 11:27:29 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1798073 | |||
|
Description
Jeff Cantrill
2020-01-29 17:00:54 UTC
Fluent logs were written to disk in order to: 1. Avoid a feedback loop of collecting its own logs 2. Allow collector investigation when there are issues by being able to review persistent logs. This strategy has proven to be problematic and can lead to node issues if fluent log rotation is now working properly and logs are not being cleaned up. Logs should be reverted to be written to stdout and excluded from collection. The pattern for the container logs for thcollector: sh-4.4# ls | grep fluent fluentd-m65p9_openshift-logging_fluentd-591768f9629810d85fbe623c4afc76bd96424e67e48ccdc2128f40cd0a94611d.log fluentd-m65p9_openshift-logging_fluentd-9872757d29cc899822c2586f9e519c8d365044ab7330371564085ba6e600fbf6.log Possibly partially responsible for https://bugzilla.redhat.com/show_bug.cgi?id=1780698 This should be backported to 4.2 and 4.3 Tested with ose-cluster-logging-operator-v4.4.0-202002050701, the fluentd pod logs are set to stdout, and aren't collected to the ES. But there still have the env vars `LOGGING_FILE_*` in the fluentd pods, should these env vars be removed ? # oc exec fluentd-6rkql env |grep LOGGING_FILE LOGGING_FILE_PATH=/var/log/fluentd/fluentd.log LOGGING_FILE_AGE=10 LOGGING_FILE_SIZE=1024000 (In reply to Qiaoling Tang from comment #4) > Tested with ose-cluster-logging-operator-v4.4.0-202002050701, the fluentd > pod logs are set to stdout, and aren't collected to the ES. > > But there still have the env vars `LOGGING_FILE_*` in the fluentd pods, > should these env vars be removed ? > > # oc exec fluentd-6rkql env |grep LOGGING_FILE > LOGGING_FILE_PATH=/var/log/fluentd/fluentd.log > LOGGING_FILE_AGE=10 > LOGGING_FILE_SIZE=1024000 We will do this as tech debt Per c4 and c5, move this bug to VERIFIED. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581 |