Bug 1796133 - Write fluent logs to stdout and exclude them from collection
Summary: Write fluent logs to stdout and exclude them from collection
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 4.4.0
Assignee: Jeff Cantrill
QA Contact: Qiaoling Tang
URL:
Whiteboard:
Depends On:
Blocks: 1798073
TreeView+ depends on / blocked
 
Reported: 2020-01-29 17:00 UTC by Jeff Cantrill
Modified: 2020-09-14 09:44 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Fluent logs were written to host disk so they would be available for analysis but were not being properly cleaned up. Consequence: This could result in the node not having enough disk to function correctly Fix: Revert container logging to stdout and exclude them from collection to avoid feedback loops Result: Disk space is no longer consumed by collector logs
Clone Of:
: 1798073 (view as bug list)
Environment:
Last Closed: 2020-05-04 11:27:29 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-logging-operator pull 348 0 None closed Bug 1796133: Exclude fluent/elasticsearch pod from log collection and write logs to STDOUT 2021-02-18 18:50:44 UTC
Red Hat Product Errata RHBA-2020:0581 0 None None None 2020-05-04 11:27:45 UTC

Description Jeff Cantrill 2020-01-29 17:00:54 UTC
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Jeff Cantrill 2020-01-29 18:23:47 UTC
Fluent logs were written to disk in order to:
1. Avoid a feedback loop of collecting its own logs
2. Allow collector investigation when there are issues by being able to review persistent logs.

This strategy has proven to be problematic and can lead to node issues if fluent log rotation is now working properly and logs are not being cleaned up.  Logs should be reverted to be written to stdout and excluded from collection.  The pattern for the container logs for thcollector:

sh-4.4# ls | grep fluent
fluentd-m65p9_openshift-logging_fluentd-591768f9629810d85fbe623c4afc76bd96424e67e48ccdc2128f40cd0a94611d.log
fluentd-m65p9_openshift-logging_fluentd-9872757d29cc899822c2586f9e519c8d365044ab7330371564085ba6e600fbf6.log

Possibly partially responsible for https://bugzilla.redhat.com/show_bug.cgi?id=1780698

Comment 2 Jeff Cantrill 2020-01-29 20:30:36 UTC
This should be backported to 4.2 and 4.3

Comment 4 Qiaoling Tang 2020-02-06 07:33:27 UTC
Tested with ose-cluster-logging-operator-v4.4.0-202002050701, the fluentd pod logs are set to stdout, and aren't collected to the ES.

But there still have the env vars `LOGGING_FILE_*` in the fluentd pods, should these env vars be removed ?

# oc exec fluentd-6rkql env |grep LOGGING_FILE
LOGGING_FILE_PATH=/var/log/fluentd/fluentd.log
LOGGING_FILE_AGE=10
LOGGING_FILE_SIZE=1024000

Comment 5 Jeff Cantrill 2020-02-17 14:00:50 UTC
(In reply to Qiaoling Tang from comment #4)
> Tested with ose-cluster-logging-operator-v4.4.0-202002050701, the fluentd
> pod logs are set to stdout, and aren't collected to the ES.
> 
> But there still have the env vars `LOGGING_FILE_*` in the fluentd pods,
> should these env vars be removed ?
> 
> # oc exec fluentd-6rkql env |grep LOGGING_FILE
> LOGGING_FILE_PATH=/var/log/fluentd/fluentd.log
> LOGGING_FILE_AGE=10
> LOGGING_FILE_SIZE=1024000

We will do this as tech debt

Comment 6 Qiaoling Tang 2020-02-18 00:04:42 UTC
Per c4 and c5, move this bug to VERIFIED.

Comment 8 errata-xmlrpc 2020-05-04 11:27:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.