Bug 1626281 - [3.9] fluentd pods are running with error logs which makes fill up disk very quickly.
Summary: [3.9] fluentd pods are running with error logs which makes fill up disk very ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.9.z
Assignee: Rich Megginson
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks: 1626552 1626553
TreeView+ depends on / blocked
 
Reported: 2018-09-07 00:12 UTC by Wang Haoran
Modified: 2018-11-20 03:12 UTC (History)
4 users (show)

Fixed In Version: openshift3/logging-fluentd:v3.9.44-1
Doc Type: Bug Fix
Doc Text:
Cause: Fluentd generates events internally for some unknown reason with the OneEventStream class. This class does not have the `empty?` method. The kubernetes metadata filter was using the `empty?` method on the EventStream object to avoid processing an empty stream. Consequence: Fluentd issues many error messages complaining about a missing `empty?` method. This overwhelms the container logging and causes disk issues. Fix: The kubernetes metadata filter was changed to only call the `empty?` method on objects which have that method. Result: fluentd logs should not be filled with this message.
Clone Of:
: 1626552 (view as bug list)
Environment:
Last Closed: 2018-11-20 03:12:03 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github fabric8io fluent-plugin-kubernetes_metadata_filter issues 146 0 None closed OneEventStream throws undefined error method 'empty?' 2021-01-29 20:24:42 UTC
Github fabric8io fluent-plugin-kubernetes_metadata_filter pull 147 0 None closed [master] test for eventstream has empty method and if class is EventStream 2021-01-29 20:24:43 UTC
Github fabric8io fluent-plugin-kubernetes_metadata_filter pull 148 0 None closed [v0.12] test for eventstream has empty method and if class is EventStream 2021-01-29 20:24:43 UTC
Red Hat Product Errata RHSA-2018:2908 0 None None None 2018-11-20 03:12:44 UTC

Description Wang Haoran 2018-09-07 00:12:22 UTC 
Description of problem:

2018-09-06 09:07:39 +0000 [warn]: emit transaction failed: error_class=NoMethodError error="undefined method `empty?' for #<Fluent::OneEventStream:0x007fbe0179ea70>" location="/usr/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-1.0.3/lib/fluent/plugin/filter_kubernetes_metadata.rb:287:in `filter_stream_from_files'" tag="kubernetes.journal.container.fluentd"
  2018-09-06 09:07:39 +0000 [warn]: suppressed same stacktrace
2018-09-06 09:07:39 +0000 [warn]: emit transaction failed: error_class=NoMethodError error="undefined method `empty?' for #<Fluent::OneEventStream:0x007fbe0179ea70>" location="/usr/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-1.0.3/lib/fluent/plugin/filter_kubernetes_metadata.rb:287:in `filter_stream_from_files'" tag="journal"
2018-09-06 09:07:39 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2018-09-06 09:07:42 +0000 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"logging-es\", :port=>9200, :scheme=>\"https\"})!" plugin_id="object:3fdf005434e4"
  2018-09-06 09:07:39 +0000 [warn]: suppressed same stacktrace
  2018-09-06 09:07:39 +0000 [warn]: suppressed same stacktrace



Version-Release number of selected component (if applicable):

v3.9.40

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 6 Anping Li 2018-10-15 05:54:44 UTC 
Verified and pass with openshift3/logging-fluentd/images/v3.9.47-1
Comment 8 errata-xmlrpc 2018-11-20 03:12:03 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2908
Note You need to log in before you can comment on or make changes to this bug.