Cause: Fluentd generates events internally for some unknown reason with the OneEventStream class. This class does not have the `empty?` method. The kubernetes metadata filter was using the `empty?` method on the EventStream object to avoid processing an empty stream.
Consequence: Fluentd issues many error messages complaining about a missing `empty?` method. This overwhelms the container logging and causes disk issues.
Fix: The kubernetes metadata filter was changed to only call the `empty?` method on objects which have that method.
Result: fluentd logs should not be filled with this message.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2018:2908
Description of problem: 2018-09-06 09:07:39 +0000 [warn]: emit transaction failed: error_class=NoMethodError error="undefined method `empty?' for #<Fluent::OneEventStream:0x007fbe0179ea70>" location="/usr/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-1.0.3/lib/fluent/plugin/filter_kubernetes_metadata.rb:287:in `filter_stream_from_files'" tag="kubernetes.journal.container.fluentd" 2018-09-06 09:07:39 +0000 [warn]: suppressed same stacktrace 2018-09-06 09:07:39 +0000 [warn]: emit transaction failed: error_class=NoMethodError error="undefined method `empty?' for #<Fluent::OneEventStream:0x007fbe0179ea70>" location="/usr/share/gems/gems/fluent-plugin-kubernetes_metadata_filter-1.0.3/lib/fluent/plugin/filter_kubernetes_metadata.rb:287:in `filter_stream_from_files'" tag="journal" 2018-09-06 09:07:39 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2018-09-06 09:07:42 +0000 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"logging-es\", :port=>9200, :scheme=>\"https\"})!" plugin_id="object:3fdf005434e4" 2018-09-06 09:07:39 +0000 [warn]: suppressed same stacktrace 2018-09-06 09:07:39 +0000 [warn]: suppressed same stacktrace Version-Release number of selected component (if applicable): v3.9.40 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: