Bug 1490395

Summary: logging-fluentd fails to start in 3.6.173.0.32 - "Unknown filter plugin 'k8s_meta_filter_for_mux_client'"
Product: OpenShift Container Platform Reporter: Mike Fiedler <mifiedle>
Component: LoggingAssignee: Rich Megginson <rmeggins>
Status: CLOSED ERRATA QA Contact: Mike Fiedler <mifiedle>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.6.1CC: anli, aos-bugs, juzhao, rmeggins
Target Milestone: ---   
Target Release: 3.6.z   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Forgot to add k8s_meta_filter_for_mux_client ruby code to Dockerfile. Consequence: Fluentd complains about missing filter. Fix: Add the file to the Dockerfile. Result: No error.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-25 13:06:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mike Fiedler 2017-09-11 13:02:06 UTC 
Description of problem:

umount: /var/lib/docker/containers/*/shm: mountpoint not found
umounts of dead containers will fail. Ignoring...
2017-09-11 12:56:54 +0000 [info]: reading config file path="/etc/fluent/fluent.conf"
2017-09-11 12:56:54 +0000 [error]: config error file="/etc/fluent/fluent.conf" error="Unknown filter plugin 'k8s_meta_filter_for_mux_client'. Run 'gem search -rd fluent-plugin' to find plugins"
2017-09-11 12:56:54 +0000 [warn]: process died within 1 second. exit.



Version-Release number of selected component (if applicable): 3.6.0.173.32


How reproducible: Always


Steps to Reproduce:
1.  Deploy logging 3.6.0.173.32


Actual results:

All logging-fluentd pods are in CrashLoopBackoff
Comment 1 Mike Fiedler 2017-09-11 13:05:17 UTC 
Version in the description should be 3.6.173.0.32:

<ops registry>/openshift3/logging-fluentd             v3.6.173.0.32       4a9adda44009
Comment 2 Rich Megginson 2017-09-11 14:15:41 UTC 
Fixed in 
koji_builds:
  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=596221
repositories:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:rhaos-3.6-rhel-7-docker-candidate-51302-20170911131400
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:latest
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6.173.0.32
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6.173.0.32-2
Comment 3 Anping Li 2017-09-15 06:37:47 UTC 
There are similar error with both v3.6.173.0.5[1] and v3.6.173.0.33[2]. Can you confirm if the fix is in v3.6.173.0.33?

[1] v3.6.173.0.5
# oc logs logging-fluentd-2mvhq
umounts of dead containers will fail. Ignoring...
umount: /var/lib/docker/containers/*/shm: mountpoint not found
2017-09-15 01:52:57 -0400 [info]: reading config file path="/etc/fluent/fluent.conf"
2017-09-15 01:53:01 -0400 [warn]: 'exclude1' parameter is deprecated: Use <exclude> section
2017-09-15 01:53:08 -0400 [warn]: 'block' action stops input process until the buffer full is resolved. Check your pipeline this action is fit or not
2017-09-15 01:53:08 -0400 [warn]: 'block' action stops input process until the buffer full is resolved. Check your pipeline this action is fit or not
2017-09-15 01:53:48 -0400 [warn]: temporarily failed to flush the buffer. next_retry=2017-09-15 01:53:10 -0400 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"logging-es\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"})!" plugin_id="object:19a1b94"
  2017-09-15 01:53:48 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/out_elasticsearch_dynamic.rb:59:in `client'
  2017-09-15 01:53:48 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/out_elasticsearch_dynamic.rb:198:in `rescue in send_bulk'
  2017-09-15 01:53:48 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/out_elasticsearch_dynamic.rb:193:in `send_bulk'
  2017-09-15 01:53:48 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/out_elasticsearch_dynamic.rb:186:in `block in write_objects'
  2017-09-15 01:53:48 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/out_elasticsearch_dynamic.rb:185:in `each'
  2017-09-15 01:53:48 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/out_elasticsearch_dynamic.rb:185:in `write_objects'
  2017-09-15 01:53:48 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/output.rb:490:in `write'
  2017-09-15 01:53:48 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/buffer.rb:354:in `write_chunk'
  2017-09-15 01:53:48 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/buffer.rb:333:in `pop'
  2017-09-15 01:53:48 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/output.rb:342:in `try_flush'
  2017-09-15 01:53:48 -0400 [warn]: /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/output.rb:149:in `run'
2017-09-15 01:53:51 -0400 [warn]: temporarily failed to flush the buffer. next_retry=2017-09-15 01:53:10 -0400 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"logging-es\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"})!" plugin_id="object:17690c0"
  2017-09-15 01:53:51 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/out_elasticsearch_dynamic.rb:59:in `client'
  2017-09-15 01:53:51 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/out_elasticsearch_dynamic.rb:198:in `rescue in send_bulk'
  2017-09-15 01:53:51 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/out_elasticsearch_dynamic.rb:193:in `send_bulk'
  2017-09-15 01:53:51 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/out_elasticsearch_dynamic.rb:186:in `block in write_objects'
  2017-09-15 01:53:51 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/out_elasticsearch_dynamic.rb:185:in `each'
  2017-09-15 01:53:51 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/out_elasticsearch_dynamic.rb:185:in 

[2]v3.6.173.0.33
[root@openshift-181 ~]# oc logs logging-fluentd-kd50w
umounts of dead containers will fail. Ignoring...
umount: /var/lib/docker/containers/*/shm: mountpoint not found
2017-09-15 02:28:58 -0400 [info]: reading config file path="/etc/fluent/fluent.conf"
2017-09-15 02:29:08 -0400 [warn]: 'block' action stops input process until the buffer full is resolved. Check your pipeline this action is fit or not
2017-09-15 02:29:08 -0400 [warn]: 'block' action stops input process until the buffer full is resolved. Check your pipeline this action is fit or not
2017-09-15 02:29:53 -0400 [warn]: Could not push logs to Elasticsearch, resetting connection and trying again. read timeout reached
2017-09-15 02:29:54 -0400 [warn]: Could not push logs to Elasticsearch, resetting connection and trying again. read timeout reached
2017-09-15 02:30:29 -0400 [warn]: Could not push logs to Elasticsearch, resetting connection and trying again. read timeout reached
2017-09-15 02:30:30 -0400 [warn]: Could not push logs to Elasticsearch, resetting connection and trying again. read timeout reached
2017-09-15 02:30:51 -0400 [warn]: temporarily failed to flush the buffer. next_retry=2017-09-15 02:29:14 -0400 error_class="Fluent::ElasticsearchOutput::BulkIndexQueueFull" error="Bulk index queue is full, retrying" plugin_id="object:12d8d30"
  2017-09-15 02:30:51 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5.1/lib/fluent/plugin/out_elasticsearch.rb:454:in `block in send_bulk'
  2017-09-15 02:30:51 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5.1/lib/fluent/plugin/out_elasticsearch.rb:449:in `each_key'
  2017-09-15 02:30:51 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5.1/lib/fluent/plugin/out_elasticsearch.rb:449:in `send_bulk'
  2017-09-15 02:30:51 -0400 [warn]: /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5.1/lib/fluent/plugin/out_elasticsearch.rb:361:in `write_objects'
Comment 4 Junqi Zhao 2017-09-15 10:06:59 UTC 
(In reply to Anping Li from comment #3)
> There are similar error with both v3.6.173.0.5[1] and v3.6.173.0.33[2]. Can
> you confirm if the fix is in v3.6.173.0.33?
> 
> [1] v3.6.173.0.5
> # oc logs logging-fluentd-2mvhq
> umounts of dead containers will fail. Ignoring...
> umount: /var/lib/docker/containers/*/shm: mountpoint not found
> 2017-09-15 01:52:57 -0400 [info]: reading config file
> path="/etc/fluent/fluent.conf"
> 2017-09-15 01:53:01 -0400 [warn]: 'exclude1' parameter is deprecated: Use
> <exclude> section
> 2017-09-15 01:53:08 -0400 [warn]: 'block' action stops input process until
> the buffer full is resolved. Check your pipeline this action is fit or not
> 2017-09-15 01:53:08 -0400 [warn]: 'block' action stops input process until
> the buffer full is resolved. Check your pipeline this action is fit or not
> 2017-09-15 01:53:48 -0400 [warn]: temporarily failed to flush the buffer.
> next_retry=2017-09-15 01:53:10 -0400
> error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not
> reach Elasticsearch cluster ({:host=>\"logging-es\", :port=>9200,
> :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"})!"
> plugin_id="object:19a1b94"
>   2017-09-15 01:53:48 -0400 [warn]:
> /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/
> out_elasticsearch_dynamic.rb:59:in `client'
>   2017-09-15 01:53:48 -0400 [warn]:
> /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/
> out_elasticsearch_dynamic.rb:198:in `rescue in send_bulk'
>   2017-09-15 01:53:48 -0400 [warn]:
> /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/
> out_elasticsearch_dynamic.rb:193:in `send_bulk'
>   2017-09-15 01:53:48 -0400 [warn]:
> /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/
> out_elasticsearch_dynamic.rb:186:in `block in write_objects'
>   2017-09-15 01:53:48 -0400 [warn]:
> /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/
> out_elasticsearch_dynamic.rb:185:in `each'
>   2017-09-15 01:53:48 -0400 [warn]:
> /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/
> out_elasticsearch_dynamic.rb:185:in `write_objects'
>   2017-09-15 01:53:48 -0400 [warn]:
> /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/output.rb:490:in `write'
>   2017-09-15 01:53:48 -0400 [warn]:
> /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/buffer.rb:354:in
> `write_chunk'
>   2017-09-15 01:53:48 -0400 [warn]:
> /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/buffer.rb:333:in `pop'
>   2017-09-15 01:53:48 -0400 [warn]:
> /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/output.rb:342:in `try_flush'
>   2017-09-15 01:53:48 -0400 [warn]:
> /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/output.rb:149:in `run'
> 2017-09-15 01:53:51 -0400 [warn]: temporarily failed to flush the buffer.
> next_retry=2017-09-15 01:53:10 -0400
> error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not
> reach Elasticsearch cluster ({:host=>\"logging-es\", :port=>9200,
> :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"})!"
> plugin_id="object:17690c0"
>   2017-09-15 01:53:51 -0400 [warn]:
> /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/
> out_elasticsearch_dynamic.rb:59:in `client'
>   2017-09-15 01:53:51 -0400 [warn]:
> /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/
> out_elasticsearch_dynamic.rb:198:in `rescue in send_bulk'
>   2017-09-15 01:53:51 -0400 [warn]:
> /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/
> out_elasticsearch_dynamic.rb:193:in `send_bulk'
>   2017-09-15 01:53:51 -0400 [warn]:
> /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/
> out_elasticsearch_dynamic.rb:186:in `block in write_objects'
>   2017-09-15 01:53:51 -0400 [warn]:
> /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/
> out_elasticsearch_dynamic.rb:185:in `each'
>   2017-09-15 01:53:51 -0400 [warn]:
> /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5/lib/fluent/plugin/
> out_elasticsearch_dynamic.rb:185:in 
> 
> [2]v3.6.173.0.33
> [root@openshift-181 ~]# oc logs logging-fluentd-kd50w
> umounts of dead containers will fail. Ignoring...
> umount: /var/lib/docker/containers/*/shm: mountpoint not found
> 2017-09-15 02:28:58 -0400 [info]: reading config file
> path="/etc/fluent/fluent.conf"
> 2017-09-15 02:29:08 -0400 [warn]: 'block' action stops input process until
> the buffer full is resolved. Check your pipeline this action is fit or not
> 2017-09-15 02:29:08 -0400 [warn]: 'block' action stops input process until
> the buffer full is resolved. Check your pipeline this action is fit or not
> 2017-09-15 02:29:53 -0400 [warn]: Could not push logs to Elasticsearch,
> resetting connection and trying again. read timeout reached
> 2017-09-15 02:29:54 -0400 [warn]: Could not push logs to Elasticsearch,
> resetting connection and trying again. read timeout reached
> 2017-09-15 02:30:29 -0400 [warn]: Could not push logs to Elasticsearch,
> resetting connection and trying again. read timeout reached
> 2017-09-15 02:30:30 -0400 [warn]: Could not push logs to Elasticsearch,
> resetting connection and trying again. read timeout reached
> 2017-09-15 02:30:51 -0400 [warn]: temporarily failed to flush the buffer.
> next_retry=2017-09-15 02:29:14 -0400
> error_class="Fluent::ElasticsearchOutput::BulkIndexQueueFull" error="Bulk
> index queue is full, retrying" plugin_id="object:12d8d30"
>   2017-09-15 02:30:51 -0400 [warn]:
> /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5.1/lib/fluent/plugin/
> out_elasticsearch.rb:454:in `block in send_bulk'
>   2017-09-15 02:30:51 -0400 [warn]:
> /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5.1/lib/fluent/plugin/
> out_elasticsearch.rb:449:in `each_key'
>   2017-09-15 02:30:51 -0400 [warn]:
> /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5.1/lib/fluent/plugin/
> out_elasticsearch.rb:449:in `send_bulk'
>   2017-09-15 02:30:51 -0400 [warn]:
> /usr/share/gems/gems/fluent-plugin-elasticsearch-1.9.5.1/lib/fluent/plugin/
> out_elasticsearch.rb:361:in `write_objects'

see this defect for logging 3.5, https://bugzilla.redhat.com/show_bug.cgi?id=1479651, we did not find the error for logging 3.6, maybe it has error now
Comment 5 Rich Megginson 2017-09-18 02:44:09 UTC 
The bug was introduced in 3.6.2 (and 3.7.0), and fixed in a later 3.6.2 (and 3.7.0) candidate image.
Comment 6 Mike Fiedler 2017-09-18 17:43:06 UTC 
Verified on 3.6.173.0.35
Comment 8 errata-xmlrpc 2017-10-25 13:06:40 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3049