Created attachment 1326316 [details]
fluentd log file

Description of problem:
Deploy logging 3.7 with mux service, "Unknown filter plugin 'k8s_meta_filter_for_mux_client' error in fluentd pod log, and fluentd can not work.
# oc get po | grep fluentd
logging-fluentd-hg5w5                     0/1       CrashLoopBackOff   9          49m
logging-fluentd-zw39h                     1/1       Running            0          49m

# oc logs logging-fluentd-hg5w5
2017-09-15 05:34:19 +0000 [info]: reading config file path="/etc/fluent/fluent.conf"
2017-09-15 05:34:21 +0000 [error]: config error file="/etc/fluent/fluent.conf" error="Unknown filter plugin 'k8s_meta_filter_for_mux_client'. Run 'gem search -rd fluent-plugin' to find plugins"
2017-09-15 05:34:21 +0000 [error]: fluentd main process died unexpectedly. restarting.
2017-09-15 05:34:22 +0000 [error]: config error file="/etc/fluent/fluent.conf" error="Unknown filter plugin 'k8s_meta_filter_for_mux_client'. Run 'gem search -rd fluent-plugin' to find plugins"
2017-09-15 05:34:22 +0000 [error]: fluentd main process died unexpectedly. restarting.
2017-09-15 05:34:24 +0000 [error]: config error file="/etc/fluent/fluent.conf" error="Unknown filter plugin 'k8s_meta_filter_for_mux_client'. Run 'gem search -rd fluent-plugin' to find plugins"
2017-09-15 05:34:24 +0000 [error]: fluentd main process died unexpectedly. restarting.

Version-Release number of selected component (if applicable):
# rpm -qa | grep openshift-ansible
openshift-ansible-3.7.0-0.126.1.git.0.0bb5b0c.el7.noarch
openshift-ansible-roles-3.7.0-0.126.1.git.0.0bb5b0c.el7.noarch
openshift-ansible-docs-3.7.0-0.126.1.git.0.0bb5b0c.el7.noarch
openshift-ansible-callback-plugins-3.7.0-0.126.1.git.0.0bb5b0c.el7.noarch
openshift-ansible-filter-plugins-3.7.0-0.126.1.git.0.0bb5b0c.el7.noarch
openshift-ansible-playbooks-3.7.0-0.126.1.git.0.0bb5b0c.el7.noarch
openshift-ansible-lookup-plugins-3.7.0-0.126.1.git.0.0bb5b0c.el7.noarch

Images
logging-kibana:v3.7.0-0.126.1.0
logging-fluentd:v3.7.0-0.126.1.0
logging-auth-proxy:v3.7.0-0.126.1.0
logging-curator:v3.7.0-0.126.1.0
logging-elasticsearch:v3.7.0-0.126.1.0

How reproducible:
Always

Steps to Reproduce:
1. Deploy logging 3.7 with mux, inventory file see the [Additional info] part
2.
3.

Actual results:
"Unknown filter plugin 'k8s_meta_filter_for_mux_client' error in fluentd pod log

Expected results:
There should not be error in fluentd pod log

Additional info:
# inventory file
[OSEv3:children]
masters
etcd

[masters]
${MASTER_URL} openshift_public_hostname=${MASTER_URL}

[etcd]
${MASTER_URL} openshift_public_hostname=${MASTER_URL}

[OSEv3:vars]
ansible_ssh_user=root
ansible_ssh_private_key_file="~/libra.pem"
deployment_type=openshift-enterprise
openshift_docker_additional_registries=registry.ops.openshift.com

# Logging
openshift_logging_install_logging=true
openshift_logging_kibana_hostname=kibana.${SUB_DOMAIN}
public_master_url=https://${MASTER_URL}:8443
openshift_logging_image_prefix=registry.ops.openshift.com/openshift3/
openshift_logging_image_version=v3.7
openshift_logging_namespace=logging
openshift_logging_elasticsearch_cpu_limit=500m
openshift_logging_elasticsearch_memory_limit=1Gi
openshift_logging_use_mux=true
openshift_logging_mux_client_mode=maximal
openshift_logging_mux_memory_limit=1Gi