Bug 1506099

+++ This bug was initially created as a clone of Bug #1505683 +++

Description of problem:
Deployed logging 3.6, fluentd pods failed to start up, error="Unknown filter plugin 'record_modifier' in fluentd pods log.
used 'gem search -rd fluent-plugin' to check, found fluent-plugin-record-modifier (0.6.0), not ***record_modifier***, see the attached file.
bash-4.2# gem search -rd fluent-plugin
# oc get po
NAME                                          READY     STATUS             RESTARTS   AGE
logging-curator-1-jhjcm                       1/1       Running            0          1h
logging-curator-ops-1-glg8f                   1/1       Running            0          1h
logging-es-data-master-myws4feq-1-zwrp5       1/1       Running            0          1h
logging-es-ops-data-master-6zs09u6z-1-lsb8h   1/1       Running            0          1h
logging-fluentd-9cmn0                         0/1       CrashLoopBackOff   17         1h
logging-fluentd-l9vhg                         0/1       CrashLoopBackOff   17         1h
logging-kibana-1-bqs5k                        2/2       Running            0          1h
logging-kibana-ops-1-c9s1t                    2/2       Running            0          1h

# oc logs logging-fluentd-9cmn0
umounts of dead containers will fail. Ignoring...
umount: /var/lib/docker/containers/1df690adcaaa59489757a69afbf20d344e341a76cddd57284103b5f801381e18/shm: not mounted
2017-10-24 01:40:48 -0400 [info]: reading config file path="/etc/fluent/fluent.conf"
2017-10-24 01:40:48 -0400 [error]: config error file="/etc/fluent/fluent.conf" error="Unknown filter plugin 'record_modifier'. Run 'gem search -rd fluent-plugin' to find plugins"
2017-10-24 01:40:48 -0400 [warn]: process died within 1 second. exit.


Version-Release number of selected component (if applicable):
logging-kibana:v3.6.173.0.59-1
logging-elasticsearch:v3.6.173.0.59-1
logging-fluentd:v3.6.173.0.59-1
logging-auth-proxy:v3.6.173.0.59-1
logging-curator:v3.6.173.0.59-1

# rpm -qa | grep openshift-ansible
openshift-ansible-filter-plugins-3.6.173.0.59-1.git.0.0e31372.el7.noarch
openshift-ansible-docs-3.6.173.0.59-1.git.0.0e31372.el7.noarch
openshift-ansible-lookup-plugins-3.6.173.0.59-1.git.0.0e31372.el7.noarch
openshift-ansible-callback-plugins-3.6.173.0.59-1.git.0.0e31372.el7.noarch
openshift-ansible-playbooks-3.6.173.0.59-1.git.0.0e31372.el7.noarch
openshift-ansible-3.6.173.0.59-1.git.0.0e31372.el7.noarch
openshift-ansible-roles-3.6.173.0.59-1.git.0.0e31372.el7.noarch


How reproducible:
Always

Steps to Reproduce:
1. Deploy logging 3.6 and check pod status
2.
3.

Actual results:
fluentd pods failed to start up, error="Unknown filter plugin 'record_modifier' in fluentd pods log.

Expected results:
fluentd pods could be started up.

Additional info:

--- Additional comment from Junqi Zhao on 2017-10-24 02:45 EDT ---



--- Additional comment from Junqi Zhao on 2017-10-24 03:27:46 EDT ---

blocks all the logging testing now

3.7 env:
# openshift version
openshift v3.7.0-0.176.0
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8

image
logging-fluentd/images/v3.7.0-0.177.0.1