+++ This bug was initially created as a clone of Bug #1505683 +++ Description of problem: Deployed logging 3.6, fluentd pods failed to start up, error="Unknown filter plugin 'record_modifier' in fluentd pods log. used 'gem search -rd fluent-plugin' to check, found fluent-plugin-record-modifier (0.6.0), not ***record_modifier***, see the attached file. bash-4.2# gem search -rd fluent-plugin # oc get po NAME READY STATUS RESTARTS AGE logging-curator-1-jhjcm 1/1 Running 0 1h logging-curator-ops-1-glg8f 1/1 Running 0 1h logging-es-data-master-myws4feq-1-zwrp5 1/1 Running 0 1h logging-es-ops-data-master-6zs09u6z-1-lsb8h 1/1 Running 0 1h logging-fluentd-9cmn0 0/1 CrashLoopBackOff 17 1h logging-fluentd-l9vhg 0/1 CrashLoopBackOff 17 1h logging-kibana-1-bqs5k 2/2 Running 0 1h logging-kibana-ops-1-c9s1t 2/2 Running 0 1h # oc logs logging-fluentd-9cmn0 umounts of dead containers will fail. Ignoring... umount: /var/lib/docker/containers/1df690adcaaa59489757a69afbf20d344e341a76cddd57284103b5f801381e18/shm: not mounted 2017-10-24 01:40:48 -0400 [info]: reading config file path="/etc/fluent/fluent.conf" 2017-10-24 01:40:48 -0400 [error]: config error file="/etc/fluent/fluent.conf" error="Unknown filter plugin 'record_modifier'. Run 'gem search -rd fluent-plugin' to find plugins" 2017-10-24 01:40:48 -0400 [warn]: process died within 1 second. exit. Version-Release number of selected component (if applicable): logging-kibana:v3.6.173.0.59-1 logging-elasticsearch:v3.6.173.0.59-1 logging-fluentd:v3.6.173.0.59-1 logging-auth-proxy:v3.6.173.0.59-1 logging-curator:v3.6.173.0.59-1 # rpm -qa | grep openshift-ansible openshift-ansible-filter-plugins-3.6.173.0.59-1.git.0.0e31372.el7.noarch openshift-ansible-docs-3.6.173.0.59-1.git.0.0e31372.el7.noarch openshift-ansible-lookup-plugins-3.6.173.0.59-1.git.0.0e31372.el7.noarch openshift-ansible-callback-plugins-3.6.173.0.59-1.git.0.0e31372.el7.noarch openshift-ansible-playbooks-3.6.173.0.59-1.git.0.0e31372.el7.noarch openshift-ansible-3.6.173.0.59-1.git.0.0e31372.el7.noarch openshift-ansible-roles-3.6.173.0.59-1.git.0.0e31372.el7.noarch How reproducible: Always Steps to Reproduce: 1. Deploy logging 3.6 and check pod status 2. 3. Actual results: fluentd pods failed to start up, error="Unknown filter plugin 'record_modifier' in fluentd pods log. Expected results: fluentd pods could be started up. Additional info: --- Additional comment from Junqi Zhao on 2017-10-24 02:45 EDT --- --- Additional comment from Junqi Zhao on 2017-10-24 03:27:46 EDT --- blocks all the logging testing now 3.7 env: # openshift version openshift v3.7.0-0.176.0 kubernetes v1.7.6+a08f5eeb62 etcd 3.2.8 image logging-fluentd/images/v3.7.0-0.177.0.1
This PR will be addressing the issue https://github.com/openshift/origin-aggregated-logging/pull/745
Commits pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/7f6b96ed040617e95ad396d3c133c86c720e7d53 Bug 1506099 - fluentd downstream vendor record_modifier Vendoring the missing record_modifier to downstream image. The #622 PR introduced record_modifier to upstream Dockerfile but not to the downstream. https://github.com/openshift/origin-aggregated-logging/pull/622 https://github.com/openshift/origin-aggregated-logging/commit/effe1d921af7ddfed4e918661bb89c1aefb11b0b Merge pull request #745 from wozniakjan/bz_1505683/fluentd/record-modifier Automatic merge from submit-queue. Bug 1506099 - fluentd downstream vendor record_modifier Vendoring the missing record_modifier to downstream image. The #622 PR introduced record_modifier to upstream Dockerfile but not to the downstream. This is porting forward the https://github.com/openshift/origin-aggregated-logging/pull/742
koji_builds: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=615161 repositories: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:rhaos-3.7-rhel-7-docker-candidate-50013-20171027132618 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:latest brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.7.0 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.7.0-0.183.0.1 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.7
The bug is fixed on v3.7.0-0.185.0.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3188