Created attachment 1342522 [details] logging environment dump Description of problem: Deployed logging 3.6, fluentd pods failed to start up, error="Unknown filter plugin 'record_modifier' in fluentd pods log. used 'gem search -rd fluent-plugin' to check, found fluent-plugin-record-modifier (0.6.0), not ***record_modifier***, see the attached file. bash-4.2# gem search -rd fluent-plugin # oc get po NAME READY STATUS RESTARTS AGE logging-curator-1-jhjcm 1/1 Running 0 1h logging-curator-ops-1-glg8f 1/1 Running 0 1h logging-es-data-master-myws4feq-1-zwrp5 1/1 Running 0 1h logging-es-ops-data-master-6zs09u6z-1-lsb8h 1/1 Running 0 1h logging-fluentd-9cmn0 0/1 CrashLoopBackOff 17 1h logging-fluentd-l9vhg 0/1 CrashLoopBackOff 17 1h logging-kibana-1-bqs5k 2/2 Running 0 1h logging-kibana-ops-1-c9s1t 2/2 Running 0 1h # oc logs logging-fluentd-9cmn0 umounts of dead containers will fail. Ignoring... umount: /var/lib/docker/containers/1df690adcaaa59489757a69afbf20d344e341a76cddd57284103b5f801381e18/shm: not mounted 2017-10-24 01:40:48 -0400 [info]: reading config file path="/etc/fluent/fluent.conf" 2017-10-24 01:40:48 -0400 [error]: config error file="/etc/fluent/fluent.conf" error="Unknown filter plugin 'record_modifier'. Run 'gem search -rd fluent-plugin' to find plugins" 2017-10-24 01:40:48 -0400 [warn]: process died within 1 second. exit. Version-Release number of selected component (if applicable): logging-kibana:v3.6.173.0.59-1 logging-elasticsearch:v3.6.173.0.59-1 logging-fluentd:v3.6.173.0.59-1 logging-auth-proxy:v3.6.173.0.59-1 logging-curator:v3.6.173.0.59-1 # rpm -qa | grep openshift-ansible openshift-ansible-filter-plugins-3.6.173.0.59-1.git.0.0e31372.el7.noarch openshift-ansible-docs-3.6.173.0.59-1.git.0.0e31372.el7.noarch openshift-ansible-lookup-plugins-3.6.173.0.59-1.git.0.0e31372.el7.noarch openshift-ansible-callback-plugins-3.6.173.0.59-1.git.0.0e31372.el7.noarch openshift-ansible-playbooks-3.6.173.0.59-1.git.0.0e31372.el7.noarch openshift-ansible-3.6.173.0.59-1.git.0.0e31372.el7.noarch openshift-ansible-roles-3.6.173.0.59-1.git.0.0e31372.el7.noarch How reproducible: Always Steps to Reproduce: 1. Deploy logging 3.6 and check pod status 2. 3. Actual results: fluentd pods failed to start up, error="Unknown filter plugin 'record_modifier' in fluentd pods log. Expected results: fluentd pods could be started up. Additional info:
Created attachment 1342526 [details] "gem search -rd fluent-plugin" result
blocks all the logging testing now
The pull request #722 [1] backported to 3.6 the instalation of the 'record-modifier' in Dockerfile.centos7 but not in Dockerfile. Other changes from this PR are shared for both images which leads to the fact that centos based image works, rhel based images doesn't. The record-modifier probably hasn't been productized yet. I was unable to find the RPM package or the dist-git repo. [2] Potential solution could be to install 'record-modifier' in the rhel based image similarly how it is installed in centos based image. [1] https://github.com/openshift/origin-aggregated-logging/pull/722 [2] http://pkgs.devel.redhat.com/cgit/?q=record-modifier
PR merged: https://github.com/openshift/origin-aggregated-logging/pull/742
koji_builds: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=615164 repositories: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:rhaos-3.6-rhel-7-docker-candidate-57336-20171027134826 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:latest brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6.173.0.61 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6.173.0.61-2 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6
Verified and pass on v3.6.173.0.62
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3389