Bug 1505683 - fluentd pods failed to start up,"Unknown filter plugin 'record_modifier' in fluentd pods log
Summary: fluentd pods failed to start up,"Unknown filter plugin 'record_modifier' in f...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.6.1
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 3.6.z
Assignee: Jan Wozniak
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks: 1490719 1506099
TreeView+ depends on / blocked
 
Reported: 2017-10-24 06:43 UTC by Junqi Zhao
Modified: 2017-12-07 07:13 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
: 1506099 (view as bug list)
Environment:
Last Closed: 2017-12-07 07:13:19 UTC
Target Upstream Version:


Attachments (Terms of Use)
logging environment dump (43.21 KB, application/x-gzip)
2017-10-24 06:43 UTC, Junqi Zhao
no flags Details
"gem search -rd fluent-plugin" result (146.02 KB, text/plain)
2017-10-24 06:45 UTC, Junqi Zhao
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3389 0 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Enterprise security, bug fix, and enhancement update 2017-12-07 12:09:10 UTC

Description Junqi Zhao 2017-10-24 06:43:00 UTC
Created attachment 1342522 [details]
logging environment dump

Description of problem:
Deployed logging 3.6, fluentd pods failed to start up, error="Unknown filter plugin 'record_modifier' in fluentd pods log.
used 'gem search -rd fluent-plugin' to check, found fluent-plugin-record-modifier (0.6.0), not ***record_modifier***, see the attached file.
bash-4.2# gem search -rd fluent-plugin
# oc get po
NAME                                          READY     STATUS             RESTARTS   AGE
logging-curator-1-jhjcm                       1/1       Running            0          1h
logging-curator-ops-1-glg8f                   1/1       Running            0          1h
logging-es-data-master-myws4feq-1-zwrp5       1/1       Running            0          1h
logging-es-ops-data-master-6zs09u6z-1-lsb8h   1/1       Running            0          1h
logging-fluentd-9cmn0                         0/1       CrashLoopBackOff   17         1h
logging-fluentd-l9vhg                         0/1       CrashLoopBackOff   17         1h
logging-kibana-1-bqs5k                        2/2       Running            0          1h
logging-kibana-ops-1-c9s1t                    2/2       Running            0          1h

# oc logs logging-fluentd-9cmn0
umounts of dead containers will fail. Ignoring...
umount: /var/lib/docker/containers/1df690adcaaa59489757a69afbf20d344e341a76cddd57284103b5f801381e18/shm: not mounted
2017-10-24 01:40:48 -0400 [info]: reading config file path="/etc/fluent/fluent.conf"
2017-10-24 01:40:48 -0400 [error]: config error file="/etc/fluent/fluent.conf" error="Unknown filter plugin 'record_modifier'. Run 'gem search -rd fluent-plugin' to find plugins"
2017-10-24 01:40:48 -0400 [warn]: process died within 1 second. exit.


Version-Release number of selected component (if applicable):
logging-kibana:v3.6.173.0.59-1
logging-elasticsearch:v3.6.173.0.59-1
logging-fluentd:v3.6.173.0.59-1
logging-auth-proxy:v3.6.173.0.59-1
logging-curator:v3.6.173.0.59-1

# rpm -qa | grep openshift-ansible
openshift-ansible-filter-plugins-3.6.173.0.59-1.git.0.0e31372.el7.noarch
openshift-ansible-docs-3.6.173.0.59-1.git.0.0e31372.el7.noarch
openshift-ansible-lookup-plugins-3.6.173.0.59-1.git.0.0e31372.el7.noarch
openshift-ansible-callback-plugins-3.6.173.0.59-1.git.0.0e31372.el7.noarch
openshift-ansible-playbooks-3.6.173.0.59-1.git.0.0e31372.el7.noarch
openshift-ansible-3.6.173.0.59-1.git.0.0e31372.el7.noarch
openshift-ansible-roles-3.6.173.0.59-1.git.0.0e31372.el7.noarch


How reproducible:
Always

Steps to Reproduce:
1. Deploy logging 3.6 and check pod status
2.
3.

Actual results:
fluentd pods failed to start up, error="Unknown filter plugin 'record_modifier' in fluentd pods log.

Expected results:
fluentd pods could be started up.

Additional info:

Comment 1 Junqi Zhao 2017-10-24 06:45:46 UTC
Created attachment 1342526 [details]
"gem search -rd fluent-plugin" result

Comment 2 Junqi Zhao 2017-10-24 07:27:46 UTC
blocks all the logging testing now

Comment 3 Jan Wozniak 2017-10-24 12:21:39 UTC
The pull request #722 [1] backported to 3.6 the instalation of the 'record-modifier' in Dockerfile.centos7 but not in Dockerfile. Other changes from this PR are shared for both images which leads to the fact that centos based image works, rhel based images doesn't.

The record-modifier probably hasn't been productized yet. I was unable to find the RPM package or the dist-git repo. [2]

Potential solution could be to install 'record-modifier' in the rhel based image similarly how it is installed in centos based image.



[1] https://github.com/openshift/origin-aggregated-logging/pull/722

[2] http://pkgs.devel.redhat.com/cgit/?q=record-modifier

Comment 4 Jan Wozniak 2017-10-25 11:04:16 UTC
PR merged: https://github.com/openshift/origin-aggregated-logging/pull/742

Comment 5 Rich Megginson 2017-10-27 14:18:10 UTC
koji_builds:
  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=615164
repositories:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:rhaos-3.6-rhel-7-docker-candidate-57336-20171027134826
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:latest
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6.173.0.61
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6.173.0.61-2
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6

Comment 6 Anping Li 2017-10-30 05:32:00 UTC
Verified and pass on v3.6.173.0.62

Comment 9 errata-xmlrpc 2017-12-07 07:13:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3389


Note You need to log in before you can comment on or make changes to this bug.