Red Hat Bugzilla – Bug 1490660
logging-fluentd 3.6.173.0.32 in non-mux mode fails to start with ConfigParseError
Last modified: 2017-11-28 17:10:32 EST
Description of problem: This might be getting too far from reality. Running the logging build from https://bugzilla.redhat.com/show_bug.cgi?id=1490395 (a 3.6.173.0.32 re-build on brew) and working around https://bugzilla.redhat.com/show_bug.cgi?id=1490647 by removing references to logging-mux secrets from the logging-fluentd daemonset. logging-fluentd still fails to start. It goes into CrashLoopBackoff with the error "/usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/literal_parser.rb:165:in `instance_eval': include error configs.d/openshift/output-pre-*.conf (Fluent::ConfigParseError) " . The full stack trace is below. Version-Release number of selected component (if applicable): v3.6.173.0.32 re-build on brew How reproducible: Always in a non-mux configuration. Steps to Reproduce: 1. Install logging using the inventory from https://bugzilla.redhat.com/show_bug.cgi?id=1490647 2. oc edit the logginf-fluentd daemonset and remove logging-mux secret and muxsecret volume references 3. re-tag the node with logging-infra-fluentd=false and then true Actual results: logging-fluentd pods fail to start with the error message below Expected results: logging-fluentd pods start ok Additional info: root@ip-172-31-24-59: ~ # oc logs logging-fluentd-lzjwt | less │···················· umount: /var/lib/docker/containers/*/shm: mountpoint not found │···················· umounts of dead containers will fail. Ignoring... │···················· 2017-09-12 00:17:42 +0000 [info]: reading config file path="/etc/fluent/fluent.conf" │···················· /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/literal_parser.rb:165:in `instance_eval': include error configs.d/openshift/output-pre-*.conf (Fluent::ConfigParseError) │···················· from (eval):1:in `open' │···················· from (eval):1:in `eval_embedded_code' │···················· from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/literal_parser.rb:165:in `instance_eval' │···················· from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/literal_parser.rb:165:in `eval_embedded_code' │···················· from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/literal_parser.rb:92:in `scan_double_quoted_string' │···················· from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/literal_parser.rb:71:in `scan_string' │···················· from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/literal_parser.rb:64:in `parse_literal' │···················· from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/v1_parser.rb:130:in `parse_element' │···················· from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/v1_parser.rb:95:in `parse_element' │···················· from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/v1_parser.rb:167:in `block in eval_include' │···················· from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/v1_parser.rb:161:in `each' │···················· from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/v1_parser.rb:161:in `eval_include' │···················· from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/v1_parser.rb:145:in `parse_include' │···················· from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/v1_parser.rb:113:in `parse_element' │···················· from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/v1_parser.rb:95:in `parse_element' │···················· from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/v1_parser.rb:43:in `parse!' │···················· from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/v1_parser.rb:33:in `parse' │···················· from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config.rb:30:in `parse' │···················· from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/supervisor.rb:480:in `read_config' │···················· from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/supervisor.rb:144:in `start' │···················· from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/command/fluentd.rb:173:in `<top (required)>' │···················· from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' │···················· from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require' │···················· from /usr/share/gems/gems/fluentd-0.12.39/bin/fluentd:5:in `<top (required)>' │···················· from /usr/bin/fluentd:23:in `load' │···················· from /usr/bin/fluentd:23:in `<main>'
I don't see why this should block. At any rate, the problem is the openshift_logging_mux_client_mode=maximal from the installation, which sets the MUX_CLIENT_MODE env. var. in the logging-fluentd daemonset which you will need to remove. Once you do that, it should not block testing. I'm removing the blocker.
e.g. oc set env ds/logging-fluentd MUX_CLIENT_MODE-
Verified comment 2 - removing the env var works around the issue. Lowering severity. We probably don't need both this bz and https://bugzilla.redhat.com/show_bug.cgi?id=1490647. Dup one of them if needed.
koji_builds: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=605535 repositories: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:rhaos-3.7-rhel-7-docker-candidate-91975-20171009193313 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:latest brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.7.0 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.7.0-0.146.0.1 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.7
Verified on 3.7.0-0.188.0. Changed Target Release to 3.7.0 since it is fixed there.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3188