Bug 1490660

Summary: logging-fluentd 3.6.173.0.32 in non-mux mode fails to start with ConfigParseError
Product: OpenShift Container Platform Reporter: Mike Fiedler <mifiedle>
Component: LoggingAssignee: Rich Megginson <rmeggins>
Status: CLOSED ERRATA QA Contact: Mike Fiedler <mifiedle>
Severity: low Docs Contact:
Priority: unspecified    
Version: 3.6.1CC: aos-bugs, rmeggins
Target Milestone: ---   
Target Release: 3.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
 Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-28 22:10:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mike Fiedler 2017-09-12 00:26:58 UTC 
Description of problem:

This might be getting too far from reality.

Running the logging build from https://bugzilla.redhat.com/show_bug.cgi?id=1490395  (a 3.6.173.0.32 re-build on brew)  and working around https://bugzilla.redhat.com/show_bug.cgi?id=1490647 by removing references to logging-mux secrets from the logging-fluentd daemonset.

logging-fluentd still fails to start.  It goes into CrashLoopBackoff with the error "/usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/literal_parser.rb:165:in `instance_eval': include error configs.d/openshift/output-pre-*.conf (Fluent::ConfigParseError) "  .  

The full stack trace is below.

Version-Release number of selected component (if applicable): v3.6.173.0.32 re-build on brew


How reproducible: Always in a non-mux configuration.


Steps to Reproduce:
1.  Install logging using the inventory from https://bugzilla.redhat.com/show_bug.cgi?id=1490647
2.  oc edit the logginf-fluentd daemonset and remove logging-mux secret and muxsecret volume references
3. re-tag the node with logging-infra-fluentd=false and then true

Actual results:

logging-fluentd pods fail to start with the error message below

Expected results:

logging-fluentd pods start ok

Additional info:

root@ip-172-31-24-59: ~ # oc logs logging-fluentd-lzjwt | less                                                                                                                                │····················
umount: /var/lib/docker/containers/*/shm: mountpoint not found                                                                                                                                │····················
umounts of dead containers will fail. Ignoring...                                                                                                                                             │····················
2017-09-12 00:17:42 +0000 [info]: reading config file path="/etc/fluent/fluent.conf"                                                                                                          │····················
/usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/literal_parser.rb:165:in `instance_eval': include error configs.d/openshift/output-pre-*.conf (Fluent::ConfigParseError)               │····················
        from (eval):1:in `open'                                                                                                                                                               │····················
        from (eval):1:in `eval_embedded_code'                                                                                                                                                 │····················
        from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/literal_parser.rb:165:in `instance_eval'                                                                                  │····················
        from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/literal_parser.rb:165:in `eval_embedded_code'                                                                             │····················
        from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/literal_parser.rb:92:in `scan_double_quoted_string'                                                                       │····················
        from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/literal_parser.rb:71:in `scan_string'                                                                                     │····················
        from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/literal_parser.rb:64:in `parse_literal'                                                                                   │····················
        from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/v1_parser.rb:130:in `parse_element'                                                                                       │····················
        from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/v1_parser.rb:95:in `parse_element'                                                                                        │····················
        from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/v1_parser.rb:167:in `block in eval_include'                                                                               │····················
        from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/v1_parser.rb:161:in `each'                                                                                                │····················
        from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/v1_parser.rb:161:in `eval_include'                                                                                        │····················
        from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/v1_parser.rb:145:in `parse_include'                                                                                       │····················
        from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/v1_parser.rb:113:in `parse_element'                                                                                       │····················
        from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/v1_parser.rb:95:in `parse_element'                                                                                        │····················
        from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/v1_parser.rb:43:in `parse!'                                                                                               │····················
        from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config/v1_parser.rb:33:in `parse'                                                                                                │····················
        from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/config.rb:30:in `parse'                                                                                                          │····················
        from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/supervisor.rb:480:in `read_config'                                                                                               │····················
        from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/supervisor.rb:144:in `start'                                                                                                     │····················
        from /usr/share/gems/gems/fluentd-0.12.39/lib/fluent/command/fluentd.rb:173:in `<top (required)>'                                                                                     │····················
        from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'                                                                                                          │····················
        from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:55:in `require'                                                                                                          │····················
        from /usr/share/gems/gems/fluentd-0.12.39/bin/fluentd:5:in `<top (required)>'                                                                                                         │····················
        from /usr/bin/fluentd:23:in `load'                                                                                                                                                    │····················
        from /usr/bin/fluentd:23:in `<main>'
Comment 1 Rich Megginson 2017-09-12 02:26:52 UTC 
I don't see why this should block.

At any rate, the problem is the

openshift_logging_mux_client_mode=maximal

from the installation, which sets the MUX_CLIENT_MODE env. var. in the logging-fluentd daemonset which you will need to remove.

Once you do that, it should not block testing.

I'm removing the blocker.
Comment 2 Rich Megginson 2017-09-12 02:27:43 UTC 
e.g.

oc set env ds/logging-fluentd MUX_CLIENT_MODE-
Comment 3 Mike Fiedler 2017-09-12 12:03:35 UTC 
Verified comment 2 - removing the env var works around the issue.  Lowering severity.

We probably don't need both this bz and https://bugzilla.redhat.com/show_bug.cgi?id=1490647.  Dup one of them if needed.
Comment 4 Rich Megginson 2017-10-09 21:28:40 UTC 
koji_builds:
  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=605535
repositories:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:rhaos-3.7-rhel-7-docker-candidate-91975-20171009193313
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:latest
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.7.0
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.7.0-0.146.0.1
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.7
Comment 5 Mike Fiedler 2017-10-31 15:08:03 UTC 
Verified on 3.7.0-0.188.0.   Changed Target Release to 3.7.0 since it is fixed there.
Comment 9 errata-xmlrpc 2017-11-28 22:10:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188