Description of problem: The logs couldn't be forward to kakfa when secret are enabled. $ oc logs fluentd-l774m Setting each total_size_limit for 1 buffers to 19242572390 bytes Setting queued_chunks_limit_size for each buffer to 2293 Setting chunk_limit_size for each buffer to 8388608 /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/config/literal_parser.rb:171:in `instance_eval': undefined local variable or method `use_nil' for #<Binding:0x00007f1e2157ef68> (NameError) from /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/config/literal_parser.rb:171:in `instance_eval' from /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/config/literal_parser.rb:171:in `eval_embedded_code' from /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/config/literal_parser.rb:93:in `scan_double_quoted_string' from /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/config/literal_parser.rb:72:in `scan_string' from /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/config/literal_parser.rb:65:in `parse_literal' from /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/config/v1_parser.rb:130:in `parse_element' from /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/config/v1_parser.rb:95:in `parse_element' from /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/config/v1_parser.rb:95:in `parse_element' from /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/config/v1_parser.rb:43:in `parse!' from /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/config/v1_parser.rb:33:in `parse' from /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/config.rb:39:in `parse' from /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/supervisor.rb:783:in `read_config' from /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/supervisor.rb:539:in `run_worker' from /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/command/fluentd.rb:330:in `<top (required)>' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require' from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require' from /usr/local/share/gems/gems/fluentd-1.7.4/bin/fluentd:8:in `<top (required)>' from /usr/local/bin/fluentd:23:in `load' from /usr/local/bin/fluentd:23:in `<main>' Version-Release number of selected component (if applicable): clusterlogging.4.6.0-202009240527.p0 ose-logging-fluentd:v4.6.0-202009231847.p0 # gem list |grep fluent fluent-config-regexp-type (1.0.0) fluent-mixin-config-placeholders (0.4.0) fluent-plugin-concat (2.4.0) fluent-plugin-elasticsearch (4.1.1) fluent-plugin-kafka (0.13.1) fluent-plugin-kubernetes_metadata_filter (2.5.2) fluent-plugin-multi-format-parser (1.0.0) fluent-plugin-prometheus (1.7.3) fluent-plugin-record-modifier (2.1.0) fluent-plugin-remote-syslog (1.1) fluent-plugin-remote_syslog (1.0.0) fluent-plugin-rewrite-tag-filter (2.3.0) fluent-plugin-splunk-hec (1.1.2) fluent-plugin-systemd (1.0.2) fluentd (1.7.4) How reproducible: always Steps to Reproduce: 1. Forward to kafka using secret: apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: instance spec: outputs: - name: kafka secret: name: kafka-fluent type: kafka url: tls://my-cluster-kafka-bootstrap.amq.svc.cluster.local:9093/topic-logging-app pipelines: - inputRefs: - application name: test-app outputRefs: - kafka 2. Check the fluentd logs Actual results: No log can be sent to kafka topic. Expected results: Additional info:
https://docs.fluentd.org/configuration/config-file Since v1.8.0, helper methods use_nil and use_default are available. But we are using fluentd (1.7.4)
Can you please set CLO to unmanaged and then edit the fluent configuration to replace 'use_nil' with 'nil' unquoted to see if it resolves the issue. I would like to add a proper test but need to understand how to setup ssl first
@jeff, the nil doesn't work. It popup the error below. $ oc logs fluentd-2zvll Setting each total_size_limit for 1 buffers to 19242572390 bytes Setting queued_chunks_limit_size for each buffer to 2293 Setting chunk_limit_size for each buffer to 8388608 2020-09-29 14:44:52 +0000 [error]: unexpected error error_class=Errno::ENOENT error="No such file or directory @ rb_sysopen - " 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluent-plugin-kafka-0.13.1/lib/fluent/plugin/kafka_plugin_util.rb:41:in `read' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluent-plugin-kafka-0.13.1/lib/fluent/plugin/kafka_plugin_util.rb:41:in `read_ssl_file' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluent-plugin-kafka-0.13.1/lib/fluent/plugin/out_kafka2.rb:109:in `refresh_client' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluent-plugin-kafka-0.13.1/lib/fluent/plugin/out_kafka2.rb:182:in `start' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/root_agent.rb:203:in `block in start' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/root_agent.rb:182:in `block (2 levels) in lifecycle' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/agent.rb:119:in `block (2 levels) in lifecycle' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/agent.rb:118:in `each' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/agent.rb:118:in `block in lifecycle' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/agent.rb:111:in `each' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/agent.rb:111:in `lifecycle' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/root_agent.rb:181:in `block in lifecycle' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/root_agent.rb:178:in `each' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/root_agent.rb:178:in `lifecycle' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/root_agent.rb:202:in `start' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/engine.rb:274:in `start' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/engine.rb:219:in `run' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/supervisor.rb:816:in `run_engine' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/supervisor.rb:559:in `block in run_worker' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/supervisor.rb:741:in `main_process' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/supervisor.rb:554:in `run_worker' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/command/fluentd.rb:330:in `<top (required)>' 2020-09-29 14:44:52 +0000 [error]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require' 2020-09-29 14:44:52 +0000 [error]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require' 2020-09-29 14:44:52 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/bin/fluentd:8:in `<top (required)>' 2020-09-29 14:44:52 +0000 [error]: /usr/local/bin/fluentd:23:in `load' 2020-09-29 14:44:52 +0000 [error]: /usr/local/bin/fluentd:23:in `<main>' 2020-09-29 14:44:52 +0000 [error]: unexpected error error_class=Errno::ENOENT error="No such file or directory @ rb_sysopen - " 2020-09-29 14:44:52 +0000 [error]: suppressed same stacktrace
The fluentd works well if all keys(ca-bundle.crt, tls.crt and tls.key) are provided in the kafka secret.
Verified on clusterserviceversion.operators.coreos.com/clusterlogging.4.6.0-202010081538.p0
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6.1 extras update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4198