Bug 1867053

Created attachment 1710769 [details]
fluent.conf extracted from cm/fluentd

Description of problem:
After the legacy forward to syslog is enabled, the logs are not sent to the default ES, only sent to the syslog server, which is different from the behavior in 4.5.

Fluentd logs:
$ oc logs fluentd-clvhx
Setting each total_size_limit for 2 buffers to 9621286195 bytes
Setting queued_chunks_limit_size for each buffer to 1146
Setting chunk_limit_size for each buffer to 8388608
2020-08-07 08:58:54 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="'Metadata' is not a designated severity"
  2020-08-07 08:58:54 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/syslog_protocol-0.9.2/lib/syslog_protocol/packet.rb:100:in `severity='
  2020-08-07 08:58:54 +0000 [warn]: /etc/fluent/plugin/out_syslog_buffered.rb:104:in `send_to_syslog'
  2020-08-07 08:58:54 +0000 [warn]: /etc/fluent/plugin/out_syslog_buffered.rb:90:in `block in write'
  2020-08-07 08:58:54 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/event.rb:327:in `each'
  2020-08-07 08:58:54 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/event.rb:327:in `block in each'
  2020-08-07 08:58:54 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/buffer/memory_chunk.rb:81:in `open'
  2020-08-07 08:58:54 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/buffer/memory_chunk.rb:81:in `open'
  2020-08-07 08:58:54 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/event.rb:326:in `each'
  2020-08-07 08:58:54 +0000 [warn]: /etc/fluent/plugin/out_syslog_buffered.rb:89:in `write'
  2020-08-07 08:58:54 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/compat/output.rb:131:in `write'
  2020-08-07 08:58:54 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:1125:in `try_flush'
  2020-08-07 08:58:54 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:1431:in `flush_thread_run'
  2020-08-07 08:58:54 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:461:in `block (2 levels) in start'
  2020-08-07 08:58:54 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-08-07 08:58:54 +0000 [warn]: bad chunk is moved to /tmp/fluent/backup/worker0/object_3f95cbb6fcec/5ac45d06cae1a66575dd699b7c59f008.log
2020-08-07 08:58:57 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="'Metadata' is not a designated severity"
  2020-08-07 08:58:57 +0000 [warn]: suppressed same stacktrace
2020-08-07 08:58:57 +0000 [warn]: bad chunk is moved to /tmp/fluent/backup/worker0/object_3f95cbb6fcec/5ac45d088b5d508f42807e9b73c103f2.log
2020-08-07 08:58:58 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="'Metadata' is not a designated severity"
  2020-08-07 08:58:58 +0000 [warn]: suppressed same stacktrace
2020-08-07 08:58:58 +0000 [warn]: bad chunk is moved to /tmp/fluent/backup/worker0/object_3f95cbb6fcec/5ac45d0a9b5e2ac03eb4386b48494cb8.log
2020-08-07 08:59:00 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="'Metadata' is not a designated severity"
  2020-08-07 08:59:00 +0000 [warn]: suppressed same stacktrace
2020-08-07 08:59:00 +0000 [warn]: bad chunk is moved to /tmp/fluent/backup/worker0/object_3f95cbb6fcec/5ac45d0c6876b7a5ed6c575704ea54af.log
2020-08-07 08:59:01 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="'Metadata' is not a designated severity"
  2020-08-07 08:59:01 +0000 [warn]: suppressed same stacktrace
2020-08-07 08:59:01 +0000 [warn]: bad chunk is moved to /tmp/fluent/backup/worker0/object_3f95cbb6fcec/5ac45d0dc05d6b0dbf1783e7ce9436c8.log
2020-08-07 08:59:54 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="'Metadata' is not a designated severity"
  2020-08-07 08:59:54 +0000 [warn]: suppressed same stacktrace
2020-08-07 08:59:54 +0000 [warn]: bad chunk is moved to /tmp/fluent/backup/worker0/object_3f95cbb6fcec/5ac45d0ef7bd4d525caa04098e4a7729.log
2020-08-07 09:00:58 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="'Metadata' is not a designated severity"
  2020-08-07 09:00:58 +0000 [warn]: suppressed same stacktrace
2020-08-07 09:00:58 +0000 [warn]: bad chunk is moved to /tmp/fluent/backup/worker0/object_3f95cbb6fcec/5ac45d4149f1a447c2b60ec44558c2ee.log
2020-08-07 09:01:58 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="'Metadata' is not a designated severity"
  2020-08-07 09:01:58 +0000 [warn]: suppressed same stacktrace
2020-08-07 09:01:58 +0000 [warn]: bad chunk is moved to /tmp/fluent/backup/worker0/object_3f95cbb6fcec/5ac45d7e52d40021bb4da7e4203d65ce.log
2020-08-07 09:02:53 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="'Metadata' is not a designated severity"
  2020-08-07 09:02:53 +0000 [warn]: suppressed same stacktrace
2020-08-07 09:02:53 +0000 [warn]: bad chunk is moved to /tmp/fluent/backup/worker0/object_3f95cbb6fcec/5ac45db78b62d8d5b0ff37f0931dbba0.log
2020-08-07 09:03:58 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="'Metadata' is not a designated severity"
  2020-08-07 09:03:58 +0000 [warn]: suppressed same stacktrace
2020-08-07 09:03:58 +0000 [warn]: bad chunk is moved to /tmp/fluent/backup/worker0/object_3f95cbb6fcec/5ac45dec3d1ab16f087256730858006d.log
2020-08-07 09:04:58 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="'Metadata' is not a designated severity"
  2020-08-07 09:04:58 +0000 [warn]: suppressed same stacktrace
2020-08-07 09:04:58 +0000 [warn]: bad chunk is moved to /tmp/fluent/backup/worker0/object_3f95cbb6fcec/5ac45e29f5ae32e33f932190ce048dfa.log
2020-08-07 09:05:56 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="'Metadata' is not a designated severity"
  2020-08-07 09:05:56 +0000 [warn]: suppressed same stacktrace
2020-08-07 09:05:56 +0000 [warn]: bad chunk is moved to /tmp/fluent/backup/worker0/object_3f95cbb6fcec/5ac45e6335f8bc9c83b54a5f240d0266.log
2020-08-07 09:06:58 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="'Metadata' is not a designated severity"
  2020-08-07 09:06:58 +0000 [warn]: suppressed same stacktrace
2020-08-07 09:06:58 +0000 [warn]: bad chunk is moved to /tmp/fluent/backup/worker0/object_3f95cbb6fcec/5ac45e9ab3dc5a98842adba4d26f083a.log


$ oc get cm syslog -oyaml
apiVersion: v1
data:
  syslog.conf: |
    <store>
         @type syslog_buffered
         remote_syslog rsyslogserver.openshift-logging.svc.cluster.local
         port 514
         hostname $hostname
         remove_tag_prefix tag
         tag_key ident,systemd.u.SYSLOG_IDENTIFIER
         facility local0
         severity info
         use_record true
         payload_key message
    </store>
kind: ConfigMap
metadata:


Version-Release number of selected component (if applicable):
clusterlogging.4.6.0-202008061759.p0

How reproducible:
Always

Steps to Reproduce:
1. deploy syslog server
2. deploy logging operators
3. create a configmap named syslog to enable the legacy forward
4. create clusterlogging
5. check data on syslog server and default ES

Actual results:


Expected results:
Logs could be sent to syslog server and default ES.

Additional info: