Description of problem: The logs collected by rsyslog are not parsed, $ oc exec elasticsearch-cdm-yb8rtczn-2-56f7b6d468-vtzgc -- indices Defaulting container name to elasticsearch. Use 'oc describe pod/elasticsearch-cdm-yb8rtczn-2-56f7b6d468-vtzgc -n openshift-logging' to see all of the containers in this pod. Mon Aug 5 02:20:20 UTC 2019 health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open .kibana QzMCdxfvRwWZO6w8bXc_8Q 1 1 5 0 0 0 green open project.json1.d3c51092-b723-11e9-8cf2-026c8b4a782a.2019.08.05 f68hN4iWQOuDWl4UIQznww 2 1 246 0 0 0 green open .kibana.647a750f1787408bf50088234ec0edd5a6a9b2ac fI6GZ4pDQbm6QIofJ5Va8w 1 1 4 1 0 0 green open .kibana.af6615b8667e56f05cfd9d4bd371847d9ece3226 4eenvyLiRLSoyFcprlcm2Q 1 1 5 1 0 0 green open project.json2.e168f72b-b723-11e9-b221-063bf3b869e2.2019.08.05 _1hoXfTqTXGMPufSRpXZQw 2 1 123 0 0 0 green open .operations.2019.08.05 RYgdkdG3SlWusm2xdyinTg 2 1 27299 0 38 19 green open .searchguard XQhazZvZQF-zRdT8ZiFHug 1 1 5 0 0 0 logs collected by rsyslog: $ oc exec elasticsearch-cdm-yb8rtczn-2-56f7b6d468-vtzgc -- es_util --query=.operations*/_search?pretty -d ' { "size": 2, "sort": [ {"@timestamp": {"order":"desc"}} ] }' Defaulting container name to elasticsearch. Use 'oc describe pod/elasticsearch-cdm-yb8rtczn-2-56f7b6d468-vtzgc -n openshift-logging' to see all of the containers in this pod. { "took" : 1, "timed_out" : false, "_shards" : { "total" : 2, "successful" : 2, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 30823, "max_score" : null, "hits" : [ { "_index" : ".operations.2019.08.05", "_type" : "com.redhat.viaq.common", "_id" : "B04D2A28BE804A64B0407CB9E503D30F", "_score" : null, "_source" : { "originalmsg" : "I0805 02:16:18.460227 1169 prober.go:125] Readiness probe for \"sdn-2bk6v_openshift-sdn(febe0495-b71a-11e9-a0f4-063bf3b869e2):sdn\" succeeded", "unparsed-data" : "I0805 02:16:18.460227 1169 prober.go:125] Readiness probe for \"sdn-2bk6v_openshift-sdn(febe0495-b71a-11e9-a0f4-063bf3b869e2):sdn\" succeeded" }, "sort" : [ -9223372036854775808 ] }, { "_index" : ".operations.2019.08.05", "_type" : "com.redhat.viaq.common", "_id" : "83C5A7C80C3D41A99AE40B008C900B8B", "_score" : null, "_source" : { "originalmsg" : "2019-08-05T02:16:18.938647884+00:00 stderr F I0805 02:16:18.938618 1 controller.go:280] event from workqueue successfully processed", "unparsed-data" : "2019-08-05T02:16:18.938647884+00:00 stderr F I0805 02:16:18.938618 1 controller.go:280] event from workqueue successfully processed" }, "sort" : [ -9223372036854775808 ] } ] } } Version-Release number of selected component (if applicable): ose-cluster-logging-operator-v4.2.0-201908041300 ose-logging-rsyslog-v4.2.0-201908041300 How reproducible: Always Steps to Reproduce: 1.Deploy logging using rsyslog as log collector 2.check logs in ES 3. Actual results: Expected results: Additional info: Fluentd doesn't have this issue.
@Qiaoling, it could be introduced by the recent changes for CDM_*, SKIP_EMPTY, USE_MMEXTERNAL and MERGE_JSON_LOG settings... :( Could you please share the value of all the environment variables set in the rsyslog pod?
I didn't set the environment variables, I was using the default values.
Verified with ose-cluster-logging-operator-v4.2.0-201908061819
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922