Bug 1737286 - The logs collected by rsyslog are not parsed
Summary: The logs collected by rsyslog are not parsed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 4.2.0
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-05 02:22 UTC by Qiaoling Tang
Modified: 2019-10-16 06:34 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-16 06:34:43 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-logging-operator pull 229 0 'None' closed Bug 1737286: The logs collected by rsyslog are not parsed 2020-07-21 09:34:29 UTC
Red Hat Product Errata RHBA-2019:2922 0 None None None 2019-10-16 06:34:53 UTC

Description Qiaoling Tang 2019-08-05 02:22:02 UTC
Description of problem:
The logs collected by rsyslog are not parsed, 

$ oc exec elasticsearch-cdm-yb8rtczn-2-56f7b6d468-vtzgc -- indices
Defaulting container name to elasticsearch.
Use 'oc describe pod/elasticsearch-cdm-yb8rtczn-2-56f7b6d468-vtzgc -n openshift-logging' to see all of the containers in this pod.
Mon Aug  5 02:20:20 UTC 2019
health status index                                                         uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana                                                       QzMCdxfvRwWZO6w8bXc_8Q   1   1          5            0          0              0
green  open   project.json1.d3c51092-b723-11e9-8cf2-026c8b4a782a.2019.08.05 f68hN4iWQOuDWl4UIQznww   2   1        246            0          0              0
green  open   .kibana.647a750f1787408bf50088234ec0edd5a6a9b2ac              fI6GZ4pDQbm6QIofJ5Va8w   1   1          4            1          0              0
green  open   .kibana.af6615b8667e56f05cfd9d4bd371847d9ece3226              4eenvyLiRLSoyFcprlcm2Q   1   1          5            1          0              0
green  open   project.json2.e168f72b-b723-11e9-b221-063bf3b869e2.2019.08.05 _1hoXfTqTXGMPufSRpXZQw   2   1        123            0          0              0
green  open   .operations.2019.08.05                                        RYgdkdG3SlWusm2xdyinTg   2   1      27299            0         38             19
green  open   .searchguard                                                  XQhazZvZQF-zRdT8ZiFHug   1   1          5            0          0              0


logs collected by rsyslog:
$ oc exec elasticsearch-cdm-yb8rtczn-2-56f7b6d468-vtzgc -- es_util --query=.operations*/_search?pretty -d '
{
   "size": 2,
   "sort": [
      {"@timestamp": {"order":"desc"}}
   ]
}'
Defaulting container name to elasticsearch.
Use 'oc describe pod/elasticsearch-cdm-yb8rtczn-2-56f7b6d468-vtzgc -n openshift-logging' to see all of the containers in this pod.
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 30823,
    "max_score" : null,
    "hits" : [
      {
        "_index" : ".operations.2019.08.05",
        "_type" : "com.redhat.viaq.common",
        "_id" : "B04D2A28BE804A64B0407CB9E503D30F",
        "_score" : null,
        "_source" : {
          "originalmsg" : "I0805 02:16:18.460227    1169 prober.go:125] Readiness probe for \"sdn-2bk6v_openshift-sdn(febe0495-b71a-11e9-a0f4-063bf3b869e2):sdn\" succeeded",
          "unparsed-data" : "I0805 02:16:18.460227    1169 prober.go:125] Readiness probe for \"sdn-2bk6v_openshift-sdn(febe0495-b71a-11e9-a0f4-063bf3b869e2):sdn\" succeeded"
        },
        "sort" : [
          -9223372036854775808
        ]
      },
      {
        "_index" : ".operations.2019.08.05",
        "_type" : "com.redhat.viaq.common",
        "_id" : "83C5A7C80C3D41A99AE40B008C900B8B",
        "_score" : null,
        "_source" : {
          "originalmsg" : "2019-08-05T02:16:18.938647884+00:00 stderr F I0805 02:16:18.938618       1 controller.go:280] event from workqueue successfully processed",
          "unparsed-data" : "2019-08-05T02:16:18.938647884+00:00 stderr F I0805 02:16:18.938618       1 controller.go:280] event from workqueue successfully processed"
        },
        "sort" : [
          -9223372036854775808
        ]
      }
    ]
  }
}

Version-Release number of selected component (if applicable):
ose-cluster-logging-operator-v4.2.0-201908041300
ose-logging-rsyslog-v4.2.0-201908041300


How reproducible:
Always

Steps to Reproduce:
1.Deploy logging using rsyslog as log collector
2.check logs in ES
3.

Actual results:


Expected results:


Additional info:
Fluentd doesn't have this issue.

Comment 1 Noriko Hosoi 2019-08-05 17:37:42 UTC
@Qiaoling, it could be introduced by the recent changes for CDM_*, SKIP_EMPTY, USE_MMEXTERNAL and MERGE_JSON_LOG settings... :(

Could you please share the value of all the environment variables set in the rsyslog pod?

Comment 2 Qiaoling Tang 2019-08-06 00:14:40 UTC
I didn't set the environment variables, I was using the default values.

Comment 5 Qiaoling Tang 2019-08-07 01:18:58 UTC
Verified with ose-cluster-logging-operator-v4.2.0-201908061819

Comment 6 errata-xmlrpc 2019-10-16 06:34:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922


Note You need to log in before you can comment on or make changes to this bug.