Bug 1632892 - [3.10] logging-eventrouter event not formatted correctly in Elasticsearch when using MUX
Summary: [3.10] logging-eventrouter event not formatted correctly in Elasticsearch whe...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.10.z
Assignee: Noriko Hosoi
QA Contact: Qiaoling Tang
URL:
Whiteboard:
Depends On: 1554293 1632895
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-09-25 18:55 UTC by Noriko Hosoi
Modified: 2018-12-13 17:09 UTC (History)
6 users (show)

Fixed In Version: openshift3/logging-fluentd:v3.10.57-1
Doc Type: Bug Fix
Doc Text:
Cause: When fluentd is configured as the combination of collectors and MUX, event logs from the event were supposed to be processed by MUX not by the collector for the both MUX_CLIENT_MODE maximal and minimal. It is because if an event log is formatted in the collector (note: the event record is put under the kubernetes key), the log is forwarded to MUX and passed to the k8s-meta plugin there and the existing kubernetes record is overwritten. It wiped out the event info from the log. Fix 1: To avoid the replacement, if the log is from event router, the tag is rewritten to ${tag}.raw in input-post-forward-mux.conf, which makes the log treated in the MUX_CLIENT_MODE=minimal way. Fix 2: There was another bug in ansible. That is, the environment variable TRANSFORM_EVENTS was not set in MUX even if openshift_logging_install_eventrouter is set to true. Result: With these 2 fixes, the event logs are correctly logged when MUX is configured with MUX_CLIENT_MODE=maximal as well as minimal.
Clone Of: 1554293
Environment:
Last Closed: 2018-12-13 17:09:08 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:3750 0 None None None 2018-12-13 17:09:14 UTC

Comment 1 Noriko Hosoi 2018-10-08 21:18:47 UTC 
For release-3.10:
https://github.com/openshift/origin-aggregated-logging/pull/1375
https://github.com/openshift/openshift-ansible/pull/10243
Comment 5 Qiaoling Tang 2018-11-29 08:02:08 UTC 
Verified in logging-fluentd-v3.10.79-1.

The logs collected when mux enabled are like this:
 {
      "_index" : ".operations.2018.11.29",
      "_type" : "com.redhat.viaq.common",
      "_id" : "b40c2230-f3ab-11e8-a8b6-0e0e8eca8496",
      "_score" : 9.291312,
      "_source" : {
        "message" : "Killing container with id docker://centos-logtest:Need to kill Pod",
        "level" : "info",
        "hostname" : "ip-172-18-1-48.ec2.internal",
        "pipeline_metadata" : {
          "collector" : {
            "ipaddr4" : "10.129.0.10",
            "ipaddr6" : "fe80::dc25:26ff:fe12:f60e",
            "inputname" : "fluent-plugin-systemd",
            "name" : "fluentd",
            "received_at" : "2018-11-29T07:52:22.348552+00:00",
            "version" : "0.12.43 1.6.0"
          },
          "normalizer" : {
            "original_raw_message" : "{\"verb\":\"ADDED\",\"event\":{\"metadata\":{\"name\":\"centos-logtest-d2m2v.156b8902b6173ec5\",\"namespace\":\"test2\",\"selfLink\":\"/api/v1/namespaces/test2/events/centos-logtest-d2m2v.156b8902b6173ec5\",\"uid\":\"b40c2230-f3ab-11e8-a8b6-0e0e8eca8496\",\"resourceVersion\":\"35861\",\"creationTimestamp\":\"2018-11-29T07:52:22Z\"},\"involvedObject\":{\"kind\":\"Pod\",\"namespace\":\"test2\",\"name\":\"centos-logtest-d2m2v\",\"uid\":\"4ded2475-f3ab-11e8-a8b6-0e0e8eca8496\",\"apiVersion\":\"v1\",\"resourceVersion\":\"35460\",\"fieldPath\":\"spec.containers{centos-logtest}\"},\"reason\":\"Killing\",\"message\":\"Killing container with id docker://centos-logtest:Need to kill Pod\",\"source\":{\"component\":\"kubelet\",\"host\":\"ip-172-18-4-161.ec2.internal\"},\"firstTimestamp\":\"2018-11-29T07:52:22Z\",\"lastTimestamp\":\"2018-11-29T07:52:22Z\",\"count\":1,\"type\":\"Normal\"}}\n",
            "ipaddr4" : "10.130.0.17",
            "ipaddr6" : "fe80::6417:59ff:fea9:cf20",
            "inputname" : "fluent-plugin-systemd",
            "name" : "fluentd",
            "received_at" : "2018-11-29T07:52:26.385415+00:00",
            "version" : "0.12.43 1.6.0"
          }
        },
        "@timestamp" : "2018-11-29T07:52:22Z",
        "viaq_msg_id" : "b40c2230-f3ab-11e8-a8b6-0e0e8eca8496",
        "docker" : {
          "container_id" : "f2a2c1a45921f25686d7116e4883d2316f99eee5676ce14bbdd2a0b9817bb347"
        },
        "kubernetes" : {
          "container_name" : "kube-eventrouter",
          "namespace_name" : "default",
          "pod_name" : "logging-eventrouter-1-qffjg",
          "pod_id" : "ecfd84bf-f3aa-11e8-a8b6-0e0e8eca8496",
          "labels" : {
            "component" : "eventrouter",
            "deployment" : "logging-eventrouter-1",
            "deploymentconfig" : "logging-eventrouter",
            "logging-infra" : "eventrouter",
            "provider" : "openshift"
          },
          "host" : "ip-172-18-1-48.ec2.internal",
          "master_url" : "https://kubernetes.default.svc.cluster.local",
          "namespace_id" : "bba8b899-f389-11e8-a8b6-0e0e8eca8496",
          "event" : {
            "metadata" : {
              "name" : "centos-logtest-d2m2v.156b8902b6173ec5",
              "namespace" : "test2",
              "selfLink" : "/api/v1/namespaces/test2/events/centos-logtest-d2m2v.156b8902b6173ec5",
              "uid" : "b40c2230-f3ab-11e8-a8b6-0e0e8eca8496",
              "resourceVersion" : "35861"
            },
            "involvedObject" : {
              "kind" : "Pod",
              "namespace" : "test2",
              "name" : "centos-logtest-d2m2v",
              "uid" : "4ded2475-f3ab-11e8-a8b6-0e0e8eca8496",
              "apiVersion" : "v1",
              "resourceVersion" : "35460",
              "fieldPath" : "spec.containers{centos-logtest}"
            },
            "reason" : "Killing",
            "source" : {
              "component" : "kubelet",
              "host" : "ip-172-18-4-161.ec2.internal"
            },
            "firstTimestamp" : "2018-11-29T07:52:22Z",
            "lastTimestamp" : "2018-11-29T07:52:22Z",
            "count" : 1,
            "type" : "Normal",
            "verb" : "ADDED"
          }
        }
      }
Comment 7 errata-xmlrpc 2018-12-13 17:09:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3750
Note You need to log in before you can comment on or make changes to this bug.