Bug 1632892
| Summary: | [3.10] logging-eventrouter event not formatted correctly in Elasticsearch when using MUX | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Noriko Hosoi <nhosoi> |
| Component: | Logging | Assignee: | Noriko Hosoi <nhosoi> |
| Status: | CLOSED ERRATA | QA Contact: | Qiaoling Tang <qitang> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 3.10.0 | CC: | anli, aos-bugs, jforrest, nhosoi, pportant, rmeggins |
| Target Milestone: | --- | ||
| Target Release: | 3.10.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openshift3/logging-fluentd:v3.10.57-1 | Doc Type: | Bug Fix |
| Doc Text: |
Cause:
When fluentd is configured as the combination of collectors and MUX, event logs from the event were supposed to be processed by MUX not by the collector for the both MUX_CLIENT_MODE maximal and minimal. It is because if an event log is formatted in the collector (note: the event record is put under the kubernetes key), the log is forwarded to MUX and passed to the k8s-meta plugin there and the existing kubernetes record is overwritten. It wiped out the event info from the log.
Fix 1:
To avoid the replacement, if the log is from event router, the tag is rewritten to ${tag}.raw in input-post-forward-mux.conf, which makes the log treated in the MUX_CLIENT_MODE=minimal way.
Fix 2:
There was another bug in ansible. That is, the environment variable TRANSFORM_EVENTS was not set in MUX even if openshift_logging_install_eventrouter is set to true.
Result:
With these 2 fixes, the event logs are correctly logged when MUX is configured with MUX_CLIENT_MODE=maximal as well as minimal.
|
Story Points: | --- |
| Clone Of: | 1554293 | Environment: | |
| Last Closed: | 2018-12-13 17:09:08 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1554293, 1632895 | ||
| Bug Blocks: | |||
|
Comment 1
Noriko Hosoi
2018-10-08 21:18:47 UTC
Verified in logging-fluentd-v3.10.79-1.
The logs collected when mux enabled are like this:
{
"_index" : ".operations.2018.11.29",
"_type" : "com.redhat.viaq.common",
"_id" : "b40c2230-f3ab-11e8-a8b6-0e0e8eca8496",
"_score" : 9.291312,
"_source" : {
"message" : "Killing container with id docker://centos-logtest:Need to kill Pod",
"level" : "info",
"hostname" : "ip-172-18-1-48.ec2.internal",
"pipeline_metadata" : {
"collector" : {
"ipaddr4" : "10.129.0.10",
"ipaddr6" : "fe80::dc25:26ff:fe12:f60e",
"inputname" : "fluent-plugin-systemd",
"name" : "fluentd",
"received_at" : "2018-11-29T07:52:22.348552+00:00",
"version" : "0.12.43 1.6.0"
},
"normalizer" : {
"original_raw_message" : "{\"verb\":\"ADDED\",\"event\":{\"metadata\":{\"name\":\"centos-logtest-d2m2v.156b8902b6173ec5\",\"namespace\":\"test2\",\"selfLink\":\"/api/v1/namespaces/test2/events/centos-logtest-d2m2v.156b8902b6173ec5\",\"uid\":\"b40c2230-f3ab-11e8-a8b6-0e0e8eca8496\",\"resourceVersion\":\"35861\",\"creationTimestamp\":\"2018-11-29T07:52:22Z\"},\"involvedObject\":{\"kind\":\"Pod\",\"namespace\":\"test2\",\"name\":\"centos-logtest-d2m2v\",\"uid\":\"4ded2475-f3ab-11e8-a8b6-0e0e8eca8496\",\"apiVersion\":\"v1\",\"resourceVersion\":\"35460\",\"fieldPath\":\"spec.containers{centos-logtest}\"},\"reason\":\"Killing\",\"message\":\"Killing container with id docker://centos-logtest:Need to kill Pod\",\"source\":{\"component\":\"kubelet\",\"host\":\"ip-172-18-4-161.ec2.internal\"},\"firstTimestamp\":\"2018-11-29T07:52:22Z\",\"lastTimestamp\":\"2018-11-29T07:52:22Z\",\"count\":1,\"type\":\"Normal\"}}\n",
"ipaddr4" : "10.130.0.17",
"ipaddr6" : "fe80::6417:59ff:fea9:cf20",
"inputname" : "fluent-plugin-systemd",
"name" : "fluentd",
"received_at" : "2018-11-29T07:52:26.385415+00:00",
"version" : "0.12.43 1.6.0"
}
},
"@timestamp" : "2018-11-29T07:52:22Z",
"viaq_msg_id" : "b40c2230-f3ab-11e8-a8b6-0e0e8eca8496",
"docker" : {
"container_id" : "f2a2c1a45921f25686d7116e4883d2316f99eee5676ce14bbdd2a0b9817bb347"
},
"kubernetes" : {
"container_name" : "kube-eventrouter",
"namespace_name" : "default",
"pod_name" : "logging-eventrouter-1-qffjg",
"pod_id" : "ecfd84bf-f3aa-11e8-a8b6-0e0e8eca8496",
"labels" : {
"component" : "eventrouter",
"deployment" : "logging-eventrouter-1",
"deploymentconfig" : "logging-eventrouter",
"logging-infra" : "eventrouter",
"provider" : "openshift"
},
"host" : "ip-172-18-1-48.ec2.internal",
"master_url" : "https://kubernetes.default.svc.cluster.local",
"namespace_id" : "bba8b899-f389-11e8-a8b6-0e0e8eca8496",
"event" : {
"metadata" : {
"name" : "centos-logtest-d2m2v.156b8902b6173ec5",
"namespace" : "test2",
"selfLink" : "/api/v1/namespaces/test2/events/centos-logtest-d2m2v.156b8902b6173ec5",
"uid" : "b40c2230-f3ab-11e8-a8b6-0e0e8eca8496",
"resourceVersion" : "35861"
},
"involvedObject" : {
"kind" : "Pod",
"namespace" : "test2",
"name" : "centos-logtest-d2m2v",
"uid" : "4ded2475-f3ab-11e8-a8b6-0e0e8eca8496",
"apiVersion" : "v1",
"resourceVersion" : "35460",
"fieldPath" : "spec.containers{centos-logtest}"
},
"reason" : "Killing",
"source" : {
"component" : "kubelet",
"host" : "ip-172-18-4-161.ec2.internal"
},
"firstTimestamp" : "2018-11-29T07:52:22Z",
"lastTimestamp" : "2018-11-29T07:52:22Z",
"count" : 1,
"type" : "Normal",
"verb" : "ADDED"
}
}
}
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3750 |