Bug 1515447
Summary: | [3.6] Aggregated Logging replacing all log levels with '3' and '6' after upgrade to 3.5 from 3.4 | |||
---|---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Rich Megginson <rmeggins> | |
Component: | Logging | Assignee: | Rich Megginson <rmeggins> | |
Status: | CLOSED CURRENTRELEASE | QA Contact: | Anping Li <anli> | |
Severity: | urgent | Docs Contact: | ||
Priority: | unspecified | |||
Version: | 3.6.1 | CC: | anli, antti.niskanen, aos-bugs, rmeggins, trankin, vlaad | |
Target Milestone: | --- | |||
Target Release: | 3.6.z | |||
Hardware: | x86_64 | |||
OS: | Linux | |||
Whiteboard: | ||||
Fixed In Version: | openshift3/logging-fluentd:v3.6.173.0.94-2 | Doc Type: | Bug Fix | |
Doc Text: |
Cause: Fluentd was adding the level field with a value of 3 or 6, overwriting any existing level field.
Consequence: The level field set by the application was being removed, and the '3' or '6' value was not useful.
Fix: If there is already a `level` field in the record, then see if it is
a "close" match to one of the canonical `level` field values at
https://github.com/ViaQ/elasticsearch-templates/blob/master/namespaces/_default_.yml#L63
e.g. if `level` is "CRITICAL", convert to "crit", if level
is "WARN" convert to 'warning', etc.
Otherwise, if we cannot use it directly or normalize it, convert
it to its string representation (ruby `to_s` method) and store
the string value in the `level` field.
Result: If the record already has a level field, the value is normalized or preserved, otherwise, a value like `info` or `err` is used.
|
Story Points: | --- | |
Clone Of: | 1514110 | |||
: | 1515448 (view as bug list) | Environment: | ||
Last Closed: | 2018-10-08 12:34:04 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | 1514110 | |||
Bug Blocks: | 1515448 |
Description
Rich Megginson
2017-11-20 19:39:29 UTC
logging-fluentd-docker-v3.6.173.0.83-3 This is fixed in logging-fluentd-docker-v3.6.173.0.83-2 which is included in https://errata.devel.redhat.com/advisory/31698 RHBA-2017:31698-04 OpenShift Container Platform 3.6.z and 3.5.z image update Failed with both logging-fluentd/images/v3.6.173.0.83-2 and logging-fluentd/images/v3.6.173.0.90-1 My test steps as following 1) oc debug logging-kibana-1-rj99r $ echo '{"message":"anli_container_message_loglevel_debug", "level":"debug"}' 2) search record in ES oc exec -c elasticsearch $es_pod_name -- curl -s -XGET --cacert /etc/elasticsearch/secret/admin-ca --cert /etc/elasticsearch/secret/admin-cert --key /etc/elasticsearch/secret/admin-key https://localhost:9200/_search?pretty -d '{ "query" : { "match" : { "message" : "anli_container_message_loglevel_warn" } } }' 3) result as following "_source" : { "message" : "anli_container_message_loglevel_debug", "docker" : { "container_id" : "3c615787a6ac7f2f9e3a9dc18835d74f7af6c19ef7b5190c256c09f5cdea0126" }, "kubernetes" : { "container_name" : "kibana", "namespace_name" : "logging", "pod_name" : "logging-kibana-1-rj99r-debug", "pod_id" : "5c683163-e576-11e7-acfe-fa163ea31fda", "host" : "host-8-241-0.host.centralci.eng.rdu2.redhat.com", "master_url" : "https://kubernetes.default.svc.cluster.local", "namespace_id" : "ea000949-e4b4-11e7-918b-fa163ea31fda" }, "level" : "info", "hostname" : "host-xxxx.example.com, koji_builds: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=636402 repositories: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:rhaos-3.6-rhel-7-docker-candidate-69601-20180104021016 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:latest brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6.173.0.94 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6.173.0.94-2 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-fluentd:v3.6 Verified and pass in v3.6.173.0.95-1 "_index" : "project.install-test.504d5cc0-f426-11e7-bd8f-fa163e48c543.2018.01.08", "_type" : "com.redhat.viaq.common", "_id" : "AWDUWp6Ql6XcCvEMVCIR", "_score" : 3.7972813, "_source" : { "message" : "anli_container_message_loglevel_debug", "level" : "debug", "docker" : { "container_id" : "69037f7e99a5ea63da3514f38179de1b9d51c22def88ec3210151c7064d56343" }, "_source" : { "message" : "anli_container_message_loglevel_warn", "level" : "warning", "docker" : { "container_id" : "69037f7e99a5ea63da3514f38179de1b9d51c22def88ec3210151c7064d56343" }, "_source" : { "message" : "anli_container_message_loglevel_nolevel", "level" : "info", |