Bug 1588829

Summary: After upgrade to 3.6 some logs are not showing in the Kibana webUI
Product: OpenShift Container Platform Reporter: Rich Megginson <rmeggins>
Component: LoggingAssignee: Rich Megginson <rmeggins>
Status: CLOSED CURRENTRELEASE QA Contact: Anping Li <anli>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.10.0CC: anli, aos-bugs, jack.ottofaro, jcantril, jfoots, openshift-bugs-escalate, peli, rkant, rmeggins, vlaad
Target Milestone: ---   
Target Release: 3.10.0   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: The incoming data has a field `record["event"]` that is a String value and not the Hash value expected by the transform_eventrouter code. Consequence: This causes the code to throw an error and fluentd to emit an error like this: ``` error_class=NoMethodError error="undefined method `key?' for \"request\":String" ``` Fix: The transform_eventrouter code has been changed to only process the `record["event"]` field if it is a Hash. Result: Records can flow through to the destination again.
 Story Points: ---
Clone Of: 1588828 Environment:
Last Closed: 2018-08-29 21:23:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1588772, 1588827, 1588828    
Bug Blocks:    

Comment 1 Rich Megginson 2018-06-07 23:53:50 UTC 
fixed: https://github.com/ViaQ/fluent-plugin-viaq_data_model/commit/1084c3f85b5d25c4f929ccb055c43fec39d87c9f
Comment 4 Anping Li 2018-06-13 06:36:58 UTC 
Reproduced with v3.9, The message is blank when event is string.
With openshift3/logging-fluentd/images/v3.10.0-0.67.0.0. The string event message are in message field.


{
  "_index": ".operations.2018.06.13",
  "_type": "com.redhat.viaq.common",
  "_id": "MTA3ZjEzYTUtMjFjNC00YzkzLWFmMWItYjI2ZDJiOTU3Y2Vi",
  "_score": null,
  "_source": {
    "event": "anlieventevent",
    "verb": "ADDED",
    "logtag": "F",
    "docker": {
      "container_id": "d3bc9346e4d9f808463536e12a8d752fd5c8cb5e96e29a77a44647a4620df92b"
    },
    "kubernetes": {
      "container_name": "kube-eventrouter",
      "namespace_name": "default",
      "pod_name": "logging-eventrouter-1-pzf8v-debug",
      "pod_id": "39729dbb-6ed2-11e8-9b3b-fa163e8298c1",
      "host": "qe-anli10node-registry-router-2",
      "master_url": "https://kubernetes.default.svc.cluster.local",
      "namespace_id": "95fe5920-6ea3-11e8-96d9-fa163e8298c1"
    },
    "message": "{ \"event\": \"anlieventevent\", \"verb\": \"ADDED\" }",
    "level": "info",
    "hostname": "qe-anli10node-registry-router-2",
    "pipeline_metadata": {
      "collector": {
        "ipaddr4": "10.130.0.14",
        "ipaddr6": "fe80::50ca:88ff:fe1d:dc7",
        "inputname": "fluent-plugin-systemd",
        "name": "fluentd",
        "received_at": "2018-06-13T06:30:39.224634+00:00",
        "version": "0.12.42 1.6.0"
      }
    },
    "@timestamp": "2018-06-13T06:30:39.043046+00:00",
    "viaq_msg_id": "MTA3ZjEzYTUtMjFjNC00YzkzLWFmMWItYjI2ZDJiOTU3Y2Vi"
  },
  "fields": {
    "@timestamp": [
      1528871439043
    ],
    "pipeline_metadata.collector.received_at": [
      1528871439224
    ]
  },
  "highlight": {
    "kubernetes.pod_name": [
      "@kibana-highlighted-field@logging-eventrouter-1-pzf8v-debug@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1528871439043
  ]
}