Bug 1420234 - illegal_argument_exception in Kibana UI.
Summary: illegal_argument_exception in Kibana UI.
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.5.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: ---
Assignee: Jeff Cantrill
QA Contact: Xia Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-02-08 09:29 UTC by Junqi Zhao
Modified: 2017-03-02 22:18 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-03-02 22:18:26 UTC
Target Upstream Version:


Attachments (Terms of Use)
illegal_argument_exception in Kibana UI (169.31 KB, image/png)
2017-02-08 09:29 UTC, Junqi Zhao
no flags Details
kibana settings tab picture (144.98 KB, image/png)
2017-02-09 09:19 UTC, Junqi Zhao
no flags Details
disable to change field type (94.41 KB, image/png)
2017-02-09 09:20 UTC, Junqi Zhao
no flags Details

Description Junqi Zhao 2017-02-08 09:29:28 UTC
Created attachment 1248567 [details]
illegal_argument_exception in Kibana UI

Description of problem:
Deployed logging stack with ansible, illegal_argument_exception shows when logon Kibana UI.


# oc get po
NAME                          READY     STATUS    RESTARTS   AGE
logging-curator-1-dslwb       1/1       Running   1          40m
logging-es-9nmvgerr-1-92ttn   1/1       Running   0          40m
logging-fluentd-qf0xc         1/1       Running   0          41m
logging-kibana-1-dmjsf        2/2       Running   0          40m


Version-Release number of selected component (if applicable):
# openshift version
openshift v3.5.0.17+c55cf2b
kubernetes v1.5.2+43a9be4
etcd 3.1.0

Image id:
openshift3/logging-elasticsearch    7605f043d232
openshift3/logging-kibana    e0ab09c2cbeb
openshift3/logging-fluentd    47057624ecab
openshift3/logging-auth-proxy    139f7943475e
openshift3/logging-curator    7f034fdf7702


How reproducible:
Always


Steps to Reproduce:
1. prepare the inventory file

[oo_first_master]
$master-public-dns ansible_user=root ansible_ssh_user=root ansible_ssh_private_key_file="~/cfile/libra.pem" openshift_public_hostname=$master-public-dns

[oo_first_master:vars]
deployment_type=openshift-enterprise
openshift_release=v3.5.0
openshift_logging_install_logging=true

openshift_logging_kibana_hostname=kibana.$sub-domain
public_master_url=https://$master-public-dns:8443

openshift_logging_image_prefix=registry.ops.openshift.com/openshift3/
openshift_logging_image_version=3.5.0

openshift_logging_namespace=juzhao


2. Running the playbook from a control machine (my laptop) which is not oo_master:
git clone https://github.com/openshift/openshift-ansible
ansible-playbook -vvv -i ~/inventory   playbooks/common/openshift-cluster/openshift_logging.yml

3. Log on Kibana UI to check logs.
Actual results:
Error shows, Discover: Expected numeric type on field [time], but got [string], click "more info" to see error trace

Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Expected numeric type on field [time], but got [string]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query_fetch","grouped":true,"failed_shards":[{"shard":0,"index":".operations.2017.02.08","node":"H_KsakBYQC2G1Y2rTScUAA","reason":{"type":"illegal_argument_exception","reason":"Expected numeric type on field [time], but got [string]"}}]}}
KbnError@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:57511:21
RequestFailure@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:57544:6
callResponseHandlers/<@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:98389:39
__WEBPACK_AMD_DEFINE_RESULT__</</Promise.try@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:101829:20
__WEBPACK_AMD_DEFINE_RESULT__</</Promise.map/<@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:101798:17
__WEBPACK_AMD_DEFINE_RESULT__</</Promise.map@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:101797:27
callResponseHandlers@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:98361:15
fetchWithStrategy/<@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:97862:17
processQueue@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:42452:29
scheduleProcessQueue/<@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:42468:28
$RootScopeProvider/this.$get</Scope.prototype.$eval@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:43696:17
$RootScopeProvider/this.$get</Scope.prototype.$digest@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:43507:16
$RootScopeProvider/this.$get</Scope.prototype.$apply@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:43804:14
done@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:38253:37
completeRequest@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:38451:8
requestLoaded@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:38392:10


Expected results:
no error in Kibana UI

Additional info:
Attached Kibana UI snapshot

Comment 1 Jeff Cantrill 2017-02-08 15:19:57 UTC
Was this error seen immediately upon log in to kibana or did you do any additional steps?

Comment 2 Junqi Zhao 2017-02-09 01:27:14 UTC
I did not do any additional steps, the error shows after logging in Kibana. It seems like the type for time is not proper

Comment 3 Peter Portante 2017-02-09 02:01:58 UTC
This is a lot like a mapping did not get setup for an index properly.

Error: Request to Elasticsearch failed:
  { "error":
    { "root_cause": [
          { "type":   "illegal_argument_exception",
            "reason": "Expected numeric type on field [time], but got [string]"
          }
        ],
      "type":    "search_phase_execution_exception",
      "reason":  "all shards failed",
      "phase":   "query_fetch",
      "grouped": true,
      "failed_shards": [
          { "shard": 0,
            "index": ".operations.2017.02.08",
            "node": "H_KsakBYQC2G1Y2rTScUAA",
            "reason": { 
              "type":   "illegal_argument_exception",
              "reason": "Expected numeric type on field [time], but got [string]"
            }
          }
        ]
      }
    }
  }

We need to look at the mappings for index .operations.2017.02.08 and what Kibana thinks that field mapping is.

What URL for kibana was used exactly?

Comment 4 Junqi Zhao 2017-02-09 02:58:45 UTC
(In reply to Peter Portante from comment #3)
> This is a lot like a mapping did not get setup for an index properly.
> 
> Error: Request to Elasticsearch failed:
>   { "error":
>     { "root_cause": [
>           { "type":   "illegal_argument_exception",
>             "reason": "Expected numeric type on field [time], but got
> [string]"
>           }
>         ],
>       "type":    "search_phase_execution_exception",
>       "reason":  "all shards failed",
>       "phase":   "query_fetch",
>       "grouped": true,
>       "failed_shards": [
>           { "shard": 0,
>             "index": ".operations.2017.02.08",
>             "node": "H_KsakBYQC2G1Y2rTScUAA",
>             "reason": { 
>               "type":   "illegal_argument_exception",
>               "reason": "Expected numeric type on field [time], but got
> [string]"
>             }
>           }
>         ]
>       }
>     }
>   }
> 
> We need to look at the mappings for index .operations.2017.02.08 and what
> Kibana thinks that field mapping is.
> 
> What URL for kibana was used exactly?

Environment was shutdown yesterday, I will provide info if this error shows today.

$SUBDOMAIN = `grep subdomain /etc/origin/master/master-config.yaml`
kibana URL is https://kibana.$SUBDOMAIN

Comment 5 Junqi Zhao 2017-02-09 03:44:54 UTC
Same error shows today:

Discover: Expected numeric type on field [time], but got [string]

Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Expected numeric type on field [time], but got [string]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query_fetch","grouped":true,"failed_shards":[{"shard":0,"index":".operations.2017.02.09","node":"5fjnOvDmQvG2Tf4X3bCc8A","reason":{"type":"illegal_argument_exception","reason":"Expected numeric type on field [time], but got [string]"}}]}}
KbnError@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:57511:21
RequestFailure@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:57544:6
callResponseHandlers/<@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:98389:39
__WEBPACK_AMD_DEFINE_RESULT__</</Promise.try@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:101829:20
__WEBPACK_AMD_DEFINE_RESULT__</</Promise.map/<@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:101798:17
__WEBPACK_AMD_DEFINE_RESULT__</</Promise.map@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:101797:27
callResponseHandlers@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:98361:15
fetchWithStrategy/<@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:97862:17
processQueue@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:42452:29
scheduleProcessQueue/<@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:42468:28
$RootScopeProvider/this.$get</Scope.prototype.$eval@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:43696:17
$RootScopeProvider/this.$get</Scope.prototype.$digest@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:43507:16
$RootScopeProvider/this.$get</Scope.prototype.$apply@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:43804:14
done@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:38253:37
completeRequest@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:38451:8
requestLoaded@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:38392:10



mappings for index .operations.2017.02.09

{
    ".operations.2017.02.09": {
        "mappings": {
            "_default_": {
                "_meta": {
                    "version": "2016.10.12.0"
                },
                "date_detection": false,
                "dynamic_templates": [
                    {
                        "message_field": {
                            "mapping": {
                                "index": "analyzed",
                                "omit_norms": true,
                                "type": "string"
                            },
                            "match": "message",
                            "match_mapping_type": "string"
                        }
                    },
                    {
                        "string_fields": {
                            "mapping": {
                                "fields": {
                                    "raw": {
                                        "ignore_above": 256,
                                        "index": "not_analyzed",
                                        "type": "string"
                                    }
                                },
                                "index": "analyzed",
                                "omit_norms": true,
                                "type": "string"
                            },
                            "match": "*",
                            "match_mapping_type": "string"
                        }
                    }
                ],
                "properties": {
                    "@timestamp": {
                        "fields": {
                            "raw": {
                                "ignore_above": 256,
                                "index": "not_analyzed",
                                "type": "string"
                            }
                        },
                        "format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime",
                        "type": "date"
                    },
                    "docker": {
                        "properties": {
                            "container_id": {
                                "index": "not_analyzed",
                                "type": "string"
                            }
                        }
                    },
                    "file": {
                        "fields": {
                            "raw": {
                                "ignore_above": 256,
                                "index": "not_analyzed",
                                "type": "string"
                            }
                        },
                        "type": "string"
                    },
                    "geoip": {
                        "dynamic": "true",
                        "properties": {
                            "location": {
                                "type": "geo_point"
                            }
                        }
                    },
                    "hostname": {
                        "index": "not_analyzed",
                        "type": "string"
                    },
                    "ipaddr4": {
                        "fields": {
                            "raw": {
                                "ignore_above": 256,
                                "index": "not_analyzed",
                                "type": "string"
                            }
                        },
                        "type": "ip"
                    },
                    "ipaddr6": {
                        "index": "not_analyzed",
                        "type": "string"
                    },
                    "kubernetes": {
                        "properties": {
                            "host": {
                                "index": "not_analyzed",
                                "norms": {
                                    "enabled": true
                                },
                                "type": "string"
                            },
                            "namespace_id": {
                                "index": "not_analyzed",
                                "norms": {
                                    "enabled": true
                                },
                                "type": "string"
                            },
                            "namespace_name": {
                                "index": "not_analyzed",
                                "norms": {
                                    "enabled": true
                                },
                                "type": "string"
                            },
                            "pod_id": {
                                "index": "not_analyzed",
                                "norms": {
                                    "enabled": true
                                },
                                "type": "string"
                            },
                            "pod_name": {
                                "index": "not_analyzed",
                                "norms": {
                                    "enabled": true
                                },
                                "type": "string"
                            }
                        }
                    },
                    "level": {
                        "index": "not_analyzed",
                        "type": "string"
                    },
                    "message": {
                        "norms": {
                            "enabled": false
                        },
                        "type": "string"
                    },
                    "offset": {
                        "type": "long"
                    },
                    "pid": {
                        "index": "not_analyzed",
                        "type": "string"
                    },
                    "pipeline_metadata": {
                        "properties": {
                            "@version": {
                                "index": "not_analyzed",
                                "type": "string"
                            },
                            "collector": {
                                "properties": {
                                    "hostname": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "inputname": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "ipaddr4": {
                                        "fields": {
                                            "raw": {
                                                "ignore_above": 256,
                                                "index": "not_analyzed",
                                                "type": "string"
                                            }
                                        },
                                        "type": "ip"
                                    },
                                    "ipaddr6": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "name": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "original_raw_message": {
                                        "fields": {
                                            "raw": {
                                                "ignore_above": 256,
                                                "index": "not_analyzed",
                                                "type": "string"
                                            }
                                        },
                                        "type": "string"
                                    },
                                    "received_at": {
                                        "format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime",
                                        "type": "date"
                                    },
                                    "version": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    }
                                }
                            },
                            "normalizer": {
                                "properties": {
                                    "hostname": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "inputname": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "ipaddr4": {
                                        "fields": {
                                            "raw": {
                                                "ignore_above": 256,
                                                "index": "not_analyzed",
                                                "type": "string"
                                            }
                                        },
                                        "type": "ip"
                                    },
                                    "ipaddr6": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "name": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "original_raw_message": {
                                        "fields": {
                                            "raw": {
                                                "ignore_above": 256,
                                                "index": "not_analyzed",
                                                "type": "string"
                                            }
                                        },
                                        "type": "string"
                                    },
                                    "received_at": {
                                        "format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime",
                                        "type": "date"
                                    },
                                    "version": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    }
                                }
                            },
                            "trace": {
                                "analyzer": "whitespace",
                                "type": "string"
                            }
                        }
                    },
                    "rsyslog": {
                        "properties": {
                            "appname": {
                                "index": "not_analyzed",
                                "type": "string"
                            },
                            "facility": {
                                "index": "not_analyzed",
                                "type": "string"
                            },
                            "msgid": {
                                "type": "long"
                            },
                            "protocol-version": {
                                "index": "not_analyzed",
                                "type": "string"
                            },
                            "structured-data": {
                                "index": "not_analyzed",
                                "type": "string"
                            }
                        }
                    },
                    "service": {
                        "index": "not_analyzed",
                        "type": "string"
                    },
                    "systemd": {
                        "properties": {
                            "k": {
                                "properties": {
                                    "KERNEL_DEVICE": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "KERNEL_SUBSYSTEM": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "UDEV_DEVLINK": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "UDEV_DEVNODE": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "UDEV_SYSNAME": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    }
                                }
                            },
                            "t": {
                                "properties": {
                                    "AUDIT_LOGINUID": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "AUDIT_SESSION": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "BOOT_ID": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "CAP_EFFECTIVE": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "CMDLINE": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "COMM": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "EXE": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "GID": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "HOSTNAME": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "MACHINE_ID": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "PID": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "SELINUX_CONTEXT": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "SOURCE_REALTIME_TIMESTAMP": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "SYSTEMD_CGROUP": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "SYSTEMD_OWNER_UID": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "SYSTEMD_SESSION": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "SYSTEMD_SLICE": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "SYSTEMD_UNIT": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "SYSTEMD_USER_UNIT": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "TRANSPORT": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "UID": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    }
                                }
                            },
                            "u": {
                                "properties": {
                                    "CODE_FILE": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "CODE_FUNCTION": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "CODE_LINE": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "ERRNO": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "MESSAGE_ID": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "RESULT": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "UNIT": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    }
                                }
                            }
                        }
                    },
                    "tags": {
                        "analyzer": "whitespace",
                        "type": "string"
                    }
                }
            },
            "com.redhat.viaq.common": {
                "_meta": {
                    "version": "2016.10.12.0"
                },
                "date_detection": false,
                "dynamic_templates": [
                    {
                        "message_field": {
                            "mapping": {
                                "index": "analyzed",
                                "omit_norms": true,
                                "type": "string"
                            },
                            "match": "message",
                            "match_mapping_type": "string"
                        }
                    },
                    {
                        "string_fields": {
                            "mapping": {
                                "fields": {
                                    "raw": {
                                        "ignore_above": 256,
                                        "index": "not_analyzed",
                                        "type": "string"
                                    }
                                },
                                "index": "analyzed",
                                "omit_norms": true,
                                "type": "string"
                            },
                            "match": "*",
                            "match_mapping_type": "string"
                        }
                    }
                ],
                "properties": {
                    "@timestamp": {
                        "fields": {
                            "raw": {
                                "ignore_above": 256,
                                "index": "not_analyzed",
                                "type": "string"
                            }
                        },
                        "format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime",
                        "type": "date"
                    },
                    "_SOURCE_MONOTONIC_TIMESTAMP": {
                        "fields": {
                            "raw": {
                                "ignore_above": 256,
                                "index": "not_analyzed",
                                "type": "string"
                            }
                        },
                        "norms": {
                            "enabled": false
                        },
                        "type": "string"
                    },
                    "docker": {
                        "properties": {
                            "container_id": {
                                "index": "not_analyzed",
                                "type": "string"
                            }
                        }
                    },
                    "file": {
                        "fields": {
                            "raw": {
                                "ignore_above": 256,
                                "index": "not_analyzed",
                                "type": "string"
                            }
                        },
                        "type": "string"
                    },
                    "geoip": {
                        "dynamic": "true",
                        "properties": {
                            "location": {
                                "type": "geo_point"
                            }
                        }
                    },
                    "hostname": {
                        "index": "not_analyzed",
                        "type": "string"
                    },
                    "ipaddr4": {
                        "fields": {
                            "raw": {
                                "ignore_above": 256,
                                "index": "not_analyzed",
                                "type": "string"
                            }
                        },
                        "type": "ip"
                    },
                    "ipaddr6": {
                        "index": "not_analyzed",
                        "type": "string"
                    },
                    "kubernetes": {
                        "properties": {
                            "host": {
                                "index": "not_analyzed",
                                "norms": {
                                    "enabled": true
                                },
                                "type": "string"
                            },
                            "namespace_id": {
                                "index": "not_analyzed",
                                "norms": {
                                    "enabled": true
                                },
                                "type": "string"
                            },
                            "namespace_name": {
                                "index": "not_analyzed",
                                "norms": {
                                    "enabled": true
                                },
                                "type": "string"
                            },
                            "pod_id": {
                                "index": "not_analyzed",
                                "norms": {
                                    "enabled": true
                                },
                                "type": "string"
                            },
                            "pod_name": {
                                "index": "not_analyzed",
                                "norms": {
                                    "enabled": true
                                },
                                "type": "string"
                            }
                        }
                    },
                    "level": {
                        "index": "not_analyzed",
                        "type": "string"
                    },
                    "message": {
                        "norms": {
                            "enabled": false
                        },
                        "type": "string"
                    },
                    "offset": {
                        "type": "long"
                    },
                    "pid": {
                        "index": "not_analyzed",
                        "type": "string"
                    },
                    "pipeline_metadata": {
                        "properties": {
                            "@version": {
                                "index": "not_analyzed",
                                "type": "string"
                            },
                            "collector": {
                                "properties": {
                                    "hostname": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "inputname": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "ipaddr4": {
                                        "fields": {
                                            "raw": {
                                                "ignore_above": 256,
                                                "index": "not_analyzed",
                                                "type": "string"
                                            }
                                        },
                                        "type": "ip"
                                    },
                                    "ipaddr6": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "name": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "original_raw_message": {
                                        "fields": {
                                            "raw": {
                                                "ignore_above": 256,
                                                "index": "not_analyzed",
                                                "type": "string"
                                            }
                                        },
                                        "type": "string"
                                    },
                                    "received_at": {
                                        "format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime",
                                        "type": "date"
                                    },
                                    "version": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    }
                                }
                            },
                            "normalizer": {
                                "properties": {
                                    "hostname": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "inputname": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "ipaddr4": {
                                        "fields": {
                                            "raw": {
                                                "ignore_above": 256,
                                                "index": "not_analyzed",
                                                "type": "string"
                                            }
                                        },
                                        "type": "ip"
                                    },
                                    "ipaddr6": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "name": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "original_raw_message": {
                                        "fields": {
                                            "raw": {
                                                "ignore_above": 256,
                                                "index": "not_analyzed",
                                                "type": "string"
                                            }
                                        },
                                        "type": "string"
                                    },
                                    "received_at": {
                                        "format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime",
                                        "type": "date"
                                    },
                                    "version": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    }
                                }
                            },
                            "trace": {
                                "analyzer": "whitespace",
                                "type": "string"
                            }
                        }
                    },
                    "rsyslog": {
                        "properties": {
                            "appname": {
                                "index": "not_analyzed",
                                "type": "string"
                            },
                            "facility": {
                                "index": "not_analyzed",
                                "type": "string"
                            },
                            "msgid": {
                                "type": "long"
                            },
                            "protocol-version": {
                                "index": "not_analyzed",
                                "type": "string"
                            },
                            "structured-data": {
                                "index": "not_analyzed",
                                "type": "string"
                            }
                        }
                    },
                    "service": {
                        "index": "not_analyzed",
                        "type": "string"
                    },
                    "systemd": {
                        "properties": {
                            "k": {
                                "properties": {
                                    "KERNEL_DEVICE": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "KERNEL_SUBSYSTEM": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "UDEV_DEVLINK": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "UDEV_DEVNODE": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "UDEV_SYSNAME": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    }
                                }
                            },
                            "t": {
                                "properties": {
                                    "AUDIT_LOGINUID": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "AUDIT_SESSION": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "BOOT_ID": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "CAP_EFFECTIVE": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "CMDLINE": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "COMM": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "EXE": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "GID": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "HOSTNAME": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "MACHINE_ID": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "PID": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "SELINUX_CONTEXT": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "SOURCE_REALTIME_TIMESTAMP": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "SYSTEMD_CGROUP": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "SYSTEMD_OWNER_UID": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "SYSTEMD_SESSION": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "SYSTEMD_SLICE": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "SYSTEMD_UNIT": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "SYSTEMD_USER_UNIT": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "TRANSPORT": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "UID": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    }
                                }
                            },
                            "u": {
                                "properties": {
                                    "CODE_FILE": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "CODE_FUNCTION": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "CODE_LINE": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "ERRNO": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "MESSAGE_ID": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "RESULT": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    },
                                    "SYSLOG_FACILITY": {
                                        "fields": {
                                            "raw": {
                                                "ignore_above": 256,
                                                "index": "not_analyzed",
                                                "type": "string"
                                            }
                                        },
                                        "norms": {
                                            "enabled": false
                                        },
                                        "type": "string"
                                    },
                                    "SYSLOG_IDENTIFIER": {
                                        "fields": {
                                            "raw": {
                                                "ignore_above": 256,
                                                "index": "not_analyzed",
                                                "type": "string"
                                            }
                                        },
                                        "norms": {
                                            "enabled": false
                                        },
                                        "type": "string"
                                    },
                                    "SYSLOG_PID": {
                                        "fields": {
                                            "raw": {
                                                "ignore_above": 256,
                                                "index": "not_analyzed",
                                                "type": "string"
                                            }
                                        },
                                        "norms": {
                                            "enabled": false
                                        },
                                        "type": "string"
                                    },
                                    "UNIT": {
                                        "index": "not_analyzed",
                                        "type": "string"
                                    }
                                }
                            }
                        }
                    },
                    "tags": {
                        "analyzer": "whitespace",
                        "type": "string"
                    },
                    "time": {
                        "fields": {
                            "raw": {
                                "ignore_above": 256,
                                "index": "not_analyzed",
                                "type": "string"
                            }
                        },
                        "norms": {
                            "enabled": false
                        },
                        "type": "string"
                    }
                }
            }
        }
    }
}

Comment 6 Peter Portante 2017-02-09 04:24:02 UTC
Thanks Junqi, can you also take a look in the "Settings" tab of Kibana to see how the default search pattern is defined, and what the "fields" are?  If you see the time field, it likely defined there as a number.  If you refresh the field patterns in Kibana, that might fix it.

That is, if that is the problem.  I am just guessing at this point.

Comment 7 Junqi Zhao 2017-02-09 09:18:29 UTC
Kibana version is 4.6.4 for logging 3.5, checked the fields according to comment 6, type is 'date' for time field, and can not change type in Kibana UI, so this error can not be fixed by changing type in Kibana UI, I think it should be fixed in mappings.

See the attached pictures.

Comment 8 Junqi Zhao 2017-02-09 09:19:47 UTC
Created attachment 1248763 [details]
kibana settings tab picture

attached kibana settings tab picture

Comment 9 Junqi Zhao 2017-02-09 09:20:40 UTC
Created attachment 1248764 [details]
disable to change field type

Comment 10 Peter Portante 2017-02-09 12:27:35 UTC
So on the settings tab, hitting "refresh" does not change the field to reflect what's in the Elasticsearch mapping?

If that is the case, can you tell me how many different .operations-* indices there are?

I suspect one or more of them has "time" in its mapping as a date field, and the rest have it as a string.  If so, I think we can promote the time field to a string in that set of indices to fix this problem.

We need to turn off automatic date detect for fields [1] that are not in our mapping.

[1] http://stackoverflow.com/questions/14813379/disable-date-detection-in-tires-elasticsearch-mapping

Comment 11 Rich Megginson 2017-02-09 21:28:01 UTC
There should be no field named "time".  There should only be a field named "@timestamp".  I suspect fluentd is having problems.  Can you provide the fluentd log?

Comment 12 Peter Portante 2017-02-09 21:44:23 UTC
Rich, regardless, if we allow elasticsearch to detect date fields from strings, then this problem can happen with any string field that is encountered.  We have to turn that off.

Comment 13 Rich Megginson 2017-02-09 21:50:12 UTC
(In reply to Peter Portante from comment #12)
> Rich, regardless, if we allow elasticsearch to detect date fields from
> strings, then this problem can happen with any string field that is
> encountered.  We have to turn that off.

Ok, how do we do that?

I suspect the problem is related to https://bugzilla.redhat.com/show_bug.cgi?id=1420219 which can also be caused if there is a top level field called "time" but no top level field called "@timestamp".

Comment 14 Peter Portante 2017-02-09 21:59:30 UTC
See comment 10.

Comment 15 Rich Megginson 2017-02-09 22:24:22 UTC
(In reply to Peter Portante from comment #14)
> See comment 10.

So, since this is an "unknown unknown" we have to disable date detection for all fields?

   "mappings" : {
      "_default_" : {
         "date_detection" : 0
      }
   }

Does that mean we can still do date detection for date valued fields such as @timestamp, pipeline_metadata.collector.received_at, etc?

Anton, is there support in the common data model for this?  If not, we need to add it when we generate index templates.

Comment 16 Rich Megginson 2017-02-09 22:41:20 UTC
The "@timestamp" field is missing because the common data model filter is missing.  Here is the fluent.conf as deployed by ansible 3.5:

  @include configs.d/openshift/filter-syslog-record-transform.conf
  @include configs.d/openshift/filter-post-*.conf

It should look like this:

  @include configs.d/openshift/filter-syslog-record-transform.conf
  @include configs.d/openshift/filter-common-data-model.conf
  @include configs.d/openshift/filter-post-*.conf

PR: https://github.com/openshift/openshift-ansible/pull/3323

Comment 17 openshift-github-bot 2017-02-10 01:01:53 UTC
Commits pushed to master at https://github.com/openshift/openshift-ansible

https://github.com/openshift/openshift-ansible/commit/f9daaa768ec415c2ae83a8c42ae35776d996083e
Bug 1420234 - illegal_argument_exception in Kibana UI.

https://bugzilla.redhat.com/show_bug.cgi?id=1420234
The problem is that the fluent.conf is missing the common data model
filter which renames the "time" field to the "@timestamp" field.

https://github.com/openshift/openshift-ansible/commit/9e92660ed86623385e48c4960f4db681d86a7a57
Merge pull request #3323 from richm/missing-common-data-model-filter

Bug 1420234 - illegal_argument_exception in Kibana UI.

Comment 18 Rich Megginson 2017-02-10 01:14:58 UTC
Moving to MODIFIED because fix committed upstream - not sure if QE will test with upstream or will wait for RH openshift-ansible package to contain fix.

Comment 19 Junqi Zhao 2017-02-10 09:30:38 UTC
blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1421011

Comment 20 Junqi Zhao 2017-02-20 06:39:39 UTC
@rmeggins
Verified with the latest logging 3.5 openshift-ansible, this defect is get fixed, please feel free to change the status to ON_QA, then I can close it. thanks.

Comment 21 Junqi Zhao 2017-02-21 01:11:19 UTC
close this defect according to Comment 20

Comment 22 Troy Dawson 2017-03-02 22:18:26 UTC
Since this bug never reached customers, I am closing it.


Note You need to log in before you can comment on or make changes to this bug.