Bug 1420234
| Summary: | illegal_argument_exception in Kibana UI. | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Junqi Zhao <juzhao> | ||||||||
| Component: | Logging | Assignee: | Jeff Cantrill <jcantril> | ||||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Xia Zhao <xiazhao> | ||||||||
| Severity: | low | Docs Contact: | |||||||||
| Priority: | low | ||||||||||
| Version: | 3.5.0 | CC: | aos-bugs, asherkho, juzhao, pportant, rmeggins, tdawson | ||||||||
| Target Milestone: | --- | ||||||||||
| Target Release: | --- | ||||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2017-03-02 22:18:26 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Attachments: |
|
||||||||||
|
Description
Junqi Zhao
2017-02-08 09:29:28 UTC
Was this error seen immediately upon log in to kibana or did you do any additional steps? I did not do any additional steps, the error shows after logging in Kibana. It seems like the type for time is not proper This is a lot like a mapping did not get setup for an index properly.
Error: Request to Elasticsearch failed:
{ "error":
{ "root_cause": [
{ "type": "illegal_argument_exception",
"reason": "Expected numeric type on field [time], but got [string]"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query_fetch",
"grouped": true,
"failed_shards": [
{ "shard": 0,
"index": ".operations.2017.02.08",
"node": "H_KsakBYQC2G1Y2rTScUAA",
"reason": {
"type": "illegal_argument_exception",
"reason": "Expected numeric type on field [time], but got [string]"
}
}
]
}
}
}
We need to look at the mappings for index .operations.2017.02.08 and what Kibana thinks that field mapping is.
What URL for kibana was used exactly?
(In reply to Peter Portante from comment #3) > This is a lot like a mapping did not get setup for an index properly. > > Error: Request to Elasticsearch failed: > { "error": > { "root_cause": [ > { "type": "illegal_argument_exception", > "reason": "Expected numeric type on field [time], but got > [string]" > } > ], > "type": "search_phase_execution_exception", > "reason": "all shards failed", > "phase": "query_fetch", > "grouped": true, > "failed_shards": [ > { "shard": 0, > "index": ".operations.2017.02.08", > "node": "H_KsakBYQC2G1Y2rTScUAA", > "reason": { > "type": "illegal_argument_exception", > "reason": "Expected numeric type on field [time], but got > [string]" > } > } > ] > } > } > } > > We need to look at the mappings for index .operations.2017.02.08 and what > Kibana thinks that field mapping is. > > What URL for kibana was used exactly? Environment was shutdown yesterday, I will provide info if this error shows today. $SUBDOMAIN = `grep subdomain /etc/origin/master/master-config.yaml` kibana URL is https://kibana.$SUBDOMAIN Same error shows today:
Discover: Expected numeric type on field [time], but got [string]
Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Expected numeric type on field [time], but got [string]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query_fetch","grouped":true,"failed_shards":[{"shard":0,"index":".operations.2017.02.09","node":"5fjnOvDmQvG2Tf4X3bCc8A","reason":{"type":"illegal_argument_exception","reason":"Expected numeric type on field [time], but got [string]"}}]}}
KbnError@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:57511:21
RequestFailure@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:57544:6
callResponseHandlers/<@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:98389:39
__WEBPACK_AMD_DEFINE_RESULT__</</Promise.try@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:101829:20
__WEBPACK_AMD_DEFINE_RESULT__</</Promise.map/<@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:101798:17
__WEBPACK_AMD_DEFINE_RESULT__</</Promise.map@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:101797:27
callResponseHandlers@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:98361:15
fetchWithStrategy/<@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:97862:17
processQueue@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:42452:29
scheduleProcessQueue/<@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:42468:28
$RootScopeProvider/this.$get</Scope.prototype.$eval@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:43696:17
$RootScopeProvider/this.$get</Scope.prototype.$digest@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:43507:16
$RootScopeProvider/this.$get</Scope.prototype.$apply@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:43804:14
done@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:38253:37
completeRequest@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:38451:8
requestLoaded@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:38392:10
mappings for index .operations.2017.02.09
{
".operations.2017.02.09": {
"mappings": {
"_default_": {
"_meta": {
"version": "2016.10.12.0"
},
"date_detection": false,
"dynamic_templates": [
{
"message_field": {
"mapping": {
"index": "analyzed",
"omit_norms": true,
"type": "string"
},
"match": "message",
"match_mapping_type": "string"
}
},
{
"string_fields": {
"mapping": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"index": "analyzed",
"omit_norms": true,
"type": "string"
},
"match": "*",
"match_mapping_type": "string"
}
}
],
"properties": {
"@timestamp": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime",
"type": "date"
},
"docker": {
"properties": {
"container_id": {
"index": "not_analyzed",
"type": "string"
}
}
},
"file": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"type": "string"
},
"geoip": {
"dynamic": "true",
"properties": {
"location": {
"type": "geo_point"
}
}
},
"hostname": {
"index": "not_analyzed",
"type": "string"
},
"ipaddr4": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"type": "ip"
},
"ipaddr6": {
"index": "not_analyzed",
"type": "string"
},
"kubernetes": {
"properties": {
"host": {
"index": "not_analyzed",
"norms": {
"enabled": true
},
"type": "string"
},
"namespace_id": {
"index": "not_analyzed",
"norms": {
"enabled": true
},
"type": "string"
},
"namespace_name": {
"index": "not_analyzed",
"norms": {
"enabled": true
},
"type": "string"
},
"pod_id": {
"index": "not_analyzed",
"norms": {
"enabled": true
},
"type": "string"
},
"pod_name": {
"index": "not_analyzed",
"norms": {
"enabled": true
},
"type": "string"
}
}
},
"level": {
"index": "not_analyzed",
"type": "string"
},
"message": {
"norms": {
"enabled": false
},
"type": "string"
},
"offset": {
"type": "long"
},
"pid": {
"index": "not_analyzed",
"type": "string"
},
"pipeline_metadata": {
"properties": {
"@version": {
"index": "not_analyzed",
"type": "string"
},
"collector": {
"properties": {
"hostname": {
"index": "not_analyzed",
"type": "string"
},
"inputname": {
"index": "not_analyzed",
"type": "string"
},
"ipaddr4": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"type": "ip"
},
"ipaddr6": {
"index": "not_analyzed",
"type": "string"
},
"name": {
"index": "not_analyzed",
"type": "string"
},
"original_raw_message": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"type": "string"
},
"received_at": {
"format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime",
"type": "date"
},
"version": {
"index": "not_analyzed",
"type": "string"
}
}
},
"normalizer": {
"properties": {
"hostname": {
"index": "not_analyzed",
"type": "string"
},
"inputname": {
"index": "not_analyzed",
"type": "string"
},
"ipaddr4": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"type": "ip"
},
"ipaddr6": {
"index": "not_analyzed",
"type": "string"
},
"name": {
"index": "not_analyzed",
"type": "string"
},
"original_raw_message": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"type": "string"
},
"received_at": {
"format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime",
"type": "date"
},
"version": {
"index": "not_analyzed",
"type": "string"
}
}
},
"trace": {
"analyzer": "whitespace",
"type": "string"
}
}
},
"rsyslog": {
"properties": {
"appname": {
"index": "not_analyzed",
"type": "string"
},
"facility": {
"index": "not_analyzed",
"type": "string"
},
"msgid": {
"type": "long"
},
"protocol-version": {
"index": "not_analyzed",
"type": "string"
},
"structured-data": {
"index": "not_analyzed",
"type": "string"
}
}
},
"service": {
"index": "not_analyzed",
"type": "string"
},
"systemd": {
"properties": {
"k": {
"properties": {
"KERNEL_DEVICE": {
"index": "not_analyzed",
"type": "string"
},
"KERNEL_SUBSYSTEM": {
"index": "not_analyzed",
"type": "string"
},
"UDEV_DEVLINK": {
"index": "not_analyzed",
"type": "string"
},
"UDEV_DEVNODE": {
"index": "not_analyzed",
"type": "string"
},
"UDEV_SYSNAME": {
"index": "not_analyzed",
"type": "string"
}
}
},
"t": {
"properties": {
"AUDIT_LOGINUID": {
"index": "not_analyzed",
"type": "string"
},
"AUDIT_SESSION": {
"index": "not_analyzed",
"type": "string"
},
"BOOT_ID": {
"index": "not_analyzed",
"type": "string"
},
"CAP_EFFECTIVE": {
"index": "not_analyzed",
"type": "string"
},
"CMDLINE": {
"index": "not_analyzed",
"type": "string"
},
"COMM": {
"index": "not_analyzed",
"type": "string"
},
"EXE": {
"index": "not_analyzed",
"type": "string"
},
"GID": {
"index": "not_analyzed",
"type": "string"
},
"HOSTNAME": {
"index": "not_analyzed",
"type": "string"
},
"MACHINE_ID": {
"index": "not_analyzed",
"type": "string"
},
"PID": {
"index": "not_analyzed",
"type": "string"
},
"SELINUX_CONTEXT": {
"index": "not_analyzed",
"type": "string"
},
"SOURCE_REALTIME_TIMESTAMP": {
"index": "not_analyzed",
"type": "string"
},
"SYSTEMD_CGROUP": {
"index": "not_analyzed",
"type": "string"
},
"SYSTEMD_OWNER_UID": {
"index": "not_analyzed",
"type": "string"
},
"SYSTEMD_SESSION": {
"index": "not_analyzed",
"type": "string"
},
"SYSTEMD_SLICE": {
"index": "not_analyzed",
"type": "string"
},
"SYSTEMD_UNIT": {
"index": "not_analyzed",
"type": "string"
},
"SYSTEMD_USER_UNIT": {
"index": "not_analyzed",
"type": "string"
},
"TRANSPORT": {
"index": "not_analyzed",
"type": "string"
},
"UID": {
"index": "not_analyzed",
"type": "string"
}
}
},
"u": {
"properties": {
"CODE_FILE": {
"index": "not_analyzed",
"type": "string"
},
"CODE_FUNCTION": {
"index": "not_analyzed",
"type": "string"
},
"CODE_LINE": {
"index": "not_analyzed",
"type": "string"
},
"ERRNO": {
"index": "not_analyzed",
"type": "string"
},
"MESSAGE_ID": {
"index": "not_analyzed",
"type": "string"
},
"RESULT": {
"index": "not_analyzed",
"type": "string"
},
"UNIT": {
"index": "not_analyzed",
"type": "string"
}
}
}
}
},
"tags": {
"analyzer": "whitespace",
"type": "string"
}
}
},
"com.redhat.viaq.common": {
"_meta": {
"version": "2016.10.12.0"
},
"date_detection": false,
"dynamic_templates": [
{
"message_field": {
"mapping": {
"index": "analyzed",
"omit_norms": true,
"type": "string"
},
"match": "message",
"match_mapping_type": "string"
}
},
{
"string_fields": {
"mapping": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"index": "analyzed",
"omit_norms": true,
"type": "string"
},
"match": "*",
"match_mapping_type": "string"
}
}
],
"properties": {
"@timestamp": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime",
"type": "date"
},
"_SOURCE_MONOTONIC_TIMESTAMP": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"norms": {
"enabled": false
},
"type": "string"
},
"docker": {
"properties": {
"container_id": {
"index": "not_analyzed",
"type": "string"
}
}
},
"file": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"type": "string"
},
"geoip": {
"dynamic": "true",
"properties": {
"location": {
"type": "geo_point"
}
}
},
"hostname": {
"index": "not_analyzed",
"type": "string"
},
"ipaddr4": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"type": "ip"
},
"ipaddr6": {
"index": "not_analyzed",
"type": "string"
},
"kubernetes": {
"properties": {
"host": {
"index": "not_analyzed",
"norms": {
"enabled": true
},
"type": "string"
},
"namespace_id": {
"index": "not_analyzed",
"norms": {
"enabled": true
},
"type": "string"
},
"namespace_name": {
"index": "not_analyzed",
"norms": {
"enabled": true
},
"type": "string"
},
"pod_id": {
"index": "not_analyzed",
"norms": {
"enabled": true
},
"type": "string"
},
"pod_name": {
"index": "not_analyzed",
"norms": {
"enabled": true
},
"type": "string"
}
}
},
"level": {
"index": "not_analyzed",
"type": "string"
},
"message": {
"norms": {
"enabled": false
},
"type": "string"
},
"offset": {
"type": "long"
},
"pid": {
"index": "not_analyzed",
"type": "string"
},
"pipeline_metadata": {
"properties": {
"@version": {
"index": "not_analyzed",
"type": "string"
},
"collector": {
"properties": {
"hostname": {
"index": "not_analyzed",
"type": "string"
},
"inputname": {
"index": "not_analyzed",
"type": "string"
},
"ipaddr4": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"type": "ip"
},
"ipaddr6": {
"index": "not_analyzed",
"type": "string"
},
"name": {
"index": "not_analyzed",
"type": "string"
},
"original_raw_message": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"type": "string"
},
"received_at": {
"format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime",
"type": "date"
},
"version": {
"index": "not_analyzed",
"type": "string"
}
}
},
"normalizer": {
"properties": {
"hostname": {
"index": "not_analyzed",
"type": "string"
},
"inputname": {
"index": "not_analyzed",
"type": "string"
},
"ipaddr4": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"type": "ip"
},
"ipaddr6": {
"index": "not_analyzed",
"type": "string"
},
"name": {
"index": "not_analyzed",
"type": "string"
},
"original_raw_message": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"type": "string"
},
"received_at": {
"format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime",
"type": "date"
},
"version": {
"index": "not_analyzed",
"type": "string"
}
}
},
"trace": {
"analyzer": "whitespace",
"type": "string"
}
}
},
"rsyslog": {
"properties": {
"appname": {
"index": "not_analyzed",
"type": "string"
},
"facility": {
"index": "not_analyzed",
"type": "string"
},
"msgid": {
"type": "long"
},
"protocol-version": {
"index": "not_analyzed",
"type": "string"
},
"structured-data": {
"index": "not_analyzed",
"type": "string"
}
}
},
"service": {
"index": "not_analyzed",
"type": "string"
},
"systemd": {
"properties": {
"k": {
"properties": {
"KERNEL_DEVICE": {
"index": "not_analyzed",
"type": "string"
},
"KERNEL_SUBSYSTEM": {
"index": "not_analyzed",
"type": "string"
},
"UDEV_DEVLINK": {
"index": "not_analyzed",
"type": "string"
},
"UDEV_DEVNODE": {
"index": "not_analyzed",
"type": "string"
},
"UDEV_SYSNAME": {
"index": "not_analyzed",
"type": "string"
}
}
},
"t": {
"properties": {
"AUDIT_LOGINUID": {
"index": "not_analyzed",
"type": "string"
},
"AUDIT_SESSION": {
"index": "not_analyzed",
"type": "string"
},
"BOOT_ID": {
"index": "not_analyzed",
"type": "string"
},
"CAP_EFFECTIVE": {
"index": "not_analyzed",
"type": "string"
},
"CMDLINE": {
"index": "not_analyzed",
"type": "string"
},
"COMM": {
"index": "not_analyzed",
"type": "string"
},
"EXE": {
"index": "not_analyzed",
"type": "string"
},
"GID": {
"index": "not_analyzed",
"type": "string"
},
"HOSTNAME": {
"index": "not_analyzed",
"type": "string"
},
"MACHINE_ID": {
"index": "not_analyzed",
"type": "string"
},
"PID": {
"index": "not_analyzed",
"type": "string"
},
"SELINUX_CONTEXT": {
"index": "not_analyzed",
"type": "string"
},
"SOURCE_REALTIME_TIMESTAMP": {
"index": "not_analyzed",
"type": "string"
},
"SYSTEMD_CGROUP": {
"index": "not_analyzed",
"type": "string"
},
"SYSTEMD_OWNER_UID": {
"index": "not_analyzed",
"type": "string"
},
"SYSTEMD_SESSION": {
"index": "not_analyzed",
"type": "string"
},
"SYSTEMD_SLICE": {
"index": "not_analyzed",
"type": "string"
},
"SYSTEMD_UNIT": {
"index": "not_analyzed",
"type": "string"
},
"SYSTEMD_USER_UNIT": {
"index": "not_analyzed",
"type": "string"
},
"TRANSPORT": {
"index": "not_analyzed",
"type": "string"
},
"UID": {
"index": "not_analyzed",
"type": "string"
}
}
},
"u": {
"properties": {
"CODE_FILE": {
"index": "not_analyzed",
"type": "string"
},
"CODE_FUNCTION": {
"index": "not_analyzed",
"type": "string"
},
"CODE_LINE": {
"index": "not_analyzed",
"type": "string"
},
"ERRNO": {
"index": "not_analyzed",
"type": "string"
},
"MESSAGE_ID": {
"index": "not_analyzed",
"type": "string"
},
"RESULT": {
"index": "not_analyzed",
"type": "string"
},
"SYSLOG_FACILITY": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"norms": {
"enabled": false
},
"type": "string"
},
"SYSLOG_IDENTIFIER": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"norms": {
"enabled": false
},
"type": "string"
},
"SYSLOG_PID": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"norms": {
"enabled": false
},
"type": "string"
},
"UNIT": {
"index": "not_analyzed",
"type": "string"
}
}
}
}
},
"tags": {
"analyzer": "whitespace",
"type": "string"
},
"time": {
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
},
"norms": {
"enabled": false
},
"type": "string"
}
}
}
}
}
}
Thanks Junqi, can you also take a look in the "Settings" tab of Kibana to see how the default search pattern is defined, and what the "fields" are? If you see the time field, it likely defined there as a number. If you refresh the field patterns in Kibana, that might fix it. That is, if that is the problem. I am just guessing at this point. Kibana version is 4.6.4 for logging 3.5, checked the fields according to comment 6, type is 'date' for time field, and can not change type in Kibana UI, so this error can not be fixed by changing type in Kibana UI, I think it should be fixed in mappings. See the attached pictures. Created attachment 1248763 [details]
kibana settings tab picture
attached kibana settings tab picture
Created attachment 1248764 [details]
disable to change field type
So on the settings tab, hitting "refresh" does not change the field to reflect what's in the Elasticsearch mapping? If that is the case, can you tell me how many different .operations-* indices there are? I suspect one or more of them has "time" in its mapping as a date field, and the rest have it as a string. If so, I think we can promote the time field to a string in that set of indices to fix this problem. We need to turn off automatic date detect for fields [1] that are not in our mapping. [1] http://stackoverflow.com/questions/14813379/disable-date-detection-in-tires-elasticsearch-mapping There should be no field named "time". There should only be a field named "@timestamp". I suspect fluentd is having problems. Can you provide the fluentd log? Rich, regardless, if we allow elasticsearch to detect date fields from strings, then this problem can happen with any string field that is encountered. We have to turn that off. (In reply to Peter Portante from comment #12) > Rich, regardless, if we allow elasticsearch to detect date fields from > strings, then this problem can happen with any string field that is > encountered. We have to turn that off. Ok, how do we do that? I suspect the problem is related to https://bugzilla.redhat.com/show_bug.cgi?id=1420219 which can also be caused if there is a top level field called "time" but no top level field called "@timestamp". See comment 10. (In reply to Peter Portante from comment #14) > See comment 10. So, since this is an "unknown unknown" we have to disable date detection for all fields? "mappings" : { "_default_" : { "date_detection" : 0 } } Does that mean we can still do date detection for date valued fields such as @timestamp, pipeline_metadata.collector.received_at, etc? Anton, is there support in the common data model for this? If not, we need to add it when we generate index templates. The "@timestamp" field is missing because the common data model filter is missing. Here is the fluent.conf as deployed by ansible 3.5: @include configs.d/openshift/filter-syslog-record-transform.conf @include configs.d/openshift/filter-post-*.conf It should look like this: @include configs.d/openshift/filter-syslog-record-transform.conf @include configs.d/openshift/filter-common-data-model.conf @include configs.d/openshift/filter-post-*.conf PR: https://github.com/openshift/openshift-ansible/pull/3323 Commits pushed to master at https://github.com/openshift/openshift-ansible https://github.com/openshift/openshift-ansible/commit/f9daaa768ec415c2ae83a8c42ae35776d996083e Bug 1420234 - illegal_argument_exception in Kibana UI. https://bugzilla.redhat.com/show_bug.cgi?id=1420234 The problem is that the fluent.conf is missing the common data model filter which renames the "time" field to the "@timestamp" field. https://github.com/openshift/openshift-ansible/commit/9e92660ed86623385e48c4960f4db681d86a7a57 Merge pull request #3323 from richm/missing-common-data-model-filter Bug 1420234 - illegal_argument_exception in Kibana UI. Moving to MODIFIED because fix committed upstream - not sure if QE will test with upstream or will wait for RH openshift-ansible package to contain fix. @rmeggins Verified with the latest logging 3.5 openshift-ansible, this defect is get fixed, please feel free to change the status to ON_QA, then I can close it. thanks. close this defect according to Comment 20 Since this bug never reached customers, I am closing it. |