Hide Forgot
Created attachment 1248567 [details] illegal_argument_exception in Kibana UI Description of problem: Deployed logging stack with ansible, illegal_argument_exception shows when logon Kibana UI. # oc get po NAME READY STATUS RESTARTS AGE logging-curator-1-dslwb 1/1 Running 1 40m logging-es-9nmvgerr-1-92ttn 1/1 Running 0 40m logging-fluentd-qf0xc 1/1 Running 0 41m logging-kibana-1-dmjsf 2/2 Running 0 40m Version-Release number of selected component (if applicable): # openshift version openshift v3.5.0.17+c55cf2b kubernetes v1.5.2+43a9be4 etcd 3.1.0 Image id: openshift3/logging-elasticsearch 7605f043d232 openshift3/logging-kibana e0ab09c2cbeb openshift3/logging-fluentd 47057624ecab openshift3/logging-auth-proxy 139f7943475e openshift3/logging-curator 7f034fdf7702 How reproducible: Always Steps to Reproduce: 1. prepare the inventory file [oo_first_master] $master-public-dns ansible_user=root ansible_ssh_user=root ansible_ssh_private_key_file="~/cfile/libra.pem" openshift_public_hostname=$master-public-dns [oo_first_master:vars] deployment_type=openshift-enterprise openshift_release=v3.5.0 openshift_logging_install_logging=true openshift_logging_kibana_hostname=kibana.$sub-domain public_master_url=https://$master-public-dns:8443 openshift_logging_image_prefix=registry.ops.openshift.com/openshift3/ openshift_logging_image_version=3.5.0 openshift_logging_namespace=juzhao 2. Running the playbook from a control machine (my laptop) which is not oo_master: git clone https://github.com/openshift/openshift-ansible ansible-playbook -vvv -i ~/inventory playbooks/common/openshift-cluster/openshift_logging.yml 3. Log on Kibana UI to check logs. Actual results: Error shows, Discover: Expected numeric type on field [time], but got [string], click "more info" to see error trace Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Expected numeric type on field [time], but got [string]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query_fetch","grouped":true,"failed_shards":[{"shard":0,"index":".operations.2017.02.08","node":"H_KsakBYQC2G1Y2rTScUAA","reason":{"type":"illegal_argument_exception","reason":"Expected numeric type on field [time], but got [string]"}}]}} KbnError@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:57511:21 RequestFailure@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:57544:6 callResponseHandlers/<@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:98389:39 __WEBPACK_AMD_DEFINE_RESULT__</</Promise.try@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:101829:20 __WEBPACK_AMD_DEFINE_RESULT__</</Promise.map/<@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:101798:17 __WEBPACK_AMD_DEFINE_RESULT__</</Promise.map@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:101797:27 callResponseHandlers@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:98361:15 fetchWithStrategy/<@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:97862:17 processQueue@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:42452:29 scheduleProcessQueue/<@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:42468:28 $RootScopeProvider/this.$get</Scope.prototype.$eval@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:43696:17 $RootScopeProvider/this.$get</Scope.prototype.$digest@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:43507:16 $RootScopeProvider/this.$get</Scope.prototype.$apply@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:43804:14 done@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:38253:37 completeRequest@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:38451:8 requestLoaded@https://kibana.0208-35z.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:38392:10 Expected results: no error in Kibana UI Additional info: Attached Kibana UI snapshot
Was this error seen immediately upon log in to kibana or did you do any additional steps?
I did not do any additional steps, the error shows after logging in Kibana. It seems like the type for time is not proper
This is a lot like a mapping did not get setup for an index properly. Error: Request to Elasticsearch failed: { "error": { "root_cause": [ { "type": "illegal_argument_exception", "reason": "Expected numeric type on field [time], but got [string]" } ], "type": "search_phase_execution_exception", "reason": "all shards failed", "phase": "query_fetch", "grouped": true, "failed_shards": [ { "shard": 0, "index": ".operations.2017.02.08", "node": "H_KsakBYQC2G1Y2rTScUAA", "reason": { "type": "illegal_argument_exception", "reason": "Expected numeric type on field [time], but got [string]" } } ] } } } We need to look at the mappings for index .operations.2017.02.08 and what Kibana thinks that field mapping is. What URL for kibana was used exactly?
(In reply to Peter Portante from comment #3) > This is a lot like a mapping did not get setup for an index properly. > > Error: Request to Elasticsearch failed: > { "error": > { "root_cause": [ > { "type": "illegal_argument_exception", > "reason": "Expected numeric type on field [time], but got > [string]" > } > ], > "type": "search_phase_execution_exception", > "reason": "all shards failed", > "phase": "query_fetch", > "grouped": true, > "failed_shards": [ > { "shard": 0, > "index": ".operations.2017.02.08", > "node": "H_KsakBYQC2G1Y2rTScUAA", > "reason": { > "type": "illegal_argument_exception", > "reason": "Expected numeric type on field [time], but got > [string]" > } > } > ] > } > } > } > > We need to look at the mappings for index .operations.2017.02.08 and what > Kibana thinks that field mapping is. > > What URL for kibana was used exactly? Environment was shutdown yesterday, I will provide info if this error shows today. $SUBDOMAIN = `grep subdomain /etc/origin/master/master-config.yaml` kibana URL is https://kibana.$SUBDOMAIN
Same error shows today: Discover: Expected numeric type on field [time], but got [string] Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Expected numeric type on field [time], but got [string]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query_fetch","grouped":true,"failed_shards":[{"shard":0,"index":".operations.2017.02.09","node":"5fjnOvDmQvG2Tf4X3bCc8A","reason":{"type":"illegal_argument_exception","reason":"Expected numeric type on field [time], but got [string]"}}]}} KbnError@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:57511:21 RequestFailure@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:57544:6 callResponseHandlers/<@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:98389:39 __WEBPACK_AMD_DEFINE_RESULT__</</Promise.try@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:101829:20 __WEBPACK_AMD_DEFINE_RESULT__</</Promise.map/<@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:101798:17 __WEBPACK_AMD_DEFINE_RESULT__</</Promise.map@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:101797:27 callResponseHandlers@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:98361:15 fetchWithStrategy/<@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:97862:17 processQueue@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:42452:29 scheduleProcessQueue/<@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:42468:28 $RootScopeProvider/this.$get</Scope.prototype.$eval@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:43696:17 $RootScopeProvider/this.$get</Scope.prototype.$digest@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:43507:16 $RootScopeProvider/this.$get</Scope.prototype.$apply@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:43804:14 done@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:38253:37 completeRequest@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:38451:8 requestLoaded@https://kibana.0209-xe4.qe.rhcloud.com/bundles/commons.bundle.js?v=10229:38392:10 mappings for index .operations.2017.02.09 { ".operations.2017.02.09": { "mappings": { "_default_": { "_meta": { "version": "2016.10.12.0" }, "date_detection": false, "dynamic_templates": [ { "message_field": { "mapping": { "index": "analyzed", "omit_norms": true, "type": "string" }, "match": "message", "match_mapping_type": "string" } }, { "string_fields": { "mapping": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "index": "analyzed", "omit_norms": true, "type": "string" }, "match": "*", "match_mapping_type": "string" } } ], "properties": { "@timestamp": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime", "type": "date" }, "docker": { "properties": { "container_id": { "index": "not_analyzed", "type": "string" } } }, "file": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "type": "string" }, "geoip": { "dynamic": "true", "properties": { "location": { "type": "geo_point" } } }, "hostname": { "index": "not_analyzed", "type": "string" }, "ipaddr4": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "type": "ip" }, "ipaddr6": { "index": "not_analyzed", "type": "string" }, "kubernetes": { "properties": { "host": { "index": "not_analyzed", "norms": { "enabled": true }, "type": "string" }, "namespace_id": { "index": "not_analyzed", "norms": { "enabled": true }, "type": "string" }, "namespace_name": { "index": "not_analyzed", "norms": { "enabled": true }, "type": "string" }, "pod_id": { "index": "not_analyzed", "norms": { "enabled": true }, "type": "string" }, "pod_name": { "index": "not_analyzed", "norms": { "enabled": true }, "type": "string" } } }, "level": { "index": "not_analyzed", "type": "string" }, "message": { "norms": { "enabled": false }, "type": "string" }, "offset": { "type": "long" }, "pid": { "index": "not_analyzed", "type": "string" }, "pipeline_metadata": { "properties": { "@version": { "index": "not_analyzed", "type": "string" }, "collector": { "properties": { "hostname": { "index": "not_analyzed", "type": "string" }, "inputname": { "index": "not_analyzed", "type": "string" }, "ipaddr4": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "type": "ip" }, "ipaddr6": { "index": "not_analyzed", "type": "string" }, "name": { "index": "not_analyzed", "type": "string" }, "original_raw_message": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "type": "string" }, "received_at": { "format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime", "type": "date" }, "version": { "index": "not_analyzed", "type": "string" } } }, "normalizer": { "properties": { "hostname": { "index": "not_analyzed", "type": "string" }, "inputname": { "index": "not_analyzed", "type": "string" }, "ipaddr4": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "type": "ip" }, "ipaddr6": { "index": "not_analyzed", "type": "string" }, "name": { "index": "not_analyzed", "type": "string" }, "original_raw_message": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "type": "string" }, "received_at": { "format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime", "type": "date" }, "version": { "index": "not_analyzed", "type": "string" } } }, "trace": { "analyzer": "whitespace", "type": "string" } } }, "rsyslog": { "properties": { "appname": { "index": "not_analyzed", "type": "string" }, "facility": { "index": "not_analyzed", "type": "string" }, "msgid": { "type": "long" }, "protocol-version": { "index": "not_analyzed", "type": "string" }, "structured-data": { "index": "not_analyzed", "type": "string" } } }, "service": { "index": "not_analyzed", "type": "string" }, "systemd": { "properties": { "k": { "properties": { "KERNEL_DEVICE": { "index": "not_analyzed", "type": "string" }, "KERNEL_SUBSYSTEM": { "index": "not_analyzed", "type": "string" }, "UDEV_DEVLINK": { "index": "not_analyzed", "type": "string" }, "UDEV_DEVNODE": { "index": "not_analyzed", "type": "string" }, "UDEV_SYSNAME": { "index": "not_analyzed", "type": "string" } } }, "t": { "properties": { "AUDIT_LOGINUID": { "index": "not_analyzed", "type": "string" }, "AUDIT_SESSION": { "index": "not_analyzed", "type": "string" }, "BOOT_ID": { "index": "not_analyzed", "type": "string" }, "CAP_EFFECTIVE": { "index": "not_analyzed", "type": "string" }, "CMDLINE": { "index": "not_analyzed", "type": "string" }, "COMM": { "index": "not_analyzed", "type": "string" }, "EXE": { "index": "not_analyzed", "type": "string" }, "GID": { "index": "not_analyzed", "type": "string" }, "HOSTNAME": { "index": "not_analyzed", "type": "string" }, "MACHINE_ID": { "index": "not_analyzed", "type": "string" }, "PID": { "index": "not_analyzed", "type": "string" }, "SELINUX_CONTEXT": { "index": "not_analyzed", "type": "string" }, "SOURCE_REALTIME_TIMESTAMP": { "index": "not_analyzed", "type": "string" }, "SYSTEMD_CGROUP": { "index": "not_analyzed", "type": "string" }, "SYSTEMD_OWNER_UID": { "index": "not_analyzed", "type": "string" }, "SYSTEMD_SESSION": { "index": "not_analyzed", "type": "string" }, "SYSTEMD_SLICE": { "index": "not_analyzed", "type": "string" }, "SYSTEMD_UNIT": { "index": "not_analyzed", "type": "string" }, "SYSTEMD_USER_UNIT": { "index": "not_analyzed", "type": "string" }, "TRANSPORT": { "index": "not_analyzed", "type": "string" }, "UID": { "index": "not_analyzed", "type": "string" } } }, "u": { "properties": { "CODE_FILE": { "index": "not_analyzed", "type": "string" }, "CODE_FUNCTION": { "index": "not_analyzed", "type": "string" }, "CODE_LINE": { "index": "not_analyzed", "type": "string" }, "ERRNO": { "index": "not_analyzed", "type": "string" }, "MESSAGE_ID": { "index": "not_analyzed", "type": "string" }, "RESULT": { "index": "not_analyzed", "type": "string" }, "UNIT": { "index": "not_analyzed", "type": "string" } } } } }, "tags": { "analyzer": "whitespace", "type": "string" } } }, "com.redhat.viaq.common": { "_meta": { "version": "2016.10.12.0" }, "date_detection": false, "dynamic_templates": [ { "message_field": { "mapping": { "index": "analyzed", "omit_norms": true, "type": "string" }, "match": "message", "match_mapping_type": "string" } }, { "string_fields": { "mapping": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "index": "analyzed", "omit_norms": true, "type": "string" }, "match": "*", "match_mapping_type": "string" } } ], "properties": { "@timestamp": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime", "type": "date" }, "_SOURCE_MONOTONIC_TIMESTAMP": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "norms": { "enabled": false }, "type": "string" }, "docker": { "properties": { "container_id": { "index": "not_analyzed", "type": "string" } } }, "file": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "type": "string" }, "geoip": { "dynamic": "true", "properties": { "location": { "type": "geo_point" } } }, "hostname": { "index": "not_analyzed", "type": "string" }, "ipaddr4": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "type": "ip" }, "ipaddr6": { "index": "not_analyzed", "type": "string" }, "kubernetes": { "properties": { "host": { "index": "not_analyzed", "norms": { "enabled": true }, "type": "string" }, "namespace_id": { "index": "not_analyzed", "norms": { "enabled": true }, "type": "string" }, "namespace_name": { "index": "not_analyzed", "norms": { "enabled": true }, "type": "string" }, "pod_id": { "index": "not_analyzed", "norms": { "enabled": true }, "type": "string" }, "pod_name": { "index": "not_analyzed", "norms": { "enabled": true }, "type": "string" } } }, "level": { "index": "not_analyzed", "type": "string" }, "message": { "norms": { "enabled": false }, "type": "string" }, "offset": { "type": "long" }, "pid": { "index": "not_analyzed", "type": "string" }, "pipeline_metadata": { "properties": { "@version": { "index": "not_analyzed", "type": "string" }, "collector": { "properties": { "hostname": { "index": "not_analyzed", "type": "string" }, "inputname": { "index": "not_analyzed", "type": "string" }, "ipaddr4": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "type": "ip" }, "ipaddr6": { "index": "not_analyzed", "type": "string" }, "name": { "index": "not_analyzed", "type": "string" }, "original_raw_message": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "type": "string" }, "received_at": { "format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime", "type": "date" }, "version": { "index": "not_analyzed", "type": "string" } } }, "normalizer": { "properties": { "hostname": { "index": "not_analyzed", "type": "string" }, "inputname": { "index": "not_analyzed", "type": "string" }, "ipaddr4": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "type": "ip" }, "ipaddr6": { "index": "not_analyzed", "type": "string" }, "name": { "index": "not_analyzed", "type": "string" }, "original_raw_message": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "type": "string" }, "received_at": { "format": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ||yyyy-MM-dd'T'HH:mm:ssZ||dateOptionalTime", "type": "date" }, "version": { "index": "not_analyzed", "type": "string" } } }, "trace": { "analyzer": "whitespace", "type": "string" } } }, "rsyslog": { "properties": { "appname": { "index": "not_analyzed", "type": "string" }, "facility": { "index": "not_analyzed", "type": "string" }, "msgid": { "type": "long" }, "protocol-version": { "index": "not_analyzed", "type": "string" }, "structured-data": { "index": "not_analyzed", "type": "string" } } }, "service": { "index": "not_analyzed", "type": "string" }, "systemd": { "properties": { "k": { "properties": { "KERNEL_DEVICE": { "index": "not_analyzed", "type": "string" }, "KERNEL_SUBSYSTEM": { "index": "not_analyzed", "type": "string" }, "UDEV_DEVLINK": { "index": "not_analyzed", "type": "string" }, "UDEV_DEVNODE": { "index": "not_analyzed", "type": "string" }, "UDEV_SYSNAME": { "index": "not_analyzed", "type": "string" } } }, "t": { "properties": { "AUDIT_LOGINUID": { "index": "not_analyzed", "type": "string" }, "AUDIT_SESSION": { "index": "not_analyzed", "type": "string" }, "BOOT_ID": { "index": "not_analyzed", "type": "string" }, "CAP_EFFECTIVE": { "index": "not_analyzed", "type": "string" }, "CMDLINE": { "index": "not_analyzed", "type": "string" }, "COMM": { "index": "not_analyzed", "type": "string" }, "EXE": { "index": "not_analyzed", "type": "string" }, "GID": { "index": "not_analyzed", "type": "string" }, "HOSTNAME": { "index": "not_analyzed", "type": "string" }, "MACHINE_ID": { "index": "not_analyzed", "type": "string" }, "PID": { "index": "not_analyzed", "type": "string" }, "SELINUX_CONTEXT": { "index": "not_analyzed", "type": "string" }, "SOURCE_REALTIME_TIMESTAMP": { "index": "not_analyzed", "type": "string" }, "SYSTEMD_CGROUP": { "index": "not_analyzed", "type": "string" }, "SYSTEMD_OWNER_UID": { "index": "not_analyzed", "type": "string" }, "SYSTEMD_SESSION": { "index": "not_analyzed", "type": "string" }, "SYSTEMD_SLICE": { "index": "not_analyzed", "type": "string" }, "SYSTEMD_UNIT": { "index": "not_analyzed", "type": "string" }, "SYSTEMD_USER_UNIT": { "index": "not_analyzed", "type": "string" }, "TRANSPORT": { "index": "not_analyzed", "type": "string" }, "UID": { "index": "not_analyzed", "type": "string" } } }, "u": { "properties": { "CODE_FILE": { "index": "not_analyzed", "type": "string" }, "CODE_FUNCTION": { "index": "not_analyzed", "type": "string" }, "CODE_LINE": { "index": "not_analyzed", "type": "string" }, "ERRNO": { "index": "not_analyzed", "type": "string" }, "MESSAGE_ID": { "index": "not_analyzed", "type": "string" }, "RESULT": { "index": "not_analyzed", "type": "string" }, "SYSLOG_FACILITY": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "norms": { "enabled": false }, "type": "string" }, "SYSLOG_IDENTIFIER": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "norms": { "enabled": false }, "type": "string" }, "SYSLOG_PID": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "norms": { "enabled": false }, "type": "string" }, "UNIT": { "index": "not_analyzed", "type": "string" } } } } }, "tags": { "analyzer": "whitespace", "type": "string" }, "time": { "fields": { "raw": { "ignore_above": 256, "index": "not_analyzed", "type": "string" } }, "norms": { "enabled": false }, "type": "string" } } } } } }
Thanks Junqi, can you also take a look in the "Settings" tab of Kibana to see how the default search pattern is defined, and what the "fields" are? If you see the time field, it likely defined there as a number. If you refresh the field patterns in Kibana, that might fix it. That is, if that is the problem. I am just guessing at this point.
Kibana version is 4.6.4 for logging 3.5, checked the fields according to comment 6, type is 'date' for time field, and can not change type in Kibana UI, so this error can not be fixed by changing type in Kibana UI, I think it should be fixed in mappings. See the attached pictures.
Created attachment 1248763 [details] kibana settings tab picture attached kibana settings tab picture
Created attachment 1248764 [details] disable to change field type
So on the settings tab, hitting "refresh" does not change the field to reflect what's in the Elasticsearch mapping? If that is the case, can you tell me how many different .operations-* indices there are? I suspect one or more of them has "time" in its mapping as a date field, and the rest have it as a string. If so, I think we can promote the time field to a string in that set of indices to fix this problem. We need to turn off automatic date detect for fields [1] that are not in our mapping. [1] http://stackoverflow.com/questions/14813379/disable-date-detection-in-tires-elasticsearch-mapping
There should be no field named "time". There should only be a field named "@timestamp". I suspect fluentd is having problems. Can you provide the fluentd log?
Rich, regardless, if we allow elasticsearch to detect date fields from strings, then this problem can happen with any string field that is encountered. We have to turn that off.
(In reply to Peter Portante from comment #12) > Rich, regardless, if we allow elasticsearch to detect date fields from > strings, then this problem can happen with any string field that is > encountered. We have to turn that off. Ok, how do we do that? I suspect the problem is related to https://bugzilla.redhat.com/show_bug.cgi?id=1420219 which can also be caused if there is a top level field called "time" but no top level field called "@timestamp".
See comment 10.
(In reply to Peter Portante from comment #14) > See comment 10. So, since this is an "unknown unknown" we have to disable date detection for all fields? "mappings" : { "_default_" : { "date_detection" : 0 } } Does that mean we can still do date detection for date valued fields such as @timestamp, pipeline_metadata.collector.received_at, etc? Anton, is there support in the common data model for this? If not, we need to add it when we generate index templates.
The "@timestamp" field is missing because the common data model filter is missing. Here is the fluent.conf as deployed by ansible 3.5: @include configs.d/openshift/filter-syslog-record-transform.conf @include configs.d/openshift/filter-post-*.conf It should look like this: @include configs.d/openshift/filter-syslog-record-transform.conf @include configs.d/openshift/filter-common-data-model.conf @include configs.d/openshift/filter-post-*.conf PR: https://github.com/openshift/openshift-ansible/pull/3323
Commits pushed to master at https://github.com/openshift/openshift-ansible https://github.com/openshift/openshift-ansible/commit/f9daaa768ec415c2ae83a8c42ae35776d996083e Bug 1420234 - illegal_argument_exception in Kibana UI. https://bugzilla.redhat.com/show_bug.cgi?id=1420234 The problem is that the fluent.conf is missing the common data model filter which renames the "time" field to the "@timestamp" field. https://github.com/openshift/openshift-ansible/commit/9e92660ed86623385e48c4960f4db681d86a7a57 Merge pull request #3323 from richm/missing-common-data-model-filter Bug 1420234 - illegal_argument_exception in Kibana UI.
Moving to MODIFIED because fix committed upstream - not sure if QE will test with upstream or will wait for RH openshift-ansible package to contain fix.
blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1421011
@rmeggins Verified with the latest logging 3.5 openshift-ansible, this defect is get fixed, please feel free to change the status to ON_QA, then I can close it. thanks.
close this defect according to Comment 20
Since this bug never reached customers, I am closing it.