Description of problem: Kibana Visualisation errors out with "Visualize: Fielddata is disabled on text fields by default. Set fielddata=true on [kubernetes.container_name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively …" MYUSER is a cluster admin Version-Release number of selected component (if applicable): OCP 3.11 How reproducible: Always on a specific user environment. Steps to Reproduce: 1- Login to Kibana with MYUSER (cluster version v3.11.170) 2- Grant access to UI objects: `oc exec logging-es-data-master-k2xkskj5-1-mcwd4 -- es_load_kibana_ui_objects MYUSER ` 3- Login to Kiba and click on "Visualise" -> Select "Kubernetes Logs by Namespace, DC, Pod, Container" 4- It fails to load, and shows error at top of screen: "Visualize: Fielddata is disabled on text fields by default. Set fielddata=true on [kubernetes.container_name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively …" Actual results: Login to Kiba and click on "Visualise" -> Select "Kubernetes Logs by Namespace, DC, Pod, Container" It fails to load, and shows error at top of screen: "Visualize: Fielddata is disabled on text fields by default. Set fielddata=true on [kubernetes.container_name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively …" Expected results: We should be able to Visualize. Additional info: I can see multiple errors like this in the es.log "[2020-04-06T02:20:12,152][DEBUG][o.e.a.s.TransportSearchAction] [logging-es-data-master-jou9bxno] All shards failed for phase: [query] org.elasticsearch.ElasticsearchException$1: Fielddata is disabled on text fields by default. Set fielddata=true on [kubernetes.container_name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead." All indices are green health also looks good Any help on this will be highly appreciated . Things we tried -> 1. Also we tried increasing ES_RAM to 32 GB and it did not help ``` resources: limits: memory: 32Gi requests: cpu: "1" memory: 32Gi ``` which was 16Gi and the new parameter: ``` env: - name: INSTANCE_RAM value: 32G 2. We also tried deleting kibana index and restarting it's pods and it did not help cat es/cluster-es/health epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent 1586139934 02:25:34 logging-es green 3 3 460 230 0 0 0 0 - 100.0% [sgaikwad@sgaikwad logging-20200405_222205]$ cat es/cluster-es/nodes ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name XX.XX.8.28 37 96 2 1.28 1.09 1.22 mdi - logging-es-data-master-tvsfasj5 XX.XX.6.33 44 80 2 2.98 2.01 1.16 mdi - logging-es-data-master-jou9bxno XX.XX.8.9 18 96 1 1.28 1.09 1.22 mdi * logging-es-data-master-w14p3jos https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html#fielddata-disabled-text-fields Delete the kibana index and restart the kibana pod and see if it works. 3. We also tried this [root@master-2 ~]# oc describe ds/logging-fluentd | grep -i json MERGE_JSON_LOG: false and it did not help. Questions -> 1. What else can we to do workaround/resolve this issue ? 2. Will enabling Fielddata help ? If yes can you please share us the steps. Also please note that this a fresh cluster.
If user needs to perform query on this field and it requires fielddata (sorting or aggregation) then they should be using `kubernetes.container_name.raw` field instead. This raw field is not tokenized [1] and is treated as a keyword [2]. [1] https://github.com/openshift/origin-aggregated-logging/blob/master/elasticsearch/index_templates/com.redhat.viaq-openshift-operations.template.json#L256-L261 [2] https://www.elastic.co/guide/en/elasticsearch/reference/5.6/keyword.html
(In reply to Lukas Vlcek from comment #3) > If user needs to perform query on this field and it requires fielddata > (sorting or aggregation) then they should be using > `kubernetes.container_name.raw` field instead. This raw field is not > tokenized [1] and is treated as a keyword [2]. > > [1] > https://github.com/openshift/origin-aggregated-logging/blob/master/ > elasticsearch/index_templates/com.redhat.viaq-openshift-operations.template. > json#L256-L261 > [2] https://www.elastic.co/guide/en/elasticsearch/reference/5.6/keyword.html Can you please tell the steps to use kubernetes.container_name.raw ?
Here are my primitive steps, and my suspicion is since this raw field is not tokenised, I shouldn't be using 'MYUSER' index: ``` [root@master-2 ~]# curl -k -H "Authorization: Bearer $(oc whoami -t)" -X DELETE https://$es_pod_ip:9200/.kibana.974602627aec0ac9c80ba400b71c38a29eca3d89 {"acknowledged":true} [root@master-2 ~]# curl -k -H "Authorization: Bearer $(oc whoami -t)" -X PUT "https://$es_pod_ip:9200/.kibana.974602627aec0ac9c80ba400b71c38a29eca3d89" -H 'Content-Type: application/json' -d'{ > "mappings": { > "type": { > "properties": { > "kubernetes.container_name.raw": { > "type": "text", > "fielddata": true > } > } > } > } > }' {"acknowledged":true,"shards_acknowledged":true,"index":".kibana.974602627aec0ac9c80ba400b71c38a29eca3d89"} [root@master-2 ~]# [root@master-2 ~]# oc exec -it -c elasticsearch $es_pod -- es_load_kibana_ui_objects MYUSER [2020-04-08 22:27:43,332][INFO ][container.run ] Adding Kibana dashboards and other UI objects for user MYUSER index .kibana.974602627aec0ac9c80ba400b71c38a29eca3d89 [2020-04-08 22:27:43,334][INFO ][container.run ] Adding the index pattern for project.* . . . [2020-04-08 22:27:43,718][INFO ][container.run ] Adding the Kibana UI objects . . . [2020-04-08 22:27:44,087][INFO ][container.run ] Success ## I see this in the output. But still get the error. "type": { "properties": { "kubernetes": { "properties": { "container_name": { "properties": { "raw": { "fielddata": true, "type": "text" } } } } } } }, ```
Can you please share the error now? I want to make sure which field it is complaining about, does it still complain about `kubernetes.container_name` or `kubernetes.container_name.raw` now?
Here are the steps we followed, but unfortunately they do not persist even after saving the file. Attached as screenshots.
Hi, I tested this and it was fixed by changing "kubernetes.container_name" to "kubernetes.container_name.raw" and now get no error and can see nice pie chart :-) Kibana --> Management --> Saved Objects --> Visualizations Click "Kubernetes Logs by Namespace, DC, Pod, Container" Down bottom of page edit "visState" textbox adjusting "kubernetes.container_name" to "kubernetes.container_name.raw" { "id": "5", "type": "terms", "schema": "segment", "params": { "field": "kubernetes.container_name.raw", "size": 50, "order": "desc", "orderBy": "1" } } Click "Save visualization Object" Visualize --> Kubernetes Logs by Namespace, DC, Pod, Container
I have applied the PR located here: https://github.com/openshift/origin-aggregated-logging/pull/1878 to an Elastic pod running in OCP v3.11.200. From a host with access to the cluster and assuming that you have the patch file (attached here as k8s-visualizations-unified.patch, you can run the following commands to validate the patch works as intended: $ export ESPOD=$(oc get pods -n openshift-logging -o=jsonpath="{range .items[*]}{.metadata.name}{\"\n\"}{end}"| grep es) $ mkdir elasticsearch.$ESPOD $ oc rsync -n openshift-logging -c elasticsearch logging-es-data-master-qm79y90e-1-kcc6k:/usr/share/elasticsearch/kibana_ui_objects elasticsearch.$ESPOD/ kibana_ui_objects/k8s-dashboard.json kibana_ui_objects/k8s-search.json kibana_ui_objects/k8s-visualizations.json $ cp -a elasticsearch.$ESPOD elasticsearch $ patch -p0 < k8s-visualizations-unified.patch patching file elasticsearch/kibana_ui_objects/k8s-dashboard.json patching file elasticsearch/kibana_ui_objects/k8s-search.json patching file elasticsearch/kibana_ui_objects/k8s-visualizations.json $ oc rsync -n openshift-logging -c elasticsearch elasticsearch/kibana_ui_objects/ $ESPOD:/usr/share/elasticsearch/kibana_ui_objects/ $ oc exec -n openshift-logging $ESPOD -- es_load_kibana_ui_objects <USER NAME> Assuming you have applications running under that user that are generating logs, you should see that the dashboards are present and working as intended.
Created attachment 1680401 [details] A patch generated from the upstream GIT PR request. Attaching the patch used in the previous comment. k8s-visualizations-unified.patch
Verified, The user can navagated to kibana without any error.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days