Kibana doesn't show project data. On fluentd pods We can see some errors: 7360f9d4b5e9605d"}, "kubernetes"=>{"container_name"=>"msjsmstoken", "namespace_name"=>"paquetes-homo", "pod_name"=>"msjsmstoken-12-nlkbg", "pod_id"=>"20bf4fe2-ee99-11e8-bc46-005056007805", "labels"=>{"app"=>"msjsmstoken", "deployment"=>"msjsmstoken-12", "deploymentconfig"=>"msjsmstoken"}, "host"=>"dookubnod05.bancogalicia.com.ar", "master_url"=>"https://kubernetes.default.svc.cluster.local", "namespace_id"=>"3db2ee7e-a951-11e8-9d7b-005056007805"}, "message"=>"{\"@timestamp\":\"2019-01-02 10:43:10,046\",\"lvl\":\"DEBUG\",\"thread\":\"http-apr-8080-exec-5\",\"logger\":\"ar.com.bancogalicia.edge.smstoken.repository.SMSTokenRepository\",\"msg\":{\"Envelope\":{\"Body\":{\"ValidarTokenCliente\":{\"ValidarTokenClienteRequest\":{\"BGBAHeader\":{\"Identificadores\":{\"IdMensaje\":{\"@idEsquema\":\"HB2\",\"$\":\"0\"},\"IdMensajeAnterior\":{\"@idEsquema\":\"HB2\",\"$\":\"0\"},\"IdOperacion\":{\"@idEsquema\":\"HB2\",\"$\":\"0\"}},\"ModuloAplicativo\":{\"IdGalicia\":\"Mod-333\",\"IdProveedor\":\"GR\"},\"Equipo\":{\"@ip\":\"1.0.0.1\",\"@nombre\":\"\"},\"Origen\":{\"ModuloAplicativo\":{\"IdGalicia\":\"Mod-333\",\"IdProveedor\":\"GRSU999\"},\"Canal\":\"GR\",\"OrganizacionInterna\":{\"@tipo\":\"DE\",\"@id\":\"4\"},\"Equipo\":{\"@ip\":\"1.0.0.1\",\"@nombre\":\"\"},\"Operador\":{\"$\":\"L9999999\"}}},\"Datos\":{\"idHost\":\"123687404\",\"token\":\"234567\"}}}}}},\"duration\":0,\"operationname\":\"validarTokenCliente\",\"session_id\":\"52036436-9432-4dc7-859d-329b50f64c0f\",\"operationtype\":\"REQ\",\"canal\":\"-\",\"dni\":\"53453453\"}\n", "level"=>"info", "hostname"=>"dookubnod05.bancogalicia.com.ar", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.254.42.238", "ipaddr6"= "container_id"=>"56f31f9415b301399c475222417d434de2b847b27b5ff90e7360f9d4b5e9605d"}, "kubernetes"=>{"container_name"=>"msjsmstoken", "namespace_name"=>"paquetes-homo", "pod_name"=>"msjsmstoken-12-nlkbg", "pod_id"=>"20bf4fe2-ee99-11e8-bc46-0050560 2019-01-02 11:34:31 -0300 [warn]: dump an error event: error_class=Fluent::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="retry_es" time=1546436591 record={"@timestamp"=>"2019-01-02T13:43:11.276949115Z", "lvl"=>"DEBUG", "thread"=>"http-apr-8080-exec-5", "logger"=>"ar.com.bancogalicia.edge.smstoken.component.SMSTokenService", "msg"=>{"ValidarTokenResponse"=>{"ResultadoOperacion"=>{"Severidad"=>"OK"}, "Datos"=>{"Mensaje"=>"V"}}}, "duration"=>1231, "operationname"=>"validarToken", "session_id"=>"52036436-9432-4dc7-859d-329b50f64c0f", "operationtype"=>"RES", "canal"=>"-", "dni"=>"53453453", "docker"=>{"container_id"=>"56f31f9415b301399c475222417d434de2b847b27b5ff90e7360f9d4b5e9605d"}, "kubernetes"=>{"container_name"=>"msjsmstoken", "namespace_name"=>"paquetes-homo", "pod_name"=>"msjsmstoken-12-nlkbg", "pod_id"=>"20bf4fe2-ee99-11e8-bc46-005056007805", "labels"=>{"app"=>"msjsmstoken", "deployment"=>"msjsmstoken-12", "deploymentconfig"=>"msjsmstoken"}, "host"=>"dookubnod05.bancogalicia.com.ar", "master_url"=>"https://kubernetes.default.svc.cluster.local", "namespace_id"=>"3db2ee7e-a951-11e8-9d7b-005056007805"}, "message"=>"{\"@timestamp\":\"2019-01-02 10:43:11,276\",\"lvl\":\"DEBUG\",\"thread\":\"http-apr-8080-exec-5\",\"logger\":\"ar.com.bancogalicia.edge.smstoken.component.SMSTokenService\",\"msg\":{\"ValidarTokenResponse\":{\"ResultadoOperacion\":{\"Severidad\":\"OK\"},\"Datos\":{\"Mensaje\":\"V\"}}},\"duration\":1231,\"operationname\":\"validarToken\",\"session_id\":\"52036436-9432-4dc7-859d-329b50f64c0f\",\"operationtype\":\"RES\",\"canal\":\"-\",\"dni\":\"53453453\"}\n", "level"=>"info", "hostname"=>"dookubnod05.bancogalicia.com.ar", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.254.42.238", "ipaddr6"=>"fe80::c4ea:1bff:fe0a:3b86", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2019-01-02T13:43:11.346703+00:00", "version"=>"0.12.43 1.6.0"}}, "viaq_index_name"=>"project.paq ~~~ [l0637033@localhost Kibana_No_Match_02257770]$ for i in $(oc exec logging-es-data-master-frpq29be-6-42l6c -c elasticsearch -- ls /elasticsearch/persistent/logging-es/logs); do oc exec logging-es-data-master-frpq29be-6-42l6c -c elasticsearch -- cat /elasticsearch/persistent/logging-es/logs/$i | grep parse; done at org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer.writeSparseMissingBitset(Lucene54DocValuesConsumer.java:399) at org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer.writeSparseMissingBitset(Lucene54DocValuesConsumer.java:399) at org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer.writeSparseMissingBitset(Lucene54DocValuesConsumer.java:399) We couldn't find any parsing error like: MapperParsingException[object mapping for [example] tried to parse field [example] as object, but found a concrete value] We need to know why es is rejecting. thx
This is most likely caused because merging of json logs is enabled by default. In our pending 4.0 release, we disabled this feature by default because of various issues related to this feature. The problem is your applications are likely logging a JSON message payload that is being added to the payload fluentd submits to Elasticsearch. The data types one application emits may conflict with the types another application emits. Elasicsearch is unable to distinguish dataypes differences for logs from different applications. We recommend disabling MERGE_JSON_LOG feature: 1. Edit the logging-fluentd configmap to add the environment variable MERGE_JSON_LOG with a value of 'false' Note: This will require reapplying this change on every upgrade. Takeaway is to fix ansible to allow you set via ansible var. Lowering the severity as there is a work around
Will require cherry-pick of https://github.com/openshift/openshift-ansible/pull/11078
release 3.9 https://github.com/openshift/openshift-ansible/pull/11096
Note this does not fix the reported issue but allows disabling of the feature that causes the error.
Fix is in openshift-ansible-3.9.70-1. bash-4.2$ grep -r "openshift_logging_fluentd_merge_json_log" roles/openshift_logging_fluentd/defaults/main.yml:# openshift_logging_fluentd_merge_json_log configures fluentd to parse roles/openshift_logging_fluentd/defaults/main.yml:openshift_logging_fluentd_merge_json_log: "true" roles/openshift_logging_fluentd/templates/2.x/fluentd.j2: value: "{{ openshift_logging_fluentd_merge_json_log }}" roles/openshift_logging_fluentd/templates/5.x/fluentd.j2: value: "{{ openshift_logging_fluentd_merge_json_log }}" bash-4.2$ rpm -qa |grep ansible openshift-ansible-playbooks-3.9.70-1.git.0.d5a98de.el7.noarch openshift-ansible-3.9.70-1.git.0.d5a98de.el7.noarch ansible-2.4.6.0-1.el7ae.noarch openshift-ansible-roles-3.9.70-1.git.0.d5a98de.el7.noarch openshift-ansible-docs-3.9.70-1.git.0.d5a98de.el7.noarch Move bug to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0403
*** Bug 1689063 has been marked as a duplicate of this bug. ***
*** Bug 1691142 has been marked as a duplicate of this bug. ***
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days