Bug 1668338
| Summary: | fluentd ElasticsearchError error="400 - Rejected by Elasticsearch | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Vítor Corrêa <vcorrea> |
| Component: | Logging | Assignee: | Jeff Cantrill <jcantril> |
| Status: | CLOSED ERRATA | QA Contact: | Qiaoling Tang <qitang> |
| Severity: | low | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.9.0 | CC: | aos-bugs, jcantril, ktadimar, misalunk, rmeggins, sgarciam, stwalter |
| Target Milestone: | --- | ||
| Target Release: | 3.9.z | ||
| Hardware: | x86_64 | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Enhancement | |
| Doc Text: |
Feature: Allow disabling of "merge json log" via ansible inventory variable
Reason: MERGE_JSON_LOG environment variable exists on the deployment configuration. Adding an inventory variable keep users from needing to reapply the setting when they perform an upgrade or re-installation.
Result: The logging-fluentd daemonset will have the MERGE_JSON_LOG variable value set to reflect what is specified in the ansible inventory.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-03-13 22:27:23 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This is most likely caused because merging of json logs is enabled by default. In our pending 4.0 release, we disabled this feature by default because of various issues related to this feature. The problem is your applications are likely logging a JSON message payload that is being added to the payload fluentd submits to Elasticsearch. The data types one application emits may conflict with the types another application emits. Elasicsearch is unable to distinguish dataypes differences for logs from different applications. We recommend disabling MERGE_JSON_LOG feature: 1. Edit the logging-fluentd configmap to add the environment variable MERGE_JSON_LOG with a value of 'false' Note: This will require reapplying this change on every upgrade. Takeaway is to fix ansible to allow you set via ansible var. Lowering the severity as there is a work around Will require cherry-pick of https://github.com/openshift/openshift-ansible/pull/11078 Note this does not fix the reported issue but allows disabling of the feature that causes the error. Fix is in openshift-ansible-3.9.70-1.
bash-4.2$ grep -r "openshift_logging_fluentd_merge_json_log"
roles/openshift_logging_fluentd/defaults/main.yml:# openshift_logging_fluentd_merge_json_log configures fluentd to parse
roles/openshift_logging_fluentd/defaults/main.yml:openshift_logging_fluentd_merge_json_log: "true"
roles/openshift_logging_fluentd/templates/2.x/fluentd.j2: value: "{{ openshift_logging_fluentd_merge_json_log }}"
roles/openshift_logging_fluentd/templates/5.x/fluentd.j2: value: "{{ openshift_logging_fluentd_merge_json_log }}"
bash-4.2$ rpm -qa |grep ansible
openshift-ansible-playbooks-3.9.70-1.git.0.d5a98de.el7.noarch
openshift-ansible-3.9.70-1.git.0.d5a98de.el7.noarch
ansible-2.4.6.0-1.el7ae.noarch
openshift-ansible-roles-3.9.70-1.git.0.d5a98de.el7.noarch
openshift-ansible-docs-3.9.70-1.git.0.d5a98de.el7.noarch
Move bug to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0403 *** Bug 1689063 has been marked as a duplicate of this bug. *** *** Bug 1691142 has been marked as a duplicate of this bug. *** The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |
Kibana doesn't show project data. On fluentd pods We can see some errors: 7360f9d4b5e9605d"}, "kubernetes"=>{"container_name"=>"msjsmstoken", "namespace_name"=>"paquetes-homo", "pod_name"=>"msjsmstoken-12-nlkbg", "pod_id"=>"20bf4fe2-ee99-11e8-bc46-005056007805", "labels"=>{"app"=>"msjsmstoken", "deployment"=>"msjsmstoken-12", "deploymentconfig"=>"msjsmstoken"}, "host"=>"dookubnod05.bancogalicia.com.ar", "master_url"=>"https://kubernetes.default.svc.cluster.local", "namespace_id"=>"3db2ee7e-a951-11e8-9d7b-005056007805"}, "message"=>"{\"@timestamp\":\"2019-01-02 10:43:10,046\",\"lvl\":\"DEBUG\",\"thread\":\"http-apr-8080-exec-5\",\"logger\":\"ar.com.bancogalicia.edge.smstoken.repository.SMSTokenRepository\",\"msg\":{\"Envelope\":{\"Body\":{\"ValidarTokenCliente\":{\"ValidarTokenClienteRequest\":{\"BGBAHeader\":{\"Identificadores\":{\"IdMensaje\":{\"@idEsquema\":\"HB2\",\"$\":\"0\"},\"IdMensajeAnterior\":{\"@idEsquema\":\"HB2\",\"$\":\"0\"},\"IdOperacion\":{\"@idEsquema\":\"HB2\",\"$\":\"0\"}},\"ModuloAplicativo\":{\"IdGalicia\":\"Mod-333\",\"IdProveedor\":\"GR\"},\"Equipo\":{\"@ip\":\"1.0.0.1\",\"@nombre\":\"\"},\"Origen\":{\"ModuloAplicativo\":{\"IdGalicia\":\"Mod-333\",\"IdProveedor\":\"GRSU999\"},\"Canal\":\"GR\",\"OrganizacionInterna\":{\"@tipo\":\"DE\",\"@id\":\"4\"},\"Equipo\":{\"@ip\":\"1.0.0.1\",\"@nombre\":\"\"},\"Operador\":{\"$\":\"L9999999\"}}},\"Datos\":{\"idHost\":\"123687404\",\"token\":\"234567\"}}}}}},\"duration\":0,\"operationname\":\"validarTokenCliente\",\"session_id\":\"52036436-9432-4dc7-859d-329b50f64c0f\",\"operationtype\":\"REQ\",\"canal\":\"-\",\"dni\":\"53453453\"}\n", "level"=>"info", "hostname"=>"dookubnod05.bancogalicia.com.ar", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.254.42.238", "ipaddr6"= "container_id"=>"56f31f9415b301399c475222417d434de2b847b27b5ff90e7360f9d4b5e9605d"}, "kubernetes"=>{"container_name"=>"msjsmstoken", "namespace_name"=>"paquetes-homo", "pod_name"=>"msjsmstoken-12-nlkbg", "pod_id"=>"20bf4fe2-ee99-11e8-bc46-0050560 2019-01-02 11:34:31 -0300 [warn]: dump an error event: error_class=Fluent::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="retry_es" time=1546436591 record={"@timestamp"=>"2019-01-02T13:43:11.276949115Z", "lvl"=>"DEBUG", "thread"=>"http-apr-8080-exec-5", "logger"=>"ar.com.bancogalicia.edge.smstoken.component.SMSTokenService", "msg"=>{"ValidarTokenResponse"=>{"ResultadoOperacion"=>{"Severidad"=>"OK"}, "Datos"=>{"Mensaje"=>"V"}}}, "duration"=>1231, "operationname"=>"validarToken", "session_id"=>"52036436-9432-4dc7-859d-329b50f64c0f", "operationtype"=>"RES", "canal"=>"-", "dni"=>"53453453", "docker"=>{"container_id"=>"56f31f9415b301399c475222417d434de2b847b27b5ff90e7360f9d4b5e9605d"}, "kubernetes"=>{"container_name"=>"msjsmstoken", "namespace_name"=>"paquetes-homo", "pod_name"=>"msjsmstoken-12-nlkbg", "pod_id"=>"20bf4fe2-ee99-11e8-bc46-005056007805", "labels"=>{"app"=>"msjsmstoken", "deployment"=>"msjsmstoken-12", "deploymentconfig"=>"msjsmstoken"}, "host"=>"dookubnod05.bancogalicia.com.ar", "master_url"=>"https://kubernetes.default.svc.cluster.local", "namespace_id"=>"3db2ee7e-a951-11e8-9d7b-005056007805"}, "message"=>"{\"@timestamp\":\"2019-01-02 10:43:11,276\",\"lvl\":\"DEBUG\",\"thread\":\"http-apr-8080-exec-5\",\"logger\":\"ar.com.bancogalicia.edge.smstoken.component.SMSTokenService\",\"msg\":{\"ValidarTokenResponse\":{\"ResultadoOperacion\":{\"Severidad\":\"OK\"},\"Datos\":{\"Mensaje\":\"V\"}}},\"duration\":1231,\"operationname\":\"validarToken\",\"session_id\":\"52036436-9432-4dc7-859d-329b50f64c0f\",\"operationtype\":\"RES\",\"canal\":\"-\",\"dni\":\"53453453\"}\n", "level"=>"info", "hostname"=>"dookubnod05.bancogalicia.com.ar", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.254.42.238", "ipaddr6"=>"fe80::c4ea:1bff:fe0a:3b86", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2019-01-02T13:43:11.346703+00:00", "version"=>"0.12.43 1.6.0"}}, "viaq_index_name"=>"project.paq ~~~ [l0637033@localhost Kibana_No_Match_02257770]$ for i in $(oc exec logging-es-data-master-frpq29be-6-42l6c -c elasticsearch -- ls /elasticsearch/persistent/logging-es/logs); do oc exec logging-es-data-master-frpq29be-6-42l6c -c elasticsearch -- cat /elasticsearch/persistent/logging-es/logs/$i | grep parse; done at org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer.writeSparseMissingBitset(Lucene54DocValuesConsumer.java:399) at org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer.writeSparseMissingBitset(Lucene54DocValuesConsumer.java:399) at org.apache.lucene.codecs.lucene54.Lucene54DocValuesConsumer.writeSparseMissingBitset(Lucene54DocValuesConsumer.java:399) We couldn't find any parsing error like: MapperParsingException[object mapping for [example] tried to parse field [example] as object, but found a concrete value] We need to know why es is rejecting. thx