Description of problem: Fluentd removed json field from logs. Customer is using secure-forward to forward logs to an external fluentd. In fluentd image 3.5.0, observed that fluentd removes/not collecting JSON field to external fluentd, where as with fluentd image version(3.4.1) is fine. Version-Release number of selected component (if applicable): OpenShift Container Platform 3.5 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Verified and pass with openshift3/logging-fluentd/images/3.5.0-53. Step as following: 1. Enable fluentd forward to external fluentd 1. oc debug $pod1 2. echo message in $pod1 { "@version":1, "category":"developer", "code":"KKMSVC004E", "@timestamp":"2018-01-18T13:35:50.415+0100", "sequence":4638, "loggerName":"at.jrap.soa.services.jaxrs.internal.exception.AbstractCheckedStatusExc eptionMapper", "loglevel":"ERROR", "threadName":"default task-23", "message":"Failed to search for Kunde by name: java.lang.NullPointerException", "threadId":270, "mdc":{ "RBG_APPLICATION":"drb_dev", "RBG_RE QUEST_CORRELATION":"S_discovery_120c17359f81139e", "RBG_IPADDRESS":"10.94.10.1" }, "exception":{ "class":"at.rsg.kokuma.model.KokumaBusinessException", "message":"{developer} KKMSVC004E: Failed to search for Kunde by name: java.lang.NullPointerException", "trace":["at.rsg.kokuma.services.impl.business.kundensuche.KundensucheBusinessBean.sucheName(195)", "... 63 filtered", "at.rsg.kokuma.services.impl.business.k undensuche.KundensucheBusinessBean$$$view20.sucheName", "... 9 filtered", "at.rsg.kokuma.services.impl.business.kundensuche.KundensucheBusinessBean$Proxy$_$$_Weld$EnterpriseProxy$.sucheName", "at.rsg.kokuma. services.impl.KundensucheServiceBean.sucheName(35)", "... 10 filtered", "at.rsg.kokuma.services.impl.logging.LoggingInterceptor.log(51)", "... 10 filtered", "at.jrap.commons.validation.internal.ExceptionInte rceptor.mapException(35)", "... 10 filtered", "at.jrap.commons.secured.internal.SecuredInterceptor.checkSecurity(51)", "... 10 filtered", "at.jrap.commons.exceptions.internal.ExceptionInterceptor.mapExceptio n(48)", "... 10 filtered", "at.jrap.commons.context.internal.ServiceContextManagingInterceptor.reset(53)", "... 10 filtered", "at.jrap.commons.logging.internal.ThreadContextResettingInterceptor.reset(53)", " ... 65 filtered", "at.rsg.kokuma.services.KundensucheService$$$view15.sucheName", "... 9 filtered", "at.rsg.kokuma.services.impl.KundensucheService$1917269646$Proxy$_$$_Weld$EnterpriseProxy$.sucheName", "at. rsg.kokuma.services.kundensucheservice.jaxrs.KundensucheServiceResource.sucheName(75)", "at.rsg.kokuma.services.kundensucheservice.jaxrs.KundensucheServiceResource$Proxy$_$$_WeldClientProxy.sucheName", "... 16 filtered", "at.jrap.soa.cors.CorsFilter.doFilter(85)", "... 29 filtered" ], "cause":{ "class":"java.lang.NullPointerException", "message":"^", "trace":["at.rsg.kokuma.services.impl.business.kundensuche.Ku ndensucheBusinessBean.createGefundenerKunde(344)", "at.rsg.kokuma.services.impl.business.kundensuche.KundensucheBusinessBean.createGefundenerKunde(380)", "... 266 more" ] } } } 3. Check the external fluentd store apps.20180224_12.log:2018-02-24T20:37:22-05:00 kubernetes.journal.container {"@version":1,"category":"developer","code":"KKMSVC004E","@timestamp":"2018-02-25T01:37:22.518192+00:00","sequence":4638,"loggerName":"at.jrap.soa.services.jaxrs.internal.exception.AbstractCheckedStatusExc eptionMapper","loglevel":"ERROR","threadName":"default task-23","message":"Failed to search for Kunde by name: java.lang.NullPointerException","threadId":270,"mdc":{"RBG_APPLICATION":"drb_dev","RBG_RE QUEST_CORRELATION":"S_discovery_120c17359f81139e","RBG_IPADDRESS":"10.94.10.1"},"exception":{"class":"at.rsg.kokuma.model.KokumaBusinessException","message":"{developer} KKMSVC004E: Failed to search for Kunde by name: java.lang.NullPointerException","trace":["at.rsg.kokuma.services.impl.business.kundensuche.KundensucheBusinessBean.sucheName(195)","... 63 filtered","at.rsg.kokuma.services.impl.business.k undensuche.KundensucheBusinessBean$$$view20.sucheName","... 9 filtered","at.rsg.kokuma.services.impl.business.kundensuche.KundensucheBusinessBean$Proxy$_$$_Weld$EnterpriseProxy$.sucheName","at.rsg.kokuma. services.impl.KundensucheServiceBean.sucheName(35)","... 10 filtered","at.rsg.kokuma.services.impl.logging.LoggingInterceptor.log(51)","... 10 filtered","at.jrap.commons.validation.internal.ExceptionInte rceptor.mapException(35)","... 10 filtered","at.jrap.commons.secured.internal.SecuredInterceptor.checkSecurity(51)","... 10 filtered","at.jrap.commons.exceptions.internal.ExceptionInterceptor.mapExceptio n(48)","... 10 filtered","at.jrap.commons.context.internal.ServiceContextManagingInterceptor.reset(53)","... 10 filtered","at.jrap.commons.logging.internal.ThreadContextResettingInterceptor.reset(53)"," ... 65 filtered","at.rsg.kokuma.services.KundensucheService$$$view15.sucheName","... 9 filtered","at.rsg.kokuma.services.impl.KundensucheService$1917269646$Proxy$_$$_Weld$EnterpriseProxy$.sucheName","at. rsg.kokuma.services.kundensucheservice.jaxrs.KundensucheServiceResource.sucheName(75)","at.rsg.kokuma.services.kundensucheservice.jaxrs.KundensucheServiceResource$Proxy$_$$_WeldClientProxy.sucheName","... 16 filtered","at.jrap.soa.cors.CorsFilter.doFilter(85)","... 29 filtered"],"cause":{"class":"java.lang.NullPointerException","message":"^","trace":["at.rsg.kokuma.services.impl.business.kundensuche.Ku ndensucheBusinessBean.createGefundenerKunde(344)","at.rsg.kokuma.services.impl.business.kundensuche.KundensucheBusinessBean.createGefundenerKunde(380)","... 266 more"]}},"docker":{"container_id":"482aa92ba988706e9ba415c2f454244c1274af229d9f7e64915bec7db671627b"},"kubernetes":{"namespace_name":"install-test","pod_id":"4062fc71-19cc-11e8-95c8-fa163e7b100f","pod_name":"nodejs-mongodb-example-1-63tnq-debug","host":"host-8-241-127.host.centralci.eng.rdu2.redhat.com","namespace_id":"7883a744-19c4-11e8-986b-fa163e7b100f","container_name":"nodejs-mongodb-example"},"level":"info","hostname":"host-8-241-127.host.centralci.eng.rdu2.redhat.com","pipeline_metadata":{"collector":{"ipaddr4":"10.129.0.10","ipaddr6":"fe80::b86d:a9ff:fe93:cd81","inputname":"fluent-plugin-systemd","name":"fluentd","received_at":"2018-02-25T01:37:24.317072+00:00","version":"0.12.39-2 unknown data version"}}}
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1107