Bug 1545081

Summary:	Fluentd removed json field from logs.
Product:	OpenShift Container Platform	Reporter:	Rajnikant <rkant>
Component:	Logging	Assignee:	Rich Megginson <rmeggins>
Status:	CLOSED ERRATA	QA Contact:	Anping Li <anli>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	3.5.0	CC:	aos-bugs, fcami, rmeggins
Target Milestone:	---
Target Release:	3.5.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openshift3/logging-fluentd:3.5.0-53	Doc Type:	Bug Fix
Doc Text:	Cause: JSON valued records with array valued fields were being stripped during fluentd processing. Consequence: Array valued fields are dropped from the records. Fix: Do not strip array valued fields from the record. Result: No fields are missing.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-04-12 06:43:19 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Rajnikant 2018-02-14 08:56:03 UTC

Description of problem:
Fluentd removed json field from logs.

Customer is using secure-forward to forward logs to an external fluentd.

In fluentd image 3.5.0, observed that fluentd removes/not collecting JSON field to external fluentd, where as with fluentd image version(3.4.1) is fine.


Version-Release number of selected component (if applicable):
OpenShift Container Platform 3.5

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 7 Anping Li 2018-02-25 01:45:49 UTC

Verified and pass with openshift3/logging-fluentd/images/3.5.0-53.

Step as following:
1. Enable fluentd forward to external fluentd
1. oc debug $pod1
2. echo message in $pod1
{ "@version":1, "category":"developer", "code":"KKMSVC004E", "@timestamp":"2018-01-18T13:35:50.415+0100", "sequence":4638, "loggerName":"at.jrap.soa.services.jaxrs.internal.exception.AbstractCheckedStatusExc    eptionMapper", "loglevel":"ERROR", "threadName":"default task-23", "message":"Failed to search for Kunde by name: java.lang.NullPointerException", "threadId":270, "mdc":{ "RBG_APPLICATION":"drb_dev", "RBG_RE    QUEST_CORRELATION":"S_discovery_120c17359f81139e", "RBG_IPADDRESS":"10.94.10.1" }, "exception":{ "class":"at.rsg.kokuma.model.KokumaBusinessException", "message":"{developer} KKMSVC004E: Failed to search for     Kunde by name: java.lang.NullPointerException", "trace":["at.rsg.kokuma.services.impl.business.kundensuche.KundensucheBusinessBean.sucheName(195)", "... 63 filtered", "at.rsg.kokuma.services.impl.business.k    undensuche.KundensucheBusinessBean$$$view20.sucheName", "... 9 filtered", "at.rsg.kokuma.services.impl.business.kundensuche.KundensucheBusinessBean$Proxy$_$$_Weld$EnterpriseProxy$.sucheName", "at.rsg.kokuma.    services.impl.KundensucheServiceBean.sucheName(35)", "... 10 filtered", "at.rsg.kokuma.services.impl.logging.LoggingInterceptor.log(51)", "... 10 filtered", "at.jrap.commons.validation.internal.ExceptionInte    rceptor.mapException(35)", "... 10 filtered", "at.jrap.commons.secured.internal.SecuredInterceptor.checkSecurity(51)", "... 10 filtered", "at.jrap.commons.exceptions.internal.ExceptionInterceptor.mapExceptio    n(48)", "... 10 filtered", "at.jrap.commons.context.internal.ServiceContextManagingInterceptor.reset(53)", "... 10 filtered", "at.jrap.commons.logging.internal.ThreadContextResettingInterceptor.reset(53)", "    ... 65 filtered", "at.rsg.kokuma.services.KundensucheService$$$view15.sucheName", "... 9 filtered", "at.rsg.kokuma.services.impl.KundensucheService$1917269646$Proxy$_$$_Weld$EnterpriseProxy$.sucheName", "at.    rsg.kokuma.services.kundensucheservice.jaxrs.KundensucheServiceResource.sucheName(75)", "at.rsg.kokuma.services.kundensucheservice.jaxrs.KundensucheServiceResource$Proxy$_$$_WeldClientProxy.sucheName", "...     16 filtered", "at.jrap.soa.cors.CorsFilter.doFilter(85)", "... 29 filtered" ], "cause":{ "class":"java.lang.NullPointerException", "message":"^", "trace":["at.rsg.kokuma.services.impl.business.kundensuche.Ku    ndensucheBusinessBean.createGefundenerKunde(344)", "at.rsg.kokuma.services.impl.business.kundensuche.KundensucheBusinessBean.createGefundenerKunde(380)", "... 266 more" ] } } }


3. Check the external fluentd store

apps.20180224_12.log:2018-02-24T20:37:22-05:00	kubernetes.journal.container	{"@version":1,"category":"developer","code":"KKMSVC004E","@timestamp":"2018-02-25T01:37:22.518192+00:00","sequence":4638,"loggerName":"at.jrap.soa.services.jaxrs.internal.exception.AbstractCheckedStatusExc    eptionMapper","loglevel":"ERROR","threadName":"default task-23","message":"Failed to search for Kunde by name: java.lang.NullPointerException","threadId":270,"mdc":{"RBG_APPLICATION":"drb_dev","RBG_RE    QUEST_CORRELATION":"S_discovery_120c17359f81139e","RBG_IPADDRESS":"10.94.10.1"},"exception":{"class":"at.rsg.kokuma.model.KokumaBusinessException","message":"{developer} KKMSVC004E: Failed to search for     Kunde by name: java.lang.NullPointerException","trace":["at.rsg.kokuma.services.impl.business.kundensuche.KundensucheBusinessBean.sucheName(195)","... 63 filtered","at.rsg.kokuma.services.impl.business.k    undensuche.KundensucheBusinessBean$$$view20.sucheName","... 9 filtered","at.rsg.kokuma.services.impl.business.kundensuche.KundensucheBusinessBean$Proxy$_$$_Weld$EnterpriseProxy$.sucheName","at.rsg.kokuma.    services.impl.KundensucheServiceBean.sucheName(35)","... 10 filtered","at.rsg.kokuma.services.impl.logging.LoggingInterceptor.log(51)","... 10 filtered","at.jrap.commons.validation.internal.ExceptionInte    rceptor.mapException(35)","... 10 filtered","at.jrap.commons.secured.internal.SecuredInterceptor.checkSecurity(51)","... 10 filtered","at.jrap.commons.exceptions.internal.ExceptionInterceptor.mapExceptio    n(48)","... 10 filtered","at.jrap.commons.context.internal.ServiceContextManagingInterceptor.reset(53)","... 10 filtered","at.jrap.commons.logging.internal.ThreadContextResettingInterceptor.reset(53)","    ... 65 filtered","at.rsg.kokuma.services.KundensucheService$$$view15.sucheName","... 9 filtered","at.rsg.kokuma.services.impl.KundensucheService$1917269646$Proxy$_$$_Weld$EnterpriseProxy$.sucheName","at.    rsg.kokuma.services.kundensucheservice.jaxrs.KundensucheServiceResource.sucheName(75)","at.rsg.kokuma.services.kundensucheservice.jaxrs.KundensucheServiceResource$Proxy$_$$_WeldClientProxy.sucheName","...     16 filtered","at.jrap.soa.cors.CorsFilter.doFilter(85)","... 29 filtered"],"cause":{"class":"java.lang.NullPointerException","message":"^","trace":["at.rsg.kokuma.services.impl.business.kundensuche.Ku    ndensucheBusinessBean.createGefundenerKunde(344)","at.rsg.kokuma.services.impl.business.kundensuche.KundensucheBusinessBean.createGefundenerKunde(380)","... 266 more"]}},"docker":{"container_id":"482aa92ba988706e9ba415c2f454244c1274af229d9f7e64915bec7db671627b"},"kubernetes":{"namespace_name":"install-test","pod_id":"4062fc71-19cc-11e8-95c8-fa163e7b100f","pod_name":"nodejs-mongodb-example-1-63tnq-debug","host":"host-8-241-127.host.centralci.eng.rdu2.redhat.com","namespace_id":"7883a744-19c4-11e8-986b-fa163e7b100f","container_name":"nodejs-mongodb-example"},"level":"info","hostname":"host-8-241-127.host.centralci.eng.rdu2.redhat.com","pipeline_metadata":{"collector":{"ipaddr4":"10.129.0.10","ipaddr6":"fe80::b86d:a9ff:fe93:cd81","inputname":"fluent-plugin-systemd","name":"fluentd","received_at":"2018-02-25T01:37:24.317072+00:00","version":"0.12.39-2 unknown data version"}}}

Comment 11 errata-xmlrpc 2018-04-12 06:43:19 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1107