Bug 1696718

Summary:	arrayIndexOutOfbounds wrapped in JsonMappingException during serializing java.util.Date fields
Product:	Red Hat Satellite	Reporter:	Jan Jansky <jjansky>
Component:	Candlepin	Assignee:	satellite6-bugs <satellite6-bugs>
Status:	CLOSED CURRENTRELEASE	QA Contact:	jcallaha
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6.4.2	CC:	bbuckingham, bcourt, hhudgeon, khowell, mjia, mmccune, nmoumoul, pdwyer, pmoravec, rajgupta, wclark, wpinheir, zhunting
Target Milestone:	Released	Keywords:	Reopened, Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	candlepin-2.5.14-1, candlepin-2.4.16-1	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1696772 1704720 (view as bug list)		Environment:
Last Closed:	2019-06-10 16:59:29 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1704720
Bug Blocks:

Description Jan Jansky 2019-04-05 13:35:28 UTC

Description of problem: 

Candlepin throwing exceptions similar per second, after few hours/days tomcat will crash.

It seems to be known issue and fixed in candlepin 2.5, but for Satellite is latest 2.4, please create patch and backport this fix.

Related bug for candlepin (https://bugzilla.redhat.com/show_bug.cgi?id=1683305)

Version-Release number of selected component (if applicable):
candlepin-2.4.8-1.el7.noarch

How reproducible: 
Not sure, customer reported this issue.


Actual results:

Apr 05, 2019 4:46:32 AM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [default] in context with path [/candlepin] threw exception
org.jboss.resteasy.spi.UnhandledException: Response is committed, can't handle exception
        at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:148)
        at org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:432)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:376)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
        at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
        at org.candlepin.servlet.filter.EventFilter.doFilter(EventFilter.java:61)
        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
        at org.candlepin.servlet.filter.ContentTypeHackFilter.doFilter(ContentTypeHackFilter.java:58)
        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
        at org.candlepin.common.filter.LoggingFilter.doFilter(LoggingFilter.java:113)
        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
        at org.candlepin.servlet.filter.CandlepinPersistFilter.doFilter(CandlepinPersistFilter.java:48)
        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
        at org.candlepin.servlet.filter.CandlepinScopeFilter.doFilter(CandlepinScopeFilter.java:68)
        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: (was java.lang.ArrayIndexOutOfBoundsException) (through reference chain: org.candlepin.policy.js.compliance.ComplianceStatus["date"])
        at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:391)
        at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:351)
        at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:316)
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFieldsFiltered(BeanSerializerBase.java:779)
        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:153)
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
        at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1396)
        at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:913)
        at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:625)
        at org.jboss.resteasy.core.interception.AbstractWriterInterceptorContext.writeTo(AbstractWriterInterceptorContext.java:129)
        at org.jboss.resteasy.core.interception.ServerWriterInterceptorContext.writeTo(ServerWriterInterceptorContext.java:62)
        at org.jboss.resteasy.core.interception.AbstractWriterInterceptorContext.proceed(AbstractWriterInterceptorContext.java:118)
        at org.jboss.resteasy.plugins.interceptors.encoding.GZIPEncodingInterceptor.aroundWriteTo(GZIPEncodingInterceptor.java:100)
        at org.jboss.resteasy.core.interception.AbstractWriterInterceptorContext.proceed(AbstractWriterInterceptorContext.java:122)
        at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:99)
        at org.jboss.resteasy.core.SynchronousDispatcher.writeResponse(SynchronousDispatcher.java:427)
        ... 43 more
Caused by: java.lang.ArrayIndexOutOfBoundsException


Expected results:

No error.

Comment 7 jcallaha 2019-04-18 14:37:35 UTC

Due to the unreproducible nature of this issue, we are marking this verified in a no-break condition.
Additionally, this fix has been implemented in stage and production cdns and has been effective in eliminating this issue.

Verified in Satellite 6.5.0 Snap 24

Comment 13 Barnaby Court 2019-05-09 20:48:20 UTC

Moved to modified with patch for 6.4.z

Comment 16 errata-xmlrpc 2019-05-14 12:40:38 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222

Comment 17 Rajan Gupta 2019-06-04 10:56:30 UTC

Customer is requesting to avail the fix in Satellite 6.4. Any plan?

Comment 18 Mike McCune 2019-06-10 16:59:29 UTC

This bug is resolved in the release of 6.5. If there are any customers on 6.4.3 or later who wish to hotfix this issue, instructions are as follows:

*** Hotfix instructions ***

1) download https://people.redhat.com/~mmccune/hotfix/1696718/candlepin-2.4.16-1.tar to your Satellite

2) Unpack

tar cvf candlepin-2.4.16-1.tar 

3) stop services

foreman-maintain service stop

4) upgrade

rpm -Uvh candlepin*.rpm

5) restart and resume operations

foreman-maintain service start