Bug 536070 (RHQ-45)

Summary: unique constraint violated (JON.RHQ_PACKAGE_IDX, JON.RHQ_PACKAGE_VERSION_IDX)
Product: [Other] RHQ Project Reporter: John Mazzitelli <mazz>
Component: ContentAssignee: John Mazzitelli <mazz>
Status: CLOSED NEXTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: high    
Version: unspecified   
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: All   
URL: http://jira.rhq-project.org/browse/RHQ-45
Whiteboard:
Fixed In Version: 1.0 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description John Mazzitelli 2008-03-07 14:28:00 UTC
During perf testing, 90 agents were all reporting in.  Got this error during package report merging:

2008-03-05 13:01:07,218 WARN  [org.hibernate.util.JDBCExceptionReporter] SQL Error: 1, SQLState: 23000
2008-03-05 13:01:07,218 ERROR [org.hibernate.util.JDBCExceptionReporter] ORA-00001: unique constraint (JON.RHQ_PACKAGE_IDX) violated
2008-03-05 13:01:07,218 WARN  [org.hibernate.util.JDBCExceptionReporter] SQL Error: 1, SQLState: 23000
2008-03-05 13:01:07,218 ERROR [org.hibernate.util.JDBCExceptionReporter] ORA-00001: unique constraint (JON.RHQ_PACKAGE_IDX) violated
2008-03-05 13:01:07,218 ERROR [org.hibernate.event.def.AbstractFlushingEventListener] Could not synchronize database state with session
org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
	at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:71)
	at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
	at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:253)
	at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:92)
	at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:87)
	at org.hibernate.jdbc.AbstractBatcher.prepareBatchStatement(AbstractBatcher.java:222)
	at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2229)
	at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2665)
	at org.hibernate.action.EntityInsertAction.execute(EntityInsertAction.java:60)
	at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:250)
	at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:234)
	at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:141)
	at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
	at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
	at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
	at org.hibernate.ejb.AbstractEntityManagerImpl.flush(AbstractEntityManagerImpl.java:296)
	at org.jboss.ejb3.entity.TransactionScopedEntityManager.flush(TransactionScopedEntityManager.java:211)
	at org.rhq.enterprise.server.content.ContentManagerBean.mergeDiscoveredPackages(ContentManagerBean.java:286)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
	at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
	at org.rhq.enterprise.server.authz.RequiredPermissionsInterceptor.checkRequiredPermissions(RequiredPermissionsInterceptor.java:153)
	at sun.reflect.GeneratedMethodAccessor154.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
	at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
	at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
	at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:214)
	at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:184)
	at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:81)
	at $Proxy188.mergeDiscoveredPackages(Unknown Source)
	at org.rhq.enterprise.server.content.ContentServerServiceImpl.mergeDiscoveredPackages(ContentServerServiceImpl.java:58)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.rhq.enterprise.communications.command.impl.remotepojo.server.RemotePojoInvocationCommandService.execute(RemotePojoInvocationCommandService.java:184)
	at sun.reflect.GeneratedMethodAccessor235.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
	at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
	at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
	at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
	at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
	at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201)
	at $Proxy370.execute(Unknown Source)
	at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:335)
	at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:230)
	at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
	at org.jboss.remoting.transport.servlet.ServletServerInvoker.processRequest(ServletServerInvoker.java:231)
	at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
	at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
	at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
	at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
	at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
	at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201)
	at $Proxy347.processRequest(Unknown Source)
	at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:128)
	at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.doPost(ServerInvokerServlet.java:157)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:109)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
	at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
	at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
	at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
	at java.lang.Thread.run(Thread.java:595)
Caused by: java.sql.BatchUpdateException: ORA-00001: unique constraint (JON.RHQ_PACKAGE_IDX) violated

	at oracle.jdbc.driver.DatabaseError.throwBatchUpdateException(DatabaseError.java:343)
	at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:10720)
	at org.jboss.resource.adapter.jdbc.CachedPreparedStatement.executeBatch(CachedPreparedStatement.java:478)
	at org.jboss.resource.adapter.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:519)
	at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:48)
	at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:246)
	... 105 more
2008-03-05 13:01:07,328 INFO  [org.rhq.enterprise.server.content.ContentManagerBean] Merging packages for resource ID [1199]. Package count [311]


Comment 1 John Mazzitelli 2008-03-07 14:34:23 UTC
And later we get this:

2008-03-05 13:05:13,281 WARN  [org.hibernate.util.JDBCExceptionReporter] SQL Error: 1, SQLState: 23000
2008-03-05 13:05:13,281 ERROR [org.hibernate.util.JDBCExceptionReporter] ORA-00001: unique constraint (JON.RHQ_PACKAGE_VERSION_IDX) violated

2008-03-05 13:05:13,281 ERROR [org.hibernate.event.def.AbstractFlushingEventListener] Could not synchronize database state with session
org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
	at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:71)
	at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
	at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:253)
	at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:92)
	at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:87)
	at org.hibernate.jdbc.AbstractBatcher.prepareBatchStatement(AbstractBatcher.java:222)
	at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2229)
	at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2665)
	at org.hibernate.action.EntityInsertAction.execute(EntityInsertAction.java:60)
	at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:250)
	at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:234)
	at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:141)
	at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
	at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
	at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
	at org.hibernate.ejb.AbstractEntityManagerImpl.flush(AbstractEntityManagerImpl.java:296)
	at org.jboss.ejb3.entity.TransactionScopedEntityManager.flush(TransactionScopedEntityManager.java:211)
	at org.rhq.enterprise.server.content.ContentManagerBean.mergeDiscoveredPackages(ContentManagerBean.java:286)
	at sun.reflect.GeneratedMethodAccessor642.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
	at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
	at org.rhq.enterprise.server.authz.RequiredPermissionsInterceptor.checkRequiredPermissions(RequiredPermissionsInterceptor.java:153)
	at sun.reflect.GeneratedMethodAccessor154.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
	at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
	at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
	at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:214)
	at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:184)
	at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:81)
	at $Proxy188.mergeDiscoveredPackages(Unknown Source)
	at org.rhq.enterprise.server.content.ContentServerServiceImpl.mergeDiscoveredPackages(ContentServerServiceImpl.java:58)
	at sun.reflect.GeneratedMethodAccessor643.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.rhq.enterprise.communications.command.impl.remotepojo.server.RemotePojoInvocationCommandService.execute(RemotePojoInvocationCommandService.java:184)
	at sun.reflect.GeneratedMethodAccessor235.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
	at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
	at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
	at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
	at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
	at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201)
	at $Proxy370.execute(Unknown Source)
	at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:335)
	at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:230)
	at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
	at org.jboss.remoting.transport.servlet.ServletServerInvoker.processRequest(ServletServerInvoker.java:231)
	at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
	at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
	at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
	at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
	at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
	at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201)
	at $Proxy347.processRequest(Unknown Source)
	at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:128)
	at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.doPost(ServerInvokerServlet.java:157)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:109)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
	at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
	at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
	at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
	at java.lang.Thread.run(Thread.java:595)
Caused by: java.sql.BatchUpdateException: ORA-00001: unique constraint (JON.RHQ_PACKAGE_VERSION_IDX) violated

	at oracle.jdbc.driver.DatabaseError.throwBatchUpdateException(DatabaseError.java:343)
	at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:10720)
	at org.jboss.resource.adapter.jdbc.CachedPreparedStatement.executeBatch(CachedPreparedStatement.java:478)
	at org.jboss.resource.adapter.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:519)
	at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:48)
	at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:246)
	... 103 more

Comment 2 Jason Dobies 2008-03-07 15:07:26 UTC
This looks like it's directly related to the fact that packages, if not already known to the system, are added if they are discovered. If we're starting 90 agents at roughly the same time, I'm not surprised the reports are hitting.

In addition to addressing this with regard to agent resource discoveries, we'll run into the same issue if a content source introduces a package into the DB at the same time as an inventory report. This will probably be a less common scenario, but still very possible.

Comment 3 John Mazzitelli 2008-03-07 21:36:37 UTC
need to write some REQUIRES_NEW methods to create new package versions.
We need to catch the exception if it fails and retry it or do something to see if it already exists to catch the fact that something else might have already created it concurrently to our create attempt.

same with package objects

same with architecture objects

Comment 4 John Mazzitelli 2008-03-10 19:19:49 UTC
i'm questioning how I should do this.  right now we have methods like this:

    @TransactionAttribute(value = TransactionAttributeType.REQUIRES_NEW)
    public PackageVersion createPackageVersion(String packageName, int packageTypeId, String version,
        int architectureId, InputStream packageBitStream) {

    public void mergeDiscoveredPackages(ContentDiscoveryReport report) {

These methods check to see if the package version exists, if it does, reuses it; otherwise, it creates it and attaches bits to it.  The content source sync stuff does similar things.

But this is where the constraint violations come into play.  If we concurrently call these methods, we get bad things happen.

So, the thought is to do this (callers should only call this if they already found that there is no PV in the DB):

@TransactionAttribute(REQUIRES_NEW)
PackageVersion findOrPersistPackageVersionSafely(PackageVersion pv)
{
try
{
   pv = entitManager.persist(pv);
   entityManager.flush(); // force it to go out to the DB
   return pv;
}
catch (Exception e)
{
   // if this exception was because the PV already exists, we should be able to find it
   // if we can find it, can we ignore this exception assuming it was because the PV existed?
   // if we can't find it, throw the original error since it needs to be reported
   List l = entityManager.createNamedQuery(PackageVersion.FIND_ME, ...).getResultList();
   if (l.count() != 1) throw e;
   if (l.count() > 1) throw new RuntimeException("Got more than 1 package version");
   return l.get(0);
}
}

If two threads end up in here (after they both found the PV is not already in the DB), one will succeed in that try block, the other will get an exception.  At which time the catch clause will find the new PV.


Comment 5 John Mazzitelli 2008-03-10 19:50:45 UTC
I think I need a second method here, to ensure our tx is committed (to avoid some concurrency issues):

@TransactionAttribute(REQUIRES_NEW)
PackageVersion persistPackageVersion(PackageVersion pv)
{
   return entityManager.persist(pv);
}

@TransactionAttribute(REQUIRES) // reuse existing tx if we are already in one)
PackageVersion persistOrMergePackageVersionSafely(PackageVersion pv)
{
try
{
   pv = persistPackageVersion();
}
catch (Exception e)
{
   // if this exception was because the PV already exists, we should be able to find it
   // if we can find it, can we ignore this exception assuming it was because the PV existed?
   // if we can't find it, throw the original error since it needs to be reported
   List l = entityManager.createNamedQuery(PackageVersion.FIND_ME, ...).getResultList();
   if (l.count() != 1) throw e;
   if (l.count() > 1) throw new RuntimeException("Got more than 1 package version");
   pv.setId(l.get(0).getId);
   entityManager.merge(pv); // the caller might have something that it wanted to update in the PV
}

return pv;

} 

Comment 6 John Mazzitelli 2008-03-11 18:20:45 UTC
rev248 implements the solution, with a unit test that shows it working

i need to go through our code that persists packages and package versions and make sure they use this as appropriate

Comment 7 John Mazzitelli 2008-03-11 21:47:16 UTC
rev254 switched our stuff to use the new persistOrMergePackage[Version]Safely methods.
this should fix the constraint violations.
some more testing needs to be done, but I tried syncing content sources, merging content reports and that stuff still works as it did before. so at least I didn't regress.  i'd need to see our 90 agent perftest run - if we can avoid the constr. violations in that test, I'll consider this fixed.

Comment 8 John Mazzitelli 2008-03-14 18:59:12 UTC
this didn't fix the problem - still getting the constr. violations :-/

Comment 9 John Mazzitelli 2008-03-14 19:09:35 UTC
HA! I scared myself... good thing I added this log message:

2008-03-14 14:54:35,093 WARN  [org.rhq.enterprise.server.content.ContentManagerBean] There was probably a very big and ugly EJB/hibernate error just above this log message - you can normally ignore that. We detected that a package was already created when we tried to do it also - we will ignore this and just use the new package that was created in the other thread

There is nothing we can do about getting that ugly message to not show up, short of us turning off all logging for the arjuna log4j category but then we wouldn't be able to see "real" errors when we get them.


Comment 10 John Mazzitelli 2008-03-14 21:06:23 UTC
this is working now, I can see content reports come in and get processed and I see the constraint violations be worked around

Comment 11 Red Hat Bugzilla 2009-11-10 21:09:44 UTC
This bug was previously known as http://jira.rhq-project.org/browse/RHQ-45