Bug 536070 (RHQ-45)
Summary: | unique constraint violated (JON.RHQ_PACKAGE_IDX, JON.RHQ_PACKAGE_VERSION_IDX) | ||
---|---|---|---|
Product: | [Other] RHQ Project | Reporter: | John Mazzitelli <mazz> |
Component: | Content | Assignee: | John Mazzitelli <mazz> |
Status: | CLOSED NEXTRELEASE | QA Contact: | |
Severity: | medium | Docs Contact: | |
Priority: | high | ||
Version: | unspecified | ||
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | All | ||
OS: | All | ||
URL: | http://jira.rhq-project.org/browse/RHQ-45 | ||
Whiteboard: | |||
Fixed In Version: | 1.0 | Doc Type: | Bug Fix |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | Type: | --- | |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
John Mazzitelli
2008-03-07 14:28:00 UTC
And later we get this: 2008-03-05 13:05:13,281 WARN [org.hibernate.util.JDBCExceptionReporter] SQL Error: 1, SQLState: 23000 2008-03-05 13:05:13,281 ERROR [org.hibernate.util.JDBCExceptionReporter] ORA-00001: unique constraint (JON.RHQ_PACKAGE_VERSION_IDX) violated 2008-03-05 13:05:13,281 ERROR [org.hibernate.event.def.AbstractFlushingEventListener] Could not synchronize database state with session org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:71) at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43) at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:253) at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:92) at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:87) at org.hibernate.jdbc.AbstractBatcher.prepareBatchStatement(AbstractBatcher.java:222) at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2229) at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2665) at org.hibernate.action.EntityInsertAction.execute(EntityInsertAction.java:60) at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:250) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:234) at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:141) at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298) at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27) at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000) at org.hibernate.ejb.AbstractEntityManagerImpl.flush(AbstractEntityManagerImpl.java:296) at org.jboss.ejb3.entity.TransactionScopedEntityManager.flush(TransactionScopedEntityManager.java:211) at org.rhq.enterprise.server.content.ContentManagerBean.mergeDiscoveredPackages(ContentManagerBean.java:286) at sun.reflect.GeneratedMethodAccessor642.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166) at org.rhq.enterprise.server.authz.RequiredPermissionsInterceptor.checkRequiredPermissions(RequiredPermissionsInterceptor.java:153) at sun.reflect.GeneratedMethodAccessor154.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118) at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79) at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77) at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:214) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:184) at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:81) at $Proxy188.mergeDiscoveredPackages(Unknown Source) at org.rhq.enterprise.server.content.ContentServerServiceImpl.mergeDiscoveredPackages(ContentServerServiceImpl.java:58) at sun.reflect.GeneratedMethodAccessor643.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.rhq.enterprise.communications.command.impl.remotepojo.server.RemotePojoInvocationCommandService.execute(RemotePojoInvocationCommandService.java:184) at sun.reflect.GeneratedMethodAccessor235.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201) at $Proxy370.execute(Unknown Source) at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:335) at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:230) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734) at org.jboss.remoting.transport.servlet.ServletServerInvoker.processRequest(ServletServerInvoker.java:231) at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201) at $Proxy347.processRequest(Unknown Source) at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:128) at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.doPost(ServerInvokerServlet.java:157) at javax.servlet.http.HttpServlet.service(HttpServlet.java:710) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:109) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104) at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) at java.lang.Thread.run(Thread.java:595) Caused by: java.sql.BatchUpdateException: ORA-00001: unique constraint (JON.RHQ_PACKAGE_VERSION_IDX) violated at oracle.jdbc.driver.DatabaseError.throwBatchUpdateException(DatabaseError.java:343) at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:10720) at org.jboss.resource.adapter.jdbc.CachedPreparedStatement.executeBatch(CachedPreparedStatement.java:478) at org.jboss.resource.adapter.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:519) at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:48) at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:246) ... 103 more This looks like it's directly related to the fact that packages, if not already known to the system, are added if they are discovered. If we're starting 90 agents at roughly the same time, I'm not surprised the reports are hitting. In addition to addressing this with regard to agent resource discoveries, we'll run into the same issue if a content source introduces a package into the DB at the same time as an inventory report. This will probably be a less common scenario, but still very possible. need to write some REQUIRES_NEW methods to create new package versions. We need to catch the exception if it fails and retry it or do something to see if it already exists to catch the fact that something else might have already created it concurrently to our create attempt. same with package objects same with architecture objects i'm questioning how I should do this. right now we have methods like this: @TransactionAttribute(value = TransactionAttributeType.REQUIRES_NEW) public PackageVersion createPackageVersion(String packageName, int packageTypeId, String version, int architectureId, InputStream packageBitStream) { public void mergeDiscoveredPackages(ContentDiscoveryReport report) { These methods check to see if the package version exists, if it does, reuses it; otherwise, it creates it and attaches bits to it. The content source sync stuff does similar things. But this is where the constraint violations come into play. If we concurrently call these methods, we get bad things happen. So, the thought is to do this (callers should only call this if they already found that there is no PV in the DB): @TransactionAttribute(REQUIRES_NEW) PackageVersion findOrPersistPackageVersionSafely(PackageVersion pv) { try { pv = entitManager.persist(pv); entityManager.flush(); // force it to go out to the DB return pv; } catch (Exception e) { // if this exception was because the PV already exists, we should be able to find it // if we can find it, can we ignore this exception assuming it was because the PV existed? // if we can't find it, throw the original error since it needs to be reported List l = entityManager.createNamedQuery(PackageVersion.FIND_ME, ...).getResultList(); if (l.count() != 1) throw e; if (l.count() > 1) throw new RuntimeException("Got more than 1 package version"); return l.get(0); } } If two threads end up in here (after they both found the PV is not already in the DB), one will succeed in that try block, the other will get an exception. At which time the catch clause will find the new PV. I think I need a second method here, to ensure our tx is committed (to avoid some concurrency issues): @TransactionAttribute(REQUIRES_NEW) PackageVersion persistPackageVersion(PackageVersion pv) { return entityManager.persist(pv); } @TransactionAttribute(REQUIRES) // reuse existing tx if we are already in one) PackageVersion persistOrMergePackageVersionSafely(PackageVersion pv) { try { pv = persistPackageVersion(); } catch (Exception e) { // if this exception was because the PV already exists, we should be able to find it // if we can find it, can we ignore this exception assuming it was because the PV existed? // if we can't find it, throw the original error since it needs to be reported List l = entityManager.createNamedQuery(PackageVersion.FIND_ME, ...).getResultList(); if (l.count() != 1) throw e; if (l.count() > 1) throw new RuntimeException("Got more than 1 package version"); pv.setId(l.get(0).getId); entityManager.merge(pv); // the caller might have something that it wanted to update in the PV } return pv; } rev248 implements the solution, with a unit test that shows it working i need to go through our code that persists packages and package versions and make sure they use this as appropriate rev254 switched our stuff to use the new persistOrMergePackage[Version]Safely methods. this should fix the constraint violations. some more testing needs to be done, but I tried syncing content sources, merging content reports and that stuff still works as it did before. so at least I didn't regress. i'd need to see our 90 agent perftest run - if we can avoid the constr. violations in that test, I'll consider this fixed. this didn't fix the problem - still getting the constr. violations :-/ HA! I scared myself... good thing I added this log message: 2008-03-14 14:54:35,093 WARN [org.rhq.enterprise.server.content.ContentManagerBean] There was probably a very big and ugly EJB/hibernate error just above this log message - you can normally ignore that. We detected that a package was already created when we tried to do it also - we will ignore this and just use the new package that was created in the other thread There is nothing we can do about getting that ugly message to not show up, short of us turning off all logging for the arjuna log4j category but then we wouldn't be able to see "real" errors when we get them. this is working now, I can see content reports come in and get processed and I see the constraint violations be worked around This bug was previously known as http://jira.rhq-project.org/browse/RHQ-45 |