Hide Forgot
Date of First Response: 2009-09-01 12:31:12 Help Desk Ticket Reference: https://enterprise.redhat.com/issue-tracker/?module=issues&action=view&tid=336913 project_key: SOA DESCRIPTION: Under the scenario reported at JBPM-1953, StaleObjectStateException is still seen after configuring new jms implementation introduced by SOA-1310 in SOA-P 4.3.0CP02. ENVIRONMENT: JBoss: SOA-P 4.3.0.GA_CP02_CR02 Database: PostgreSQL 8.3.4 JDBC Driver: postgresql-8.1-410.jdbc3.jar HOW TO REPRODUCE: 1. create a jBPM database on PosgreSQL. 2. make sure that target database is changed to PostgreSQL. - cd <SOA_HOME>/tools/schema/ - ant 3. copy jbpm.esb/config/jmsscheduler/*.config to ones under jbpm.esb. - copy jbm-queue-service.xml.config to jbpm.esb/jbm-queue-service.xml - copy jbpm.cfg.xml.config to jbpm.esb/jbpm.cfg.xml - copy jbpm-service.xml.config to jbpm.esb/jbpm 4. make sure DatabaseInitializer has jbpm.jbdl.postgresql.sql in SqlFiles attribute of jbpm-service.xml. For example) <mbean code="org.jboss.internal.soa.esb.dependencies.DatabaseInitializer" name="jboss.esb:service=JBPMDatabaseInitializer"> <attribute name="Datasource">java:/JbpmDS</attribute> <attribute name="ExistsSql">select count(*) from JBPM_ID_USER</attribute> <attribute name="SqlFiles"> jbpm-sql/jbpm.jpdl.postgresql.sql </attribute> <depends>jboss.jca:service=DataSourceBinding,name=JbpmDS</depends> <attribute name="UseEOL">true</attribute> </mbean> 5. run SOA-P instance then jBPM related tables etc. are created. 6. make sure no errors are found in server.log. 7. run bpm_orchestration2 exapmle. - cd <soa_home>/examples/quickstarts/bpm_orchestration2 - ant deploy - ant deployProcess - ant startProcess Then you will see org.hibernate.StaleObjectStateException in server log after repeating 'ant startProcess'. For example) 2009-09-01 12:20:32,923 ERROR [org.jbpm.graph.def.GraphElement] action threw exception: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [org.jbpm.graph.exe.Token#167] org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [org.jbpm.graph.exe.Token#167] at org.hibernate.dialect.lock.SelectLockingStrategy.lock(SelectLockingStrategy.java:78) at org.hibernate.persister.entity.AbstractEntityPersister.lock(AbstractEntityPersister.java:1338) at org.hibernate.event.def.AbstractLockUpgradeEventListener.upgradeLock(AbstractLockUpgradeEventListener.java:88) at org.hibernate.event.def.DefaultLockEventListener.onLock(DefaultLockEventListener.java:64) at org.hibernate.impl.SessionImpl.fireLock(SessionImpl.java:584) at org.hibernate.impl.SessionImpl.lock(SessionImpl.java:576) at org.jbpm.graph.node.Join.execute(Join.java:121) at org.jbpm.graph.def.Node.enter(Node.java:375) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:173) at org.jbpm.graph.def.Node_$$_javassist_49.enter(Node_$$_javassist_49.java) at org.jbpm.graph.def.Transition.take(Transition.java:167) at org.jbpm.graph.def.Node.leave(Node.java:453) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:173) at org.jbpm.graph.def.Node_$$_javassist_49.leave(Node_$$_javassist_49.java) at org.jbpm.graph.exe.Token.signal(Token.java:221) at org.jbpm.graph.exe.Token.signal(Token.java:149) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:173) at org.jbpm.graph.exe.Token_$$_javassist_64.signal(Token_$$_javassist_64.java) at org.jbpm.command.SignalCommand.execute(SignalCommand.java:97) at org.jboss.soa.esb.services.jbpm.cmd.AsyncProcessSignal$AsyncSignalAction.execute(AsyncProcessSignal.java:304) at org.jbpm.graph.def.Action.execute(Action.java:137) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:173) at org.jbpm.graph.def.Action_$$_javassist_63.execute(Action_$$_javassist_63.java) at org.jbpm.graph.def.GraphElement.executeAction(GraphElement.java:284) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:173) at org.jbpm.graph.def.Node_$$_javassist_49.executeAction(Node_$$_javassist_49.java) at org.jbpm.job.ExecuteActionJob.execute(ExecuteActionJob.java:33) at org.jboss.soa.esb.services.jbpm.integration.job.ExecuteJobCommand.executeJob(ExecuteJobCommand.java:137) at org.jboss.soa.esb.services.jbpm.integration.job.ExecuteJobCommand.execute(ExecuteJobCommand.java:125) at org.jboss.soa.esb.services.jbpm.integration.command.AbstractMessageListener.onMessage(AbstractMessageListener.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.internal.soa.esb.dependencies.JCAInflowAdapter$1.invoke(JCAInflowAdapter.java:276) at org.jboss.soa.esb.listeners.jca.EndpointProxy.delivery(EndpointProxy.java:242) at org.jboss.soa.esb.listeners.jca.EndpointProxy.invoke(EndpointProxy.java:145) at $Proxy60.onMessage(Unknown Source) at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178) at org.jboss.jms.client.container.ClientConsumer.callOnMessageStatic(ClientConsumer.java:160) at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:831) at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect16.invoke(SessionAspect16.java) at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105) at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java) at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java) at org.jboss.jms.client.JBossSession.run(JBossSession.java:199) at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237) at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:213) at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756) at java.lang.Thread.run(Thread.java:595)
Link: Added: This issue related SOA-1310
Attaching log.
Attachment: Added: server_JBPM-1953_bpm_orchestration2.log
Link: Added: This issue is related to JBPM-1953
The JBPM-1952 test case reproduces the situation where asynchronous forked nodes reach a join. When the join node executes, it acquires a pessimistic lock on the common parent of the arriving tokens. Before issuing a SELECT... FOR UPDATE statement tough, Hibernate does a version check and may throw a SOSE. Unfortunately, this Hibernate 'feature' kinda defeats the point of locking the parent token. I cannot think of a way around it. JBPM1952Test completes successfully by retrying the jobs after SOSEs occur in the join node. By setting async='exclusive' in the join node, SOSEs occur less frequently, although they do not fade away completely. <fork name='fork'> <transition to='c1' name='to_c1'/> <transition to='c2' name='to_c2'/> <transition to='c3' name='to_c3'/> </fork> <node name='c1' async='true'> <transition to='join' /> </node> <node name='c2' async='true'> <transition to='join' /> </node> <node name='c3' async='true'> <transition to='join' /> </node> <join name='join' async='exclusive'> <transition to='d' /> </join> Note that the message service must implement the async='exclusive' mode in order to mitigate SOSEs in the join node. 'Exclusive' means to acquire all other exclusive jobs for the current process instance and execute them serially.
This is no surprise and has been stated on a number of occasions, the work only removes the conflict during job acquisition. Any concurrency in the process execution, which remains unchanged, can still generate stale object exceptions
Try using <!-- JMS scheduler --> <service name="scheduler"> <factory> <bean class="org.jboss.soa.esb.services.jbpm.integration.timer.JmsSchedulerServiceFactory"> <field name="connectionFactoryJndiName"><string value="XAConnectionFactory"/></field> </bean> </factory> </service> <!-- End of JMS scheduler -->
Link: Added: This issue related JBPM-1952
Setting reference to related support case.
Help Desk Ticket Reference: Added: https://enterprise.redhat.com/issue-tracker/?module=issues&action=view&tid=336913
has anyone tried Jirka's suggestion?
I can't try it, I don't have the platform build. Would anyone be so kind to point me to the download location? Apart from that, I don't see how switching to the JMS scheduler will make any difference here. We are discussing conflicts during token join, not job acquisition.
We have been debugging this issue internally for the last couple of weeks, but had not come across this ticket, nor the related JBPM tickets referenced above. However, a little more reading, and some tinkering, and we have found a solution. I thought I would share it with you. In the execute() method of the Join, the parent token is locked using the following code: if (session!=null) { // force version increment by default (LockMode.FORCE) LockMode lockMode = parentLockMode != null ? LockMode.parse(parentLockMode) : LockMode.FORCE; log.debug("acquiring " + lockMode + " lock on " + parentToken); // lock updates as appropriate, no need to flush here session.lock(parentToken, lockMode); } The problem with this snippet, is that in Hibernate, session.lock(Object, LockMode), when used on a versioned object, locks the row not by ID, but by ID AND VERSION. This prevents a second thread from executing concurrently, but will always throw an SOSE in the subsequent threads when LockMode.FORCE is used, OR when the locked object is modified in any way, thereby changing it's version. To obtain a non-versioned lock on the object, we instead used session.load(Class, Id, LockMode), which locks the row based on ID alone, and, when successful at owning the lock, returns the current version. In our particular case, we have implemented a custom action which replaces the Join, and also performs some work of aggregating list variables from the multiple child token variable sets. The resulting variable is then saved as a variable on the parentToken. Because this causes a version bump on the parentToken, we have altered our lock to be on the ProcessInstance, rather than on the parentToken. These changes - locking on the ProcessInstance, and using the versionless lock via session.load() appears to have eliminated the SOSEs and related deadlocking that we have been seeing. I am attaching two source files, to illustrate. One is our custom Join action. The second is a util class which we used to extract the locking code during debugging, which is referenced in the Join action itself. Most of the join logic was originally cribbed from the standard Join source code (working off of JBPM version 3.2.5 at the time), but does not include special case joins such as nOfM, or scripted joins.
Source files for custom join.
Attachment: Added: VariableMergingJoinAction.java Attachment: Added: ProcessSynchronizationUtil.java
Tim, I tried your non-versioned lock suggestion and it does seem to eliminate a great deal of SOSEs even without async="exclusive". Thank you for this hint.
StaleObjectStateException is still seen after configuring jms scheduler, but BPMOrchestration2Test runs OK. (It passes every time). Verified on CP02 CR4
The changes which disable the logging contention will also be required, as will changes to the BPMOrchestraion2 QS. The process definition, as it currently stands, contains a contention on the variable assignment when handling the service responses.
added to 4.3.CP02 release notes as resolved: SOA-1476 The JMS implementation of the ESB to jBPM integration could produce Stale Object State Exceptions in many circumstances. Several of these have been eliminated.