Bug 780646 (SOA-3084)

Summary: Use MVCC locking scheme for jBPM
Product: [JBoss] JBoss Enterprise SOA Platform 5 Reporter: Martin Weiler <mweiler>
Component: JBPM - within SOAAssignee: Douglas Palmer <dpalmer>
Status: CLOSED NEXTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: 5.1.0 GACC: kevin.conner
Target Milestone: ---   
Target Release: 5.2.0 ER1   
Hardware: Unspecified   
OS: Unspecified   
URL: http://jira.jboss.org/jira/browse/SOA-3084
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Clustered SOA-P 5.1.0 instance using JBoss Cache
Last Closed: 2011-10-07 07:27:25 UTC Type: Task
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Martin Weiler 2011-05-24 08:55:44 UTC
Help Desk Ticket Reference: https://na7.salesforce.com/500A0000007B3Hg
project_key: SOA

Using the default JBoss Cache configuration for jBPM in a clustered SOA-P 5.1.0 instance, the following WARN messages can sometimes be found in the logs:

WARN  [org.jboss.cache.interceptors.OptimisticTxInterceptor] (pool-90-thread-3) Caught exception, will now set transaction to roll back
org.jboss.cache.optimistic.DataVersioningException: Transaction attempted to create /org/jbpm/graph/def/Node/ENTITY/org.jbpm.graph.def.Node#9812 anew.
It has already been created since this transaction started, by another (possibly remote) transaction.  We have a concurrent creation event.
     at org.jboss.cache.interceptors.OptimisticValidatorInterceptor.visitOptimisticPrepareCommand(OptimisticValidatorInterceptor.java:116)
     at org.jboss.cache.commands.tx.OptimisticPrepareCommand.acceptVisitor(OptimisticPrepareCommand.java:55)
     at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
     at org.jboss.cache.interceptors.OptimisticLockingInterceptor.visitOptimisticPrepareCommand(OptimisticLockingInterceptor.java:89)


As per the JBoss Cache User Guide [1], the optimistic node locking scheme (which is used here) is deprecated in EAP 5.

Therefore, the following two lines should be added to the jbpm.esb/hibernate.cfg.xml file in the clustered configurations to use MVCC instead of OPTIMISTIC node locking:
 
    <property name="hibernate.cache.region.jbc2.cfg.entity">mvcc-entity</property>
    <property name="hibernate.cache.region.jbc2.cfg.query">local-query</property>
 
[1] http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/5/html-single/JBoss_Cache_User_Guide/index.html# transactions.locks.dep

Comment 1 (please assign to mrietvel@redhat.com) 2011-07-06 10:31:32 UTC
Link: Added: This issue Cloned to SOA-3152


Comment 2 (please assign to mrietvel@redhat.com) 2011-07-06 10:32:21 UTC
Link: Removed: This issue Cloned to SOA-3152 


Comment 3 (please assign to mrietvel@redhat.com) 2011-07-06 10:36:25 UTC
Link: Added: This issue is related to JBPM-3281


Comment 4 (please assign to mrietvel@redhat.com) 2011-07-06 10:37:10 UTC
Link: Added: This issue Cloned to JBPM-3281


Comment 5 (please assign to mrietvel@redhat.com) 2011-07-06 10:37:20 UTC
Link: Removed: This issue is related to JBPM-3281 


Comment 6 (please assign to mrietvel@redhat.com) 2011-07-06 10:37:54 UTC
Link: Removed: This issue Cloned to JBPM-3281 


Comment 7 (please assign to mrietvel@redhat.com) 2011-07-06 10:38:19 UTC
Link: Added: This issue incorporates JBPM-3281


Comment 8 (please assign to mrietvel@redhat.com) 2011-07-06 10:51:39 UTC
Link: Removed: This issue incorporates JBPM-3281 


Comment 9 Kevin Conner 2011-07-12 08:56:48 UTC
This configuration is provided by the platform integration so should be handled by someone from the platform team.

Comment 11 (please assign to mrietvel@redhat.com) 2011-07-13 15:55:12 UTC
The short answer is: with regards to the that particular file, we only need to _add_ the lines specified above (hibernate properties).
But at the moment, there are some things that don't make sense to me. 

As of Hibernate-Caching 3.3, the default configuration for the hibernate mechanism that supports jBoss Cache in a clustered environment is such that it defaults to optimistic locking: the property in question is "{{hibernate.cache.region.jbc2.cfg.entity}}": ([3.3 doc|http://docs.jboss.org/jbossclustering/hibernate-caching/3.3/en-US/html_single/#sessionfactory-multiplexed]). In 3.5, it's been updated to MVCC ([3.5 doc|http://docs.jboss.org/jbossclustering/hibernate-caching/3.5/en-US/html_single/#sessionfactory-multiplexed]). 

I'm guessing that SOA 5.2.0  will still depend on Hibernate-Caching 3.3 (or less)? Otherwise, it doesn't seem like this would be necessary, given that Hibernate-Caching 3.5 has MVCC as the default. 

By adding the lines above, we're ensuring that hibernate-caching runs in a clustered environment with the MVCC as the default locking scheme. This will then support the jBoss Cache MVCC mechanism the way it wants to be supported (I think?). 

----

But, what I don't understand is that while we're modifying the hibernate properties, there's still the jBoss Cache (and *it's* configuration) on top of that:
{noformat}
---------------------
| Application (jBPM)|
---------------------
| jBoss Cache       |
---------------------
| Hibernate-Caching |
---------------------
| Database          |
---------------------
{noformat}

*Configuration*:
See [here|http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/5/html-single/JBoss_Cache_User_Guide/index.html#deployment.microcontainer] for an example -- in the xml there (scroll down a bit), you could add 

{{<attribute name="NodeLockingScheme">OPTIMISTIC</attribute>}}

to the config, and it would be optimistically locked. [The most recent documentation about optimistic (cache) node locking configuration is [here (old documentation)|http://docs.jboss.org/jbosscache/2.1.1.GA/userguide_en/html_single/index.html#d0e3708]
(scroll down to 10.1.3.3. Configuration).]

There is indeed a jBoss cache configuration file in the distribution module of jbpm: 
http://anonsvn.jboss.org/repos/jbpm/jbpm3/branches/jbpm-3.2-soa/distribution/src/main/resources/cache/jbpm-jbc-service.xml

*Old version of jBoss Cache:*
This file ({{jbpm-jbc-service.xml}}) -- of course -- does exactly what I outline above and sets NodeLockingScheme to Optimistic. Furthermore, it's not possible to use MVCC because we're using the (jBoss Cache) TreeCache object, which 1) doesn't have other NodeLockingScheme's besides Optimistic and  Pessimistic and 2) is discontinued after jBoss Cache 1.4.1 (<-> jBoss AS 4.0.5]
(See this [compatibility matrix|http://www.jboss.org/jbosscache/compatibility.html] for jBoss Cache <-> AS info). 

Of course, we're using this version of jBoss Cache because it's what dictated to us by the hibernate-jbc-cache dependency in distribution/pom.xml. We can not update the version of hibernate-jbc-cache in order to fix this, because we're already using the last version. 

*Possible fix:*
- It seems like it would also be necessary (or advisable) to update the jBPM jBoss Cache config so that it uses a Pojo Cache. 
- Obviously, that would also mean that we would update dependencies so that jBoss Cache 3.0.1 would be used instead of 1.4.1.
- Lastly, there are also possibly other dependencies that need to be updated, seeing as we would no longer use the hibernate-jbc-cache dependency (which limits us to jboss cache 1.4.1), but roll our own set. 

----

*Conclusion*
Most of the stuff above is for documentation purposes, should this come up again. 

At the time being, given that Martin knows what he's doing (and is unfortunately on vacation), we'll just go with adding those two lines. (It seems to already have been shown that just adding those two lines is enough to make things work.)


Comment 12 Jiri Pechanec 2011-10-07 07:27:25 UTC
Verified in ER4