Bug 1141960

Created attachment 937721 [details]
Server.log, error is at tail.

This appears to be similar to this issue:

https://bugzilla.redhat.com/show_bug.cgi?id=852044

I will inquire over there.

I was not able to reproduce in single node configuration (standalone.xml) - OOTB installation worked just fine. User needs to run FSW in *-ha profile to reproduce, but this might be more of a configuration problem since S-RAMP is not configured to run in clustered environment ootb, see: https://bugzilla.redhat.com/show_bug.cgi?id=1035315

Tip from the sidelines: this error is almost always caused when a query/find/get/retrieval operation is done without a transaction. 

The problem is that postgres's Large Object Facility uses two tables when storing (B)LOBs where other databases use 1 table. Because two tables are used, a transaction is required and the query will fail if it isn't used. Needless to say, this is a PITA. 8/  

For example, let's say we have an entity: 

@Entity
@Table(name="IMAGE")
@SequenceGenerator(name="imageIdSeq", 
  sequenceName="IMAGE_ID_SEQ", 
  allocationSize=1)
public class Image {
    
    @Id
    @GeneratedValue(strategy = GenerationType.AUTO, generator="imageIdSeq")
    private Long   id;
    
    @Lob
    @Column(length=2147483647)
    // see the length attr? 
    // Trust me, that will save you time when dealing with MySQL.. 
    private byte[] content;    
}

In any other database, the actual record/structure of the table will look like this: 

Imgae table: 
  [ id ] [ byte array ] 

However, in Postgres, we get this: 

Image table: 
  [ id ] [ pointer/id to actual byte array ]

Large Object facility table: 
  [ byte array id ][ byte array ]

Oh sure, Postgres won't show this if you user Squirrel of your database GUI of choice, but this is what's happening under the hood. See the Postgres documentation for more info. 

Hope this helps! jBPM has run into this problem far too many times..

Marco - thanks for this information.  This definitely helps understand the root of the problem.

However, all of the TX management stuff is (I *think*) all done within the embedded jBPM- we don't do any transaction management in our code.  It's possible this is a jBPM bug that was fixed in 5.x but never applied to 6, I suppose.

Should I be trying to get access to the same TX manager that jBPM is using and starting the TX myself, wrapping our call to jBPM's TaskService?  Currently we simply @inject the TaskService into our code and use it as-is, with the expectation that all the JPA/TX stuff is handled internally to jBPM.

Do we by any chance have a smaller reproducer, for example reproducable using BPMS jars only?

As Marco mentioned, it seems a query (looking at the log getContentById) is executed outside a transaction.  

Could you share the code how the task service that is being used is created?

The task service is injected into a jax-rs service.  All usage of the task service goes through that jax-rs api.

https://github.com/Governance/dtgov/blob/dtgov-1.3.0.Final/dtgov-war/src/main/java/org/overlord/dtgov/taskapi/TaskApi.java#L84-L85

The call to getTask happens here:

https://github.com/Governance/dtgov/blob/dtgov-1.3.0.Final/dtgov-war/src/main/java/org/overlord/dtgov/taskapi/TaskApi.java#L238

What version of jBPM is being included (and at what time was it branched from the BPMS 6.0.x stream)?  There were significant changes related to task service transaction management during the 6.0 product life cycle.  It seems the version used here is not yet using those improvements (it's a 6.0.0-redhatx version, which seems to indicate it was branched somewhere between our ER5 and ER6 6.0 product builds).

Would it be possible to verify if the issue still exists in BPMS 6.0.0.GA (or maybe even 6.0.3.GA).

Another solution I believe might be to try and control the transaction boundary yourself (using UserTransaction or transaction annotation), as there is an issue with jBPM doing it automatically here it seems.

The version of jbpm in FSW 6.0 GA is:

6.0.0-redhat-9

I do not believe this has been modified by the patch(es).

I can't answer the *other* question regarding when it was branched from BRMS 6.0.x.  Kris, if you give me something specific to look for in one of the jbpm JARs I could decompile and find out what's there.

If an upgraded jbpm would fix the problem I think that's the best way to go.  If I were told what specific version to try in the product branch I'd be happy to do it.

Eric,

I don't know exactly what version of jBPM in the community that corresponds with, but I believe it's likely not to include the following commit:
https://github.com/droolsjbpm/jbpm/commit/cc430e3ccdf0bcb6fb0224950c1611c2c62aa351

This was a large change to align transaction mgmt for human tasks with the rest of the modules.

When upgrading, I believe the best candidate to test with would be the latest stable from the same branch, ie. 6.0.3-SNAPSHOT in the community, or 6.0.3.GA in the product (available from customer portal or the product maven repo).

Kris

Thanks Kris.

Stefan - do you think you'd be able to try this newer version of jbpm (6.0.3.GA)?  You could try replacing the jbpm JARs in standalone/deployments/dtgov-war/WEB-INF/lib with newer versions.

I will try to verify that version of jbpm works (in general) in the dtgov product branch.

Update:  Kevin and I are working on this issue (he's helping me reproduce).  I was looking at the wrong version of TaskApi - the one in FSW 6.0 uses bean managed transactions.  So I have added a transaction to the TaskApi::getTask() method in the hopes of solving this problem.

@Stefan - please do not try what I suggested in my last post. :)

Sure, I will those jars with the 6.0.3.GA from product maven repo and report back.

Eric, after fesh installation of FSW 6.0.0.GA and subsequent replacement of jbpm libraries in dtgov-war/WEB-INF/lib caused the following exception in server.log:

13:14:08,034 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 58) MSC000001: Failed to start service jboss.persistenceunit."dtgov.war#org.overlord.dtgov.jbpm": org.jboss.msc.service.StartException in service jboss.persistenceunit."dtgov.war#org.overlord.dtgov.jbpm": javax.persistence.PersistenceException: [PersistenceUnit: org.overlord.dtgov.jbpm] class or package not found
	at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1.run(PersistenceUnitServiceImpl.java:103)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.0.Final-redhat-1.jar:2.1.0.Final-redhat-1]
Caused by: javax.persistence.PersistenceException: [PersistenceUnit: org.overlord.dtgov.jbpm] class or package not found
	at org.hibernate.ejb.Ejb3Configuration.addNamedAnnotatedClasses(Ejb3Configuration.java:1416)
	at org.hibernate.ejb.Ejb3Configuration.addClassesToSessionFactory(Ejb3Configuration.java:1199)
	at org.hibernate.ejb.Ejb3Configuration.configure(Ejb3Configuration.java:1063)
	at org.hibernate.ejb.Ejb3Configuration.configure(Ejb3Configuration.java:701)
	at org.hibernate.ejb.HibernatePersistence.createContainerEntityManagerFactory(HibernatePersistence.java:91)
	at org.jboss.as.jpa.service.PersistenceUnitServiceImpl.createContainerEntityManagerFactory(PersistenceUnitServiceImpl.java:200)
	at org.jboss.as.jpa.service.PersistenceUnitServiceImpl.access$600(PersistenceUnitServiceImpl.java:57)
	at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1.run(PersistenceUnitServiceImpl.java:99)
	... 4 more
Caused by: java.lang.ClassNotFoundException: org.jbpm.services.task.impl.model.BAMTaskSummaryImpl from [Module "org.hibernate:main" from local module loader @171e7af3 (finder: local module finder @5219f360 (roots: /home/sbunciak/runtimes/fsw6-pgsql/jboss-eap-6.1/modules,/home/sbunciak/runtimes/fsw6-pgsql/jboss-eap-6.1/modules/system/layers/soa,/home/sbunciak/runtimes/fsw6-pgsql/jboss-eap-6.1/modules/system/layers/sramp,/home/sbunciak/runtimes/fsw6-pgsql/jboss-eap-6.1/modules/system/layers/base))]
	at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:196) [jboss-modules.jar:1.2.2.Final-redhat-1]
	at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:444) [jboss-modules.jar:1.2.2.Final-redhat-1]
	at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:432) [jboss-modules.jar:1.2.2.Final-redhat-1]
	at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:374) [jboss-modules.jar:1.2.2.Final-redhat-1]
	at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:119) [jboss-modules.jar:1.2.2.Final-redhat-1]
	at java.lang.Class.forName0(Native Method) [rt.jar:1.7.0_65]
	at java.lang.Class.forName(Class.java:270) [rt.jar:1.7.0_65]
	at org.hibernate.internal.util.ReflectHelper.classForName(ReflectHelper.java:170)
	at org.hibernate.ejb.Ejb3Configuration.classForName(Ejb3Configuration.java:1333)
	at org.hibernate.ejb.Ejb3Configuration.addNamedAnnotatedClasses(Ejb3Configuration.java:1405)
	... 11 more

Does it work for you?

Sorry Stefan - I tried canceling that request I made of you because we have some other things we're looking into.  But BZ won't let me delete a comment.

To answer your question - no it doesn't work for me, because there are other differences between the two jbpm versions, which would require updates to our persistence.xml at the very least.

For now, I believe we are trying to reproduce the problem reliably so that we can test some changes I made to one of the .class files in dtgov.war.

I will attach the updated class file to this bug for anyone who wants to try it.  Simply replace the one found in dtgov.war with the attached one.

Created attachment 941539 [details]
Updated TaskApi.class which adds TX management to "getTask()"

Created attachment 941716 [details]
Cust console shot

I have briefly tested Eric's patch and it seems working. After updating the TaskApi.class I was able to view task summary and complete the task.

Created attachment 943346 [details]
TaskApi.class v2

Here is the version of TaskApi.class that Stefan successfully tested on 10/2/2014.  I am still in the process of testing it out myself (thanks to Stefan reproducing the problem and setting up a test environment for me!).  But Stefan's initial tests indicate that it may be working, so perhaps we can get the user to give it a try?

Verified against PgSQL 9.2 (postgresql-9.2-1004.jdbc41.jar)

This product has been discontinued or is no longer tracked in Red Hat Bugzilla.