Bug 779479 (SOA-1863) - jBPM 3.2.8 SP4 and SOA-P 5 - seeing likely deadlocks on database level for multiple databases, multiple tests
Summary: jBPM 3.2.8 SP4 and SOA-P 5 - seeing likely deadlocks on database level for mu...
Keywords:
Status: CLOSED NEXTRELEASE
Alias: SOA-1863
Product: JBoss Enterprise SOA Platform 5
Classification: JBoss
Component: JBPM - within SOA
Version: 5.0.0 ER7
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: ---
Assignee: Default User
QA Contact:
URL: http://jira.jboss.org/jira/browse/SOA...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-01-22 06:26 UTC by Jiri Pechanec
Modified: 2010-07-29 14:19 UTC (History)
0 users

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-07-29 14:19:53 UTC
Type: Bug


Attachments (Terms of Use)
jbpm.cfg.xml (2.85 KB, text/xml)
2010-01-22 06:27 UTC, Jiri Pechanec
no flags Details
hibernate.cfg.xml (13.47 KB, text/xml)
2010-01-22 06:28 UTC, Jiri Pechanec
no flags Details
mssql2005-testxml.zip (374.73 KB, application/zip)
2010-01-22 08:40 UTC, Jiri Pechanec
no flags Details
mssql2005-test.zip (159.37 KB, application/zip)
2010-01-22 08:41 UTC, Jiri Pechanec
no flags Details
non-SunJDKconfigfiles.zip (6.37 KB, application/zip)
2010-01-22 13:06 UTC, Jiri Pechanec
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker JBPM-2047 0 None Resolved PersistenceDbServiceTest and PersistenceServiceDbTest are dependent 2012-05-16 13:18:36 UTC
Red Hat Issue Tracker SOA-1863 0 None Closed jBPM 3.2.8 SP4 and SOA-P 5 - seeing likely deadlocks on database level for multiple databases, multiple tests 2012-05-16 13:18:36 UTC

Description Jiri Pechanec 2010-01-22 06:26:33 UTC
Date of First Response: 2010-01-22 11:17:35
project_key: SOA

See Hudson results

http://hudson.qa.jboss.com/hudson/view/SOA-Release/job/soa-jbpm-db-OpenJDK-maven/4/
http://hudson.qa.jboss.com/hudson/view/SOA-Release/job/soa-jbpm-db-jdk16-maven/48/

Tests org.jbpm.graph.node.JoinDbTest.testParentLockMode and org.jbpm.perf.SimplePerformanceTest.testAsyncCall are failing during each attempt of certification. There is no patter that can be identified - tests are failing on different DBs, platforms and JDKs. So it looks like a timing issue.

We are using http://svn.jboss.org/repos/jbpm/jbpm3/tags/jbpm-3.2.8.CR4 sorce codes to get tests for the certification.
The tests are executed with datasource not with JDBC connection and we are not using the JARs that are in pom.xml but those that comes with SOA-P.

The distribution used for the certification can be found at http://jawa05.englab.brq/candidate/soa-5.0.0_ER7/soa-5.0.0.ER7.zip. Each build has log file from the test attached - logging was set to DEBUG.

Please disregard Exception in thread "Thread-22" java.lang.StackOverflowError messages in job consoles log - it is known bug in Groovy and does not have any impact on test execution.

Test results relevant for certification are mysql51, postgresql83, oracle10g, mssql2005, db2.

We have also spotted test suite hanging on MySQL and SQLServer - an example is hanging on job http://hudson.qa.jboss.com/hudson/view/SOA-Release/job/soa-jbpm-db-jdk16-maven/DATABASE=mssql2005,jdk=java16_default,label=RHEL_any/48/ test JBPM1071Test - the snapshot of test.log and test results is added.
 
When embedded hsqldb is used then we sometimes observe on OpenJDK, HP and Solaris builds following failing test results http://hudson.qa.jboss.com/hudson/view/SOA-Release/job/soa-jbpm-os-jdk16-maven/jdk=java16_default,label=sol10_x86/19/ The failing tests almost every time same

Comment 1 Jiri Pechanec 2010-01-22 06:27:04 UTC
Attachment: Added: jbpm.cfg.xml


Comment 2 Jiri Pechanec 2010-01-22 06:28:24 UTC
Examples of hibernate.cfg.xml and jbpm.cfg.xml files used for testing are attached.

Comment 3 Jiri Pechanec 2010-01-22 06:28:24 UTC
Attachment: Added: hibernate.cfg.xml


Comment 4 Jiri Pechanec 2010-01-22 08:27:55 UTC
Regarding SimplePerformanceTest - check http://hudson.qa.jboss.com/hudson/view/SOA-Release/job/soa-jbpm-db-OpenJDK-maven/DATABASE=mssql2005,jdk=openjdk-local,label=RHEL5_x86_64_res/4/artifact/jbpm.3/modules/core/test.log
Last log message from the test is
00:45:50,773 DEBUG (main) [org.jbpm.perf.SimplePerformanceTest] waiting 5000 ms for 564 jobs to be executed
It seems that the executions has hanged in method getNbrOfJobsAvailable

Investigation of testParentLockMode shows that the exception is thrown during the update of db schema. Can this be caused by concurrent access by another thread to the database table? Or is it a bug in Hibernate?

Comment 5 Jiri Pechanec 2010-01-22 08:40:41 UTC
Attachment: Added: mssql2005-testxml.zip


Comment 6 Jiri Pechanec 2010-01-22 08:41:24 UTC
Attachment: Added: mssql2005-test.zip


Comment 7 Jiri Pechanec 2010-01-22 13:06:51 UTC
Config files used for Solaris tests with hsql database

Comment 8 Jiri Pechanec 2010-01-22 13:06:51 UTC
Attachment: Added: non-SunJDKconfigfiles.zip


Comment 9 Len DiMaggio 2010-01-22 16:17:35 UTC
Added regression flag

Comment 10 Len DiMaggio 2010-01-22 16:17:35 UTC
Affects Testing: Added: [Regression]


Comment 12 Martin Vecera 2010-01-27 15:00:09 UTC
A testing environment has been configured at jawa06.englab.brq.redhat.com, user jbpm, password - ask mvecera or tkirby at #jbosssoa or #jbosssoaqa. Enter the directory SOA-1863 and run the script run.sh.

Comment 13 Alejandro Guizar 2010-01-27 20:59:49 UTC
The Hibernate version is different: 3.3.2.GA vs. the declared dependency on 3.2.4.sp1. As Jiri noted, several failing tests are related to database metadata acquisition. I doubt this is a concurrency problem; instead I suspect a backwards incompatibility in Hibernate. Let me confirm with my local jbpm-3.2-soa branch copy.

Comment 14 Alejandro Guizar 2010-01-28 07:52:33 UTC
Link: Added: This issue related JBPM-2047


Comment 15 Alejandro Guizar 2010-01-28 07:52:34 UTC
The failure in org.jbpm.persistence.db.PersistenceServiceDbTest.testDefaults is related to JBPM-2047. I described the problem and attached a patch to that issue.

Comment 16 Alejandro Guizar 2010-01-28 08:27:48 UTC
Assesment of other failures:

1. org.jbpm.SerializabilityTest.testForNonSerializableClasses not finding org.jbpm.util.IoUtil$1 can be explained because the test console does not show it being extracted from resources-jbpm.zip.

2. org.jbpm.jbpm2574.JBPM2574Test.testLoadInexistentClass throwing JbpmException instead of ClassNotFoundException seems as though as JBPM-2574 had not been fixed, or as if a jBPM version was prior to 3.2.8, since JBPM-2574 was fixed in version 3.2.8.

3. org.jbpm.perf.SimplePerformanceTest.testAsyncCall and org.jbpm.jbpm2094.JBPM2094Test.testJobExecutorDeadlock may need their timeouts raised.

4. I thought org.jbpm.graph.node.JoinDbTest and org.jbpm.db.JbpmSchemaDbTest.testCreateSchema showed regressions in Hibernate 3.3.2 but after upgrading the version locally I found this was not the case.
This issue's description says "The tests are executed with datasource not with JDBC connection". It is possible that the data source provides wrapped connections that do not respond correctly to connection.getMetaData().getTables(). This is the code that throws java.lang.ClassCastException: $Proxy6 cannot be cast to java.sql.ResultSet. Note that getTables() returns ResultSet so it is not a bad cast in the calling code.

I will keep digging tomorrow, let me know if the above observations ring a bell to you.

Comment 17 Alejandro Guizar 2010-01-28 22:12:30 UTC
The Hudson jobs are downloading the jBPM distribution from:
http://hudson.qa.jboss.com/hudson/view/SOA-Release/job/soa-5.0/lastSuccessfulBuild/artifact/resources-jbpm.zip

If you download that zip file and check the Maven POMs you'll notice it's version 3.2.7. The jobs check out and run the 3.2.8 test suite, leading to various failures.

* org.jbpm.SerializabilityTest tries to load org.jbpm.util.IoUtil$1 and fails since that anonymous class was introduced in 3.2.8
* org.jbpm.jbpm2574.JBPM2574Test fails as JBPM-2574 was fixed in 3.2.8

Comment 18 Jiri Pechanec 2010-01-29 06:02:46 UTC
Hi,

let me explain the confusion. In the meantime we have received special build ER7 but with 3.2.7 jBPM - this is know is job soa-5.0.

I have also created job soa-5.0-exp that contains original ER7 build. If you start the build you are asked for entering version. You will use 5.0 to get ER7 with 3.2.7 and tested against 3.2.7 testsuite and 5.0-exp to get ER7 with 3.2.8 and tested against version 3.2.8

It is unfortunate that this happened. I suggest you to make a copy of job soa-jbpm-single-maven so you can repeatedly re-run the tests. The job is parametrized based on database (postgresql83,mysql51,oracle10g,mssql2005,db2-97 and you can choose yourself the machine or label on which it should be executed. As version you can use 5.0 (for 3.2.7) and 5.0-exp (for 3.2.8) to compare test results. Also you can try using option 'Execute concurrent builds if necessary' to run multiple configs in parallel.

Once again sorry for confusion - just one more thing jobs mentioned here were executed with 3.2.8 jBPM and with correct test suite version.

Thanks for pointing out of this weak spot - in future we will record project pom files as build artifacts to guarantee the traceability.

Comment 19 Jiri Pechanec 2010-01-29 07:13:41 UTC
There are two significant differentiators in this test against regular execution of testuite
1) The data-source configured is using H2 database that provides real transactions and transaction isolation. We are using read commited in SOA-P, hsqldb in default test suite is read uncommited
2) Data-source configuration by default caches connections - the unmodified test suite hibernate.cfg.xml does not use connection pooling at all, it is configured to create new connection every time a connection is needed.

Comment 20 Alejandro Guizar 2010-01-29 07:29:18 UTC
I can't reproduce the failures on either JoinDbTest or JbpmSchemaDbTest.

Since the host where Martin V set up the test environment was inaccessible today, I downloaded http://hudson.qa.jboss.com/hudson/job/soa-5.0-exp/lastSuccessfulBuild/artifact/binaries/soa-5.0.0.ER7.zip and tested it with a local MySQL and the one in vmg02 without seeing the failure.

Comment 21 Alejandro Guizar 2010-02-05 10:01:48 UTC
I've determined that the failures related to updating the database schema, e.g. JoinDbTest and JbpmSchemaDbTest derive from the use of a data source outside the container. If you try to invoke getMetaData().getTables() on a direct JDBC connection it works, likewise for a connection acquired from a data source and used within the container. However on remote use, the call throws ClassCastException: $Proxy6 cannot be cast to java.sql.ResultSet, indicating that the dynamic proxy does not implement the ResultSet interface. This seems to be driver specific.

Comment 22 John Graham 2010-07-29 14:19:53 UTC
Work completed to resolve this as far as possible. Future, specific cases of this, or similar, defects should be recorded as a separate JIRA.


Note You need to log in before you can comment on or make changes to this bug.