Bug 1009931

Summary: EAP fails to :reload with transactions configured to be run with JDBCObject store
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Ondrej Chaloupka <ochaloup>
Component: Transaction ManagerAssignee: tom.jenkinson
Status: CLOSED CURRENTRELEASE QA Contact: Ondrej Chaloupka <ochaloup>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 6.2.0CC: brian.stansberry, smaestri, tom.jenkinson
Target Milestone: ER6Keywords: TestBlocker
Target Release: EAP 6.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-12-15 16:12:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1012986    
Bug Blocks:    
Attachments:
Description Flags
Oracle driver none

Description Ondrej Chaloupka 2013-09-19 14:00:05 UTC
I have problem to reload server when I have configured transactions to use the JDBC store. 
After reload I'm getting following error message and the TM stops to work and the following transactions are rollbacked.

WARN  [com.arjuna.ats.arjuna] (Transaction Expired Entry Monitor) ARJUNA012251: allObjUids caught exception: java.sql.SQLException: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:jboss/datasources/jdbc-store
 at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:147)
 at com.arjuna.ats.internal.arjuna.objectstore.jdbc.accessors.DataSourceJDBCAccess.getConnection(DataSourceJDBCAccess.java:51) [jbossjts-jacorb-4.17.9.Final-redhat-1.jar:4.17.9.Final-redhat-1]
 at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCImple_driver.allObjUids(JDBCImple_driver.java:316) [jbossjts-jacorb-4.17.9.Final-redhat-1.jar:4.17.9.Final-redhat-1]
 at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCStore.allObjUids(JDBCStore.java:179) [jbossjts-jacorb-4.17.9.Final-redhat-1.jar:4.17.9.Final-redhat-1]
 at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCStore.allObjUids(JDBCStore.java:119) [jbossjts-jacorb-4.17.9.Final-redhat-1.jar:4.17.9.Final-redhat-1]
 at com.arjuna.ats.internal.arjuna.recovery.ExpiredTransactionStatusManagerScanner.scan(ExpiredTransactionStatusManagerScanner.java:81) [jbossjts-jacorb-4.17.9.Final-redhat-1.jar:4.17.9.Final-redhat-1]
 at com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor.run(ExpiredEntryMonitor.java:171) [jbossjts-jacorb-4.17.9.Final-redhat-1.jar:4.17.9.Final-redhat-1]
Caused by: javax.resource.ResourceException: IJ000451: The connection manager is shutdown: java:jboss/datasources/jdbc-store
 at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:321)
 at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:302)
 at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)
 at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139)
 ... 6 more

Comment 2 Ondrej Chaloupka 2013-09-19 14:02:23 UTC
Created attachment 799966 [details]
Oracle driver

Comment 5 Ondrej Chaloupka 2013-09-19 14:15:11 UTC
This could influence the customer as well. Supposing not being just testing blocker. Setting blocker to ?.

Comment 6 tom.jenkinson 2013-09-19 14:44:37 UTC
Hi Ondra,

Which version of EAP 6.2 did you try this on and where can I download it from please?

Thanks,
Tom

Comment 8 tom.jenkinson 2013-09-19 16:17:19 UTC
Hi Ondra,

Wouldn't we need:
https://bugzilla.redhat.com/show_bug.cgi?id=971358

It doesn't look like stefano has resolved that yet?

Tom

Comment 9 tom.jenkinson 2013-09-19 22:21:25 UTC
OK, so I have had a look at this. The problem is that the JDBCstore caches the lookup of the datasource before calling getconnection. Assuming there isn't a performance issue with re-resolving the JDBC store every time we call getConnection I can change getConnection to lookup the datasource each time.

I have solved this in JBTM-1938

Comment 10 Ondrej Chaloupka 2013-09-20 05:57:59 UTC
Hi Tom,

thanks for the fix.

From my point of view the bz
https://bugzilla.redhat.com/show_bug.cgi?id=971358
was fixed. But Stefano could have some more input on it. I'm not sure.

Ondra

Comment 11 tom.jenkinson 2013-09-20 14:45:20 UTC
So, having made the fix in the narayana repo,

Comment 12 tom.jenkinson 2013-09-27 10:34:42 UTC
For information: this has the same severity as 1011379, both are fixed in 4.17.11.

Comment 13 tom.jenkinson 2013-10-01 11:15:47 UTC
https://github.com/jbossas/jboss-eap/pull/478

Comment 14 JBoss JIRA Server 2013-10-01 16:07:42 UTC
Michael Musgrove <mmusgrov> updated the status of jira JBTM-1938 to Closed

Comment 18 Ondrej Chaloupka 2013-10-17 12:18:37 UTC
Verified on EAP 6.2.0.ER6. It's possible to reload server and no error messages are received on the console.

Comment 19 Ondrej Chaloupka 2013-10-17 14:03:31 UTC
Hi,

I'm sorry I've found out that my previous statement is not true. 

The problem with reloading the server set with JDBC driver still occur in some cases. But there is a new error now. I'm running on EAP 6.2.0.ER6.

From the log it seems to me as a timing issue. Time to time the data source could be bound after the transactions subsystem is started.

Console log:
15:05:13,272 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-4) JBAS010403: Deploying JDBC-compliant driver class oracle.jdbc.OracleDriver (version 11.2)
15:05:13,283 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-3) JBAS010400: Bound data source [java:jboss/datasources/jdbc-store]
15:05:13,284 ERROR [stderr] (Transaction Expired Entry Monitor) Exception in thread "Transaction Expired Entry Monitor" com.arjuna.ats.arjuna.exceptions.FatalError: com.arjuna.ats.internal.arjuna.objectstore.jdbc.accessors.DataSourceJDBCAccess@1fc261da : javax.naming.NameNotFoundException: Error looking up datasources/jdbc-store, service service jboss.naming.context.java.jboss.datasources.jdbc-store is not started
15:05:13,285 ERROR [stderr] (Transaction Expired Entry Monitor)         at com.arjuna.ats.internal.arjuna.objectstore.jdbc.accessors.DataSourceJDBCAccess.getConnection(DataSourceJDBCAccess.java:55)
15:05:13,285 ERROR [stderr] (Transaction Expired Entry Monitor)         at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCImple_driver.allObjUids(JDBCImple_driver.java:316)
15:05:13,286 ERROR [stderr] (Transaction Expired Entry Monitor)         at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCStore.allObjUids(JDBCStore.java:179)
15:05:13,286 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
15:05:13,286 ERROR [stderr] (Transaction Expired Entry Monitor)         at com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCStore.allObjUids(JDBCStore.java:119)
15:05:13,287 ERROR [stderr] (Transaction Expired Entry Monitor)         at com.arjuna.ats.internal.arjuna.recovery.ExpiredTransactionStatusManagerScanner.scan(ExpiredTransactionStatusManagerScanner.java:81)
15:05:13,287 ERROR [stderr] (Transaction Expired Entry Monitor)         at com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor.run(ExpiredEntryMonitor.java:171)
15:05:13,288 ERROR [stderr] (Transaction Expired Entry Monitor) Caused by: javax.naming.NameNotFoundException: Error looking up datasources/jdbc-store, service service jboss.naming.context.java.jboss.datasources.jdbc-store is not started
15:05:13,288 ERROR [stderr] (Transaction Expired Entry Monitor)         at org.jboss.as.naming.ServiceBasedNamingStore.lookup(ServiceBasedNamingStore.java:132)
15:05:13,288 ERROR [stderr] (Transaction Expired Entry Monitor)         at org.jboss.as.naming.ServiceBasedNamingStore.lookup(ServiceBasedNamingStore.java:80)
15:05:13,288 ERROR [stderr] (Transaction Expired Entry Monitor)         at org.jboss.as.naming.NamingContext.lookup(NamingContext.java:197)
15:05:13,289 ERROR [stderr] (Transaction Expired Entry Monitor)         at org.jboss.as.naming.NamingContext.lookup(NamingContext.java:174)
15:05:13,289 ERROR [stderr] (Transaction Expired Entry Monitor)         at org.jboss.as.naming.InitialContext.lookup(InitialContext.java:122)
15:05:13,289 ERROR [stderr] (Transaction Expired Entry Monitor)         at org.jboss.as.naming.NamingContext.lookup(NamingContext.java:183)
15:05:13,289 ERROR [stderr] (Transaction Expired Entry Monitor)         at org.jboss.as.naming.NamingContext.lookup(NamingContext.java:179)
15:05:13,289 ERROR [stderr] (Transaction Expired Entry Monitor)         at javax.naming.InitialContext.lookup(InitialContext.java:411)
15:05:13,290 ERROR [stderr] (Transaction Expired Entry Monitor)         at com.arjuna.ats.internal.arjuna.objectstore.jdbc.accessors.DataSourceJDBCAccess.getConnection(DataSourceJDBCAccess.java:53)
15:05:13,290 ERROR [stderr] (Transaction Expired Entry Monitor)         ... 5 more
15:05:13,317 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 26) JBAS018559: Deployed "oracle11gR2-jdbc-driver.jar" (runtime-name : "oracle11gR2-jdbc-driver.jar")

Comment 20 tom.jenkinson 2013-10-18 09:17:11 UTC
Hi Ondra,

I think it is a different cause. The last issue was that the JDBC object store caches connections, this new issue seems to be related to there not being a dependency between the TM and JCA services (i.e. from a BZ point of view, this new issue could affect the initial load, not just a :reload)?

I will look further...

Tom

Comment 21 Ondrej Chaloupka 2013-10-18 10:35:42 UTC
Hi Tom,

I agree this is different issue which is related to module dependency management in AS rather than to transactions subsystem.
Just to clarify the problem happens to me just after the reload. I didn't experience the problem after the initial load. But it could be a coincidence.

I will report this problem as a new bz and I will verify this one.

Thanks
Ondra

Comment 23 Ondrej Chaloupka 2013-10-18 12:37:57 UTC
Verifying this issue.

I created new bugzilla as mentioned above.
https://bugzilla.redhat.com/show_bug.cgi?id=1020881