Bug 1035216

Summary: [GSS](6.3.0) ArrayIndexOutOfBoundsException during periodic recovery on EJBTransactionRecoveryService
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Ondrej Chaloupka <ochaloup>
Component: EJBAssignee: David M. Lloyd <david.lloyd>
Status: CLOSED CURRENTRELEASE QA Contact: Jan Martiska <jmartisk>
Severity: medium Docs Contact: Russell Dickenson <rdickens>
Priority: unspecified    
Version: 6.2.0, 6.2.1CC: bmaxwell, jkudrnac, kkhan, mbabacek, mmiura, pjelinek, rsvoboda
Target Milestone: ER2   
Target Release: EAP 6.3.0   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
An intermittent issue between the periodic recovery and EJBTransactionRecoveryService resulted in an ArrayIndexOutOfBoundsException. ---- [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016009: Caught:: java.lang.ArrayIndexOutOfBoundsException: 0 at org.jboss.as.ejb3.remote.EJBTransactionRecoveryService.getXAResources(EJBTransactionRecoveryService.java:112) .... ---- The root cause of the issue was that the creation of an 'XAResource' element was not contained within the associated logic loop and this led to the array index error. This element has now been moved so that it is within the logic loop and so the periodic recovery no longer conflicts with the EJBTransactionRecoveryService.
Story Points: ---
Clone Of:
: 1084872 (view as bug list) Environment:
Last Closed: 2014-06-28 15:42:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1084872    
Description Flags
Server.log running with standalone-ha none

Description Ondrej Chaloupka 2013-11-27 10:15:54 UTC
Created attachment 829633 [details]
Server.log running with standalone-ha

During running JBoss AS testsuite on testing EAP 6.2.0.CR3 we hit an error [1] in the log.

As the periodic recovery runs asynchronously this error is intermittent and is hard to reproduce it.

But I've checked the code of the EJBTransactionRecoveryService which looks in this way:
final XAResource[] xaResources = new XAResource[receiverContexts.size()];
synchronized (receiverContexts) {
  for (int i = 0; i < receiverContexts.size(); i++) {
    xaResources[i] = EJBClientManagedTransactionContext.getEJBXAResourceForRecovery(receiverContexts.get(i), arjunaTxCoreEnvironmentBean.getValue().getNodeIdentifier());
and at the first look I wonder whether the problem couldn't be that the xaResources array is not created under synchronized block?

I'm adding attachement of the testsuite run.

[com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016009: Caught:: java.lang.ArrayIndexOutOfBoundsException: 0
 at org.jboss.as.ejb3.remote.EJBTransactionRecoveryService.getXAResources(EJBTransactionRecoveryService.java:112)
 at com.arjuna.ats.internal.jbossatx.jta.XAResourceRecoveryHelperWrapper.getXAResources(XAResourceRecoveryHelperWrapper.java:51) [jbossjts-integration.jar:4.17.15.Final-redhat-4]
 at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecoveryForRecoveryHelpers(XARecoveryModule.java:516) [jbossjts-jacorb.jar:4.17.15.Final-redhat-4]
 at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass(XARecoveryModule.java:182) [jbossjts-jacorb.jar:4.17.15.Final-redhat-4]
 at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:743) [jbossjts-jacorb.jar:4.17.15.Final-redhat-4]
 at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-jacorb.jar:4.17.15.Final-redhat-4]

Comment 2 Masafumi Miura 2014-04-03 09:02:21 UTC
I agree to Ondrej. I think the line 109 "final XAResource[] xaResources = new XAResource[receiverContexts.size()];" should be placed inside synchronized block:

107     @Override
108     public XAResource[] getXAResources() {
109         final XAResource[] xaResources = new XAResource[receiverContexts.size()];
110         synchronized (receiverContexts) {
111             for (int i = 0; i < receiverContexts.size(); i++) {
112                 xaResources[i] = EJBClientManagedTransactionContext.getEJBXAResourceForRecovery(receiverContexts.get(i), arjunaTxCoreEnvironmentBean.getValue().getNodeIdentifier());
113             }
114         }
115         return xaResources;
116     }

I guess this "java.lang.ArrayIndexOutOfBoundsException: 0" will happen in the following scenario:

1. receiverContexts.size() = 0 at the line 109 
   xaResources is created with new XAResource[0]

2. receiverRegistered method is called from other thread between the line 109 and 110
   Now receiverContexts.size() = 1

3. As receiverContexts.size() = 1, for-loop is executed
   Then the line 112 "xaResources[0] = ..." will get ArrayIndexOutOfBoundsException because xaResources is empty array

Comment 3 Rostislav Svoboda 2014-04-04 11:33:27 UTC
Noticed in EAP 6.3.0 ER1 too.

David, could you please take a look at description and comment 2 about line 109?
If it is the case, the fix will be easy.

Comment 7 Ondrej Chaloupka 2014-05-05 10:50:22 UTC
Verified for EAP 6.3.0.ER3.

Run like:
export JAVA_OPTS="-Djboss.modules.system.pkgs=org.jboss.byteman -Dorg.jboss.byteman.transform.all -javaagent:${BYTEMAN_HOME}/lib/byteman.jar=script:/path/to/bz1035216-byteman-script.btm,boot:${BYTEMAN_HOME}/lib/byteman.jar,listener:false"
./bin/standalone.sh -c standalone-ha.xml

For ER1 the error occurs after few server restarts (issue is reproducible with byteman script and the server which is restarted several times). For ER3 the failure was not spotted.