It looks more like a JCA issue as the root cause is coming from inside there where we request a connection?

I think there was a bug with ER6 that meant :reload didn;t work at all so that could have masked an underlying issue?

Jesper can you shed light on the IllegalStateException please?

Looks like that <driver> has been set correctly, so post your entire configuration of the :datasources: subsystem, include the content of the module itself.

on EAP 6.2.0.CR3 same behavior

standalone.xml - datasources subsystem:
*********************

<subsystem xmlns="urn:jboss:domain:datasources:1.1">
            <datasources>
                <datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true">
                    <connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url>
                    <driver>h2</driver>
                    <security>
                        <user-name>sa</user-name>
                        <password>sa</password>
                    </security>
                </datasource>
                <xa-datasource jndi-name="java:jboss/xa-datasources/CrashRecoveryDS" pool-name="CrashRecoveryDS" enabled="true">
                   <xa-datasource-property name="ServerName">
                        db01.mw.lab.eng.bos.redhat.com
                   </xa-datasource-property>  
                   <xa-datasource-property name="PortNumber">  
                        3306
                   </xa-datasource-property>  
                   <xa-datasource-property name="DatabaseName">  
                        dballo01
                   </xa-datasource-property>  
                   <xa-datasource-class>com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</xa-datasource-class>  
                   <driver>mysql-connector-java-5.1.23-bin.jar</driver>  
                   <security>  
                       <user-name>dballo01</user-name>  
                       <password>dballo01</password>  
                   </security>  
                </xa-datasource>
                <drivers>
                    <driver name="h2" module="com.h2database.h2">
                        <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
                    </driver>
                </drivers>
            </datasources>
        </subsystem>

*********************

Driver mysql-connector-java-5.1.23-bin.jar is deployed in standalone/deployments directory.

Use a proper module deployment of the MySQL JDBC driver.

Same behavior when I add jdbc driver as a module. 

jboss-eap-6.2$ ls modules/org/mysql/jdbc/main/
module.xml  mysql-connector-java-5.1.23-bin.jar

jboss-eap-6.2$ cat modules/org/mysql/jdbc/main/module.xml 
<?xml version="1.0" encoding="UTF-8"?>
<module xmlns="urn:jboss:module:1.1" name="org.mysql.jdbc">
    <resources>
        <resource-root path="mysql-connector-java-5.1.23-bin.jar"/>
    </resources>

    <dependencies>
        <module name="javax.transaction.api" />
        <module name="javax.api" />
    </dependencies>
</module>

standalone.xml - datasources subsystem:
*********************
<subsystem xmlns="urn:jboss:domain:datasources:1.1">
            <datasources>
                <datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true">
                    <connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url>
                    <driver>h2</driver>
                    <security>
                        <user-name>sa</user-name>
                        <password>sa</password>
                    </security>
                </datasource>
                <xa-datasource jndi-name="java:jboss/xa-datasources/CrashRecoveryDS" pool-name="CrashRecoveryDS" enabled="true">
                        <xa-datasource-property name="ServerName">     
                                db01.mw.lab.eng.bos.redhat.com
                        </xa-datasource-property>     
                        <xa-datasource-property name="PortNumber">     
                                3306
                        </xa-datasource-property>     
                        <xa-datasource-property name="DatabaseName">     
                                dballo00
                        </xa-datasource-property>     
                        <driver>mysql</driver>     
                        <security>     
                                <user-name>dballo00</user-name>
                                <password>dballo00</password>
                        </security>     
                </xa-datasource>
                <drivers>
                    <driver name="h2" module="com.h2database.h2">
                        <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
                    </driver>
                    <driver name="mysql" module="org.mysql.jdbc">
                        <xa-datasource-class>com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</xa-datasource-class>
                    </driver>
                </drivers>
            </datasources>

I have double checked and JCA (XaDataSourceService) is removing correctly recovery resource during its stop(). And I've double checked the ISE appear after a full stop() happen correctly calling this code

XAResourceRecoveryRegistry rr = transactionIntegrationValue.getValue().getRecoveryRegistry();

            for (XAResourceRecovery recovery : deploymentMD.getRecovery()) {
                if (recovery != null) {
                    try {
                        recovery.shutdown();
                    } catch (Exception e) {
                        ConnectorLogger.SUBSYSTEM_DATASOURCES_LOGGER.error("Error during recovery shutdown", e);
                    } finally {
                        rr.removeXAResourceRecovery(recovery);
                    }
                }
            }

I think problem is on TS side, and I'm reassigning the issue to Tom. Of course I'm more than open to work together on this one.
I have also a commit, w/ few changes to have a more correct services up/down order in border situation, but I'm not rising PR for this, since it's not related IMHO to this issue and it's not eligible for a backporting.

Does this happen always and repeatedly or just once? 

It may be possible to interleave:
https://github.com/jbosstm/narayana/blob/master/ArjunaJTA/jta/classes/com/arjuna/ats/internal/jta/recovery/arjunacore/XARecoveryModule.java#L109

With:
https://github.com/jbosstm/narayana/blob/master/ArjunaJTA/jta/classes/com/arjuna/ats/internal/jta/recovery/arjunacore/XARecoveryModule.java#L506

Assigning to Mike to take a look once you have answered.

Thanks,
Tom

removeXAResourceRecovery must be a blocking call, if 'recovery' is currently active for recovery. Then an exclusive lock must be used for the removal of the instance. So, yes, race condition atm

Ondrej,

I fixed some locking issues in this area in revision 4.17.23 (JBTM-2241, JBTM-2242). Would it be possible to retest using this version of jbossts?

Hi Mike,

I've tested it on EAP 6.4.0.DR7 where the 4.17.23 is shipped in and I haven't been able to reproduce the issue yet.
(The issue is reproducable with 4.17.22)

Setting as verified.

Thanks
Ondra

Michael Musgrove <mmusgrov@redhat.com> updated the status of jira JBTM-2282 to Resolved

Tom Jenkinson <tom.jenkinson@redhat.com> updated the status of jira JBTM-2282 to Closed