Bug 1388911

Created attachment 1214283 [details]
Example application to be deployed on the client

Description of problem:

After an EAP JMS Server is restarted, threads on an EAP JMS Client threads attempting to re-connect to the server via RemoteConnectionFactory will hang. 

Version-Release number of selected component (if applicable): Reproduced on 6.4.6. I was able to reproduce the issue also on the latest CP (10).


How reproducible: 
Attached a sample project which contains a JMS Consumer which is triggered every minute by an EJB Timer. The JMS Consumer is an Asynchronous Stateless EJB. In order to show evidence of the issue, we have configured to fire 5 Consumers and the Async Thread size is set to 5.
When the issue happens, all EJB Threads will be stuck in org.hornetq.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:265)
Additionally, I have noted that the client EAP will be not manageable any more (the native management port will stop listening).


Steps to Reproduce:
1. start the server: ./standalone.sh -c standalone-full-server.xml
2. start the client  ./standalone.sh -c standalone-full-client.xml
3. deploy the jmsclient.war in the client EAP
4. The class Poller.java will schedule 5 Timers
5. When fired the EJB timers will log:

(EJB default - 5) Attempt to Read message
(EJB default - 4) Attempt to Read message
(EJB default - 1) Attempt to Read message
(EJB default - 2) Attempt to Read message
(EJB default - 3) Attempt to Read message

6. Since now, just before the receive timeout elapses (20 seconds) shut down the server and restart it a few seconds later. 

I was able to reproduce it based on the following timeline:
Timer fired [1 min]        Attempt to read message (20 sec)  Timer fired
---------------------------|----------------------|-----------------------------
                                              [Ko] [Restart]
Actual results:
EJB threads are stuck on:
java.lang.Thread.State: TIMED_WAITING
	at java.lang.Object.wait(Native Method)
	- waiting on <5617704b> (a org.hornetq.core.client.impl.ClientConsumerImpl)
	at org.hornetq.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:259)
	at org.hornetq.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:401)
	at org.hornetq.jms.client.HornetQMessageConsumer.getMessage(HornetQMessageConsumer.java:220)
	at org.hornetq.jms.client.HornetQMessageConsumer.receive(HornetQMessageConsumer.java:129)
	at demo.ProcessQueue.getQueuedRequest(ProcessQueue.java:80)
	at demo.ProcessQueue.process(ProcessQueue.java:44)

You won't see any other:
(EJB default - X) Attempt to Read message 

Expected results:
The EJB threads should not be locked. You should continue to see logging:
(EJB default - X) Attempt to Read message

Additional info: See https://c.na7.visual.force.com/apex/Case_View?id=500A000000VVsxs&sfdc.override=1

Please Note: standalone-full-server.xml and standalone-full-client.xml are attached to the case (01720588)