Bug 924755

Summary: Failed to deliver OOB message: java.lang.IllegalStateException: channel is not connected
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Ladislav Thon <lthon>
Component: ClusteringAssignee: Paul Ferraro <paul.ferraro>
Status: CLOSED DEFERRED QA Contact: Jitka Kozana <jkudrnac>
Severity: high Docs Contact: Russell Dickenson <rdickens>
Priority: unspecified    
Version: 6.1.1CC: bmaxwell, cdewolf, jkudrnac, lcosti, mbabacek, myarboro, paul.ferraro, rhatlapa, rhusar
Target Milestone: ---Keywords: Reopened
Target Release: TBD EAP 7   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
Cause: ?? Consequence: Workaround (if any): Result:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-03-19 11:09:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ladislav Thon 2013-03-22 12:55:54 UTC
During EAP 6.1.0.ER3 testing, I've seen this exception:

14:04:41,965 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 87) ISPN000094: Received new cluster view: [perf19/ejb|5] [perf19/ejb, perf21/ejb, perf20/ejb, perf18/ejb]
14:04:41,966 ERROR [org.jgroups.protocols.pbcast.NAKACK2] (OOB-3,shared=udp) failed to deliver OOB message [dst: <null>, src: perf21/ejb (3 headers), size=5819 bytes, flags=OOB|DONT_BUNDLE, transient_flags=OOB_DELIVERED]: java.lang.IllegalStateException: channel is not connected
	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:615)
	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:544)
	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391)
	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249)
	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:598)
	at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130)
	at org.jgroups.JChannel.up(JChannel.java:707)
	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1020)
	at org.jgroups.protocols.RSVP.up(RSVP.java:188)
	at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
	at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
	at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896)
	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245)
	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:453)
	at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:721)
	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:574)
	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
	at org.jgroups.protocols.FD.up(FD.java:253)
	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
	at org.jgroups.protocols.MERGE3.up(MERGE3.java:290)
	at org.jgroups.protocols.Discovery.up(Discovery.java:359)
	at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2616)
	at org.jgroups.protocols.TP.passMessageUp(TP.java:1263)
	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1825)
	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1798)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_43]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_43]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]

See https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbremote-undeploy-repl-sync/21/console-perf18/

At the same time, perf21 logs

14:04:41,962 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-1,shared=udp) ISPN000094: Received new cluster view: [perf19/ejb|5] [perf19/ejb, perf21/ejb, perf20/ejb, perf18/ejb]
14:04:59,465 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (EJB default - 2) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf18/ejb
	at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:113)
	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:541)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234)
	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)
	at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:208)
	at org.infinispan.interceptors.ReplicationInterceptor.broadcastPrepare(ReplicationInterceptor.java:109)
	at org.infinispan.interceptors.ReplicationInterceptor.visitPrepareCommand(ReplicationInterceptor.java:101)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
	at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:108)
	at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:135)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
	at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
	at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128)
	at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
	at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
	at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189)
	at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
	at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:92)
	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124)
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:141)
	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123)
	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
	at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:230)
	at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:242)
	at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:86)
	at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)
	at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123)
	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105)
	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86)
	at org.infinispan.CacheImpl.endBatch(CacheImpl.java:684)
	at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:80)
	at org.jboss.as.clustering.infinispan.invoker.BatchCacheInvoker.invoke(BatchCacheInvoker.java:53)
	at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:81)
	at org.jboss.as.clustering.ejb3.cache.backing.infinispan.InfinispanBackingCacheEntryStore.update(InfinispanBackingCacheEntryStore.java:214)
	at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.update(SerializationGroupMemberContainer.java:280) [jboss-as-ejb3-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.update(SerializationGroupMemberContainer.java:51) [jboss-as-ejb3-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.as.ejb3.cache.impl.backing.PassivatingBackingCacheImpl.release(PassivatingBackingCacheImpl.java:227) [jboss-as-ejb3-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.release(AbstractCache.java:92) [jboss-as-ejb3-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.as.ejb3.cache.spi.impl.AbstractCache.release(AbstractCache.java:39) [jboss-as-ejb3-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor.releaseInstance(StatefulSessionSynchronizationInterceptor.java:199) [jboss-as-ejb3-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor.access$200(StatefulSessionSynchronizationInterceptor.java:50) [jboss-as-ejb3-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.as.ejb3.component.stateful.StatefulSessionSynchronizationInterceptor$StatefulSessionSynchronization.afterCompletion(StatefulSessionSynchronizationInterceptor.java:273) [jboss-as-ejb3-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402)
	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103)
	at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
	at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
	at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
	at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:91) [jboss-as-ejb3-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:252) [jboss-as-ejb3-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:315) [jboss-as-ejb3-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:214) [jboss-as-ejb3-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79) [jboss-as-ejb3-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:65) [jboss-as-ejb3-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:321) [jboss-as-ejb3-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:69) [jboss-as-ejb3-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:202) [jboss-as-ejb3-7.2.0.Final-redhat-3.jar:7.2.0.Final-redhat-3]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) [rt.jar:1.6.0_43]
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_43]
	at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_43]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_43]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_43]
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)

See https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbremote-undeploy-repl-sync/21/console-perf21/

So all nodes are up and they know about each other (they receive new view at the same time). Maybe the timeout is caused by some network glitch, maybe because ISPN overuses the OOB threadpool (I've seen some messages about that on the mailing list, but I highly doubt that it would cause this).

BTW, I've already seen the "failed to deliver OOB message" exception in the past, see https://bugzilla.redhat.com/show_bug.cgi?id=901016#c7, but I thought that it was connected to that issue. This time, it happened alone.

Comment 1 Radoslav Husar 2013-03-22 16:38:27 UTC
Unfortunately this is an Infinispan issue https://issues.jboss.org/browse/ISPN-2960

Comment 2 Ladislav Thon 2013-05-16 08:42:00 UTC
Update from EAP 6.1.0.ER8 testing cycle: This issue was not seen during this cycle.
This can mean that it was either fixed unintentionally (maybe as a by-product of the Infinispan upgrade to 5.2.6.Final) or this issue just became rarer.
Either way, we decided not to close this issue.

Comment 3 Ladislav Thon 2013-08-26 11:51:18 UTC
Still seeing this with EAP 6.1.1.ER7. For example:

https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbservlet-undeploy-repl-sync/32/

Comment 4 Paul Ferraro 2013-08-29 22:13:29 UTC
ISPN-2752 is still unresolved.

Comment 6 Radoslav Husar 2013-12-16 23:29:53 UTC
Graceful shutdown is not support in the affected version of EAP 6. The 
described behavior is to be expected.

The RFE to implement graceful shutdown is 
https://issues.jboss.org/browse/EAP6-7

Issues relating to HTTP can be avoided using mod_cluster and session 
draining prior to shutdown/undeploy.

Comment 7 Michal Karm Babacek 2014-03-12 17:00:57 UTC
Can I safely believe that these exceptions in EAP 6.3 DR2 are actually of the same nature? Three nodes, clustered webapp, restarting nodes. Nothing rapid, no load, nothing extraordinary. standalone-ha profile.

{noformat}
INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-10) ISPN000082: Stopping the RpcDispatcher
ERROR [org.jgroups.protocols.pbcast.NAKACK] (OOB-14,shared=udp) failed to deliver OOB message [dst: <null>, src: jboss-eap-6.3-3/web (4 headers), size=45 bytes, flags=OOB|DONT_BUNDLE|RSVP, transient_flags=OOB_DELIVERED]: java.lang.IllegalStateException: channel is not connected
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:617)
        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:544)
        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:391)
        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:249)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:600)
        at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130)
        at org.jgroups.JChannel.up(JChannel.java:707)
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1025)
        at org.jgroups.protocols.RSVP.up(RSVP.java:172)
        at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896)
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245)
        at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:453)
        at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:751)
        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:609)
        at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
        at org.jgroups.protocols.FD.up(FD.java:253)
        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
        at org.jgroups.protocols.MERGE3.up(MERGE3.java:290)
        at org.jgroups.protocols.Discovery.up(Discovery.java:359)
        at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2607)
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1260)
        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1822)
        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1795)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0]
        at java.lang.Thread.run(Thread.java:744) [rt.jar:1.8.0]

{noformat}

The node shows such an exception quite often when it " ISPN000094: Received new cluster view" closely prior to shutdown (via CLI).

Comment 8 Paul Ferraro 2014-05-21 13:26:13 UTC
This bug pertains to clean shutdown which is not scheduled to be implemented in EAP 6.x - and is targeted to be addressed in 7.0.  Setting resolution to WONTFIX.

Comment 10 Michal Karm Babacek 2015-01-20 14:41:35 UTC
Dear Paul, could you create an EAP 7 RFE JIRA for this issue so as it won't slip through the cracks? Thanks.

Comment 12 Carlo de Wolf 2015-03-19 11:09:53 UTC
Moved to https://issues.jboss.org/browse/EAP7-86

Comment 13 JBoss JIRA Server 2015-08-11 01:39:42 UTC
Jason Greene <jason.greene> updated the status of jira EAP7-86 to Resolved

Comment 14 JBoss JIRA Server 2015-08-13 12:59:27 UTC
Radim Hatlapatka <rhatlapa> updated the status of jira EAP7-86 to Reopened

Comment 15 JBoss JIRA Server 2015-12-09 16:23:51 UTC
Jason Greene <jason.greene> updated the status of jira EAP7-86 to Resolved

Comment 16 JBoss JIRA Server 2015-12-16 13:13:03 UTC
Radim Hatlapatka <rhatlapa> updated the status of jira EAP7-86 to Reopened

Comment 17 JBoss JIRA Server 2016-02-28 16:47:39 UTC
Jason Greene <jason.greene> updated the status of jira EAP7-86 to Resolved