Bug 1204055

We are seeing a behavioral regression in 6.4.0.CR1. This only manifests in a single test case: https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-2clusters-ejbremote-shutdown-repl-async/

In this test, a standalone client invokes an EJB in a cluster ("forwarder") and that EJB itself invokes another EJB in a different cluster ("target"). During that test, we repeatedly bring one node down (:shutdown), wait a while, bring it back up, and continue with another node.

Specifically, in our setting, perf17 is the standalone client, perf18-19 are the "target" cluster and perf20-21 are the "forwarder" cluster.

See https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-2clusters-ejbremote-shutdown-repl-async/58/artifact/report/failover.txt for a summary of the test from perf17 point of view, running against 6.4.0.ER3, and https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-2clusters-ejbremote-shutdown-repl-async/59/artifact/report/failover.txt for the same summary against CR1.

You can see that with CR1, when perf19 is down, the standalone client stops receiving responses. We set a 2-minute invocation timeout on the standalone client (the "invocation.timeout" property in jboss-ejb-client.properties), and you can see in the perf17 log that this deadline is not met frequently (TimeoutException: No invocation response received in 120000 milliseconds).

During our investigation, we concluded that this is caused by the fix for bug 1192471, because when we reverted the commit (https://github.com/jbossas/jboss-ejb-client/commit/61bd6290), things went back to normal (see https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/lthon___eap-6x-failover-ejb-2clusters-ejbremote-shutdown-repl-async/15/artifact/report/failover.txt). Note that even though we only reverted that commit in the in-server EJB client library (and the standalone client still runs vanilla 1.0.29), the problem indeed disappeared.

The commit itself looks suspicious, because it doesn't hold a lock while copying synchronized collections. Yet, we didn't see any ConcurrentModificationExceptions, and when we tried to copy those collections properly (i.e., under the respective lock), it didn't make a difference.

(What follows is a huge speculation based on my very limited understanding of the EJB client library. It can easily be wrong or misleading.)

I believe that while the commit fixes problematic behavior in one situation, it causes problematic behavior in an opposite situation. Originally, when a single invocation figured out that a cluster node is not available, even before topology change was propagated, that single invocation deleted that cluster node from the collections and the other invocations didn't bother trying that node. This doesn't happen anymore. Until a topology change is propagated, all invocations will keep trying that node, even if it is down. This will cause a reconnection process to start in the background, and all the invocations will wait for that reconnection to happen. It only happens when the node is back up, which effectively means that client failover won't happen when it should.