Bug 745845 (EDG-115)

A Hudson job showing the failure can be found at:

https://hudson.qa.jboss.com/hudson/view/EDG/job/JBoss-EAP-5.1.x-datagrid/65/

Sorry Richard, I had missed this issue. So, it'd appear that failover in the Hot Rod client is not working as expected? I'll check the test in the EDG source code and try to replicate it in Infinispan.

Attachment: Added: Configuration_changes_for_JBoss_mbean_server_lookup.patch

Attachment: Removed: Configuration_changes_for_JBoss_mbean_server_lookup.patch 

Richard, I had a look to this and I think your tests need improving to verify that a cluster actually forms between the two nodes that you start. I've replicated your issue but the issue it fails is cos the cluster does not form, and hence the client ends up knowing about only one of the servers. So, if you close the only server it knows, it's not gonna be able to failover:

datagrid-failover-0:
GMS: address=dnb-2-52617, cluster=DatagridPartition-Infinispan, physical address=127.0.0.2:55200
...
2011-02-07 18:37:08,467 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Received new cluster view: [dnb-2-52617|0] [dnb-2-52617]
2011-02-07 18:37:08,817 DEBUG [org.infinispan.server.hotrod.HotRodServer] (main) First member to start, topology view is {0}
2011-02-07 18:40:40,664 DEBUG [org.infinispan.server.hotrod.HotRodServer] (JBoss Shutdown Hook) Removed {0} from topology view, new view is {1}

datagrid-failover-1:
GMS: address=dnb-2-44413, cluster=DatagridPartition-Infinispan, physical address=127.0.0.3:55200
...
2011-02-07 18:37:35,491 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Received new cluster view: [dnb-2-44413|0] [dnb-2-44413]
...
GMS: address=dnb-2-3447, cluster=DatagridPartition-Infinispan, physical address=127.0.0.3:55200
...
2011-02-07 18:40:13,172 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Received new cluster view: [dnb-2-3447|0] [dnb-2-3447]
2011-02-07 18:40:18,080 WARN  [org.jgroups.protocols.pbcast.NAKACK] (OOB-9,null) dnb-2-3447: dropped message from dnb-2-52617 (not in table [dnb-2-3447]), view=[dnb-2-3447|0] [dnb-2-3447]
2011-02-07 18:40:18,505 WARN  [org.jgroups.protocols.pbcast.NAKACK] (OOB-10,null) dnb-2-3447: dropped message from dnb-2-52617 (not in table [dnb-2-3447]), view=[dnb-2-3447|0] [dnb-2-3447]
2011-02-07 18:40:31,566 WARN  [org.jgroups.protocols.pbcast.NAKACK] (OOB-19,null) dnb-2-3447: dropped message from dnb-2-52617 (not in table [dnb-2-3447]), view=[dnb-2-3447|0] [dnb-2-3447]
2011-02-07 18:40:31,628 WARN  [org.jgroups.protocols.pbcast.NAKACK] (OOB-19,null) dnb-2-3447: dropped message from dnb-2-52617 (not in table [dnb-2-3447]), view=[dnb-2-3447|0] [dnb-2-3447]

I think you should base your tests in the clustering ones that verify that the cluster forms in the first place.

In the client log, I only see:
test.log:60:2011-02-07 18:37:47,761 INFO  [org.infinispan.client.hotrod.impl.operations.HotRodOperation] New topology: {/127.0.0.3:11222=7359}
test.log:409:2011-02-07 18:40:22,068 INFO  [org.infinispan.client.hotrod.impl.operations.HotRodOperation] New topology: {/127.0.0.3:11222=6146}

So, since 127.0.0.2 and 127.0.0.3 did not cluster, the client never knows about them.

Also, there's another bug in your test which is highlighted by the fact that the test thinks that server list is: 127.0.0.3 and 127.0.0.1:

2011-02-07 18:37:45,928 TRACE [org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy] New server list is: [/127.0.0.3:11222, localhost/127.0.0.1:11222]. Resetting index to 0

That should clearly not be the case cos after I uncommented the following in local.properties:
node0=${env.MYTESTIP_1}
node1=${env.MYTESTIP_2}

I ran:
./build.sh -Denv.MYTESTIP_1=127.0.0.2 -Denv.MYTESTIP_2=127.0.0.3 tests-clustering-datagrid-failover-tests

I think your test should have worked even if the cluster did not form but the server list had been the correct one: 127.0.0.2 and 127.0.0.3. It's up to you to decide what to test but make sure that the test is clear about it.

Should have done some more investigation before logging this bug - when I start up the failing host, I neglected to add in the -u option to specify the multicast address used by the testsuite, so the hosts will not cluster as you rightly pointed out. 

Appologies - i'll update the test case.

Test case has been updated and the issue does not appear.

This issue was due to a test framework problem, and so was not a bug.