When I run ReplicatedColocatedClusterFailoverTestCase#testFailbackTransAckQueue I don't see an IndexOutOfBoundsException.  Are you sure this is the correct test?  I noticed it's the same one referenced on BZ1169418.

Unfortunately this is one of the "random" failures. It's really rare and I don't have 100% reproducer. Test must be run multiple times. Issue in bz#1169418 is also one of the sometimes problems but based on observation of Jenkins runs, there it occurs more often.

Hi Mirek,

My first run of the test didn't get the IndexOutOfBoundsException, however I've seen the timeout exception as you reported in bz#1169418.

I'll do some more investigation.

Howard

Hi Mirek,

Today I just found there might be a problem in the test case. If I am right the test starts three producers to produce messages. Each producer takes a messageBuilder. The problem is that each producer is running on its own thread, and the messageBuilder is shared among producers. However the methods in messageBuilder such as createMessage(session) are not synchronized, which may cause potential problem.

Please see QueueClientsTransAck.startClients():

...
for (int producerNumber = 0; producerNumber < getNumberOfProducersPerQueueu(); producerNumber++) {

p = new ProducerTransAck(container, getHostnameForProducers(), getJndiPort(), getQueueJndiNamePrefix() + destinationNumber, getMessages(), this);

if (messageBuilder != null) {
     p.setMessageBuilder(messageBuilder);
}
.....

You can see all ProducerTransAck uses the same messageBuilder ( in the test it is an instance of ClientMixMessageBuilder class).

What do you think? 

Howard

Hi Howard,

messageBuilder is really shared among producers. Here each producer has its own session created from its own connection. When messageBuilder.createMessage(session) is called in producer thread then it's passing its session to messageBuilder and there is used to create message. I went through ClientMixMessageBuilder.createMessage class and this could only affect "counter" variable so 2 messages would have same property counter. Which is harmless.
I don't think this could cause IndexOutOfBoundException because it's thrown on receiver side. Receiver has its own connection and session.

Thanks,
Mirek

Hi Mirek,

How about this method getNextMapKey() ? It uses a member mapMessageKey which is changed possible concurrently.

Howard

Hi Howard,

yes, string variable mapMessageKey can be also changed concurrently. I'm not sure whether it can influence how receiver is allocating buffer. If producer sets message.setObject(mapMessageKey, stringContent) and then before send another thread changes mapMessageKey. 

Mirek

Hi Mirek,

I'm still not clear about how's going wrong. I've been chasing the possible issues for the last few days and didn't get much progress. The only thing I'm a bit sure about is that there is some exception out of the client message builder class.

I'd suggest we keep the code synchronized and run the test again, and see what can we get.

Howard

Hi Mirek,

Just for your info, I've just seen the 'out of index' exception at this line:

int numericValueOfLastCharacter = mapMessageKey.codePointAt(mapMessageKey.length() - 1);

That probably is cause by concurrent modification.

I'll try synchronize the createMessage() method and re-run the test. I'll give you some update tomorrow. 


Howard

Hi Howard,

ok, I'll do the same thing and run our jobs in Jenkins again. Unfortunately there is rare occurrence of this. The other bz#1169418 is much easier to hit.

Mirek

That's great Mirek. For me I'm still seeing strange IndexOutOfBound error after I synchronized createMessage() method. I'm investigating...

Howard