Bug 997890

Summary:	consumer.receive() thows IndexOutOfBoundException in standalone JMS client
Product:	[JBoss] JBoss Enterprise Application Platform 6	Reporter:	Miroslav Novak <mnovak>
Component:	HornetQ	Assignee:	Clebert Suconic <csuconic>
Status:	CLOSED NOTABUG	QA Contact:	Miroslav Novak <mnovak>
Severity:	high	Docs Contact:	Russell Dickenson <rdickens>
Priority:	unspecified
Version:	6.1.1	CC:	ataylor, csuconic, mnovak, msvehla, myarboro
Target Milestone:	ER9
Target Release:	EAP 6.3.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-07-07 07:37:03 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Miroslav Novak 2013-08-16 11:55:20 UTC

Description of problem:

IndexOutOfBoundException crashed standalone JMS client in one of failover tests for replicated journal.
This can have negative impact to customer's production environment.

Test scenario:

1. Start two EAP 6.1.1.ER6 in colocated topology with replicated journal
2. Start publishers and one subscriber sending/receiving messages to/from topic
3. Kill first server
4. Wait for failover to occur
5. Start first server again
6. Wait for failback
7. Stop publishers and wait for subcriber to read all messages

Result:
In step 7. IndexOutOfBoundException was thrown to subscriber [1]:

03:39:04,654 Thread-253 ERROR [org.jboss.qa.hornetq.apps.clients.SubscriberTransAck:138] Exception was thrown during receiving messages:
java.lang.IndexOutOfBoundsException: Not enough readable bytes - Need 409600, maximum is 204795
at org.jboss.netty.buffer.AbstractChannelBuffer.checkReadableBytes(AbstractChannelBuffer.java:661)
at org.jboss.netty.buffer.AbstractChannelBuffer.readBytes(AbstractChannelBuffer.java:338)
at org.jboss.netty.buffer.AbstractChannelBuffer.readBytes(AbstractChannelBuffer.java:344)
at org.hornetq.core.buffers.impl.ChannelBufferWrapper.readSimpleStringInternal(ChannelBufferWrapper.java:71)
at org.hornetq.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:49)
at org.hornetq.jms.client.HornetQTextMessage.doBeforeReceive(HornetQTextMessage.java:141)
at org.hornetq.jms.client.HornetQMessageConsumer.getMessage(HornetQMessageConsumer.java:232)
at org.hornetq.jms.client.HornetQMessageConsumer.receive(HornetQMessageConsumer.java:129)
at org.jboss.qa.hornetq.apps.clients.SubscriberTransAck.receiveMessage(SubscriberTransAck.java:391)
at org.jboss.qa.hornetq.apps.clients.SubscriberTransAck.run(SubscriberTransAck.java:110)

Client is receiving mix of small and large messages of various types (Stream, Object, Byte, Text and Map messages). Exception was thrown after succesful failback of client from backup to live during receiving large TextMessage with body of size 400KB.

This issue appeared only once. There is no reproducer.

[1] https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-HornetQ/job/eap-60-hornetq-ha-failover-colocated-replicated-journal-rhel-ipv4/IOTYPE=aio,jdk=java16_default,label=messaging-10/lastCompletedBuild/testReport/org.jboss.qa.hornetq.test.failover/ReplicatedColocatedClusterFailoverTestCase/testFailbackTransAckTopic/

Comment 1 Clebert Suconic 2013-08-20 15:55:11 UTC

It seems you just need a retry mechanism on the client for large messages...


I would mark this as a wont-fix or works for me.. I have no substantial information to work here

Comment 3 Miroslav Novak 2013-09-06 08:15:34 UTC

Hi Clebert,

thanks for your feedback. 

There is IndexOutOfBoundException thrown from consumer.receive() method which is unexpected and unchecked. Consumer can retry receive only if JMSException is thrown. 

In this case buffer for receiving large message was not allocated properly.

From the log it appears that IndexOutOfBoundException is not related to failover/failback scenario. It was thrown when no kind of server failure was in progress. Subscriber is using transacted session and commits every 100th message. 
I could not replicate the issue but I suspect that certain sequence of large and small messages can lead to this exception.

We should not close this bz.

Thanks,
Mirek

Adding updated link to "raw" log from jenkins job:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-HornetQ/job/eap-60-hornetq-ha-failover-colocated-replicated-journal-rhel-ipv4/IOTYPE=aio,jdk=java16_default,label=messaging-10/33/consoleText

Comment 7 Miroslav Novak 2013-10-10 09:04:35 UTC

Updating url:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-HornetQ/job/eap-60-hornetq-ha-failover-colocated-replicated-journal-rhel-ipv4/35/IOTYPE=nio,jdk=java16_default,label=messaging-10/testReport/org.jboss.qa.hornetq.test.failover/ReplicatedColocatedClusterFailoverTestCase/testFailbackTransAckTopic/

Comment 8 Miroslav Novak 2014-07-07 07:37:03 UTC

Closing as I don't see IndexOutOfBoundException any more.