Bug 1369322

Summary: UnsupportedPacketException thrown from FileWrapperJournal.appendRollbackRecord
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Erich Duda <eduda>
Component: HornetQAssignee: jboss-set
Status: CLOSED EOL QA Contact: Peter Mackay <pmackay>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.4.10CC: csuconic, msvehla
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-19 12:47:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Erich Duda 2016-08-23 06:57:44 UTC 
Customer impact:

In HA scenario after the failover/failback the clients are not able to continue in sending/receiving messages, because the server stops responding them. There are no lost messages.

This is not regression against previous EAP 6 version.

Description of problem:

In HA scenario with replicated journal I can see the exception [1]. After that the server stops responding to clients and warnings [2] are printed to the server log.

The exception is thrown from the FileWrapperJournal.appendRollbackRecord. This method is claimed as unsupported. The same issue was hit in ActiveMQ Artemis and it was already fixed. See https://issues.apache.org/jira/browse/ARTEMIS-422.

[1]
07:52:49,236 WARN  [org.hornetq.core.server] (Old I/O client worker ([id: 0x2ce9cc57, /127.0.0.1:35838 => localhost/127.0.0.1:5445]), runningThread=Thread-1 (HornetQ-client-global-threads-45175226)) HQ222086: er
ror handling packet ReplicationCommitMessage[type=96, channel=2, journalID=1, txAction='rollback'] for replication: HornetQException[errorType=UNSUPPORTED_PACKET message=null]
        at org.hornetq.core.journal.impl.FileWrapperJournal.appendRollbackRecord(FileWrapperJournal.java:209) [hornetq-journal-2.3.25.SP13-redhat-1.jar:2.3.25.SP13-redhat-1]
        at org.hornetq.core.journal.impl.JournalBase.appendRollbackRecord(JournalBase.java:151) [hornetq-journal-2.3.25.SP13-redhat-1.jar:2.3.25.SP13-redhat-1]
        at org.hornetq.core.journal.impl.FileWrapperJournal.appendRollbackRecord(FileWrapperJournal.java:35) [hornetq-journal-2.3.25.SP13-redhat-1.jar:2.3.25.SP13-redhat-1]
        at org.hornetq.core.replication.ReplicationEndpoint.handleCommitRollback(ReplicationEndpoint.java:708) [hornetq-server-2.3.25.SP13-redhat-1.jar:2.3.25.SP13-redhat-1]
        at org.hornetq.core.replication.ReplicationEndpoint.handlePacket(ReplicationEndpoint.java:192) [hornetq-server-2.3.25.SP13-redhat-1.jar:2.3.25.SP13-redhat-1]
        at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:652) [hornetq-core-client-2.3.25.SP13-redhat-1.jar:2.3.25.SP13-redhat-1]
        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:615) [hornetq-core-client-2.3.25.SP13-redhat-1.jar:2.3.25.SP13-redhat-1]
        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:574) [hornetq-core-client-2.3.25.SP13-redhat-1.jar:2.3.25.SP13-redhat-1]
        at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1695) [hornetq-core-client-2.3.25.SP13-redhat-1.jar:2.3.25.SP13-redhat-1]
        at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:72) [hornetq-core-client-2.3.25.SP13-redhat-1.jar:2.3.25.SP13-redhat-1]
        at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]
        at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169) [hornetq-core-client-2.3.25.SP13-redhat-1.jar:2.3.25.SP13-redhat-1]
        at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134) [hornetq-core-client-2.3.25.SP13-redhat-1.jar:2.3.25.SP13-redhat-1]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]
        at org.jboss.netty.channel.socket.oio.OioWorker.process(OioWorker.java:71) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]
        at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]
        at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]
        at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101]

[2]
07:53:19,456 WARN  [org.hornetq.core.server] (Old I/O server worker (parentId: -1612310000, [id: 0x9fe61a10, /127.0.0.1:5445]), runningThread=Thread-4 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=86cf2f36-68f5-11e6-83c6-138d94a4e5f9-1300803308-1874445692)) HQ222059: Duplicate message detected - message will not be routed. Message information:
ServerMessage[messageID=2147485193,userID=e4b2383b-68f5-11e6-9cda-fda33d19066a,priority=4, bodySize=41279,expiration=0, durable=true, address=jms.queue.testQueue,properties=TypedProperties[__HQ_CID=99ae459d-68f5-11e6-9cda-cb49fd50e973,count=870,counter=870,_HQ_DUPL_ID=516fc68d-a0ab-4a1c-b4ea-dcceb3717520]]@577021911

Actual results:
In HA scenario after the failover/failback the clients are not able to continue in sending/receiving messages.

Expected results:
In HA scenario no messages are lost and clients are able to continue in sending/receiving messages even when one server is killed.