Bug 900024 (JBPAPP6-935)

Summary: Clean shutdown often results in "Problems unmarshalling remote command from byte buffer"
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Radoslav Husar <rhusar>
Component: ClusteringAssignee: Paul Ferraro <paul.ferraro>
Status: CLOSED CURRENTRELEASE QA Contact: Jitka Kozana <jkudrnac>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.1.0CC: brian.stansberry, jkudrnac, lthon, rhusar, rsvoboda, tomaz.cerar
Target Milestone: ---   
Target Release: EAP 6.1.1   
Hardware: Unspecified   
OS: Unspecified   
URL: http://jira.jboss.org/jira/browse/JBPAPP6-935
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-08-26 11:37:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 900611    

Description Radoslav Husar 2012-03-12 17:06:25 UTC 
Affects: Release Notes
project_key: JBPAPP6

Reopened AS7-3180

Technical details tracked here https://issues.jboss.org/browse/AS7-3180

Excerpt from current logging https://issues.jboss.org/browse/AS7-3180?focusedCommentId=12676146&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12676146
Comment 1 Shelly McGowan 2012-04-09 18:32:52 UTC 
Link: Added: This issue is related to JBPAPP-7577
Comment 2 Radoslav Husar 2012-05-11 15:05:35 UTC 
Link: Added: This issue Cloned from AS7-3180
Comment 3 Radoslav Husar 2012-05-11 15:06:26 UTC 
If anyone looms in here: The logging has now changed after adding the ExternalizerTable into:

{noformat}
[JBossINF] 10:53:32,192 WARN  [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] Problems unmarshalling remote command from byte buffer: java.lang.NullPointerException
[JBossINF] 	at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222)
[JBossINF] 	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)
[JBossINF] 	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
[JBossINF] 	at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37) [jboss-marshalling-1.3.14.GA.jar:1.3.14.GA]
[JBossINF] 	at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:154)
[JBossINF] 	at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:114)
[JBossINF] 	at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:85)
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50)
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:200)
[JBossINF] 	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.protocols.RSVP.up(RSVP.java:192) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:706) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:566) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.protocols.BARRIER.up(BARRIER.java:126) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.protocols.Discovery.up(Discovery.java:359) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.stack.Protocol.up(Protocol.java:363) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.protocols.TP.passMessageUp(TP.java:1180) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710) [jgroups-3.0.9.Final.jar:3.0.9.Final]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
[JBossINF] 	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
{noformat}
Comment 4 Misty Stanley-Jones 2012-06-01 06:11:47 UTC 
Release Notes Docs Status: Added: Documented as Known Issue
Release Notes Text: Added: If you shut down your cluster cleanly, you may still receive error messages such as <code>Problems unmarshalling remote command from byte buffer</code>. This will be fixed with an update to the Infinispan component.
Affects: Added: Release Notes
Comment 5 Rajesh Rajasekaran 2012-06-06 20:43:28 UTC 
Link: Added: This issue is a dependency of JBPAPP-9290
Comment 6 Rajesh Rajasekaran 2012-06-06 20:51:03 UTC 
Link: Removed: This issue is related to JBPAPP-7577
Comment 7 Brian Stansberry 2012-08-08 14:07:23 UTC 
This was resolved in upstream master and 7.1 quite a while ago.
Comment 8 Dana Mison 2012-10-16 05:58:48 UTC 
Writer: Added: mistysj
Comment 9 Dana Mison 2012-10-17 22:51:17 UTC 
reopening to update release notes (these are still documented as known issues from the previous release)
Comment 10 Dana Mison 2012-10-17 22:52:02 UTC 
Writer: Removed: mistysj Added: Darrin
Comment 11 Dana Mison 2012-10-17 23:04:38 UTC 
Setting these to "Not Yet Documented" since they required updating from known to resolved issues.
Comment 12 Dana Mison 2012-10-17 23:04:38 UTC 
Release Notes Docs Status: Removed: Documented as Known Issue Added: Not Yet Documented
Comment 13 Dana Mison 2012-10-22 06:00:45 UTC 
Hi Paul, 

I hunting for some additional information so I can document this as a resolved issue for EAP6.0.1.

Can you provide any additional information about what caused this problem and how it was fixed?

Thanks.
Comment 14 Dana Mison 2012-10-22 06:01:15 UTC 
Release Notes Docs Status: Removed: Not Yet Documented Added: Needs More Info
Comment 15 Dana Mison 2012-10-22 06:01:40 UTC 
Release Notes Text: Removed: If you shut down your cluster cleanly, you may still receive error messages such as <code>Problems unmarshalling remote command from byte buffer</code>. This will be fixed with an update to the Infinispan component. Added: If you shut down your cluster cleanly, you may still receive error messages such as "Problems unmarshalling remote command from byte buffer". This will be fixed with an update to the Infinispan component.
Comment 17 Radoslav Husar 2012-10-22 11:09:47 UTC 
Release Notes Text: Removed: If you shut down your cluster cleanly, you may still receive error messages such as "Problems unmarshalling remote command from byte buffer". This will be fixed with an update to the Infinispan component. Added: If you shut down a cluster node cleanly, you may have received error messages such as "Problems unmarshalling remote command from byte buffer".
Comment 18 Dana Mison 2012-10-24 05:57:29 UTC 
Release Notes Text: Removed: If you shut down a cluster node cleanly, you may have received error messages such as "Problems unmarshalling remote command from byte buffer". Added: A clean shutdown of JBoss Enterprise Application Platform 6 cluster node would result in several warning messages being logged.  The log messages warned of: "Problems unmarshalling remote command from byte buffer: java.lang.NullPointerException" but there was no other effect apart from the log messages.  This was due to the some of the components being shutdown in the wrong order.    The clustering  components have been updated to correct this. A clean shutdown of a JBoss Enterprise Application Platform 6 cluster node no longer causes these warnings to be issued.
Comment 19 Dana Mison 2012-10-24 05:58:48 UTC 
Radoslav, Thanks for the info.  Release Notes updated.
Comment 20 Dana Mison 2012-10-24 05:58:48 UTC 
Release Notes Docs Status: Removed: Needs More Info Added: Documented as Resolved Issue
Comment 21 Jitka Kozana 2012-11-01 13:14:19 UTC 
I just saw it again in our 6.0.1.ER3 runs, please have a look at the logs here:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Failover/job/eap-6x-failover-ejb-ejbservlet-shutdown-repl-async/26/console-perf18/

Reopening.
Comment 22 Anne-Louise Tangring 2012-11-13 20:07:51 UTC 
Release Notes Docs Status: Removed: Documented as Resolved Issue 
Writer: Removed: Darrin 
Release Notes Text: Removed: A clean shutdown of JBoss Enterprise Application Platform 6 cluster node would result in several warning messages being logged.  The log messages warned of: "Problems unmarshalling remote command from byte buffer: java.lang.NullPointerException" but there was no other effect apart from the log messages.  This was due to the some of the components being shutdown in the wrong order.    The clustering  components have been updated to correct this. A clean shutdown of a JBoss Enterprise Application Platform 6 cluster node no longer causes these warnings to be issued. 
Docs QE Status: Removed: NEW
Comment 23 Ladislav Thon 2013-02-06 09:35:23 UTC 
Also affects 6.1.0.DR3.
Comment 24 Ladislav Thon 2013-03-01 13:33:19 UTC 
Also affects 6.1.0.ER1.

Seen in https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/EAP6/view/EAP6-Clustering/view/EAP6-Failover/job/eap-6x-failover-ejb-ejbservlet-undeploy-repl-async/25/console-perf21/
Comment 26 Radoslav Husar 2013-08-26 11:37:24 UTC 
Closing as per last comment. Thanks.