Bug 1034619

Summary: Errors on domain shutdown with deployed application with temporary queues and topics
Product: [JBoss] JBoss Enterprise Application Platform 6 Reporter: Jitka Kozana <jkudrnac>
Component: HornetQAssignee: Enrique Gonzalez Martinez <egonzale>
Status: CLOSED NOTABUG QA Contact: Miroslav Novak <mnovak>
Severity: high Docs Contact: Russell Dickenson <rdickens>
Priority: unspecified    
Version: 6.2.0CC: ataylor, egonzale, jbertram, jkudrnac, jmesnil, mnovak, msvehla, tanabe.yoshimasa, tkonishi
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-02-19 11:20:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Jitka Kozana 2013-11-26 08:46:03 UTC 
EAP 6.2.0.CR3.

When testing hornetq-clustering quickstart and verifying BZ #1029076, I have seen the following NPE in domain log on server shutdown.
How to reproduce: 
Follow the instructions in the hornetq-clustering quickstart readme, just after the application is accessed in domain (line 172 in the README.md). Then shutdown the domain (do not undeploy the application), simply by using the control+C.


[Server:jdf-hornetqcluster-node1] 08:35:35,101 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015877: Stopped deployment jboss-helloworld-mdb.war (runtime-name: jboss-helloworld-mdb.war) in 356ms
[Server:jdf-hornetqcluster-node2] 08:35:35,114 INFO  [org.hornetq.ra] (ServerService Thread Pool -- 29) HQ151003: HornetQ resource adaptor stopped
[Server:jdf-hornetqcluster-node2] 08:35:35,131 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015877: Stopped deployment jboss-helloworld-mdb.war (runtime-name: jboss-helloworld-mdb.war) in 344ms
[Server:jdf-hornetqcluster-node2] 08:35:35,132 WARN  [org.hornetq.core.client] (Thread-4 (HornetQ-client-global-threads-1144140494)) HQ212037: Connection failure has been detected: HQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
[Server:jdf-hornetqcluster-node2] 08:35:35,140 WARN  [org.hornetq.core.client] (Thread-6 (HornetQ-client-global-threads-1144140494)) HQ212037: Connection failure has been detected: HQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
[Server:jdf-hornetqcluster-node1] 08:35:35,144 INFO  [org.hornetq.core.server] (Thread-6 (HornetQ-server-HornetQServerImpl::serverUUID=c490cc55-566b-11e3-9a34-5f04c5cc51dd-644679284)) HQ221029: stopped bridge sf.my-cluster.c74edb61-566b-11e3-b001-7d234d3fdcb7
[Server:jdf-hornetqcluster-node2] 08:35:35,146 INFO  [org.hornetq.core.server] (Thread-9 (HornetQ-server-HornetQServerImpl::serverUUID=c74edb61-566b-11e3-b001-7d234d3fdcb7-1321323955)) HQ221029: stopped bridge sf.my-cluster.c490cc55-566b-11e3-9a34-5f04c5cc51dd
[Server:jdf-hornetqcluster-node2] 08:35:35,147 ERROR [org.hornetq.core.server] (Old I/O server worker (parentId: 1032080984, [id: 0x3d844e58, /127.0.0.1:6545])) HQ224048: Failed to remove temporary queue notif.4ba4005b-566d-11e3-bfd8-7dc40b970cad.HornetQServerImpl::serverUUID=c490cc55-566b-11e3-9a34-5f04c5cc51dd: java.lang.NullPointerException
[Server:jdf-hornetqcluster-node2] 	at org.hornetq.core.server.management.impl.ManagementServiceImpl.unregisterQueue(ManagementServiceImpl.java:279) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.hornetq.core.postoffice.impl.PostOfficeImpl.removeBinding(PostOfficeImpl.java:507) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.hornetq.core.server.impl.QueueImpl.deleteQueue(QueueImpl.java:1303) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.hornetq.core.server.impl.HornetQServerImpl.destroyQueue(HornetQServerImpl.java:1181) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.hornetq.core.server.impl.ServerSessionImpl$TempQueueCleanerUpper.run(ServerSessionImpl.java:492) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.hornetq.core.server.impl.ServerSessionImpl$TempQueueCleanerUpper.connectionClosed(ServerSessionImpl.java:513) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callClosingListeners(RemotingConnectionImpl.java:595) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.destroy(RemotingConnectionImpl.java:367) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.hornetq.core.remoting.server.impl.RemotingServiceImpl.connectionDestroyed(RemotingServiceImpl.java:490) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.hornetq.core.remoting.impl.netty.NettyAcceptor$Listener.connectionDestroyed(NettyAcceptor.java:735) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.channelDisconnected(HornetQChannelHandler.java:82) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:120) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:208) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.jboss.netty.channel.socket.oio.AbstractOioWorker.close(AbstractOioWorker.java:216) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:98) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1]
[Server:jdf-hornetqcluster-node2] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_04]
[Server:jdf-hornetqcluster-node2] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_04]
[Server:jdf-hornetqcluster-node2] 	at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_04]
[Server:jdf-hornetqcluster-node2] 
[Server:jdf-hornetqcluster-node1] 08:35:35,236 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 28) HQ221002: HornetQ Server version 2.3.12.Final (2.3.12, 123) [c490cc55-566b-11e3-9a34-5f04c5cc51dd] stopped
[Server:jdf-hornetqcluster-node1] 08:35:35,249 INFO  [org.jboss.as] (MSC service thread 1-7) JBAS015950: JBoss EAP 6.2.0.GA (AS 7.3.0.Final-redhat-14) stopped in 490ms
[Server:jdf-hornetqcluster-node2] 08:35:35,253 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 29) HQ221002: HornetQ Server version 2.3.12.Final (2.3.12, 123) [c74edb61-566b-11e3-b001-7d234d3fdcb7] stopped
[Host Controller] 08:35:35,263 INFO  [org.jboss.as.host.controller] (Host Controller Service Threads - 33) JBAS010923: Stopping server jdf-hornetqcluster-node1
[Host Controller] 08:35:35,264 INFO  [org.jboss.as.host.controller] (Host Controller Service Threads - 33) JBAS010923: Stopping server server-two
[Host Controller] 08:35:35,265 INFO  [org.jboss.as.host.controller] (Host Controller Service Threads - 33) JBAS010923: Stopping server server-one
[Host Controller] 08:35:35,265 INFO  [org.jboss.as.host.controller] (Host Controller Service Threads - 33) JBAS010923: Stopping server jdf-hornetqcluster-node2
 
In other shutdown, the following error was seen:
09:38:34,022 INFO  [org.hornetq.core.server] (Thread-12 (HornetQ-server-HornetQServerImpl::serverUUID=c490cc55-566b-11e3-9a34-5f04c5cc51dd-663290997)) HQ221027: Bridge ClusterConnectionBridge@220554d4 [name=sf.my-cluster.c74edb61-566b-11e3-b001-7d234d3fdcb7, queue=QueueImpl[name=sf.my-cluster.c74edb61-566b-11e3-b001-7d234d3fdcb7, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=c490cc55-566b-11e3-9a34-5f04c5cc51dd]]@c043bbf targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@220554d4 [name=sf.my-cluster.c74edb61-566b-11e3-b001-7d234d3fdcb7, queue=QueueImpl[name=sf.my-cluster.c74edb61-566b-11e3-b001-7d234d3fdcb7, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=c490cc55-566b-11e3-9a34-5f04c5cc51dd]]@c043bbf targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=6545&host=localhost-localdomain], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1203226328[nodeUUID=c490cc55-566b-11e3-9a34-5f04c5cc51dd, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=6445&host=localhost-localdomain, address=jms, server=HornetQServerImpl::serverUUID=c490cc55-566b-11e3-9a34-5f04c5cc51dd])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=6545&host=localhost-localdomain], discoveryGroupConfiguration=null]] is connected
09:38:40,857 INFO  [org.apache.coyote.http11] (MSC service thread 1-4) JBWEB003075: Coyote HTTP/1.1 pausing on: http-localhost.localdomain/127.0.0.1:9080
09:38:40,868 INFO  [org.apache.coyote.http11] (MSC service thread 1-4) JBWEB003077: Coyote HTTP/1.1 stopping on : http-localhost.localdomain/127.0.0.1:9080
09:38:40,877 INFO  [org.jboss.web] (ServerService Thread Pool -- 28) JBAS018224: Unregister web context: /jboss-helloworld-mdb
09:38:40,906 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-4) JBAS010410: Unbound JCA ConnectionFactory [java:/JmsXA]
09:38:40,964 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-6) JBAS010409: Unbound data source [java:jboss/datasources/ExampleDS]
09:38:40,966 INFO  [org.jboss.jaxr] (MSC service thread 1-1) JBAS014002: UnBinding JAXR ConnectionFactory: java:jboss/jaxr/ConnectionFactory
09:38:40,994 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 27) MODCLUSTER000002: Initiating mod_cluster shutdown
09:38:41,027 INFO  [org.apache.coyote.ajp] (MSC service thread 1-1) JBWEB003048: Pausing Coyote AJP/1.3 on ajp-localhost.localdomain/127.0.0.1:9009
09:38:41,050 INFO  [org.apache.coyote.ajp] (MSC service thread 1-1) JBWEB003051: Stopping Coyote AJP/1.3 on ajp-localhost.localdomain/127.0.0.1:9009
09:38:41,064 INFO  [org.apache.catalina.core] (MSC service thread 1-8) JBWEB001079: Container org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/] has not been started
09:38:41,286 INFO  [org.jboss.as.messaging] (ServerService Thread Pool -- 31) JBAS011605: Unbound messaging object to jndi name java:/ConnectionFactory
09:38:41,295 INFO  [org.hornetq.ra] (ServerService Thread Pool -- 26) HQ151003: HornetQ resource adaptor stopped
09:38:41,311 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015877: Stopped deployment jboss-helloworld-mdb.war (runtime-name: jboss-helloworld-mdb.war) in 417ms
09:38:41,316 ERROR [org.hornetq.core.server] (Thread-6 (HornetQ-client-global-threads-1290997545)) HQ224037: cluster connection Failed to handle message: java.lang.IllegalStateException: Cannot find binding for jms.queue.HELLOWORLDMDBQueuec74edb61-566b-11e3-b001-7d234d3fdcb7
	at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.doConsumerClosed(ClusterConnectionImpl.java:1602) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]
	at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.onMessage(ClusterConnectionImpl.java:1295) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]
	at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1117) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]
	at org.hornetq.core.client.impl.ClientConsumerImpl.access$500(ClientConsumerImpl.java:57) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]
	at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1252) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]
	at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:107) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_04]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_04]
	at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_04]

09:38:41,360 INFO  [org.hornetq.core.server] (Thread-16 (HornetQ-server-HornetQServerImpl::serverUUID=c490cc55-566b-11e3-9a34-5f04c5cc51dd-663290997)) HQ221029: stopped bridge sf.my-cluster.c74edb61-566b-11e3-b001-7d234d3fdcb7
09:38:41,395 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 26) HQ221002: HornetQ Server version 2.3.12.Final (2.3.12, 123) [c490cc55-566b-11e3-9a34-5f04c5cc51dd] stopped
09:38:41,417 INFO  [org.jboss.as] (MSC service thread 1-6) JBAS015950: JBoss EAP 6.2.0.GA (AS 7.3.0.Final-redhat-14) stopped in 502ms
Comment 1 Miroslav Novak 2014-07-08 10:11:46 UTC 
This is still problem in EAP 6.3.0.ER9 (HQ 2.3.20).
Comment 2 Justin Bertram 2014-07-23 18:30:35 UTC 
Couple of questions...

  1) Where can I get the latest build of EAP where this issue is observed?
  2) Where can I get the "hornetq-clustering quickstart" referenced here to reproduce this issue?
Comment 5 Enrique Gonzalez Martinez 2016-02-17 14:51:49 UTC 
@Miroslav can you set the proper z-stream ?. Target release + milestone are set when the BZ is included in a CP (they don't mean when you aim to fix the issue). Streams flags are used instead. 

Setting/asking for more than one stream in a BZ is not allowed by procedure.
is it 6.3.z or 6.4.z ?