EAP 6.2.0.CR3. When testing hornetq-clustering quickstart and verifying BZ #1029076, I have seen the following NPE in domain log on server shutdown. How to reproduce: Follow the instructions in the hornetq-clustering quickstart readme, just after the application is accessed in domain (line 172 in the README.md). Then shutdown the domain (do not undeploy the application), simply by using the control+C. [Server:jdf-hornetqcluster-node1] 08:35:35,101 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015877: Stopped deployment jboss-helloworld-mdb.war (runtime-name: jboss-helloworld-mdb.war) in 356ms [Server:jdf-hornetqcluster-node2] 08:35:35,114 INFO [org.hornetq.ra] (ServerService Thread Pool -- 29) HQ151003: HornetQ resource adaptor stopped [Server:jdf-hornetqcluster-node2] 08:35:35,131 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015877: Stopped deployment jboss-helloworld-mdb.war (runtime-name: jboss-helloworld-mdb.war) in 344ms [Server:jdf-hornetqcluster-node2] 08:35:35,132 WARN [org.hornetq.core.client] (Thread-4 (HornetQ-client-global-threads-1144140494)) HQ212037: Connection failure has been detected: HQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED] [Server:jdf-hornetqcluster-node2] 08:35:35,140 WARN [org.hornetq.core.client] (Thread-6 (HornetQ-client-global-threads-1144140494)) HQ212037: Connection failure has been detected: HQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED] [Server:jdf-hornetqcluster-node1] 08:35:35,144 INFO [org.hornetq.core.server] (Thread-6 (HornetQ-server-HornetQServerImpl::serverUUID=c490cc55-566b-11e3-9a34-5f04c5cc51dd-644679284)) HQ221029: stopped bridge sf.my-cluster.c74edb61-566b-11e3-b001-7d234d3fdcb7 [Server:jdf-hornetqcluster-node2] 08:35:35,146 INFO [org.hornetq.core.server] (Thread-9 (HornetQ-server-HornetQServerImpl::serverUUID=c74edb61-566b-11e3-b001-7d234d3fdcb7-1321323955)) HQ221029: stopped bridge sf.my-cluster.c490cc55-566b-11e3-9a34-5f04c5cc51dd [Server:jdf-hornetqcluster-node2] 08:35:35,147 ERROR [org.hornetq.core.server] (Old I/O server worker (parentId: 1032080984, [id: 0x3d844e58, /127.0.0.1:6545])) HQ224048: Failed to remove temporary queue notif.4ba4005b-566d-11e3-bfd8-7dc40b970cad.HornetQServerImpl::serverUUID=c490cc55-566b-11e3-9a34-5f04c5cc51dd: java.lang.NullPointerException [Server:jdf-hornetqcluster-node2] at org.hornetq.core.server.management.impl.ManagementServiceImpl.unregisterQueue(ManagementServiceImpl.java:279) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.hornetq.core.postoffice.impl.PostOfficeImpl.removeBinding(PostOfficeImpl.java:507) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.hornetq.core.server.impl.QueueImpl.deleteQueue(QueueImpl.java:1303) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.hornetq.core.server.impl.HornetQServerImpl.destroyQueue(HornetQServerImpl.java:1181) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.hornetq.core.server.impl.ServerSessionImpl$TempQueueCleanerUpper.run(ServerSessionImpl.java:492) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.hornetq.core.server.impl.ServerSessionImpl$TempQueueCleanerUpper.connectionClosed(ServerSessionImpl.java:513) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callClosingListeners(RemotingConnectionImpl.java:595) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.destroy(RemotingConnectionImpl.java:367) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.hornetq.core.remoting.server.impl.RemotingServiceImpl.connectionDestroyed(RemotingServiceImpl.java:490) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.hornetq.core.remoting.impl.netty.NettyAcceptor$Listener.connectionDestroyed(NettyAcceptor.java:735) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.channelDisconnected(HornetQChannelHandler.java:82) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:120) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:208) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.jboss.netty.channel.socket.oio.AbstractOioWorker.close(AbstractOioWorker.java:216) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:98) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175) [netty-3.6.6.Final-redhat-1.jar:3.6.6.Final-redhat-1] [Server:jdf-hornetqcluster-node2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_04] [Server:jdf-hornetqcluster-node2] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_04] [Server:jdf-hornetqcluster-node2] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_04] [Server:jdf-hornetqcluster-node2] [Server:jdf-hornetqcluster-node1] 08:35:35,236 INFO [org.hornetq.core.server] (ServerService Thread Pool -- 28) HQ221002: HornetQ Server version 2.3.12.Final (2.3.12, 123) [c490cc55-566b-11e3-9a34-5f04c5cc51dd] stopped [Server:jdf-hornetqcluster-node1] 08:35:35,249 INFO [org.jboss.as] (MSC service thread 1-7) JBAS015950: JBoss EAP 6.2.0.GA (AS 7.3.0.Final-redhat-14) stopped in 490ms [Server:jdf-hornetqcluster-node2] 08:35:35,253 INFO [org.hornetq.core.server] (ServerService Thread Pool -- 29) HQ221002: HornetQ Server version 2.3.12.Final (2.3.12, 123) [c74edb61-566b-11e3-b001-7d234d3fdcb7] stopped [Host Controller] 08:35:35,263 INFO [org.jboss.as.host.controller] (Host Controller Service Threads - 33) JBAS010923: Stopping server jdf-hornetqcluster-node1 [Host Controller] 08:35:35,264 INFO [org.jboss.as.host.controller] (Host Controller Service Threads - 33) JBAS010923: Stopping server server-two [Host Controller] 08:35:35,265 INFO [org.jboss.as.host.controller] (Host Controller Service Threads - 33) JBAS010923: Stopping server server-one [Host Controller] 08:35:35,265 INFO [org.jboss.as.host.controller] (Host Controller Service Threads - 33) JBAS010923: Stopping server jdf-hornetqcluster-node2 In other shutdown, the following error was seen: 09:38:34,022 INFO [org.hornetq.core.server] (Thread-12 (HornetQ-server-HornetQServerImpl::serverUUID=c490cc55-566b-11e3-9a34-5f04c5cc51dd-663290997)) HQ221027: Bridge ClusterConnectionBridge@220554d4 [name=sf.my-cluster.c74edb61-566b-11e3-b001-7d234d3fdcb7, queue=QueueImpl[name=sf.my-cluster.c74edb61-566b-11e3-b001-7d234d3fdcb7, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=c490cc55-566b-11e3-9a34-5f04c5cc51dd]]@c043bbf targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@220554d4 [name=sf.my-cluster.c74edb61-566b-11e3-b001-7d234d3fdcb7, queue=QueueImpl[name=sf.my-cluster.c74edb61-566b-11e3-b001-7d234d3fdcb7, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=c490cc55-566b-11e3-9a34-5f04c5cc51dd]]@c043bbf targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=6545&host=localhost-localdomain], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1203226328[nodeUUID=c490cc55-566b-11e3-9a34-5f04c5cc51dd, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=6445&host=localhost-localdomain, address=jms, server=HornetQServerImpl::serverUUID=c490cc55-566b-11e3-9a34-5f04c5cc51dd])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=6545&host=localhost-localdomain], discoveryGroupConfiguration=null]] is connected 09:38:40,857 INFO [org.apache.coyote.http11] (MSC service thread 1-4) JBWEB003075: Coyote HTTP/1.1 pausing on: http-localhost.localdomain/127.0.0.1:9080 09:38:40,868 INFO [org.apache.coyote.http11] (MSC service thread 1-4) JBWEB003077: Coyote HTTP/1.1 stopping on : http-localhost.localdomain/127.0.0.1:9080 09:38:40,877 INFO [org.jboss.web] (ServerService Thread Pool -- 28) JBAS018224: Unregister web context: /jboss-helloworld-mdb 09:38:40,906 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-4) JBAS010410: Unbound JCA ConnectionFactory [java:/JmsXA] 09:38:40,964 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-6) JBAS010409: Unbound data source [java:jboss/datasources/ExampleDS] 09:38:40,966 INFO [org.jboss.jaxr] (MSC service thread 1-1) JBAS014002: UnBinding JAXR ConnectionFactory: java:jboss/jaxr/ConnectionFactory 09:38:40,994 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 27) MODCLUSTER000002: Initiating mod_cluster shutdown 09:38:41,027 INFO [org.apache.coyote.ajp] (MSC service thread 1-1) JBWEB003048: Pausing Coyote AJP/1.3 on ajp-localhost.localdomain/127.0.0.1:9009 09:38:41,050 INFO [org.apache.coyote.ajp] (MSC service thread 1-1) JBWEB003051: Stopping Coyote AJP/1.3 on ajp-localhost.localdomain/127.0.0.1:9009 09:38:41,064 INFO [org.apache.catalina.core] (MSC service thread 1-8) JBWEB001079: Container org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/] has not been started 09:38:41,286 INFO [org.jboss.as.messaging] (ServerService Thread Pool -- 31) JBAS011605: Unbound messaging object to jndi name java:/ConnectionFactory 09:38:41,295 INFO [org.hornetq.ra] (ServerService Thread Pool -- 26) HQ151003: HornetQ resource adaptor stopped 09:38:41,311 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015877: Stopped deployment jboss-helloworld-mdb.war (runtime-name: jboss-helloworld-mdb.war) in 417ms 09:38:41,316 ERROR [org.hornetq.core.server] (Thread-6 (HornetQ-client-global-threads-1290997545)) HQ224037: cluster connection Failed to handle message: java.lang.IllegalStateException: Cannot find binding for jms.queue.HELLOWORLDMDBQueuec74edb61-566b-11e3-b001-7d234d3fdcb7 at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.doConsumerClosed(ClusterConnectionImpl.java:1602) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1] at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.onMessage(ClusterConnectionImpl.java:1295) [hornetq-server-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1] at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1117) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1] at org.hornetq.core.client.impl.ClientConsumerImpl.access$500(ClientConsumerImpl.java:57) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1] at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1252) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1] at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:107) [hornetq-core-client-2.3.12.Final-redhat-1.jar:2.3.12.Final-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_04] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_04] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_04] 09:38:41,360 INFO [org.hornetq.core.server] (Thread-16 (HornetQ-server-HornetQServerImpl::serverUUID=c490cc55-566b-11e3-9a34-5f04c5cc51dd-663290997)) HQ221029: stopped bridge sf.my-cluster.c74edb61-566b-11e3-b001-7d234d3fdcb7 09:38:41,395 INFO [org.hornetq.core.server] (ServerService Thread Pool -- 26) HQ221002: HornetQ Server version 2.3.12.Final (2.3.12, 123) [c490cc55-566b-11e3-9a34-5f04c5cc51dd] stopped 09:38:41,417 INFO [org.jboss.as] (MSC service thread 1-6) JBAS015950: JBoss EAP 6.2.0.GA (AS 7.3.0.Final-redhat-14) stopped in 502ms
This is still problem in EAP 6.3.0.ER9 (HQ 2.3.20).
Couple of questions... 1) Where can I get the latest build of EAP where this issue is observed? 2) Where can I get the "hornetq-clustering quickstart" referenced here to reproduce this issue?
@Miroslav can you set the proper z-stream ?. Target release + milestone are set when the BZ is included in a CP (they don't mean when you aim to fix the issue). Streams flags are used instead. Setting/asking for more than one stream in a BZ is not allowed by procedure. is it 6.3.z or 6.4.z ?