There is problem with classloading when EAP 6.3.0.ER9 server is started by Arquillian. Problem was hit in following test scenario: - Start two EAP 6.3.0.ER9 servers - HornetQ is configured in collocated HA topology with shared store (EAP servers contains HornetQ backup for each other - so there are 2 live/backup pairs) - Start producer which sends messages to "jms/queue/testQueue0" to first EAP server - Start consumer which receives messages from "jms/queue/testQueue0" to first EAP server - Kill -9 "first EAP server" - Producer and consumer failovers to backup Problem: When producer and consumer failovers to backup then it fails with: 13:58:40,146 Thread-2 (HornetQ-client-global-threads-2060674296) ERROR [org.hornetq.core.client.impl.ClientSessionImpl:1198] HQ214003: Failed to handle failover HornetQException[errorType=SECURITY_EXCEPTION message=HQ119031: Unable to validate user: null] at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:394) at org.hornetq.core.client.impl.ClientSessionImpl.handleFailover(ClientSessionImpl.java:1087) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:1075) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:717) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:587) at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionException(ClientSessionFactoryImpl.java:441) at org.hornetq.core.remoting.impl.netty.NettyConnector$Listener$2.run(NettyConnector.java:924) at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Second EAP server logs: 13:58:40,144 ERROR [org.hornetq.core.server] (Old I/O server worker (parentId: 223510377, [id: 0x0d527f69, /192.168.40.2:5446])) HQ224018: Failed to create session: HornetQException[errorType=SECURITY_EXCEPTION message=HQ119031: Unable to validate user: null] at org.hornetq.core.security.impl.SecurityStoreImpl.authenticate(SecurityStoreImpl.java:146) [hornetq-server-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.hornetq.core.server.impl.HornetQServerImpl.createSession(HornetQServerImpl.java:979) [hornetq-server-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.hornetq.core.protocol.core.impl.HornetQPacketHandler.handleCreateSession(HornetQPacketHandler.java:150) [hornetq-server-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.hornetq.core.protocol.core.impl.HornetQPacketHandler.handlePacket(HornetQPacketHandler.java:78) [hornetq-server-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:636) [hornetq-core-client-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547) [hornetq-core-client-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523) [hornetq-core-client-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:564) [hornetq-server-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:72) [hornetq-core-client-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169) [hornetq-core-client-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134) [hornetq-core-client-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.channel.socket.oio.OioWorker.process(OioWorker.java:71) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_15] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_15] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_15] It failed because it was not authenticated on second EAP server. When debugging the problem I figured that root cause for this failure is that javax.security.auth.login.LoginContext.invoke(String methodName):746 is using reflection to load class - org.jboss.as.security.RealmDirectLoginModule which was not found in provided classloader: Class c = Class.forName(moduleStack[i].entry.getLoginModuleName(), true, contextClassLoader); When I try the same test scenario manually then producer and consumer failovers without problem (=authenticates). Is there a way how EAP 6 classloader can be influenced by Arquillian? I'm assigning this to Aslak as he could know? :-)
Adding David to cc as well.
Is this running with the Arquillian JMX Protocol or Servlet Protocol? If JMX, try using the Servlet to see if that does something else. It's less magical then the JMX one. Tho I'm not quite seeing how an Arquillian deployed 'client' should effect the classloading of a Module for a background failover.
It's running with Servlet Protocol (not JMX). All servers are running with @RunAsClient annotation.
Sorry, not servers but tests.
You're running this with @Deployment.testable=false as well? If so, Arquillian is not really doing anything magical at all. It just deploys the defined @Deployment and runs everything as if it were a client. Could there be missing some command line options when starting via the Managed Container?
Yes, I'm running it with @Deployment.testable=false. I've checked the command line options and did some tries and could not find the property which would cause this problem.