Bug 1119751

Summary:	LoginContext cannot load login modules in certain scenarios when EAP server is started by Arquillian
Product:	[JBoss] JBoss Enterprise Application Platform 6	Reporter:	Miroslav Novak <mnovak>
Component:	Class Loading, Testsuite, JMS	Assignee:	Aslak Knutsen <aknutsen>
Status:	CLOSED EOL	QA Contact:	Miroslav Novak <mnovak>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	6.3.0	CC:	david.lloyd, jmesnil, msvehla, pkremens
Target Milestone:	---
Target Release:	EAP 6.4.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-08-02 07:33:43 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Miroslav Novak 2014-07-15 12:36:03 UTC

There is problem with classloading when EAP 6.3.0.ER9 server is started by Arquillian. Problem was hit in following test scenario:
- Start two EAP 6.3.0.ER9 servers
	- HornetQ is configured in collocated HA topology with shared store (EAP servers contains HornetQ backup for each other - so there are 2 live/backup pairs)
- Start producer which sends messages to "jms/queue/testQueue0" to first EAP server
- Start consumer which receives messages from "jms/queue/testQueue0" to first EAP server
- Kill -9 "first EAP server"
- Producer and consumer failovers to backup

Problem:
When producer and consumer failovers to backup then it fails with:
13:58:40,146 Thread-2 (HornetQ-client-global-threads-2060674296) ERROR [org.hornetq.core.client.impl.ClientSessionImpl:1198] HQ214003: Failed to handle failover
HornetQException[errorType=SECURITY_EXCEPTION message=HQ119031: Unable to validate user: null]
	at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:394)
	at org.hornetq.core.client.impl.ClientSessionImpl.handleFailover(ClientSessionImpl.java:1087)
	at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:1075)
	at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:717)
	at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:587)
	at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionException(ClientSessionFactoryImpl.java:441)
	at org.hornetq.core.remoting.impl.netty.NettyConnector$Listener$2.run(NettyConnector.java:924)
	at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:722)

Second EAP server logs:
13:58:40,144 ERROR [org.hornetq.core.server] (Old I/O server worker (parentId: 223510377, [id: 0x0d527f69, /192.168.40.2:5446])) HQ224018: Failed to create session: HornetQException[errorType=SECURITY_EXCEPTION message=HQ119031: Unable to validate user: null]
	at org.hornetq.core.security.impl.SecurityStoreImpl.authenticate(SecurityStoreImpl.java:146) [hornetq-server-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1]
	at org.hornetq.core.server.impl.HornetQServerImpl.createSession(HornetQServerImpl.java:979) [hornetq-server-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1]
	at org.hornetq.core.protocol.core.impl.HornetQPacketHandler.handleCreateSession(HornetQPacketHandler.java:150) [hornetq-server-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1]
	at org.hornetq.core.protocol.core.impl.HornetQPacketHandler.handlePacket(HornetQPacketHandler.java:78) [hornetq-server-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1]
	at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:636) [hornetq-core-client-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1]
	at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547) [hornetq-core-client-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1]
	at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523) [hornetq-core-client-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1]
	at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:564) [hornetq-server-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1]
	at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:72) [hornetq-core-client-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1]
	at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1]
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1]
	at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169) [hornetq-core-client-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1]
	at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134) [hornetq-core-client-2.3.20.Final-redhat-1.jar:2.3.20.Final-redhat-1]
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1]
	at org.jboss.netty.channel.socket.oio.OioWorker.process(OioWorker.java:71) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1]
	at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1]
	at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1]
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1]
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1]
	at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175) [netty-3.6.9.Final-redhat-1.jar:3.6.9.Final-redhat-1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_15]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_15]
	at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_15]

It failed because it was not authenticated on second EAP server. When debugging the problem I figured that root cause for this failure is that javax.security.auth.login.LoginContext.invoke(String methodName):746 is using reflection to load class - org.jboss.as.security.RealmDirectLoginModule which was not found in provided classloader:
Class c = Class.forName(moduleStack[i].entry.getLoginModuleName(), true, contextClassLoader);

When I try the same test scenario manually then producer and consumer failovers without problem (=authenticates). Is there a way how EAP 6 classloader can be influenced by Arquillian?

I'm assigning this to Aslak as he could know? :-)

Comment 1 Miroslav Novak 2014-07-16 10:34:29 UTC

Adding David to cc as well.

Comment 2 Aslak Knutsen 2014-08-15 13:30:45 UTC

Is this running with the Arquillian JMX Protocol or Servlet Protocol?

If JMX, try using the Servlet to see if that does something else. It's less magical then the JMX one. Tho I'm not quite seeing how an Arquillian deployed 'client' should effect the classloading of a Module for a background failover.

Comment 3 Miroslav Novak 2014-08-18 11:12:46 UTC

It's running with Servlet Protocol (not JMX). All servers are running with @RunAsClient annotation.

Comment 4 Miroslav Novak 2014-08-18 11:13:40 UTC

Sorry, not servers but tests.

Comment 5 Aslak Knutsen 2014-08-18 11:40:18 UTC

You're running this with @Deployment.testable=false as well?

If so, Arquillian is not really doing anything magical at all. It just deploys the defined @Deployment and runs everything as if it were a client.

Could there be missing some command line options when starting via the Managed Container?

Comment 6 Miroslav Novak 2014-08-18 12:29:09 UTC

Yes, I'm running it with @Deployment.testable=false. 

I've checked the command line options and did some tries and could not find the property which would cause this problem.