This has the potential to being very bad. It looks like the plugin container is sharing plugin configurations across multiple resources. See the attached flash demo to see the bug and how to replicate it. This shows a Jopr installation because its illustrating the bug with JBossAS resources, but I did not file this bug in the Jopr JIRA because I think this may be a problem with the plugin container as a whole (it looks like its sharing plugin configurations across resources). In short: I have two resources (JBossAS servers) with two different plugin configurations (the security credentials to connect to them are both different). Only one resource can connect, the other cannot. The last resource whose plugin configuraiton was edited/save goes GREEN, the other RED . See the demo, it makes it more clear what the bug is.
Looks like an EMS issue. Put a breakpoint on the first line of JBossASServerComponent.getAvailability. As it gets the availability for the default1 (127.0.0.1), the EMS connection has this connection settings: EmsConnection [JBoss] Initial Context = org.jnp.interfaces.NamingContextFactory JNDI Name = jmx/rmi/RMIAdaptor Server URL = jnp://127.0.0.1:1099 Principle = admin1 Credentials = admin1 Advanced Property [java.naming.factory.initial] = org.jboss.security.jndi.JndiLoginInitialContextFactory Advanced Property [java.naming.security.principal] = admin1 Advanced Property [jnp.sotimeout] = 15000 Advanced Property [jnp.timeout] = 30000 Advanced Property [jnp.disableDiscovery] = true Advanced Property [mc4j.ems.DefaultDomainSearch] = jboss Advanced Property [java.naming.provider.url] = jnp://127.0.0.1:1099 Advanced Property [java.naming.factory.url.pkgs] = org.jboss.naming:org.jnp.interfaces Advanced Property [java.naming.security.credentials] = admin1 Control Property [mc4j.ems.CopyJarsToTemp] = true Control Property [mc4j.ems.JarTempDir] = C:\mazz\source\rhq\trunk\modules\enterprise\agent\target\rhq-agent-1.2.0-SNAPSHOT\data\tmp This connection allows for this to be successful: bean.getAttribute("Started").refresh(); return AvailabilityType.UP; Next, when the default2 server's avail is checked (192.168.0.7), there is the EMS connection settings: EmsConnection [JBoss] Initial Context = org.jnp.interfaces.NamingContextFactory JNDI Name = jmx/rmi/RMIAdaptor Server URL = jnp://192.168.0.7:1099 Principle = admin2 Credentials = admin2 Advanced Property [java.naming.factory.initial] = org.jboss.security.jndi.JndiLoginInitialContextFactory Advanced Property [java.naming.security.principal] = admin2 Advanced Property [jnp.sotimeout] = 15000 Advanced Property [jnp.timeout] = 30000 Advanced Property [jnp.disableDiscovery] = true Advanced Property [mc4j.ems.DefaultDomainSearch] = jboss Advanced Property [java.naming.provider.url] = jnp://192.168.0.7:1099 Advanced Property [java.naming.factory.url.pkgs] = org.jboss.naming:org.jnp.interfaces Advanced Property [java.naming.security.credentials] = admin2 Control Property [mc4j.ems.CopyJarsToTemp] = true Control Property [mc4j.ems.JarTempDir] = C:\mazz\source\rhq\trunk\modules\enterprise\agent\target\rhq-agent-1.2.0-SNAPSHOT\data\tmp This connection however causes this to throw an exception: bean.getAttribute("Started").refresh(); The exception is (note: why is EMS using admin1 principal? you can see the connection's settings has admin2, from above): org.mc4j.ems.connection.EmsException: Could not load attribute value Connection failure Failed to authenticate principal=admin1, securityDomain=jmx-console at org.mc4j.ems.impl.jmx.connection.bean.attribute.DAttribute.refresh(DAttribute.java:235) at org.rhq.plugins.jbossas.JBossASServerComponent.getAvailability(JBossASServerComponent.java:292) at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:450) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) Caused by: org.mc4j.ems.connection.EmsConnectException: Connection failure Failed to authenticate principal=admin1, securityDomain=jmx-console at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:144) at $Proxy44.getAttribute(Unknown Source) at org.mc4j.ems.impl.jmx.connection.bean.attribute.DAttribute.refresh(DAttribute.java:199) ... 10 more Caused by: java.lang.SecurityException: Failed to authenticate principal=admin1, securityDomain=jmx-console at org.jboss.jmx.connector.invoker.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:97) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.invocation.jrmp.server.JRMPProxyFactory.invoke(JRMPProxyFactory.java:179) at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.invocation.jrmp.server.JRMPInvoker$MBeanServerAction.invoke(JRMPInvoker.java:818) at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:419) at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142) at org.jboss.invocation.jrmp.server.JRMPInvoker_Stub.invoke(Unknown Source) at org.jboss.invocation.jrmp.interfaces.JRMPInvokerProxy.invoke(JRMPInvokerProxy.java:133) at org.jboss.invocation.InvokerInterceptor.invokeInvoker(InvokerInterceptor.java:365) at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:197) at org.jboss.jmx.connector.invoker.client.InvokerAdaptorClientInterceptor.invoke(InvokerAdaptorClientInterceptor.java:66) at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:70) at org.jboss.proxy.ClientMethodInterceptor.invoke(ClientMethodInterceptor.java:74) at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100) at $Proxy43.getAttribute(Unknown Source) at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBean ServerProxy.invoke(GenericMBeanServerProxy.java:115) ... 12 more
renaming the subject since it looks like plugin configurations *are* correct, however, the underlying EMS library is somehow getting confused. This may be a problem with EMS or the JMX plugin, I am not sure.
greg found the issue - its inside the JBoss JNP code. Will need to talk to the JBossAS devs to find out how to fix. In the meantime, the only workaround is to make sure all of your credentials for all of your JBossAS isntances running on the same box are the same. This doesn't mean ALL user/passwords are the same across ALL boxes - only has to be same for those instances running on the same box.
Here's code to replciate the problem, to prove there is nothing specific to RHQ or EMS that causes the problem. Note this code only uses javax.* classes - but you need the jboss client jar in the classpath when you run this. Set up two JBossAS servers, each bound to 127.0.0.1 and .2, each with different user/passwords and run this test: package org.jboss.on.server.other; import javax.naming.Context; import javax.naming.InitialContext; import javax.naming.NamingException; import javax.management.MBeanServer; import javax.management.MBeanServerConnection; import java.util.Properties; public class MultiJBossConnectTest { public static void main(String[] args) throws Exception { MBeanServerConnection con1 = (MBeanServerConnection)connect("admin","admin2", "jnp://127.0.0.1"); System.out.println("con1 done: " + con1.getMBeanCount()); MBeanServerConnection con2 = (MBeanServerConnection) connect("admin", "admin3", "jnp://127.0.0.2"); System.out.println("con2 done: " + con2.getMBeanCount()); System.out.println("con1 done: " + con1.getMBeanCount()); } private static Object connect(String user, String pass, String url) throws ClassNotFoundException, NamingException { Properties props = new Properties(); Class.forName("org.jboss.security.jndi.JndiLoginInitialContextFactory"); props.put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.security.jndi.JndiLoginInitialContextFactory"); props.put(Context.SECURITY_PRINCIPAL, user); props.put(Context.SECURITY_CREDENTIALS, pass); props.put(Context.PROVIDER_URL, url); props.put(Context.URL_PKG_PREFIXES, "org.jboss.naming:org.jnp.interfaces" ); props.put("jnp.disableDiscovery", "True"); InitialContext context = new InitialContext(props); Object rmiAdaptor = context.lookup("jmx/rmi/RMIAdaptor"); return rmiAdaptor; } }
This is not an RHQ bug. The config content is correctly being passed into the plugin.... in this case the naming context is storing stuff to thread local uncontrollably. Problem moved to JOPR-9 (https://jira.jboss.org/jira/browse/JOPR-9).
This bug was previously known as http://jira.rhq-project.org/browse/RHQ-1030 Imported an attachment (id=368482) This bug is duplicated by RHQ-945
*** Bug 536613 has been marked as a duplicate of this bug. ***