Bug 564514

Summary: NPE in Agent when scanning JVM with password-protected JMX
Product: [Other] RHQ Project Reporter: Ondřej Žižka <ozizka>
Component: PluginsAssignee: Stefan Negrea <snegrea>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: medium Docs Contact:
Priority: high    
Version: 1.3CC: cwelton, mazz, snegrea
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-09-02 03:25:32 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Bug Depends On:    
Bug Blocks: 683054, 565628, 567481    
Attachments:
Description Flags
Agent log. none

Description Ondřej Žižka 2010-02-12 17:45:17 EST
StR:
====

1) Install Tomcat 5 and make it be monitored in JON.
2) Secure it's JMX with a password. *
3) Agent: discovery -f
4) Check in JON. The JVM is red.
5) Check Agent's log. Contains the ERRORs as below.

*) I've put this to the startup.bat:

JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.port=12121"
JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.authenticate=true"
JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.ssl=false"
JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.access.file=/mnt/jqa/jdk1.6.0_17-x86_64/jre/lib/management/jmxremote.access"
JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.password.file=/mnt/jqa/jdk1.6.0_17-x86_64/jre/lib/management/jmxremote.password"

 ...then create and edit the .password file accordingly - see
 http://java.sun.com/j2se/1.5.0/docs/guide/management/agent.html




Full log attached.


2010-02-12 23:13:25,224 ERROR [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Failed to start component for Resource[id=10360, type=VM Class Loading System, key=java.lang:type=ClassLoading, name=Class Loading, parent=Tomcat Server JVM] from synchronized merge.
org.rhq.core.clientapi.agent.PluginContainerException: Failed to start component for resource Resource[id=10360, type=VM Class Loading System, key=java.lang:type=ClassLoading, name=Class Loading, parent=Tomcat Server JVM].
	at org.rhq.core.pc.inventory.InventoryManager.activateResource(InventoryManager.java:1283)
	at org.rhq.core.pc.inventory.InventoryManager.refreshResourceComponentState(InventoryManager.java:2256)
	at org.rhq.core.pc.inventory.InventoryManager.processSyncInfo(InventoryManager.java:2057)
	at org.rhq.core.pc.inventory.InventoryManager.processSyncInfo(InventoryManager.java:2063)
	at org.rhq.core.pc.inventory.InventoryManager.processSyncInfo(InventoryManager.java:2063)
	at org.rhq.core.pc.inventory.InventoryManager.processSyncInfo(InventoryManager.java:2063)
	at org.rhq.core.pc.inventory.InventoryManager.synchInventory(InventoryManager.java:807)
	at org.rhq.core.pc.inventory.InventoryManager.handleReport(InventoryManager.java:787)
	at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.call(AutoDiscoveryExecutor.java:121)
	at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.run(AutoDiscoveryExecutor.java:92)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:146)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:170)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
	at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.NullPointerException
	at org.rhq.plugins.jmx.MBeanResourceComponent.loadBean(MBeanResourceComponent.java:183)
	at org.rhq.plugins.jmx.MBeanResourceComponent.loadBean(MBeanResourceComponent.java:169)
	at org.rhq.plugins.jmx.MBeanResourceComponent.start(MBeanResourceComponent.java:107)
	at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:525)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
	at java.util.concurrent.FutureTask.run(FutureTask.java:123)
	... 3 more
2010-02-12 23:13:25,225 ERROR [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Failed to start component for Resource[id=10356, type=Operating System, key=java.lang:type=OperatingSystem, name=Operating system information, parent=Tomcat Server JVM] from synchronized merge.
Comment 1 Ondřej Žižka 2010-02-12 17:48:37 EST
Created attachment 390602 [details]
Agent log.
Comment 2 Ondřej Žižka 2010-02-12 17:56:56 EST
I've removed " of Tomcat 5" from the bug's title.
Perhaps it's not Tomcat-related, but I found it when testing EWS.
Comment 3 Ondřej Žižka 2010-02-12 18:00:30 EST
Tomcat 5.5.28 from EWS 1.0.1 CR1 RHEL x64, download here:
http://download.devel.redhat.com/devel/candidates/JBEWS/1.0.1_CR1/RHEL5/
Comment 4 Ondřej Žižka 2010-02-12 18:02:49 EST
Note that other Tomcat sub-nodes are OK;
and JVM monitoring works with unprotected JMX.
Comment 5 wes hayutin 2010-02-16 11:57:52 EST
Temporarily adding the keyword "SubBug" so we can be sure we have accounted for all the bugs.

keyword:
new = Tracking + FutureFeature + SubBug
Comment 6 wes hayutin 2010-02-16 12:02:49 EST
making sure we're not missing any bugs in rhq_triage
Comment 7 Corey Welton 2010-09-30 09:56:43 EDT
mazz, jshaughn - any thoughts?
Comment 8 John Mazzitelli 2010-09-30 10:11:05 EDT
looks like an NPE in the jmx plugin:

Caused by: java.lang.NullPointerException
 at
org.rhq.plugins.jmx.MBeanResourceComponent.loadBean(MBeanResourceComponent.java:183)


and this might be something jay s fixed a while ago. i don't have the bz handy, but there was an issue where the EMS bean cache was invalid and I think caused an NPE. In other words, this might be fixed alreaedy.
Comment 9 Charles Crouch 2011-03-08 07:54:37 EST
Needs to be reproduced to see if it is still a problem. In particular the following information would be helpful:

a) Were the jmxremote settings enabled in Tomcat prior to it being initially discovered or added as part of enabling security? 
b) Is this a problem if Tomcat 5 is secured before its discovered.
Comment 10 Ondřej Žižka 2011-03-15 20:22:33 EDT
Still happens. Actually I reported this as a new JIRA, but not BZ.
https://issues.jboss.org/browse/JBQA-4380

Summary:

1) Have a secured Tomcat
2) *Don't* set principal and credentials in Connection tab
3) discovery [-f]

To answer Charles' questions:

a) No, jmx were disabled when initially discovered.
b) Not tried yet. Will try.
Comment 11 Ondřej Žižka 2011-03-15 22:07:10 EDT
To clarify, the sequence was:

1) Instalation with EWS default - no JMX
2) Inventorized
3) Enabled JMX without security
4) discovery -f
5) Secured JMX
6) discovery -f => NPE
7) set credentials in Connection tab => OK
Comment 12 Stefan Negrea 2011-10-12 15:57:02 EDT
The errors reported above are displayed only in the logs, they are not visible
to the users. The users will sees the resources without correct credentials as Down until the correct credentials are provided.
Comment 13 Stefan Negrea 2011-10-13 21:59:27 EDT
Updated the JMX plugin code to gracefully handle the case when the ems connection is null due to missing JMX credentials. Also, updated mod_cluster plugin code that had a similar loadBean method implementation.

When retesting this issue, the null pointer exception is replaced by an Ems exception. This new exception is normal since the resource requires authentication but no credentials have been provided yet by the user. The exception is visible only in the agent logs; in UI the resource is down until the user provides a correct set of username/password.

Here is a snippet from the new exception:
org.mc4j.ems.connection.EmsConnectException: Could not connect [service:jmx:rmi:///jndi/rmi://localhost:12345/jmxrmi] java.lang.SecurityException: Authentication failed! Username or password is null
Comment 14 Heiko W. Rupp 2013-09-02 03:25:32 EDT
Bulk closing of issues that were VERIFIED, had no target release and where the status changed more than a year ago.