Bug 582614

Summary: Intermittent NullPointerException on clicking on any resource name in inventory
Product: [Other] RHQ Project Reporter: Sunil Kondkar <skondkar>
Component: Core ServerAssignee: Lukas Krejci <lkrejci>
Status: CLOSED CURRENTRELEASE QA Contact: Corey Welton <cwelton>
Severity: high Docs Contact:
Priority: high    
Version: 3.0.0CC: cwelton, jmarques, rtimaniy
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: 2.4 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-08-12 16:51:07 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 582609    
Attachments:
Description Flags
Screenshot for nullpointer exception
none
rhq-server-log4j.log
none
Stack trace for the error in UI
none
Screenshot for exception
none
Server log for exception
none
UI stack trace none

Description Sunil Kondkar 2010-04-15 12:47:00 UTC
Created attachment 406766 [details]
Screenshot for nullpointer exception

Description of problem:

Installed jon build#96 (Revision: 10609), enabled embedded agent
After importing resources, clicking on any resource like 'RHQ Agent' gives a NullPointerException.

Version-Release number of selected component (if applicable):

jon build#96 (Revision: 10609)

How reproducible:

Intermittent

Steps to Reproduce:

1. Login to jon after installation
2. Click on any resource name link in 'Recently Added Resources ' section
3. It gives the NullPointerException intermittently.

Actual results:
It gives the NullPointerException intermittently.

Expected results:
It should not give exception and should display the resource details.

Additional info:

Below is stack trace as observed in rhq-server-log4j.log:

2010-04-15 17:41:20,424 ERROR [org.rhq.enterprise.gui.common.error.GenericErrorUIBean] Error processing user request
java.lang.NullPointerException
        at org.richfaces.model.StackingTreeModel.getInternalModelById(StackingTreeModel.java:354)
        at org.richfaces.model.StackingTreeModel.doSetupKey(StackingTreeModel.java:154)
        at org.richfaces.model.StackingTreeModel.setupKey(StackingTreeModel.java:191)
        at org.richfaces.model.StackingTreeModel.setRowKey(StackingTreeModel.java:497)
        at org.richfaces.model.StackingTreeModel.setRowKey(StackingTreeModel.java:493)
        at org.ajax4jsf.component.UIDataAdaptorBase.setRowKey(UIDataAdaptorBase.java:351)
        at org.ajax4jsf.component.UIDataAdaptorBase.setRowKey(UIDataAdaptorBase.java:324)
        at org.richfaces.renderkit.TreeRendererBase.getSelectionValue(TreeRendererBase.java:528)
        at org.richfaces.renderkit.TreeRendererBase.encodeSelectionStateInput(TreeRendererBase.java:553)
        at org.richfaces.renderkit.html.TreeRenderer.doEncodeEnd(TreeRenderer.java:236)
        at org.richfaces.renderkit.html.TreeRenderer.doEncodeEnd(TreeRenderer.java:253)
        at org.ajax4jsf.renderkit.RendererBase.encodeEnd(RendererBase.java:134)
        at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:861)
        at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:244)
        at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:241)
        at com.sun.faces.renderkit.html_basic.GroupRenderer.encodeChildren(GroupRenderer.java:106)
        at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:837)
        at javax.faces.component.UIComponent.encodeAll(UIComponent.java:930)
        at javax.faces.render.Renderer.encodeChildren(Renderer.java:148)
        at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:837)
        at javax.faces.component.UIComponent.encodeAll(UIComponent.java:930)
        at javax.faces.component.UIComponent.encodeAll(UIComponent.java:933)
        at com.sun.facelets.FaceletViewHandler.renderView(FaceletViewHandler.java:592)
        at org.rhq.enterprise.gui.common.framework.FaceletRedirectionViewHandler.renderView(FaceletRedirectionViewHandler.java:64)
        at org.ajax4jsf.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:100)
        at org.ajax4jsf.application.AjaxViewHandler.renderView(AjaxViewHandler.java:176)
        at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:110)
        at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:100)
        at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:139)
        at javax.faces.webapp.FacesServlet.service(FacesServlet.java:266)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:530)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.rhq.enterprise.gui.legacy.WebUserTrackingFilter.doFilter(WebUserTrackingFilter.java:50)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.rhq.enterprise.gui.legacy.AuthenticationFilter.doFilter(AuthenticationFilter.java:129)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.rhq.enterprise.gui.common.upload.MultipartFilter.doFilter(MultipartFilter.java:63)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
        at org.jboss.seam.web.IdentityFilter.doFilter(IdentityFilter.java:38)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:90)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:206)
        at org.ajax4jsf.webapp.BaseFilter.handleRequest(BaseFilter.java:290)
        at org.ajax4jsf.webapp.BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:388)
        at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:515)
        at org.jboss.seam.web.Ajax4jsfFilter.doFilter(Ajax4jsfFilter.java:56)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:58)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.jboss.seam.debug.hot.HotDeployFilter.doFilter(HotDeployFilter.java:51)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:124)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
        at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
        at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
        at java.lang.Thread.run(Thread.java:619)

Comment 1 Charles Crouch 2010-04-15 17:51:05 UTC
Hi Sunil
Some questions, things to try, while we investigate further...

1) Can you paste in the exceptions you get from the server log, and the exception you see when clicking on "view stacktrace" in the UI.
2) You are only seeing this from links on 'Recently Added Resources' portlet correct? So this should not be blocking testing in other areas?
3) Don't use the embedded agent *ever* in your testing, its not supported.
4) Try out this older build:
http://hudson-qe.rhq.rdu.redhat.com:8080/view/JON/job/jon/88/
And see if you can reproduce the problem.

Comment 2 Sunil Kondkar 2010-04-16 12:51:26 UTC
Created attachment 407089 [details]
rhq-server-log4j.log

Comment 3 Sunil Kondkar 2010-04-16 12:52:33 UTC
Created attachment 407090 [details]
Stack trace for the error in UI

Comment 4 Sunil Kondkar 2010-04-16 12:53:05 UTC
1) I have attached the server log (rhq-server-log4j.log) and the stack trace displayed in UI.

2) This issue is hitting intermittently.  Today I installed an agent on other box for same build#96. The exception is reproduced. 

 Sometimes clicking on inventory resource works as expected but sometimes it gives exception. I also observed the same exception when clicked on inventory tab and 'Connection' sub tab of the resource. I haven't experienced this exception in other areas. 

3) Installed old build#88. The exception is not reproducible as tested on build#88.

Comment 5 Joseph Marques 2010-04-19 20:21:51 UTC
commit b2150fe05b725ebb11ec01eb42c9c2db644523f3

possible fix for the intermittent NPE caused by the resource/group tree advisors;
    
* my suspicion was that the tree component does not properly support concurrent access, and that one AJAX request can destroy the state/context of another;
* added eventsQueue and ignoreDupResponses attributes to both resource and group trees to effectively serialize the tree re-rendering after user interaction;

Comment 6 Joseph Marques 2010-04-19 21:36:12 UTC
OK, after some more testing, I realize that the above fix does not resolve all the issues.  Although I can expand/collapse nodes quickly now, I still see the intermittent NPEs.

Comment 7 Joseph Marques 2010-04-20 01:15:48 UTC
commit c7b5ea00d71b0df431e97f5a65e64079371152a4

fix for intermittent NPE originating from the left nav trees:

* only retrieve row key from the tree if the selected node is not null; 
* only retrieve component state from the tree if the row key is not null;
* improve error handling (and message display) around IOExceptions during redirection;

Comment 8 Joseph Marques 2010-04-20 01:19:58 UTC
To test:

1) expand/collapse nodes in both resource/group trees in quick succession
2) click on different nodes in the tree, alternating between resources, autogroups, and subcategories (for the resource tree) and between groups, clustergroups (for the group tree)
3) click vigorously between the various tabs / sub-tabs for resources and groups
4) finally, alternate back and forth between the actions laid out in #1, #2, and #3 in quick succession

Comment 9 Sunil Kondkar 2010-04-21 12:28:56 UTC
Verified on jon build # 105 (Revision: 10609)

The NullPointerException is observed intermittently when clicked on the resource name link in 'Recently Added Resources' section and inventory tab and 'Connection' sub tab of the resource.

Please refer the screenshots and logs attached.

Comment 10 Sunil Kondkar 2010-04-21 12:30:01 UTC
Created attachment 408053 [details]
Screenshot for exception

Comment 11 Sunil Kondkar 2010-04-21 12:31:00 UTC
Created attachment 408054 [details]
Server log for exception

Comment 12 Sunil Kondkar 2010-04-21 12:32:28 UTC
Created attachment 408055 [details]
UI stack trace

Comment 13 Sunil Kondkar 2010-04-21 15:46:59 UTC
Also Verified in rhq build (Revision: 80e3403ada9b0b215715dc3d6ff15d9c016fd451)

The NullPointerException is reproducible in this build.

Comment 14 Corey Welton 2010-04-21 19:59:27 UTC
QA Verified:  I don't see NPEs:

1) clicking the items in Recently Added Resources 
2) Clicking a bunch of nodes 
3) expanding contracting left nav items 
4) clicking back and forth between tabs 
5) various combinations of these.

Comment 15 Corey Welton 2010-08-12 16:51:07 UTC
Mass-closure of verified bugs against JON.