Description of problem: An unknown series of events has somehow left scheduled operations in an inconsistent state. These scheduled operations are listed when displaying the dashboard. The scheduled operations are well past their expected dates of execution, but attempting to select any of these defunct operations causes exceptions in the browser and exception information to be piped into the server log. By navigating to the specific resource which the operations should be scheduled against, no scheduled operations are listed or appear in operation history. See exception message from debug log. Version-Release number of selected component (if applicable): 1.3.1 How reproducible: On machine with these defunct events, every time. Steps to Reproduce: 1. Navigate to the Dashboard view so that the defunct operations are listed. Exceptions will be intermittently written to the server log each time that page is refreshed. Actual results: Following exception thrown: 2010-03-12 08:22:39,244 DEBUG [org.rhq.enterprise.server.discovery.DiscoveryServerServiceImpl] Processed AV:[dummy-server-49][1][changesOnly] - need full=[false] in (13)ms 2010-03-12 08:22:40,822 ERROR [org.rhq.enterprise.server.operation.OperationManagerBean] A scheduled operation has an invalid name - did a plugin change its operation metadata? : ResourceOperationScheduleComposite: operation-job-id=[rhq-resource-525179-1097506319-1245122699560_=_rhq-resource-525179], operation-name=[null], operation-next-fire-time=[Mon Jun 15 22:24:59 CDT 2009], resource-id=[525179], resource-name=[dummy-server-03 RHQ Agent], resource-type-name=[RHQ Agent] java.lang.NullPointerException at org.rhq.enterprise.server.operation.OperationManagerBean.getResourceOperationSchedule(OperationManagerBean.java:420) at org.rhq.enterprise.server.operation.OperationManagerBean.getResourceOperationSchedule(OperationManagerBean.java:459) at org.rhq.enterprise.server.operation.OperationManagerBean.findCurrentlyScheduledResourceOperations(OperationManagerBean.java:1418) at sun.reflect.GeneratedMethodAccessor2175.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166) at org.rhq.enterprise.server.common.TransactionInterruptInterceptor.addCheckedActionToTransactionManager(TransactionInterruptInterceptor.java:77) at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118) at org.rhq.enterprise.server.authz.RequiredPermissionsInterceptor.checkRequiredPermissions(RequiredPermissionsInterceptor.java:153) at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118) at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79) at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:95) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77) at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:240) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:210) at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:84) at $Proxy336.findCurrentlyScheduledResourceOperations(Unknown Source) at org.rhq.enterprise.gui.legacy.portlet.controlactions.ViewAction.execute(ViewAction.java:91) at org.apache.struts.tiles.actions.TilesAction.execute(TilesAction.java:73) at org.apache.struts.tiles.actions.TilesAction.execute(TilesAction.java:73) at org.rhq.enterprise.gui.legacy.action.BaseRequestProcessor.processActionPerform(BaseRequestProcessor.java:46) at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:236) at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196) at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414) at javax.servlet.http.HttpServlet.service(HttpServlet.java:690) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:654) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:557) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:481) at org.apache.struts.tiles.UrlController.execute(UrlController.java:89) at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:875) at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:462) at org.apache.jsp.portal.DashboardLayout_jsp._jspx_meth_tiles_005finsert_005f0(DashboardLayout_jsp.java:1341) at org.apache.jsp.portal.DashboardLayout_jsp._jspx_meth_c_005fforEach_005f1(DashboardLayout_jsp.java:1183) at org.apache.jsp.portal.DashboardLayout_jsp._jspx_meth_c_005fforEach_005f0(DashboardLayout_jsp.java:751) at org.apache.jsp.portal.DashboardLayout_jsp._jspService(DashboardLayout_jsp.java:273) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:336) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:654) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:557) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:481) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968) at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:609) at org.apache.struts.tiles.TilesUtilImpl.doInclude(TilesUtilImpl.java:99) at org.apache.struts.tiles.TilesUtil.doInclude(TilesUtil.java:135) at org.apache.struts.taglib.tiles.InsertTag.doInclude(InsertTag.java:760) at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:892) at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:462) at org.apache.jsp.portal.MainLayout_jsp._jspx_meth_tiles_005finsert_005f3(MainLayout_jsp.java:496) at org.apache.jsp.portal.MainLayout_jsp._jspx_meth_html_005fhtml_005f0(MainLayout_jsp.java:171) at org.apache.jsp.portal.MainLayout_jsp._jspService(MainLayout_jsp.java:82) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:336) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:654) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:445) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:654) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:445) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:379) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:292) at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1085) at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:263) at org.apache.struts.tiles.TilesRequestProcessor.processTilesDefinition(TilesRequestProcessor.java:239) at org.apache.struts.tiles.TilesRequestProcessor.internalModuleRelativeForward(TilesRequestProcessor.java:341) at org.apache.struts.action.RequestProcessor.processForward(RequestProcessor.java:572) at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:221) at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196) at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414) at javax.servlet.http.HttpServlet.service(HttpServlet.java:690) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.rhq.enterprise.gui.legacy.WebUserTrackingFilter.doFilter(WebUserTrackingFilter.java:47) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.rhq.enterprise.gui.legacy.AuthenticationFilter.doFilter(AuthenticationFilter.java:129) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:654) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:557) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:481) at org.apache.struts.tiles.UrlController.execute(UrlController.java:89) at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:875) at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:462) at org.apache.jsp.portal.DashboardLayout_jsp._jspx_meth_tiles_005finsert_005f0(DashboardLayout_jsp.java:1341) at org.apache.jsp.portal.DashboardLayout_jsp._jspx_meth_c_005fforEach_005f1(DashboardLayout_jsp.java:1183) at org.apache.jsp.portal.DashboardLayout_jsp._jspx_meth_c_005fforEach_005f0(DashboardLayout_jsp.java:751) at org.apache.jsp.portal.DashboardLayout_jsp._jspService(DashboardLayout_jsp.java:273) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:336) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:654) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:557) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:481) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:557) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:481) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968) at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:609) at org.apache.struts.tiles.TilesUtilImpl.doInclude(TilesUtilImpl.java:99) at org.apache.struts.tiles.TilesUtil.doInclude(TilesUtil.java:135) at org.apache.struts.taglib.tiles.InsertTag.doInclude(InsertTag.java:760) at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:892) at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:462) at org.apache.jsp.portal.MainLayout_jsp._jspx_meth_tiles_005finsert_005f3(MainLayout_jsp.java:496) at org.apache.jsp.portal.MainLayout_jsp._jspx_meth_html_005fhtml_005f0(MainLayout_jsp.java:171) at org.apache.jsp.portal.MainLayout_jsp._jspService(MainLayout_jsp.java:82) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:336) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:654) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:445) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:379) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:292) at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1085) at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:263) at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1085) at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:263) at org.apache.struts.tiles.TilesRequestProcessor.processTilesDefinition(TilesRequestProcessor.java:239) at org.apache.struts.tiles.TilesRequestProcessor.internalModuleRelativeForward(TilesRequestProcessor.java:341) at org.apache.struts.action.RequestProcessor.processForward(RequestProcessor.java:572) at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:221) at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196) at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414) at javax.servlet.http.HttpServlet.service(HttpServlet.java:690) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.rhq.enterprise.gui.legacy.WebUserTrackingFilter.doFilter(WebUserTrackingFilter.java:47) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.rhq.enterprise.gui.legacy.AuthenticationFilter.doFilter(AuthenticationFilter.java:129) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:124) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) java:206) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446) at java.lang.Thread.run(Thread.java:595) 2010-03-12 08:22:40,824 ERROR [org.rhq.enterprise.server.operation.OperationManagerBean] A scheduled operation has an invalid name - did a plugin change its operation metadata? : ResourceOperationScheduleComposite: operation-job-id=[rhq-resource-535012-1097506319-1245118436269_=_rhq-resource-535012], operation-name=[null], operation-next-fire-time=[Mon Jun 15 21:13:56 CDT 2009], resource-id=[535012], resource-name=[dummy-server-26 RHQ Agent], resource-type-name=[RHQ Agent] java.lang.NullPointerException at org.rhq.enterprise.server.operation.OperationManagerBean.getResourceOperationSchedule(OperationManagerBean.java:420) at org.rhq.enterprise.server.operation.OperationManagerBean.getResourceOperationSchedule(OperationManagerBean.java:459) at org.rhq.enterprise.server.operation.OperationManagerBean.getResourceOperationSchedule(OperationManagerBean.java:420) at org.rhq.enterprise.server.operation.OperationManagerBean.getResourceOperationSchedule(OperationManagerBean.java:459) at org.rhq.enterprise.server.operation.OperationManagerBean.findCurrentlyScheduledResourceOperations(OperationManagerBean.java:1418) at sun.reflect.GeneratedMethodAccessor2175.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166) at org.rhq.enterprise.server.common.TransactionInterruptInterceptor.addCheckedActionToTransactionManager(TransactionInterruptInterceptor.java:77) at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118) at org.rhq.enterprise.server.authz.RequiredPermissionsInterceptor.checkRequiredPermissions(RequiredPermissionsInterceptor.java:153) at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118) at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) Expected results: No defunct operations shown and no exceptions thrown in the gui or server log. Additional info: defunct scheduled operation links from browser contained following additional information: http://lxpmoa06:7080/rhq/resource/operation/resourceOperationScheduleDetails.xhtml?id=525179&jobId=rhq-resource-525179-1097506319-1245122699560_=_rhq-resource-525179 http://lxpmoa06:7080/rhq/resource/operation/resourceOperationScheduleDetails.xhtml?id=535012&jobId=rhq-resource-535012-1097506319-1245118436269_=_rhq-resource-535012 http://lxpmoa06:7080/rhq/resource/operation/resourceOperationScheduleDetails.xhtml?id=500196&jobId=rhq-resource-500196-1097506319-1245118195999_=_rhq-resource-500196 http://lxpmoa06:7080/rhq/resource/operation/resourceOperationScheduleDetails.xhtml?id=503201&jobId=rhq-resource-503201-1097506319-1245092639741_=_rhq-resource-503201 http://lxpmoa06:7080/rhq/resource/operation/resourceOperationScheduleDetails.xhtml?id=500196&jobId=rhq-resource-500196-1097506319-1245092629641_=_rhq-resource-500196
The saved scheduled operations information is in an inconsistent state. Defunct scheduled operations are queried and shown in the Dashboard but do not exist in the (selected resource)->Operations tab. Still investigating how the defunct data is being stored. Short term suggestion is to disable the "Operations" tab completely or not to display scheduled operations from being displayed to prevent annoying defunct schedule messages in the sever log. i) Use <server:port>/dashboard/Admin.do?mode=controlActions and set "Display on Dashboard" to No. OR ii)Use <server:port>/dashboard/Admin.do?mode=controlActions and uncheck the "Next" checkbox from the "Operation Range".
##### Snippet from selecting the defunct schedule entries from Dashboard. An Error Occurred: javax.el.ELException: /rhq/resource/operation/resourceOperationScheduleDetails.xhtml @40,74 value="#{item.operationDisplayName}": /rhq/resource/operation/resourceOperationScheduleDetails.xhtml @21,87 value="#{ResourceOperationScheduleDetailsUIBean.schedule}": Error reading 'schedule' on type org.rhq.enterprise.gui.operation.schedule.resource.ResourceOperationScheduleDetailsUIBean +- Stack Trace javax.faces.FacesException: javax.el.ELException: /rhq/resource/operation/resourceOperationScheduleDetails.xhtml @40,74 value="#{item.operationDisplayName}": /rhq/resource/operation/resourceOperationScheduleDetails.xhtml @21,87 value="#{ResourceOperationScheduleDetailsUIBean.schedule}": Error reading 'schedule' on type org.rhq.enterprise.gui.operation.schedule.resource.ResourceOperationScheduleDetailsUIBean at javax.faces.component.UIOutput.getValue(UIOutput.java:187) at com.sun.faces.renderkit.html_basic.HtmlBasicInputRenderer.getValue(HtmlBasicInputRenderer.java:201) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.getCurrentValue(HtmlBasicRenderer.java:284) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeEnd(HtmlBasicRenderer.java:154) at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:861) at org.ajax4jsf.renderkit.RendererBase.renderChild(RendererBase.java:286) at org.ajax4jsf.renderkit.RendererBase.renderChildren(RendererBase.java:262) at org.richfaces.renderkit.html.PanelRenderer.doEncodeChildren(PanelRenderer.java:220) at org.richfaces.renderkit.html.PanelRenderer.doEncodeChildren(PanelRenderer.java:215) at org.ajax4jsf.renderkit.RendererBase.encodeChildren(RendererBase.java:121) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:837) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:930) at javax.faces.render.Renderer.encodeChildren(Renderer.java:148) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:837) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:930) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:933) at com.sun.facelets.FaceletViewHandler.renderView(FaceletViewHandler.java:592) at org.rhq.enterprise.gui.common.framework.FaceletRedirectionViewHandler.renderView(FaceletRedirectionViewHandler.java:64) at org.ajax4jsf.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:100) at org.ajax4jsf.application.AjaxViewHandler.renderView(AjaxViewHandler.java:176) at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:110) at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:100) at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:139) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:266) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:532) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.rhq.enterprise.gui.legacy.WebUserTrackingFilter.doFilter(WebUserTrackingFilter.java:47) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.rhq.enterprise.gui.legacy.AuthenticationFilter.doFilter(AuthenticationFilter.java:129) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.rhq.enterprise.gui.common.upload.MultipartFilter.doFilter(MultipartFilter.java:63) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83) at org.jboss.seam.web.IdentityFilter.doFilter(IdentityFilter.java:38) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:90) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:178) at org.ajax4jsf.webapp.BaseFilter.handleRequest(BaseFilter.java:290) at org.ajax4jsf.webapp.BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:390) at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:517) at org.jboss.seam.web.Ajax4jsfFilter.doFilter(Ajax4jsfFilter.java:56) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:58) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:124) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446) at java.lang.Thread.run(Thread.java:595) Caused by: javax.el.ELException: /rhq/resource/operation/resourceOperationScheduleDetails.xhtml @40,74 value="#{item.operationDisplayName}": /rhq/resource/operation/resourceOperationScheduleDetails.xhtml @21,87 value="#{ResourceOperationScheduleDetailsUIBean.schedule}": Error reading 'schedule' on type org.rhq.enterprise.gui.operation.schedule.resource.ResourceOperationScheduleDetailsUIBean at com.sun.facelets.el.TagValueExpression.getValue(TagValueExpression.java:76) at javax.faces.component.UIOutput.getValue(UIOutput.java:184) ... 76 more Caused by: java.lang.IllegalStateException: java.lang.NullPointerException at org.rhq.enterprise.gui.operation.schedule.OperationScheduleDetailsUIBean.init(OperationScheduleDetailsUIBean.java:54) at org.rhq.enterprise.gui.operation.schedule.OperationScheduleDetailsUIBean.getSchedule(OperationScheduleDetailsUIBean.java:77) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at javax.el.BeanELResolver.getValue(BeanELResolver.java:62) at javax.el.CompositeELResolver.getValue(CompositeELResolver.java:53) at com.sun.faces.el.FacesCompositeELResolver.getValue(FacesCompositeELResolver.java:72) at org.jboss.el.parser.AstPropertySuffix.getValue(AstPropertySuffix.java:53) at org.jboss.el.parser.AstValue.getValue(AstValue.java:67) at org.jboss.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:186) at com.sun.facelets.el.TagValueExpression.getValue(TagValueExpression.java:71) at org.jboss.el.parser.AstIdentifier.getValue(AstIdentifier.java:40) at org.jboss.el.parser.AstValue.getValue(AstValue.java:63) at org.jboss.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:186) at com.sun.facelets.el.TagValueExpression.getValue(TagValueExpression.java:71) ... 77 more Caused by: javax.ejb.EJBException: java.lang.NullPointerException at org.jboss.ejb3.tx.Ejb3TxPolicy.handleExceptionInOurTx(Ejb3TxPolicy.java:63) at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:83) at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:95) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77) at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:240) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:210) at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:84) at $Proxy336.getResourceOperationSchedule(Unknown Source) at org.rhq.enterprise.gui.operation.schedule.resource.ResourceOperationScheduleDetailsUIBean.getOperationSchedule(ResourceOperationScheduleDetailsUIBean.java:29) at org.rhq.enterprise.gui.operation.schedule.OperationScheduleDetailsUIBean.init(OperationScheduleDetailsUIBean.java:51) ... 93 more Caused by: java.lang.NullPointerException at org.rhq.enterprise.server.operation.OperationManagerBean.getResourceOperationSchedule(OperationManagerBean.java:420) at org.rhq.enterprise.server.operation.OperationManagerBean.getResourceOperationSchedule(OperationManagerBean.java:459) at sun.reflect.GeneratedMethodAccessor1234.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166) at org.rhq.enterprise.server.common.TransactionInterruptInterceptor.addCheckedActionToTransactionManager(TransactionInterruptInterceptor.java:77) at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118) at org.rhq.enterprise.server.authz.RequiredPermissionsInterceptor.checkRequiredPermissions(RequiredPermissionsInterceptor.java:153) at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118) at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
After further investigation, it has been determined that due to some series of events in the past, the database has been left in an inconsistent state such that scheduled operation job details were deleted from the database only without the accompanying RHQ table information being updated. The simplest and fastest fix is to run the attached sql to reinsert temporary job details using the http://(jon host):7080/admin/test/sql.jsp form. Immediately after reinserted the schedule relationship data, the Dashboard.do page should be refreshed and for each of the previously stale schedules, each one should be selected and unscheduled. Essentially you are rebuilding the scheduled operation details so that the JON UI can clean up the stale schedules appropriately. The longer term fix will likely involve adding a new MeasurementScheduleManagerBean.errorCorrectOperationSchedules method for JON administrators to use to clean up such scheduling data if it were to arrive in an inconsistent state. Additional: deleting job details from a healthy scheduled operation results correctly reproduces the error log and ui messages that the customer is reporting.
Created attachment 402410 [details] sql to correct stale schedules.
Upon further inspection I think the easiest resolution to the problem is to eliminate the problematic schedules via direct SQL. The problem results when the underlying quartz job details for a scheduled operation are, for some reason, missing. Quartz tables are defined such that an rhq_qrtz_job_details row outwardly exposes the job. If it is removed all other referring quartz entries are cascade deleted at the db level, due the FK definitions. This means that if we're missing the job details record we can assume that there is basically no trace of the original quartz job in existence. The only invalid entries are what we have in our own tables. Given that that is the case we should be able to just remove any schedule that does not have an underlying quartz job_details record. It should be necessary to 1) recreate the record and 2) unschedule the operation. That should work but requires additional manual steps. Instead, executing this SQL should do the trick: DELETE FROM rhq_operation_schedule sched WHERE NOT EXISTS ( SELECT * FROM rhq_qrtz_job_details job WHERE job.job_name = sched.job_name ) This statement should remove all of the invalid schedules. p.s. I didn't look in depth but I'm not sure why we don't have a FK defined between rhq_operation_schedule and rhq_qrtz_job_details. Perhaps it has to do with not tying our tables to quartz tables.
Joe made the good point that the above SQL can be improved. It's not using the complete unique key. So, the enhanced SQL: DELETE FROM rhq_operation_schedule sched WHERE NOT EXISTS ( SELECT * FROM rhq_qrtz_job_details job WHERE job.job_name = sched.job_name AND job.job_group = sched.job_group ) Use this as opposed the SQL in the previous comment.
With sql attached that can be used to repair the defunct scheduled operations so that they can be manually removed via the GUI, this issue is being closed. The cause and more correct schedule clean up is being addressed in the following BZ issue: https://bugzilla.redhat.com/show_bug.cgi?id=580175 No source code changes were made to deal with this issue so there is nothing to be QA'd or verified.
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: No source code changed. See https://bugzilla.redhat.com/show_bug.cgi?id=580175 for more details on how situation will be better avoided/rectified.