Bug 574037

Summary: stale scheduled operations cause exceptions in UI and server log.
Product: [Other] RHQ Project Reporter: Simeon Pinder <spinder>
Component: Core UIAssignee: Simeon Pinder <spinder>
Status: CLOSED NOTABUG QA Contact: Corey Welton <cwelton>
Severity: medium Docs Contact:
Priority: low    
Version: 1.3.1CC: jshaughn, tao
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
No source code changed. See https://bugzilla.redhat.com/show_bug.cgi?id=580175 for more details on how situation will be better avoided/rectified.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2010-04-09 21:05:32 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 574133, 574605, 576714    
Attachments:
Description Flags
sql to correct stale schedules. none

Description Simeon Pinder 2010-03-16 13:17:38 UTC 
Description of problem:
An unknown series of events has somehow left scheduled operations in an inconsistent state.  These scheduled operations are listed when displaying the dashboard.  The scheduled operations are well past their expected dates of execution, but attempting to select any of these defunct operations causes exceptions in the browser and exception information to be piped into the server log. By navigating to the specific resource which the operations should be scheduled against, no scheduled operations are listed or appear in operation history.  See exception message from debug log. 

Version-Release number of selected component (if applicable):
1.3.1

How reproducible:
On machine with these defunct events, every time.

Steps to Reproduce:
1. Navigate to the Dashboard view so that the defunct operations are listed. Exceptions will be intermittently written to the server log each time that page is refreshed.
  
Actual results:
Following exception thrown:

2010-03-12 08:22:39,244 DEBUG [org.rhq.enterprise.server.discovery.DiscoveryServerServiceImpl] Processed AV:[dummy-server-49][1][changesOnly] - need full=[false] in (13)ms
2010-03-12 08:22:40,822 ERROR [org.rhq.enterprise.server.operation.OperationManagerBean] A scheduled operation has an invalid name - did a plugin change its operation metadata? : ResourceOperationScheduleComposite: operation-job-id=[rhq-resource-525179-1097506319-1245122699560_=_rhq-resource-525179], operation-name=[null], operation-next-fire-time=[Mon Jun 15 22:24:59 CDT 2009], resource-id=[525179], resource-name=[dummy-server-03 RHQ Agent], resource-type-name=[RHQ Agent]
java.lang.NullPointerException
        at org.rhq.enterprise.server.operation.OperationManagerBean.getResourceOperationSchedule(OperationManagerBean.java:420)
        at org.rhq.enterprise.server.operation.OperationManagerBean.getResourceOperationSchedule(OperationManagerBean.java:459)
        at org.rhq.enterprise.server.operation.OperationManagerBean.findCurrentlyScheduledResourceOperations(OperationManagerBean.java:1418)
        at sun.reflect.GeneratedMethodAccessor2175.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
        at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
        at org.rhq.enterprise.server.common.TransactionInterruptInterceptor.addCheckedActionToTransactionManager(TransactionInterruptInterceptor.java:77)
        at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
        at org.rhq.enterprise.server.authz.RequiredPermissionsInterceptor.checkRequiredPermissions(RequiredPermissionsInterceptor.java:153)
        at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
        at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
        at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:95)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
        at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:240)
        at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:210)
        at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:84)
        at $Proxy336.findCurrentlyScheduledResourceOperations(Unknown Source)
        at org.rhq.enterprise.gui.legacy.portlet.controlactions.ViewAction.execute(ViewAction.java:91)
        at org.apache.struts.tiles.actions.TilesAction.execute(TilesAction.java:73)
        at org.apache.struts.tiles.actions.TilesAction.execute(TilesAction.java:73)
        at org.rhq.enterprise.gui.legacy.action.BaseRequestProcessor.processActionPerform(BaseRequestProcessor.java:46)
        at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:236)
        at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
        at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:654)
        at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:557)
        at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:481)
        at org.apache.struts.tiles.UrlController.execute(UrlController.java:89)
        at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:875)
        at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:462)
        at org.apache.jsp.portal.DashboardLayout_jsp._jspx_meth_tiles_005finsert_005f0(DashboardLayout_jsp.java:1341)
        at org.apache.jsp.portal.DashboardLayout_jsp._jspx_meth_c_005fforEach_005f1(DashboardLayout_jsp.java:1183)
        at org.apache.jsp.portal.DashboardLayout_jsp._jspx_meth_c_005fforEach_005f0(DashboardLayout_jsp.java:751)
        at org.apache.jsp.portal.DashboardLayout_jsp._jspService(DashboardLayout_jsp.java:273)
        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:336)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:654)
        at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:557)
        at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:481)
        at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968)
        at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:609)
        at org.apache.struts.tiles.TilesUtilImpl.doInclude(TilesUtilImpl.java:99)
        at org.apache.struts.tiles.TilesUtil.doInclude(TilesUtil.java:135)
        at org.apache.struts.taglib.tiles.InsertTag.doInclude(InsertTag.java:760)
        at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:892)
        at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:462)
        at org.apache.jsp.portal.MainLayout_jsp._jspx_meth_tiles_005finsert_005f3(MainLayout_jsp.java:496)
        at org.apache.jsp.portal.MainLayout_jsp._jspx_meth_html_005fhtml_005f0(MainLayout_jsp.java:171)
        at org.apache.jsp.portal.MainLayout_jsp._jspService(MainLayout_jsp.java:82)
        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:336)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:654)
        at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:445)
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:654)
        at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:445)
        at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:379)
        at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:292)
        at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1085)
        at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:263)
        at org.apache.struts.tiles.TilesRequestProcessor.processTilesDefinition(TilesRequestProcessor.java:239)
        at org.apache.struts.tiles.TilesRequestProcessor.internalModuleRelativeForward(TilesRequestProcessor.java:341)
        at org.apache.struts.action.RequestProcessor.processForward(RequestProcessor.java:572)
        at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:221)
        at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
        at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.rhq.enterprise.gui.legacy.WebUserTrackingFilter.doFilter(WebUserTrackingFilter.java:47)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.rhq.enterprise.gui.legacy.AuthenticationFilter.doFilter(AuthenticationFilter.java:129)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:654)
        at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:557)
        at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:481)
        at org.apache.struts.tiles.UrlController.execute(UrlController.java:89)
        at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:875)
        at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:462)
        at org.apache.jsp.portal.DashboardLayout_jsp._jspx_meth_tiles_005finsert_005f0(DashboardLayout_jsp.java:1341)
        at org.apache.jsp.portal.DashboardLayout_jsp._jspx_meth_c_005fforEach_005f1(DashboardLayout_jsp.java:1183)
        at org.apache.jsp.portal.DashboardLayout_jsp._jspx_meth_c_005fforEach_005f0(DashboardLayout_jsp.java:751)
        at org.apache.jsp.portal.DashboardLayout_jsp._jspService(DashboardLayout_jsp.java:273)
        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:336)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:654)
        at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:557)
        at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:481)
        at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:557)
        at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:481)
        at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968)
        at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:609)
        at org.apache.struts.tiles.TilesUtilImpl.doInclude(TilesUtilImpl.java:99)
        at org.apache.struts.tiles.TilesUtil.doInclude(TilesUtil.java:135)
        at org.apache.struts.taglib.tiles.InsertTag.doInclude(InsertTag.java:760)
        at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:892)
        at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:462)
        at org.apache.jsp.portal.MainLayout_jsp._jspx_meth_tiles_005finsert_005f3(MainLayout_jsp.java:496)
        at org.apache.jsp.portal.MainLayout_jsp._jspx_meth_html_005fhtml_005f0(MainLayout_jsp.java:171)
        at org.apache.jsp.portal.MainLayout_jsp._jspService(MainLayout_jsp.java:82)
        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:336)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:654)
        at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:445)
        at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:379)
        at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:292)
        at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1085)
        at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:263)
        at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1085)
        at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:263)
        at org.apache.struts.tiles.TilesRequestProcessor.processTilesDefinition(TilesRequestProcessor.java:239)
        at org.apache.struts.tiles.TilesRequestProcessor.internalModuleRelativeForward(TilesRequestProcessor.java:341)
        at org.apache.struts.action.RequestProcessor.processForward(RequestProcessor.java:572)
        at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:221)
        at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
        at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.rhq.enterprise.gui.legacy.WebUserTrackingFilter.doFilter(WebUserTrackingFilter.java:47)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.rhq.enterprise.gui.legacy.AuthenticationFilter.doFilter(AuthenticationFilter.java:129)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:124)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
java:206)
        at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
        at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
        at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
        at java.lang.Thread.run(Thread.java:595)
2010-03-12 08:22:40,824 ERROR [org.rhq.enterprise.server.operation.OperationManagerBean] A scheduled operation has an invalid name - did a plugin change its operation metadata? : ResourceOperationScheduleComposite: operation-job-id=[rhq-resource-535012-1097506319-1245118436269_=_rhq-resource-535012], operation-name=[null], operation-next-fire-time=[Mon Jun 15 21:13:56 CDT 2009], resource-id=[535012], resource-name=[dummy-server-26 RHQ Agent], resource-type-name=[RHQ Agent]
java.lang.NullPointerException
        at org.rhq.enterprise.server.operation.OperationManagerBean.getResourceOperationSchedule(OperationManagerBean.java:420)
        at org.rhq.enterprise.server.operation.OperationManagerBean.getResourceOperationSchedule(OperationManagerBean.java:459)
        at org.rhq.enterprise.server.operation.OperationManagerBean.getResourceOperationSchedule(OperationManagerBean.java:420)
        at org.rhq.enterprise.server.operation.OperationManagerBean.getResourceOperationSchedule(OperationManagerBean.java:459)
        at org.rhq.enterprise.server.operation.OperationManagerBean.findCurrentlyScheduledResourceOperations(OperationManagerBean.java:1418)
        at sun.reflect.GeneratedMethodAccessor2175.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
        at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
        at org.rhq.enterprise.server.common.TransactionInterruptInterceptor.addCheckedActionToTransactionManager(TransactionInterruptInterceptor.java:77)
        at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
        at org.rhq.enterprise.server.authz.RequiredPermissionsInterceptor.checkRequiredPermissions(RequiredPermissionsInterceptor.java:153)
        at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
        at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
        at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
        at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)


Expected results:
No defunct operations shown and no exceptions thrown in the gui or server log.

Additional info:
defunct scheduled operation links from browser contained following additional information:

http://lxpmoa06:7080/rhq/resource/operation/resourceOperationScheduleDetails.xhtml?id=525179&jobId=rhq-resource-525179-1097506319-1245122699560_=_rhq-resource-525179

http://lxpmoa06:7080/rhq/resource/operation/resourceOperationScheduleDetails.xhtml?id=535012&jobId=rhq-resource-535012-1097506319-1245118436269_=_rhq-resource-535012

http://lxpmoa06:7080/rhq/resource/operation/resourceOperationScheduleDetails.xhtml?id=500196&jobId=rhq-resource-500196-1097506319-1245118195999_=_rhq-resource-500196

http://lxpmoa06:7080/rhq/resource/operation/resourceOperationScheduleDetails.xhtml?id=503201&jobId=rhq-resource-503201-1097506319-1245092639741_=_rhq-resource-503201

http://lxpmoa06:7080/rhq/resource/operation/resourceOperationScheduleDetails.xhtml?id=500196&jobId=rhq-resource-500196-1097506319-1245092629641_=_rhq-resource-500196
Comment 1 Simeon Pinder 2010-03-18 13:40:28 UTC 
The saved scheduled operations information is in an inconsistent state.  Defunct scheduled operations are queried and shown in the Dashboard but do not exist in the (selected resource)->Operations tab. 

Still investigating how the defunct data is being stored.

Short term suggestion is to disable the "Operations" tab completely or not to display scheduled operations from being displayed to prevent annoying defunct schedule messages in the sever log.
  
i) Use <server:port>/dashboard/Admin.do?mode=controlActions and set "Display on Dashboard" to No.  

OR

ii)Use <server:port>/dashboard/Admin.do?mode=controlActions and uncheck the "Next" checkbox from the "Operation Range".
Comment 2 Simeon Pinder 2010-03-18 20:07:52 UTC 
##### Snippet from selecting the defunct schedule entries from Dashboard.

An Error Occurred:
javax.el.ELException: /rhq/resource/operation/resourceOperationScheduleDetails.xhtml @40,74 value="#{item.operationDisplayName}": /rhq/resource/operation/resourceOperationScheduleDetails.xhtml @21,87 value="#{ResourceOperationScheduleDetailsUIBean.schedule}": Error reading 'schedule' on type org.rhq.enterprise.gui.operation.schedule.resource.ResourceOperationScheduleDetailsUIBean
+- Stack Trace

javax.faces.FacesException: javax.el.ELException: /rhq/resource/operation/resourceOperationScheduleDetails.xhtml @40,74 value="#{item.operationDisplayName}": /rhq/resource/operation/resourceOperationScheduleDetails.xhtml @21,87 value="#{ResourceOperationScheduleDetailsUIBean.schedule}": Error reading 'schedule' on type org.rhq.enterprise.gui.operation.schedule.resource.ResourceOperationScheduleDetailsUIBean
	at javax.faces.component.UIOutput.getValue(UIOutput.java:187)
	at com.sun.faces.renderkit.html_basic.HtmlBasicInputRenderer.getValue(HtmlBasicInputRenderer.java:201)
	at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.getCurrentValue(HtmlBasicRenderer.java:284)
	at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeEnd(HtmlBasicRenderer.java:154)
	at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:861)
	at org.ajax4jsf.renderkit.RendererBase.renderChild(RendererBase.java:286)
	at org.ajax4jsf.renderkit.RendererBase.renderChildren(RendererBase.java:262)
	at org.richfaces.renderkit.html.PanelRenderer.doEncodeChildren(PanelRenderer.java:220)
	at org.richfaces.renderkit.html.PanelRenderer.doEncodeChildren(PanelRenderer.java:215)
	at org.ajax4jsf.renderkit.RendererBase.encodeChildren(RendererBase.java:121)
	at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:837)
	at javax.faces.component.UIComponent.encodeAll(UIComponent.java:930)
	at javax.faces.render.Renderer.encodeChildren(Renderer.java:148)
	at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:837)
	at javax.faces.component.UIComponent.encodeAll(UIComponent.java:930)
	at javax.faces.component.UIComponent.encodeAll(UIComponent.java:933)
	at com.sun.facelets.FaceletViewHandler.renderView(FaceletViewHandler.java:592)
	at org.rhq.enterprise.gui.common.framework.FaceletRedirectionViewHandler.renderView(FaceletRedirectionViewHandler.java:64)
	at org.ajax4jsf.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:100)
	at org.ajax4jsf.application.AjaxViewHandler.renderView(AjaxViewHandler.java:176)
	at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:110)
	at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:100)
	at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:139)
	at javax.faces.webapp.FacesServlet.service(FacesServlet.java:266)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:532)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.rhq.enterprise.gui.legacy.WebUserTrackingFilter.doFilter(WebUserTrackingFilter.java:47)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.rhq.enterprise.gui.legacy.AuthenticationFilter.doFilter(AuthenticationFilter.java:129)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.rhq.enterprise.gui.common.upload.MultipartFilter.doFilter(MultipartFilter.java:63)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
	at org.jboss.seam.web.IdentityFilter.doFilter(IdentityFilter.java:38)
	at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
	at org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:90)
	at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
	at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
	at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
	at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
	at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
	at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:178)
	at org.ajax4jsf.webapp.BaseFilter.handleRequest(BaseFilter.java:290)
	at org.ajax4jsf.webapp.BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:390)
	at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:517)
	at org.jboss.seam.web.Ajax4jsfFilter.doFilter(Ajax4jsfFilter.java:56)
	at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
	at org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:58)
	at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
	at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:124)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
	at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
	at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
	at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
	at java.lang.Thread.run(Thread.java:595)
Caused by: javax.el.ELException: /rhq/resource/operation/resourceOperationScheduleDetails.xhtml @40,74 value="#{item.operationDisplayName}": /rhq/resource/operation/resourceOperationScheduleDetails.xhtml @21,87 value="#{ResourceOperationScheduleDetailsUIBean.schedule}": Error reading 'schedule' on type org.rhq.enterprise.gui.operation.schedule.resource.ResourceOperationScheduleDetailsUIBean
	at com.sun.facelets.el.TagValueExpression.getValue(TagValueExpression.java:76)
	at javax.faces.component.UIOutput.getValue(UIOutput.java:184)
	... 76 more
Caused by: java.lang.IllegalStateException: java.lang.NullPointerException
	at org.rhq.enterprise.gui.operation.schedule.OperationScheduleDetailsUIBean.init(OperationScheduleDetailsUIBean.java:54)
	at org.rhq.enterprise.gui.operation.schedule.OperationScheduleDetailsUIBean.getSchedule(OperationScheduleDetailsUIBean.java:77)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at javax.el.BeanELResolver.getValue(BeanELResolver.java:62)
	at javax.el.CompositeELResolver.getValue(CompositeELResolver.java:53)
	at com.sun.faces.el.FacesCompositeELResolver.getValue(FacesCompositeELResolver.java:72)
	at org.jboss.el.parser.AstPropertySuffix.getValue(AstPropertySuffix.java:53)
	at org.jboss.el.parser.AstValue.getValue(AstValue.java:67)
	at org.jboss.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:186)
	at com.sun.facelets.el.TagValueExpression.getValue(TagValueExpression.java:71)
	at org.jboss.el.parser.AstIdentifier.getValue(AstIdentifier.java:40)
	at org.jboss.el.parser.AstValue.getValue(AstValue.java:63)
	at org.jboss.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:186)
	at com.sun.facelets.el.TagValueExpression.getValue(TagValueExpression.java:71)
	... 77 more
Caused by: javax.ejb.EJBException: java.lang.NullPointerException
	at org.jboss.ejb3.tx.Ejb3TxPolicy.handleExceptionInOurTx(Ejb3TxPolicy.java:63)
	at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:83)
	at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:95)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
	at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:240)
	at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:210)
	at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:84)
	at $Proxy336.getResourceOperationSchedule(Unknown Source)
	at org.rhq.enterprise.gui.operation.schedule.resource.ResourceOperationScheduleDetailsUIBean.getOperationSchedule(ResourceOperationScheduleDetailsUIBean.java:29)
	at org.rhq.enterprise.gui.operation.schedule.OperationScheduleDetailsUIBean.init(OperationScheduleDetailsUIBean.java:51)
	... 93 more
Caused by: java.lang.NullPointerException
	at org.rhq.enterprise.server.operation.OperationManagerBean.getResourceOperationSchedule(OperationManagerBean.java:420)
	at org.rhq.enterprise.server.operation.OperationManagerBean.getResourceOperationSchedule(OperationManagerBean.java:459)
	at sun.reflect.GeneratedMethodAccessor1234.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
	at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
	at org.rhq.enterprise.server.common.TransactionInterruptInterceptor.addCheckedActionToTransactionManager(TransactionInterruptInterceptor.java:77)
	at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
	at org.rhq.enterprise.server.authz.RequiredPermissionsInterceptor.checkRequiredPermissions(RequiredPermissionsInterceptor.java:153)
	at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
	at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
Comment 3 Simeon Pinder 2010-03-24 20:33:31 UTC 
After further investigation, it has been determined that due to some series of events in the past, the database has been left in an inconsistent state such that scheduled operation job details were deleted from the database only without the accompanying RHQ table information being updated.

The simplest and fastest fix is to run the attached sql to reinsert temporary job details using the http://(jon host):7080/admin/test/sql.jsp form.  Immediately after reinserted the schedule relationship data, the Dashboard.do page should be refreshed and for each of the previously stale schedules, each one should be selected and unscheduled.  

Essentially you are rebuilding the scheduled operation details so that the JON UI can clean up the stale schedules appropriately.

The longer term fix will likely involve adding a new MeasurementScheduleManagerBean.errorCorrectOperationSchedules method for JON administrators to use to clean up such scheduling data if it were to arrive in an inconsistent state.

Additional:
deleting job details from a healthy scheduled operation results correctly reproduces the error log and ui messages that the customer is reporting.
Comment 4 Simeon Pinder 2010-03-24 20:34:34 UTC 
Created attachment 402410 [details]
sql to correct stale schedules.
Comment 5 Jay Shaughnessy 2010-03-25 18:09:57 UTC 
Upon further inspection I think the easiest resolution to the problem is to eliminate the problematic schedules via direct SQL.

The problem results when the underlying quartz job details for a scheduled operation are, for some reason, missing.

Quartz tables are defined such that an rhq_qrtz_job_details row outwardly exposes the job. If it is removed all other referring quartz entries are cascade deleted at the db level, due the FK definitions.  This means that if we're missing the job details record we can assume that there is basically no trace of the original quartz job in existence.  The only invalid entries are what we have in our own tables.  Given that that is the case we should be able to just remove any schedule that does not have an underlying quartz job_details record.  It should be necessary to 1) recreate the record and 2) unschedule the operation.  That should work but requires additional manual steps.

Instead, executing this SQL should do the trick:

DELETE FROM rhq_operation_schedule sched
WHERE NOT EXISTS
( SELECT * FROM rhq_qrtz_job_details job 
  WHERE job.job_name = sched.job_name
)

This statement should remove all of the invalid schedules.

p.s. I didn't look in depth but I'm not sure why we don't have a FK defined between rhq_operation_schedule and rhq_qrtz_job_details.  Perhaps it has to do with not tying our tables to quartz tables.
Comment 6 Jay Shaughnessy 2010-03-25 18:32:05 UTC 
Joe made the good point that the above SQL can be improved. It's not using the complete unique key. So, the enhanced SQL:

DELETE FROM rhq_operation_schedule sched
WHERE NOT EXISTS
( SELECT * FROM rhq_qrtz_job_details job
  WHERE job.job_name = sched.job_name AND job.job_group = sched.job_group
)


Use this as opposed the SQL in the previous comment.
Comment 7 Simeon Pinder 2010-04-09 21:05:32 UTC 
With sql attached that can be used to repair the defunct scheduled operations so that they can be manually removed via the GUI, this issue is being closed. 

The cause and more correct schedule clean up is being addressed in the following BZ issue:

https://bugzilla.redhat.com/show_bug.cgi?id=580175

No source code changes were made to deal with this issue so there is nothing to be QA'd or verified.
Comment 8 Simeon Pinder 2010-04-09 21:05:32 UTC 
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.

New Contents:
No source code changed. See https://bugzilla.redhat.com/show_bug.cgi?id=580175 for more details on how situation will be better avoided/rectified.