Not sure how to repeat, but this platform resource is a member of my group which I was testing group alerts on. The error took a while to start showing up, I think long after I moved on from testing group alerts (1 day?) Here's the stack: javax.el.ELException: /rhq/resource/summary/table/recentAlerts.xhtml @16,62 value="#{ResourceOverviewUIBean.recentAlerts}": Error reading 'recentAlerts' on type org.rhq.enterprise.gui.inventory.resource.ResourceOverviewUIBean javax.faces.el.EvaluationException: javax.el.ELException: /rhq/resource/summary/table/recentAlerts.xhtml @16,62 value="#{ResourceOverviewUIBean.recentAlerts}": Error reading 'recentAlerts' on type org.rhq.enterprise.gui.inventory.resource.ResourceOverviewUIBean at javax.faces.component.ValueBindingValueExpressionAdapter.getValue(ValueBindingValueExpressionAdapter.java:117) at com.sun.facelets.component.UIRepeat.getValue(UIRepeat.java:143) at com.sun.facelets.component.UIRepeat.getDataModel(UIRepeat.java:121) at com.sun.facelets.component.UIRepeat.setIndex(UIRepeat.java:305) at com.sun.facelets.component.UIRepeat.process(UIRepeat.java:333) at com.sun.facelets.component.UIRepeat.encodeChildren(UIRepeat.java:617) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:930) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:933) at com.sun.facelets.FaceletViewHandler.renderView(FaceletViewHandler.java:592) at org.rhq.enterprise.gui.common.framework.FaceletRedirectionViewHandler.renderView(FaceletRedirectionViewHandler.java:64) at org.ajax4jsf.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:100) at org.ajax4jsf.application.AjaxViewHandler.renderView(AjaxViewHandler.java:176) at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:110) at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:100) at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:139) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:266) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:532) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.rhq.enterprise.gui.legacy.WebUserTrackingFilter.doFilter(WebUserTrackingFilter.java:47) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.rhq.enterprise.gui.legacy.AuthenticationFilter.doFilter(AuthenticationFilter.java:129) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.rhq.enterprise.gui.common.upload.MultipartFilter.doFilter(MultipartFilter.java:63) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83) at org.jboss.seam.web.IdentityFilter.doFilter(IdentityFilter.java:38) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:90) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:178) at org.ajax4jsf.webapp.BaseFilter.handleRequest(BaseFilter.java:290) at org.ajax4jsf.webapp.BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:390) at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:517) at org.jboss.seam.web.Ajax4jsfFilter.doFilter(Ajax4jsfFilter.java:56) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:58) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:124) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446) at java.lang.Thread.run(Thread.java:619) Caused by: javax.el.ELException: /rhq/resource/summary/table/recentAlerts.xhtml @16,62 value="#{ResourceOverviewUIBean.recentAlerts}": Error reading 'recentAlerts' on type org.rhq.enterprise.gui.inventory.resource.ResourceOverviewUIBean at com.sun.facelets.el.TagValueExpression.getValue(TagValueExpression.java:76) at javax.faces.component.ValueBindingValueExpressionAdapter.getValue(ValueBindingValueExpressionAdapter.java:113) ... 68 more Caused by: javax.ejb.EJBException: javax.persistence.EntityNotFoundException: Unable to find org.rhq.core.domain.alert.AlertDefinition with id 10017 at org.jboss.ejb3.tx.Ejb3TxPolicy.handleExceptionInOurTx(Ejb3TxPolicy.java:63) at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:83) at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:95) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77) at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:240) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:210) at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:84) at $Proxy298.findAlerts(Unknown Source) at org.rhq.enterprise.gui.inventory.resource.ResourceOverviewUIBean.getAlerts(ResourceOverviewUIBean.java:41) at org.rhq.enterprise.gui.inventory.resource.ResourceOverviewUIBean.getRecentAlerts(ResourceOverviewUIBean.java:88) at sun.reflect.GeneratedMethodAccessor881.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at javax.el.BeanELResolver.getValue(BeanELResolver.java:62) at javax.el.CompositeELResolver.getValue(CompositeELResolver.java:53) at com.sun.faces.el.FacesCompositeELResolver.getValue(FacesCompositeELResolver.java:72) at org.jboss.el.parser.AstPropertySuffix.getValue(AstPropertySuffix.java:53) at org.jboss.el.parser.AstValue.getValue(AstValue.java:67) at org.jboss.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:186) at com.sun.facelets.el.TagValueExpression.getValue(TagValueExpression.java:71) ... 69 more Caused by: javax.persistence.EntityNotFoundException: Unable to find org.rhq.core.domain.alert.AlertDefinition with id 10017 at org.hibernate.ejb.Ejb3Configuration$Ejb3EntityNotFoundDelegate.handleEntityNotFound(Ejb3Configuration.java:107) at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:145) at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:195) at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:103) at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:878) at org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:846) at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:557) at org.hibernate.type.EntityType.resolve(EntityType.java:379) at org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:120) at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:854) at org.hibernate.loader.Loader.doQuery(Loader.java:729) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236) at org.hibernate.loader.Loader.loadEntity(Loader.java:1860) at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:48) at org.hibernate.loader.entity.AbstractEntityLoader.load(AbstractEntityLoader.java:42) at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:3049) at org.hibernate.event.def.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:399) at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:375) at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:139) at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:195) at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:103) at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:878) at org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:846) at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:557) at org.hibernate.type.EntityType.resolve(EntityType.java:379) at org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:120) at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:854) at org.hibernate.loader.Loader.doQuery(Loader.java:729) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236) at org.hibernate.loader.Loader.doList(Loader.java:2213) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2104) at org.hibernate.loader.Loader.list(Loader.java:2099) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:378) at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:338) at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:172) at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1121) at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79) at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:64) at org.rhq.enterprise.server.alert.AlertManagerBean.findAlerts(AlertManagerBean.java:452) at sun.reflect.GeneratedMethodAccessor1617.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166) at org.rhq.enterprise.server.common.TransactionInterruptInterceptor.addCheckedActionToTransactionManager(TransactionInterruptInterceptor.java:77) at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118) at org.rhq.enterprise.server.authz.RequiredPermissionsInterceptor.checkRequiredPermissions(RequiredPermissionsInterceptor.java:153) at sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118) at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79) ... 98 more
This bug was previously known as http://jira.rhq-project.org/browse/RHQ-2361
oops, accidental need-info by me.
Suggested next steps to try... (9:16:36 PM) ccrouch: (6:19:35 PM) joseph: I might ask the customer for more info about when this occurred. and then get them to execute and upload the results of a few native sql statements, so we can see if there's any issue with the relationships (9:16:36 PM) ccrouch: what sql statements might those be? (9:17:07 PM) joseph: the sql that gets the alert records that would have been displayed for that ui page (9:17:17 PM) joseph: and then the sql that retrieves the alert defs for those alerts (9:17:46 PM) joseph: the exception seems to indicate that the alert definition did not exist for so-and-so alert, but that's tough to swallow since alert->alertDef is a FK relationship (9:18:06 PM) joseph: so this is not easy to diagnose remotely
Issue seems to have been introduced by use of group alert definitions being added and removed. The exact steps are unknown as the user added and removed group definitions many times in attempt to get OR and NOT criteria to work for EAP group alerts.
Larry, thanks for the additional information. I think I know what's going on here. An alert definition is made up of several alert conditions. When you edit/update an alert definition, we remove all current alert conditions and add a brand new set of conditions to reflect the change. This has to do with what data is persisted to the database. On the other side of the world, we have the in-memory data - namely the alert condition cache. In older versions of the product, this cache used to reload synchronously with respect to editing alert definitions. However, with the addition of the high availability infrastructure (the ability to run multiple RHQ Servers to handle the incoming agent load and provide failover/redundancy), this was no longer possible because the cache was partitioned across the RHQ Servers. So if you edit an alert definition on RHQ Server 1, it may have to reload the alert condition cache on RHQ Server 2. As a result, the reloading mechanism was made asynchronous with respect to updating an alert definition. The cache should only be at most 30 seconds behind the updates, but there is still that small period of staleness. ----- Given that, what I believe is happening here is that data is coming across the wire from the agent and matching one of the stale alert conditions. This data *should* be discarded because it now refers to non-existent alert conditions, but there's a possibility of a small timing bug in that logic, which then allows alerts to be created against orphaned alert conditions (those which no longer refer to a real alert definition). The investigation should start by looking at the logic that sends data to and sinks data from AlertConditionConsumerBean.
Temporarily adding the keyword "SubBug" so we can be sure we have accounted for all the bugs. keyword: new = Tracking + FutureFeature + SubBug
making sure we're not missing any bugs in rhq_triage
moving any remaining Alert related bugs to rhq_chainsaw
This bug has now been triaged by Chainsaw on 2/18. The expectation is the bug to be addressed by the end of sprint06 roughly 3/10/10.
BZ-535692: improve short-cutting logic for alert condition processing * if an alert definition has been deleted, looking up by alert condition id should return no results * this will prevent downstream out-of-band alert processing from throwing exceptions later
Per dev, there does not appear to be/have been a reliable way to replicate this. Dev has put in a fix for the timing issues thought to be the root cause. It amounts to a code-change. We can reopen if it shows up again.
Mass-closure of verified bugs against JON.