Bug 1071330 - Server fails to complete operations with concurrent access timeout on org.jboss.invocation.InterceptorContext$Invocation - could not obtain lock within 5000MILLISECONDS
Summary: Server fails to complete operations with concurrent access timeout on org.jbo...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: JBoss Operations Network
Classification: JBoss
Component: Core Server
Version: JON 3.2
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: CR01
: JON 3.2.2
Assignee: Lukas Krejci
QA Contact: Mike Foley
URL:
Whiteboard:
: 1038605 1077755 1083497 1084481 (view as bug list)
Depends On:
Blocks: 1123389
TreeView+ depends on / blocked
 
Reported: 2014-02-28 13:58 UTC by Radim Hatlapatka
Modified: 2018-12-06 15:58 UTC (History)
12 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2014-07-29 00:17:25 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
jon server log (245.55 KB, application/x-zip-compressed)
2014-02-28 13:58 UTC, Radim Hatlapatka
no flags Details
UI screenshot with warning (157.29 KB, image/png)
2014-02-28 14:15 UTC, Radim Hatlapatka
no flags Details
JON 3.2.0 update 02 server log and screenshot (1022.98 KB, application/zip)
2014-07-17 08:07 UTC, Radim Hatlapatka
no flags Details
JON server version - about screenshot (117.90 KB, image/png)
2014-07-17 10:22 UTC, Radim Hatlapatka
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1038605 0 unspecified CLOSED Tx timeout on SystemManager.getObfuscatedSettings 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1123389 0 unspecified CLOSED JON server becomes unresponsive during hourly maintainance if it is under load 2021-02-22 00:41:40 UTC
Red Hat Knowledge Base (Solution) 876403 0 None None None Never

Internal Links: 1038605 1123389

Description Radim Hatlapatka 2014-02-28 13:58:54 UTC
Created attachment 869083 [details]
jon server log

Description of problem:
When I am running tests on about 14 agents (each managing one EAP 6 in standalone mode and one in domain mode), after some time it stops being able to complete requests in time and this error occurs in JON server log:

11:00:25,996 ERROR [org.jboss.as.ejb3.invocation] (http-/0.0.0.0:7080-51) JBAS014134: EJB Invocation failed on component SystemManagerBean for method public abstract org.rhq.core.domain.common.composite.SystemSettings org.rhq.enterpr      ise.server.system.SystemManagerLocal.getUnmaskedSystemSettings(boolean): javax.ejb.ConcurrentAccessTimeoutException: JBAS014373: EJB 3.1 PFD2 4.8.5.5.1 concurrent access timeout on org.jboss.invocation.InterceptorContext$Invocation@6      a3d8d62 - could not obtain lock within 5000MILLISECONDS
30603         at org.jboss.as.ejb3.concurrency.ContainerManagedConcurrencyInterceptor.processInvocation(ContainerManagedConcurrencyInterceptor.java:100) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
30604         at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
30605         at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
30606         at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
30607         at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
30608         at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
30609         at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
30610         at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat      -10]
30611         at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
30612         at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:248) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
30613         at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:315) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
30614         at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:214) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
30615         at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
30616         at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
30617         at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
30618         at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
30619         at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
30620         at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
30621         at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
30622         at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
30623         at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
30624         at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
30625         at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
30626         at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
30627         at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
30628         at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
30629         at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
30630         at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
30631         at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
30632         at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
30633         at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
30634         at org.rhq.enterprise.server.system.SystemManagerLocal$$$view164.getUnmaskedSystemSettings(Unknown Source) [rhq-server.jar:4.9.0.JON320GA]
30635         at org.rhq.coregui.server.gwt.SystemGWTServiceImpl.getSessionTimeout(SystemGWTServiceImpl.java:77)
30636         at sun.reflect.GeneratedMethodAccessor508.invoke(Unknown Source) [:1.6.0_45]
30637         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
30638         at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
30639         at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:561)
30640         at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
30641         at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
30642         at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
30643         at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1]
30644         at org.rhq.coregui.server.gwt.AbstractGWTServiceImpl.service(AbstractGWTServiceImpl.java:88)
30645         at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1]
30646         at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown Source) [:1.6.0_45]
30647         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
30648         at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
30649         at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:263)
30650         at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:261)
30651         at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.6.0_45]
30652         at javax.security.auth.Subject.doAsPrivileged(Subject.java:517) [rt.jar:1.6.0_45]
30653         at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:295)
30654         at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:155)
30655         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:288)
30656         at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:59)
30657         at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:197)
30658         at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.6.0_45]
30659         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
30660         at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:125) [rhq-rtfilter-4.9.0.JON320GA.jar:4.9.0.JON320GA]
30661         at sun.reflect.GeneratedMethodAccessor324.invoke(Unknown Source) [:1.6.0_45]
30662         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
30663         at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
30664         at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:263)
30665         at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:261)
30666         at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.6.0_45]
30667         at javax.security.auth.Subject.doAsPrivileged(Subject.java:517) [rt.jar:1.6.0_45]
30668         at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:295)
30669         at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:236)
30670         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
30671         at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:59)
30672         at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:197)
30673         at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.6.0_45]
30674         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
30675         at org.rhq.coregui.server.filter.CacheControlFilter.doFilter(CacheControlFilter.java:70)
30676         at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source) [:1.6.0_45]
30677         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
30678         at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
30679         at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:263)
30680         at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:261)
30681         at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.6.0_45]
30682         at javax.security.auth.Subject.doAsPrivileged(Subject.java:517) [rt.jar:1.6.0_45]
30683         at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:295)
30684         at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:236)
30685         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
30686         at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:59)
30687         at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:197)
30688         at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.6.0_45]
30689         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
30690         at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
30691         at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
30692         at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
30693         at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
30694         at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
30695         at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145)
30696         at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
30697         at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
30698         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336)
30699         at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
30700         at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653)
30701         at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920)
30702         at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]


Version-Release number of selected component (if applicable): JON 3.2.0.GA


How reproducible: for me always


Steps to Reproduce:
1. producing greater stress on JON server by managing multiple instances should lead to this error

Actual results:
Warning in JON UI is shown and request is not completed

Expected results:
request is successfully completed


Additional info:
I haven't encountered this behavior with JON 3.2.0.CR1 before even though I was using the same setup (even used more agents in parallel

I am also attaching JON server full log

JON server machine attributes:

cpuinfo:
cat /proc/cpuinfo 
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 13
model name	: QEMU Virtual CPU version (cpu64-rhel6)
stepping	: 3
cpu MHz		: 2400.084
cache size	: 4096 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 4
wp		: yes
flags		: fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm up unfair_spinlock pni cx16 hypervisor lahf_lm
bogomips	: 4800.16
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


memory info:
[hudson@dhcp180-190 logs]$ free
             total       used       free     shared    buffers     cached
Mem:       4957136    4826992     130144          0     159252    1063560
-/+ buffers/cache:    3604180    1352956
Swap:      5095416      85848    5009568


running processes:
[hudson@dhcp180-190 logs]$ ps x
  PID TTY      STAT   TIME COMMAND
 1132 ?        S      0:04 /qa/hudson_workspace/workspace/jon-32-server-install/dhcp180-190.mw.lab.eng.bos.redhat.com/pgsql/bin/postgres -D /home/hudson/hudson_workspace/workspace/jon-32-server-install/dhcp180-190.mw.lab.eng.bos.redhat.com
 1135 ?        Ss     1:10 postgres: writer process                                                                                                                                                                                           
 1136 ?        Ss     0:46 postgres: wal writer process                                                                                                                                                                                       
 1137 ?        Ss     0:13 postgres: autovacuum launcher process                                                                                                                                                                              
 1138 ?        Ss     0:38 postgres: stats collector process                                                                                                                                                                                  
 1823 ?        Ss     3:22 postgres: rhqadmin rhq 127.0.0.1(44557) idle                                                                                                                                                                       
 1844 ?        Ss     3:04 postgres: rhqadmin rhq 127.0.0.1(44560) idle                                                                                                                                                                       
 2187 ?        Ss     3:14 postgres: rhqadmin rhq 127.0.0.1(44580) idle                                                                                                                                                                       
 2647 ?        Sl    25:20 /qa/tools/opt/amd64/jdk1.6.0_last/bin/java -ea -javaagent:./../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms512M -Xmx512M -Xmn128M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:+UseP
 2822 ?        S      0:00 /bin/sh /qa/hudson_workspace/workspace/jon-32-server-install/dhcp180-190.mw.lab.eng.bos.redhat.com/jon-server-3.2.0.GA/jbossas/bin/standalone.sh -P /qa/hudson_workspace/workspace/jon-32-server-install/dhcp180-190
 2920 ?        Sl   334:01 /qa/tools/opt/amd64/jdk1.6.0_last/bin/java -D[Standalone] -XX:+UseCompressedOops -Dapp.name=rhq-server -Xms1024M -Xmx1024M -XX:PermSize=256M -XX:MaxPermSize=256M -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolv
 3186 ?        Ss     3:29 postgres: rhqadmin rhq 127.0.0.1(39115) idle                                                                                                                                                                       
 3858 ?        Ss     0:01 postgres: rhqadmin rhq 127.0.0.1(39122) idle                                                                                                                                                                       
 3963 ?        Ss     3:21 postgres: rhqadmin rhq 127.0.0.1(39123) idle                                                                                                                                                                       
 4255 ?        Ss     0:37 postgres: rhqadmin rhq 127.0.0.1(46865) idle                                                                                                                                                                       
 4256 ?        Ss     0:46 postgres: rhqadmin rhq 127.0.0.1(46866) idle                                                                                                                                                                       
 4257 ?        Ss     0:35 postgres: rhqadmin rhq 127.0.0.1(46867) idle                                                                                                                                                                       
 4258 ?        Ss     0:31 postgres: rhqadmin rhq 127.0.0.1(46868) idle                                                                                                                                                                       
 4264 ?        Ss     0:39 postgres: rhqadmin rhq 127.0.0.1(46870) idle                                                                                                                                                                       
 4510 ?        Ss     0:42 postgres: rhqadmin rhq 127.0.0.1(39124) idle                                                                                                                                                                       
 4516 ?        Ss     0:42 postgres: rhqadmin rhq 127.0.0.1(39125) idle                                                                                                                                                                       
 4574 ?        S      0:00 /bin/sh ./rhq-agent.sh --input=/home/hudson/hudson_workspace/workspace/jon-32-server-install/dhcp180-190.mw.lab.eng.bos.redhat.com/rhq-agent/bin/agentSetting.txt --nonative --pref=dhcp180-190.mw.lab.eng.bos.redha
 4585 ?        Sl    11:06 /qa/tools/opt/amd64/jdk1.6.0_last/bin/java -Djava.endorsed.dirs=/qa/hudson_workspace/workspace/jon-32-server-install/dhcp180-190.mw.lab.eng.bos.redhat.com/rhq-agent/lib/endorsed -Djava.library.path=/qa/hudson_wor
 4640 ?        Ss     0:42 postgres: rhqadmin rhq 127.0.0.1(39131) idle                                                                                                                                                                       
 4674 ?        Ss     3:01 postgres: rhqadmin rhq 127.0.0.1(39137) idle                                                                                                                                                                       
 4687 ?        Ss     3:08 postgres: rhqadmin rhq 127.0.0.1(39146) idle                                                                                                                                                                       
 4898 ?        Ss     2:53 postgres: rhqadmin rhq 127.0.0.1(44759) idle                                                                                                                                                                       
 4899 ?        Ss     2:56 postgres: rhqadmin rhq 127.0.0.1(44760) idle                                                                                                                                                                       
 4903 ?        Ss     3:03 postgres: rhqadmin rhq 127.0.0.1(44762) idle                                                                                                                                                                       
 4904 ?        Ss     2:57 postgres: rhqadmin rhq 127.0.0.1(44763) idle                                                                                                                                                                       
 5126 ?        Ss     3:09 postgres: rhqadmin rhq 127.0.0.1(39215) idle                                                                                                                                                                       
 5127 ?        Ss     3:27 postgres: rhqadmin rhq 127.0.0.1(39216) idle                                                                                                                                                                       
 5310 ?        Ss     2:44 postgres: rhqadmin rhq 127.0.0.1(44806) idle                                                                                                                                                                       
 5478 ?        Ss     2:48 postgres: rhqadmin rhq 127.0.0.1(44822) idle                                                                                                                                                                       
 5479 ?        Ss     3:09 postgres: rhqadmin rhq 127.0.0.1(44823) idle                                                                                                                                                                       
 5480 ?        Ss     3:06 postgres: rhqadmin rhq 127.0.0.1(44824) idle                                                                                                                                                                       
 5481 ?        Ss     2:54 postgres: rhqadmin rhq 127.0.0.1(44825) idle                                                                                                                                                                       
 5482 ?        Ss     3:05 postgres: rhqadmin rhq 127.0.0.1(44826) idle                                                                                                                                                                       
 6601 ?        Ssl    7:20 /qa/tools/opt/amd64/jdk1.6.0_last/bin/java -Djava.net.preferIPv4Stack=true -Djava.home=/qa/tools/opt/amd64/jdk1.6.0_last/jre -Xmx700m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -jar /qa/services/hudson
 8766 ?        Ss     0:41 postgres: rhqadmin rhq 127.0.0.1(39512) idle                                                                                                                                                                       
 9194 pts/2    S      0:00 -bash
11637 ?        Ss     2:42 postgres: rhqadmin rhq 127.0.0.1(45146) idle                                                                                                                                                                       
11638 ?        Ss     2:32 postgres: rhqadmin rhq 127.0.0.1(45147) idle                                                                                                                                                                       
11639 ?        Ss     2:24 postgres: rhqadmin rhq 127.0.0.1(45148) idle                                                                                                                                                                       
11640 ?        Ss     2:40 postgres: rhqadmin rhq 127.0.0.1(45149) idle                                                                                                                                                                       
11641 ?        Ss     2:27 postgres: rhqadmin rhq 127.0.0.1(45150) idle                                                                                                                                                                       
11642 ?        Ss     2:16 postgres: rhqadmin rhq 127.0.0.1(45151) idle                                                                                                                                                                       
11643 ?        Ss     2:16 postgres: rhqadmin rhq 127.0.0.1(45152) idle                                                                                                                                                                       
11644 ?        Ss     2:39 postgres: rhqadmin rhq 127.0.0.1(45154) idle                                                                                                                                                                       
11645 ?        Ss     2:22 postgres: rhqadmin rhq 127.0.0.1(45155) idle                                                                                                                                                                       
11646 ?        Ss     2:24 postgres: rhqadmin rhq 127.0.0.1(45156) idle                                                                                                                                                                       
13481 ?        Ss     2:33 postgres: rhqadmin rhq 127.0.0.1(45286) idle                                                                                                                                                                       
13485 ?        Ss     2:05 postgres: rhqadmin rhq 127.0.0.1(45287) idle                                                                                                                                                                       
13486 ?        Ss     2:16 postgres: rhqadmin rhq 127.0.0.1(45288) idle                                                                                                                                                                       
13487 ?        Ss     2:00 postgres: rhqadmin rhq 127.0.0.1(45289) idle                                                                                                                                                                       
13492 ?        Ss     2:18 postgres: rhqadmin rhq 127.0.0.1(45290) idle                                                                                                                                                                       
13493 ?        Ss     2:15 postgres: rhqadmin rhq 127.0.0.1(45291) idle                                                                                                                                                                       
13494 ?        Ss     2:07 postgres: rhqadmin rhq 127.0.0.1(45292) idle                                                                                                                                                                       
13495 ?        Ss     1:59 postgres: rhqadmin rhq 127.0.0.1(45294) idle                                                                                                                                                                       
13496 ?        Ss     2:09 postgres: rhqadmin rhq 127.0.0.1(45295) idle                                                                                                                                                                       
14022 pts/2    R+     0:00 ps x
23770 ?        Ss     1:07 postgres: rhqadmin rhq 127.0.0.1(46066) idle                                                                                                                                                                       
23917 ?        Ss     1:23 postgres: rhqadmin rhq 127.0.0.1(46082) idle

Comment 1 Radim Hatlapatka 2014-02-28 14:15:25 UTC
Created attachment 869089 [details]
UI screenshot with warning

Comment 3 Larry O'Leary 2014-05-13 00:11:13 UTC
From the looks of it, this occurs when hourly or daily maintenance is running. In all the instances I see it appears to deal with reading or deobfuscating system properties perhaps related to the storage node(s).

Comment 4 Simeon Pinder 2014-06-13 14:39:08 UTC
Moving to DR03 as didn't get included for earlier payload.

Comment 5 Jay Shaughnessy 2014-06-19 14:17:03 UTC
*** Bug 1038605 has been marked as a duplicate of this bug. ***

Comment 6 Jay Shaughnessy 2014-06-19 14:27:54 UTC
*** Bug 1084481 has been marked as a duplicate of this bug. ***

Comment 7 Jay Shaughnessy 2014-06-19 14:40:41 UTC
This is a duplicate of Bug 1083497, which has a lot of analysis regarding this exact problem.  I'm not sure why we didn't continue with the fix there.  I recommend reading through the comments in that BZ.  I am marking it a duplicate of this bug now, because this is seeing some work be done, but there is good information there.

Comment 8 Jay Shaughnessy 2014-06-19 14:42:15 UTC
*** Bug 1083497 has been marked as a duplicate of this bug. ***

Comment 9 Jay Shaughnessy 2014-06-19 14:56:03 UTC
*** Bug 1077755 has been marked as a duplicate of this bug. ***

Comment 10 Lukas Krejci 2014-06-30 22:59:47 UTC
This is now in master

commit 4610b91f945e9503fc6291d82a9d50973a840f22
Merge: f0a5cc7 5df5508
Author: Lukas Krejci <lkrejci>
Date:   Mon Jun 30 23:21:23 2014 +0200

    Merged pull request #57 from metlos/bug/1071330
    
     [BZ 1071330] Try to avoid timeouts on retrieving system settings

Comment 11 Lukas Krejci 2014-07-01 14:22:27 UTC
commit b9fe054fc5bf8cef491b82339226096d877013ea
Author: Lukas Krejci <lkrejci>
Date:   Tue Jul 1 16:20:43 2014 +0200

    [BZ 1071330] No NPE in SystemManagerBean when no last update time in DB.

Comment 12 Libor Zoubek 2014-07-02 12:14:39 UTC
commit ff05c55654115db59cece608bd5cf635353fa928
Author: Lukas Krejci <lkrejci>
Date:   Tue Jun 17 21:11:03 2014 +0200

    [BZ 1071330] Minimize the duration of the locks
    
    (cherry picked from commit 5df5508ac3415ec1d9b2eeb45c1b67b396673fd6)
    Signed-off-by: Libor Zoubek <lzoubek>

commit 913e4e9a5e675294b43d57b26584579c11493926
in 3.2 branch 

Author: Lukas Krejci <lkrejci>
Date:   Tue Jun 17 21:01:41 2014 +0200

    [BZ 1071330] Try to avoid timeouts on retrieving system settings
    
    This is a bit of a shot in the dark to reduce the possibility of
    timeouts when reading system settings in the server under load.
    
    The concurrency on the bean has been changed from the default
    container managed with a write lock on each method to bean managed
    with a single critical section in the cache loading and writing methods.
    
    Also, the deobfuscation of the password fields happens only on the
    cache reload and only if the settings were changed.
    
    All this to minimize the contention on the SystemManagerBean and to
    speed up the retrieval of the settings as much as possible.
    
    (cherry picked from commit 2d896974f527317fe418b88fcab59c3e19d1230d)
    Signed-off-by: Libor Zoubek <lzoubek>
    
    Conflicts:
        modules/core/domain/src/main/java/org/rhq/core/domain/common/composite/SystemSetting.java

Comment 13 Libor Zoubek 2014-07-02 13:16:56 UTC
also in product branch

commit 2a6ad0d556d49e69a6728dd6fc6d9b982160c93e
Author: Lukas Krejci <lkrejci>
Date:   Tue Jul 1 16:20:43 2014 +0200

    [BZ 1071330] No NPE in SystemManagerBean when no last update time in DB.
    
    (cherry picked from commit b9fe054fc5bf8cef491b82339226096d877013ea)
    Signed-off-by: Libor Zoubek <lzoubek>

Comment 14 Simeon Pinder 2014-07-11 13:43:05 UTC
Moving to ON_QA as available for test with build:
http://jon01.mw.lab.eng.bos.redhat.com:8042/dist/release/jon/3.2.2.GA/7-11-14_0600/

Comment 16 Radim Hatlapatka 2014-07-17 08:05:12 UTC
I am still seeing the error every hour as you can see from log, I am attaching.

08:00:54,255 ERROR [org.jboss.as.ejb3.invocation] (http-/0.0.0.0:7080-118) JBAS014134: EJB Invocation failed on component SystemManagerBean for method public abstract org.rhq.core.domain.common.ProductInfo org.rhq.enterprise.server.system.SystemManagerRemote.getProductInfo(org.rhq.core.domain.auth.Subject): javax.ejb.ConcurrentAccessTimeoutException: JBAS014373: EJB 3.1 PFD2 4.8.5.5.1 concurrent access timeout on org.jboss.invocation.InterceptorContext$Invocation@71718165 - could not obtain lock within 5000MILLISECONDS
	at org.jboss.as.ejb3.concurrency.ContainerManagedConcurrencyInterceptor.processInvocation(ContainerManagedConcurrencyInterceptor.java:100) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:248) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:315) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:214) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
	at org.rhq.enterprise.server.system.SystemManagerLocal$$$view165.getProductInfo(Unknown Source) [rhq-server.jar:4.9.0.JON320GA-redhat-2]
	at org.rhq.coregui.server.gwt.SystemGWTServiceImpl.getProductInfo(SystemGWTServiceImpl.java:59)
	at sun.reflect.GeneratedMethodAccessor506.invoke(Unknown Source) [:1.6.0_45]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
	at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
	at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:561)
	at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
	at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
	at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1]
	at org.rhq.coregui.server.gwt.AbstractGWTServiceImpl.service(AbstractGWTServiceImpl.java:88)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1]
	at sun.reflect.GeneratedMethodAccessor160.invoke(Unknown Source) [:1.6.0_45]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
	at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:263)
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:261)
	at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.6.0_45]
	at javax.security.auth.Subject.doAsPrivileged(Subject.java:517) [rt.jar:1.6.0_45]
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:295)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:155)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:288)
	at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:59)
	at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:197)
	at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.6.0_45]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
	at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:125) [rhq-rtfilter-4.9.0.JON320GA.jar:4.9.0.JON320GA]
	at sun.reflect.GeneratedMethodAccessor197.invoke(Unknown Source) [:1.6.0_45]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
	at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:263)
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:261)
	at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.6.0_45]
	at javax.security.auth.Subject.doAsPrivileged(Subject.java:517) [rt.jar:1.6.0_45]
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:295)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:236)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:59)
	at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:197)
	at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.6.0_45]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
	at org.rhq.coregui.server.filter.CacheControlFilter.doFilter(CacheControlFilter.java:70)
	at sun.reflect.GeneratedMethodAccessor198.invoke(Unknown Source) [:1.6.0_45]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
	at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:263)
	at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:261)
	at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.6.0_45]
	at javax.security.auth.Subject.doAsPrivileged(Subject.java:517) [rt.jar:1.6.0_45]
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:295)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:236)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:59)
	at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:197)
	at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.6.0_45]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
	at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
	at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920)
	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]

Comment 17 Radim Hatlapatka 2014-07-17 08:07:06 UTC
Created attachment 918634 [details]
JON 3.2.0 update 02 server log and screenshot

Comment 18 Lukas Krejci 2014-07-17 10:01:22 UTC
I am strongly convinced your server isn't of the latest version.

The stacktrace you provided contains this stack element:
org.jboss.as.ejb3.concurrency.ContainerManagedConcurrencyInterceptor.processInvocation(ContainerManagedConcurrencyInterceptor.java:100) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]

but the SystemManagerBean has been converted to use bean managed concurrency. This leads me to believe your server must be running old version of the code.

Either that or the latest build somehow doesn't contain the fix. Could you please provide the build numbers of your server? Go to Help -> Product -> About.

Comment 19 Radim Hatlapatka 2014-07-17 10:22:08 UTC
Created attachment 918660 [details]
JON server version - about screenshot

Version : 3.2.0.GA Update 02, see attachment

Comment 21 Lukas Krejci 2014-07-17 10:43:32 UTC
K, the build number 2fd33e4:ced7c70 tells me you're on 3.2.0 update 02 DR4 while this bug is supposed to be fixed in CR01.

Comment 23 Radim Hatlapatka 2014-07-18 11:26:08 UTC
I am no longer seeing the error, verified with JON 3.2.2 CR1
Version 3.2.0.GA Update 02
Build Number : dfe3e4b:cf4474c

Comment 24 Radim Hatlapatka 2014-07-25 12:53:11 UTC
Actually I have find out that the state only improved a little bit. The issue is still valid, (during hourly maintenance server can stop responding for a short while) just the exception changed:

08:00:00,681 INFO  [org.rhq.enterprise.server.scheduler.jobs.DataPurgeJob] (RHQScheduler_Worker-2) Performing hourly database maintenance
08:00:15,477 WARN  [org.rhq.enterprise.communications.command.server.CommandProcessor] (http-/0.0.0.0:7080-17) {CommandProcessor.failed-authentication}Command failed to be authenticated!  This command will be ignored and not processed: Command: type=[remotepojo]; cmd-in-response=[false]; config=[{rhq.agent-name=dhcp180-200.mw.lab.eng.bos.redhat.com-agent-16163, rhq.externalizable-strategy=AGENT, rhq.send-throttle=true, rhq.guaranteed-delivery=true}]; params=[{invocation=NameBasedInvocation[mergeMeasurementReport], targetInterfaceName=org.rhq.core.clientapi.server.measurement.MeasurementServerService}]
08:00:17,909 INFO  [org.rhq.enterprise.server.scheduler.jobs.DataPurgeJob] (RHQScheduler_Worker-2) Database maintenance completed in [17228]ms
08:00:17,909 INFO  [org.rhq.enterprise.server.scheduler.jobs.DataPurgeJob] (RHQScheduler_Worker-2) Auto-calculation of baselines starting at Mon Jul 21 08:00:17 EDT 2014
08:00:17,918 INFO  [org.rhq.enterprise.server.measurement.MeasurementBaselineManagerBean] (RHQScheduler_Worker-2) Calculating auto baselines
08:00:17,918 INFO  [org.rhq.enterprise.server.measurement.MeasurementBaselineManagerBean] (RHQScheduler_Worker-2) Deleting baselines computations older than Fri Jul 18 08:00:17 EDT 2014
08:00:17,918 INFO  [org.rhq.enterprise.server.measurement.MeasurementBaselineManagerBean] (RHQScheduler_Worker-2) Inserting new baselines using last 7 days of 1H data
08:00:17,927 INFO  [org.rhq.enterprise.server.measurement.MeasurementBaselineManagerBean] (RHQScheduler_Worker-2) Removed [0] old baselines - they will now be recalculated (8)ms
08:00:30,252 WARN  [org.rhq.enterprise.communications.command.server.CommandProcessor] (http-/0.0.0.0:7080-73) {CommandProcessor.failed-authentication}Command failed to be authenticated!  This command will be ignored and not processed: Command: type=[remotepojo]; cmd-in-response=[false]; config=[{rhq.agent-name=dhcp180-200.mw.lab.eng.bos.redhat.com-agent-16163, rhq.externalizable-strategy=AGENT, rhq.send-throttle=true}]; params=[{invocation=NameBasedInvocation[ping], targetInterfaceName=org.rhq.core.clientapi.server.core.CoreServerService}]
08:00:37,015 WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) SQL Error: 0, SQLState: 23503
08:00:37,015 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) Batch entry 0 insert into RHQ_MEASUREMENT_BLINE (BL_MAX, BL_MEAN, BL_MIN, BL_COMPUTE_TIME, SCHEDULE_ID, BL_USER_ENTERED, id) values (20.0, 20.0, 20.0, 1405944036998, 93051, '0', 13614) was aborted.  Call getNextException to see the cause.
08:00:37,017 WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) SQL Error: 0, SQLState: 23503
08:00:37,017 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) ERROR: insert or update on table "rhq_measurement_bline" violates foreign key constraint "rhq_measurement_bline_schedule_id_fkey"
  Detail: Key (schedule_id)=(93051) is not present in table "rhq_measurement_sched".
08:00:37,022 ERROR [org.hibernate.engine.jdbc.batch.internal.BatchingBatch] (RHQScheduler_Worker-2) HHH000315: Exception executing batch [could not perform addBatch]
08:00:37,022 WARN  [com.arjuna.ats.arjuna] (RHQScheduler_Worker-2) ARJUNA012125: TwoPhaseCoordinator.beforeCompletion - failed for SynchronizationImple< 0:ffff0a10b4c8:-63f06805:53c915ec:892f29, org.hibernate.engine.transaction.synchronization.internal.RegisteredSynchronization@2e93aa16 >: javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not perform addBatch
        at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387) [hibernate-entitymanager-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310) [hibernate-entitymanager-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1316) [hibernate-entitymanager-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        at org.hibernate.ejb.AbstractEntityManagerImpl$CallbackExceptionMapperImpl.mapManagedFlushFailure(AbstractEntityManagerImpl.java:1510) [hibernate-entitymanager-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorImpl.beforeCompletion(SynchronizationCallbackCoordinatorImpl.java:117) [hibernate-core-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        at org.hibernate.engine.transaction.synchronization.internal.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:53) [hibernate-core-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76) [jbossjts-jacorb-4.17.7.Final-redhat-4.jar:4.17.7.Final-redhat-4]
        at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:273) [jbossjts-jacorb-4.17.7.Final-redhat-4.jar:4.17.7.Final-redhat-4]
        at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:93) [jbossjts-jacorb-4.17.7.Final-redhat-4.jar:4.17.7.Final-redhat-4]
        at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162) [jbossjts-jacorb-4.17.7.Final-redhat-4.jar:4.17.7.Final-redhat-4]
        at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165) [jbossjts-jacorb-4.17.7.Final-redhat-4.jar:4.17.7.Final-redhat-4]
        at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126) [jbossjts-jacorb-4.17.7.Final-redhat-4.jar:4.17.7.Final-redhat-4]
        at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
     at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:91) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:252) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:344) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:216) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.rhq.enterprise.server.measurement.MeasurementBaselineManagerLocal$$$view12.saveNewBaselines(Unknown Source) [rhq-server.jar:4.9.0.JON320GA-redhat-2]
        at org.rhq.enterprise.server.measurement.MeasurementBaselineManagerBean.calculateBaselines(MeasurementBaselineManagerBean.java:280) [rhq-server.jar:4.9.0.JON320GA-redhat-2]
        at sun.reflect.GeneratedMethodAccessor717.invoke(Unknown Source) [:1.6.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
        at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
        at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.rhq.enterprise.server.common.PerformanceMonitorInterceptor.monitorHibernatePerformance(PerformanceMonitorInterceptor.java:32) [rhq-server.jar:4.9.0.JON320GA-redhat-2]
        at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source) [:1.6.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
        at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
        at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.rhq.enterprise.server.common.TransactionInterruptInterceptor.addCheckedActionToTransactionManager(TransactionInterruptInterceptor.java:78) [rhq-server.jar:4.9.0.JON320GA-redhat-2]
        at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source) [:1.6.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
        at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
        at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.rhq.enterprise.server.authz.RequiredPermissionsInterceptor.checkRequiredPermissions(RequiredPermissionsInterceptor.java:156) [rhq-server.jar:4.9.0.JON320GA-redhat-2]
        at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source) [:1.6.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
        at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
        at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
       at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47) [jboss-as-jpa-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:235) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.notSupported(CMTTxInterceptor.java:299) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:212) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.rhq.enterprise.server.measurement.MeasurementBaselineManagerLocal$$$view12.calculateBaselines(Unknown Source) [rhq-server.jar:4.9.0.JON320GA-redhat-2]
        at org.rhq.enterprise.server.measurement.MeasurementBaselineManagerBean.calculateAutoBaselines(MeasurementBaselineManagerBean.java:204) [rhq-server.jar:4.9.0.JON320GA-redhat-2]
        at sun.reflect.GeneratedMethodAccessor1053.invoke(Unknown Source) [:1.6.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
        at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
        at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.rhq.enterprise.server.common.PerformanceMonitorInterceptor.monitorHibernatePerformance(PerformanceMonitorInterceptor.java:32) [rhq-server.jar:4.9.0.JON320GA-redhat-2]
        at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source) [:1.6.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
        at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
        at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.rhq.enterprise.server.common.TransactionInterruptInterceptor.addCheckedActionToTransactionManager(TransactionInterruptInterceptor.java:78) [rhq-server.jar:4.9.0.JON320GA-redhat-2]
        at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source) [:1.6.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
        at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
        at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.rhq.enterprise.server.authz.RequiredPermissionsInterceptor.checkRequiredPermissions(RequiredPermissionsInterceptor.java:156) [rhq-server.jar:4.9.0.JON320GA-redhat-2]
        at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source) [:1.6.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
        at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
        at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
       at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47) [jboss-as-jpa-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:235) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.never(CMTTxInterceptor.java:285) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:210) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.rhq.enterprise.server.measurement.MeasurementBaselineManagerLocal$$$view12.calculateAutoBaselines(Unknown Source) [rhq-server.jar:4.9.0.JON320GA-redhat-2]
        at org.rhq.enterprise.server.measurement.MeasurementBaselineManagerBean.calculateAutoBaselines(MeasurementBaselineManagerBean.java:131) [rhq-server.jar:4.9.0.JON320GA-redhat-2]
        at sun.reflect.GeneratedMethodAccessor1052.invoke(Unknown Source) [:1.6.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
        at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
        at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.rhq.enterprise.server.common.PerformanceMonitorInterceptor.monitorHibernatePerformance(PerformanceMonitorInterceptor.java:32) [rhq-server.jar:4.9.0.JON320GA-redhat-2]
        at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source) [:1.6.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
        at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
        at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.rhq.enterprise.server.common.TransactionInterruptInterceptor.addCheckedActionToTransactionManager(TransactionInterruptInterceptor.java:78) [rhq-server.jar:4.9.0.JON320GA-redhat-2]
        at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source) [:1.6.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
        at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
        at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.rhq.enterprise.server.authz.RequiredPermissionsInterceptor.checkRequiredPermissions(RequiredPermissionsInterceptor.java:156) [rhq-server.jar:4.9.0.JON320GA-redhat-2]
        at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source) [:1.6.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
        at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
        at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47) [jboss-as-jpa-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:235) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.never(CMTTxInterceptor.java:285) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:210) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.rhq.enterprise.server.measurement.MeasurementBaselineManagerLocal$$$view12.calculateAutoBaselines(Unknown Source) [rhq-server.jar:4.9.0.JON320GA-redhat-2]
        at org.rhq.enterprise.server.scheduler.jobs.DataPurgeJob.calculateAutoBaselines(DataPurgeJob.java:352) [rhq-server.jar:4.9.0.JON320GA-redhat-2]
        at org.rhq.enterprise.server.scheduler.jobs.DataPurgeJob.executeJobCode(DataPurgeJob.java:95) [rhq-server.jar:4.9.0.JON320GA-redhat-2]
        at org.rhq.enterprise.server.scheduler.jobs.AbstractStatefulJob.execute(AbstractStatefulJob.java:48) [rhq-server.jar:4.9.0.JON320GA-redhat-2]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-1.6.5.jar:1.6.5]
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525) [quartz-1.6.5.jar:1.6.5]
Caused by: org.hibernate.exception.ConstraintViolationException: could not perform addBatch
        at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:128) [hibernate-core-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49) [hibernate-core-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125) [hibernate-core-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.performExecution(BatchingBatch.java:114) [hibernate-core-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.doExecuteBatch(BatchingBatch.java:101) [hibernate-core-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        at org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl.execute(AbstractBatchImpl.java:149) [hibernate-core-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.executeBatch(JdbcCoordinatorImpl.java:198) [hibernate-core-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:357) [hibernate-core-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:275) [hibernate-core-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:328) [hibernate-core-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:52) [hibernate-core-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1234) [hibernate-core-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:404) [hibernate-core-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorImpl.beforeCompletion(SynchronizationCallbackCoordinatorImpl.java:112) [hibernate-core-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        ... 238 more
Caused by: java.sql.BatchUpdateException: Batch entry 0 insert into RHQ_MEASUREMENT_BLINE (BL_MAX, BL_MEAN, BL_MIN, BL_COMPUTE_TIME, SCHEDULE_ID, BL_USER_ENTERED, id) values (20.0, 20.0, 20.0, 1405944036998, 93051, '0', 13614) was aborted.  Call getNextException to see the cause.
        at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2746)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1887)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2893)
        at sun.reflect.GeneratedMethodAccessor122.invoke(Unknown Source) [:1.6.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45]
        at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45]
        at org.postgresql.ds.jdbc23.AbstractJdbc23PooledConnection$StatementHandler.invoke(AbstractJdbc23PooledConnection.java:453)
        at com.sun.proxy.$Proxy106.executeBatch(Unknown Source)
        at org.jboss.jca.adapters.jdbc.CachedPreparedStatement.executeBatch(CachedPreparedStatement.java:708)
        at org.jboss.jca.adapters.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:1077)
        at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.performExecution(BatchingBatch.java:110) [hibernate-core-4.2.0.SP1-redhat-1.jar:4.2.0.SP1-redhat-1]
        ... 248 more

Comment 25 Lukas Krejci 2014-07-25 13:18:29 UTC
Please raise a new BZ for the latest stacktrace because the symptoms are unrelated.

As far as I can tell, this time you stumbled upon a foreign key violation when inserting baselines into the DB. That is a serious issue but is in no way related to any kind of invocation timeout this BZ is about.

Comment 26 Radim Hatlapatka 2014-07-25 13:35:37 UTC
Raised new BZ#1123389

Comment 27 Larry O'Leary 2014-07-29 00:17:25 UTC
This has been verified and released in Red Hat JBoss Operations Network 3.2 Update 02 (3.2.2) available from the Red Hat Customer Portal[1].



[1]: https://access.redhat.com/jbossnetwork/restricted/softwareDetail.html?softwareId=31783


Note You need to log in before you can comment on or make changes to this bug.