Created attachment 409856 [details] stack trace Description of problem: After creating a compatible group of Datasources, I try to use the group CONFIGURATION to update the config for the members of the group and the update times out. It appears to update the first member of the group, but times out on the rest. Version-Release number of selected component (if applicable): RHQ version: 3.0.0-SNAPSHOT build number: 20fe0ec How reproducible: Steps to Reproduce: 1. on an inventoried JBossAS Server, create two Datasources: a. navigate to the server's INVENTORY tab b. Create New: - Datasource OK default - the default template CONTINUE Resource Name: JohnsDatasource1 Type: No TX Datasource JNDI Name: JohnsDatasource1 Driver Class: ${rhq.server.database.driver-class} Connection Url: ${rhq.server.database.connection-url} SUBMIT c. Create New: - Datasource OK default - the default template CONTINUE Resource Name: JohnsDatasource2 Type: Local TX Datasource JNDI Name: JohnsDatasource2 Driver Class: ${rhq.server.database.driver-class} Connection Url: ${rhq.server.database.connection-url} SUBMIT 2. Groups > New Group Name: JohnsDatasources Contains: Compatible Resources - Datasource OK 3. ADD TO GROUP check JohnDatasource1 check JohnDatasource2 click the right arrow OK 4. Click CONFIGURATION tab EDIT set Max Pool Size 21 SAVE 3. Click to the CURRENT subtab Actual results: Configuration update is currently in progress. Please wait a few moments and refresh the page. After ten minutes of refreshing, the individual datasource configuration change fails on the second datasource. If I open the datasource's individiually and look at its CONFIGURATION/HISTORY, I find a Failure with a stack trace: Timed%20out%20-%20did%20not%20complete%20after%201184070%20ms%20%28the%20timeout%20period%20was%20600000%20ms%29 Timed out - did not complete after 872741 ms (the timeout period was 600000 ms) See the attached screen shot. PS. Nice stack trace formatting... https://bugzilla.redhat.com/show_bug.cgi?id=535805
This sounds like a regression.
Here is the underlying exception from the server log, javax.persistence.PersistenceException: org.hibernate.HibernateException: A collection with cascade="all-delete-orphan" was no longer referenced by the owning entity instance: org.rhq.core.domain.configuration.PropertyMap.map at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:629) at org.hibernate.ejb.AbstractEntityManagerImpl$1.beforeCompletion(AbstractEntityManagerImpl.java:524) at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImpl e.java:114) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:247) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:86) at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:177) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1389) at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:135) at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:87) at org.jboss.aspects.tx.TxPolicy.endTransaction(TxPolicy.java:175) at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:87) at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:95) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77) at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:240) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:210) at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:84) at $Proxy338.executeResourceConfigurationUpdate(Unknown Source) at org.rhq.enterprise.server.configuration.job.GroupResourceConfigurationUpdateJob.executeConfigurationUpdate(GroupResourceConfigurationUpdateJob.java:62) at org.rhq.enterprise.server.configuration.job.AbstractGroupConfigurationUpdateJob.processGroupConfigurationUpdate(AbstractGroupConfigurationUpdateJob.java:85) at org.rhq.enterprise.server.configuration.job.AbstractGroupConfigurationUpdateJob.execute(AbstractGroupConfigurationUpdateJob.java:64) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525) Caused by: org.hibernate.HibernateException: A collection with cascade="all-delete-orphan" was no longer referenced by the owning entity instance: org.rhq.core.domain.configuration.PropertyMap.map at org.hibernate.engine.Collections.processDereferencedCollection(Collections.java:96) at org.hibernate.engine.Collections.processUnreachableCollection(Collections.java:39) at org.hibernate.event.def.AbstractFlushingEventListener.flushCollections(AbstractFlushingEventListener.java:218) at org.hibernate.event.def.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:77) ... [org.rhq.enterprise.server.configuration.job.GroupResourceConfigurationUpdateJob] Failed to execute one or more Resource Configuration updates that were part of a group update - details: java.lang.RuntimeException:javax.transaction.RollbackException: [com.arjuna.ats.internal.jta.transaction.arjunacore.commitwhenaborted] [com.arjuna.ats.internal.jta.transaction.arjunacore.commitwhenaborted] Can't commit because the transaction is in aborted state -> javax.transaction.RollbackException:[com.arjuna.ats.internal.jta.transaction.arjunacore.commitwhenaborted] [com.arjuna.ats.internal.jta.transaction.arjunacore.commitwhenaborted] Can't commit because the transaction is in aborted state -> javax.persistence.PersistenceException:org.hibernate.HibernateException: A collection with cascade="all-delete-orphan" was no longer referenced by the owning entity instance: org.rhq.core.domain.configuration.PropertyMap.map -> org.hibernate.HibernateException:A collection with cascade="all-delete-orphan" was no longer referenced by the owning entity instance: org.rhq.core.domain.configuration.PropertyMap.map
After further testing I found that this bug is not specific to a particular resource type, and it will happen even with a group containing only a single resource. It does not happen though when you try to update the resource configuration outside of a group. I was also able to produce the same hibernate exception for PropertyList.list. Hibernate exceptions are occurring because the underlying collections in PropertyMap and PropertyList which include a cascade style of delete_orphan, are getting dereferenced. I have not been able to determine when/where the PropertyMap.map and PropertyList.list collections are getting dereferenced. I have pushed a commit to master that works around the problem by creating a deep copy of the Configuration object, stripped of any hibernate proxies. commit hash - b75773e304ac6a5fc6ac6f76fb7ab8301499d91b I am moving this to ON_QA, but I am going to continue investigating to see if I can figure out where those collections are getting dereferenced. I am also going to try to get some automated tests in place for this. This should be reproducible via test code.
Going to bounce this back to dev. I created a group of rhq agents and attempted to change a timeout value somewhere in the Miscellaneous pane. First one works, subsequent ones fail.
I saw this happen on Corey's server earlier today; however, there was no hibernate exception. The failed updates just reported a timeout. The configuration update appears to fail consistently for a group of five agents. I also reproduced the timeout with a group of two agents as well as a group of two data sources. I have not yet been able to reproduce locally yet though.
I tested this again on Corey's server today with a more recent build and did not run into the timeout. Moving back to ON_QA so that Corey can retest
I verified this on Build# 182 Version: 2.4 SNAPSHOT. Configuration update is working as expected. I can see both the datasources configuration getting updated correctly. There is no timeout happening with this build. Marking this bug as verified.
Mass-closure of verified bugs against JON.