Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 596480 - Data source configuration update crashes RHQ server
Data source configuration update crashes RHQ server
Status: CLOSED CURRENTRELEASE
Product: RHQ Project
Classification: Other
Component: Configuration (Show other bugs)
3.0.0
All Linux
high Severity urgent (vote)
: ---
: ---
Assigned To: Heiko W. Rupp
Heiko W. Rupp
:
Depends On:
Blocks: jon-sprint10-bugs jon-sprint11-bugs
  Show dependency treegraph
 
Reported: 2010-05-26 15:41 EDT by John Sanda
Modified: 2010-08-12 12:52 EDT (History)
2 users (show)

See Also:
Fixed In Version: 2.4
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-08-12 12:52:16 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
diff of rhq-ds.xml before and after the configuration update (4.50 KB, application/octet-stream)
2010-05-27 09:59 EDT, John Sanda
no flags Details
server log (30.73 KB, text/x-log)
2010-06-28 06:43 EDT, Rajan Timaniya
no flags Details
rhq-ds.xml (1.85 KB, text/xml)
2010-06-28 07:21 EDT, Rajan Timaniya
no flags Details

  None (edit)
Description John Sanda 2010-05-26 15:41:23 EDT
Description of problem:
Attempting to do a data configuration update on my rhq data source crashed my rhq server.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Log into your rhq server which should also be in inventory
2. Go the configuration tab for the RHQDS data source
3. Change the value of the prepared statement cache size
  
Actual results:
2010-05-26 15:28:47,119 WARN  [org.hibernate.util.JDBCExceptionReporter] SQL Error: 0, SQLState: null
2010-05-26 15:28:47,127 ERROR [org.hibernate.util.JDBCExceptionReporter] Could not create connection; - nested throwable: (org.postgresql.util.PSQLException: FATAL: database "null" does not exist); - nested throwable: (org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (org.postgresql.util.PSQLException: FATAL: database "null" does not exist))
2010-05-26 15:28:47,127 WARN  [org.rhq.enterprise.communications.command.server.CommandProcessor] {CommandProcessor.failed-authentication}Command failed to be authenticated!  This command will be ignored and not processed: Command: type=[remotepojo]; cmd-in-response=[false]; config=[{rhq.agent-name=localhost, rhq.security-token=lQUqy56WLrLbxLPX0uAKjGzFOZaI3qryTRvTOTTg2Whrapn/oS0N8Ago3mkGPtZdCeA=, rhq.externalizable-strategy=AGENT, rhq.send-throttle=true, rhq.guaranteed-delivery=true}]; params=[{invocation=NameBasedInvocation[persistUpdatedResourceConfiguration], targetInterfaceName=org.rhq.core.clientapi.server.configuration.ConfigurationServerService}]
2010-05-26 15:28:47,138 WARN  [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] Throwable while attempting to get a new connection: null
org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (org.postgresql.util.PSQLException: FATAL: database "null" does not exist)
	at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.createManagedConnection(XAManagedConnectionFactory.java:155)
	at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.createConnectionEventListener(InternalManagedConnectionPool.java:619)
	at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:264)
	at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:613)
	at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:347)
	at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:330)
	at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:402)
	at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:849)
	at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)
	at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:47)
	at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
	at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
	at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:140)
	at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1547)
	at org.hibernate.loader.Loader.doQuery(Loader.java:673)
	at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)
	at org.hibernate.loader.Loader.doList(Loader.java:2213)
	at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2104)
	at org.hibernate.loader.Loader.list(Loader.java:2099)
	at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:378)
	at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:338)
	at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:172)
	at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1121)
	at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79)
	at org.hibernate.ejb.QueryImpl.getSingleResult(QueryImpl.java:80)
	at org.rhq.enterprise.server.core.AgentManagerBean.getAgentByAgentToken(AgentManagerBean.java:306)
	at sun.reflect.GeneratedMethodAccessor801.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
	at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
	at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
	at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:95)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
	at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
	at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:240)
	at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:210)
	at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:84)
	at $Proxy272.getAgentByAgentToken(Unknown Source)
	at org.rhq.enterprise.server.core.comm.SecurityTokenCommandAuthenticator.isAuthenticated(SecurityTokenCommandAuthenticator.java:96)
	at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:246)
	at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:184)
	at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
	at org.jboss.remoting.transport.servlet.ServletServerInvoker.processRequest(ServletServerInvoker.java:232)
	at sun.reflect.GeneratedMethodAccessor211.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
	at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
	at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
	at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
	at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
	at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:288)
	at $Proxy431.processRequest(Unknown Source)
	at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:128)
	at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.doPost(ServerInvokerServlet.java:157)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:124)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
	at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
	at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
	at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
	at java.lang.Thread.run(Thread.java:619)
Caused by: org.postgresql.util.PSQLException: FATAL: database "null" does not exist
	at org.postgresql.core.v3.ConnectionFactoryImpl.readStartupMessages(ConnectionFactoryImpl.java:464)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:112)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
	at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
	at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
	at org.postgresql.jdbc3.Jdbc3Connection.<init>(Jdbc3Connection.java:24)
	at org.postgresql.Driver.makeConnection(Driver.java:393)
	at org.postgresql.Driver.connect(Driver.java:267)
	at java.sql.DriverManager.getConnection(DriverManager.java:582)
	at java.sql.DriverManager.getConnection(DriverManager.java:185)
	at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:87)
	at org.postgresql.xa.PGXADataSource.getXAConnection(PGXADataSource.java:47)
	at org.postgresql.xa.PGXADataSource.getXAConnection(PGXADataSource.java:32)
	at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.createManagedConnection(XAManagedConnectionFactory.java:137)
	... 94 more


Expected results:
Update should proceed without error

Additional info:
I initially ran into this while trying to perform a group config update on a group of data sources. Subsequent testing revealed that the problem happens for an individual resource update as well.

The rhq server may still actually be running, but it is virtually inaccessible do to the above database exception that seems to reoccur on every subsequent http request I send to the server.
Comment 1 John Sanda 2010-05-26 16:02:13 EDT
Further testing reveals that the rhq-ds.xml is getting screwed up. I realized this after bouncing both of my database and rhq server and still having the problem. It wasn't until I replaced rhq-ds.xml with a clean version that my server started up properly.
Comment 2 John Sanda 2010-05-27 09:59:32 EDT
Created attachment 417252 [details]
diff of rhq-ds.xml before and after the configuration update
Comment 3 John Sanda 2010-05-27 12:42:52 EDT
Here is a diff ignoring whitespace,

1,2c1
< <?xml version="1.0"?>
< 
---
> <?xml version="1.0" encoding="UTF-8"?>
4d2
< 
6d3
< 
15,18c12
<     
<     <track-connection-by-tx/>
<     <user-name>${rhq.server.database.user-name}</user-name>
<     <password>${rhq.server.database.password}</password>
---
>     <track-connection-by-tx />
25a20
>     <xa-datasource-property name="URL">jdbc:postgresql://127.0.0.1:5432/rhqdev</xa-datasource-property>
27d21
< 
44d37
< 
45a39
>
Comment 4 Charles Crouch 2010-06-13 23:31:58 EDT
So the problem is we're adding <xa-datasource-property name="URL"> and losing username/password properties
Comment 5 Heiko W. Rupp 2010-06-15 07:31:02 EDT
John: do you recall the change you wanted to make?

In general messing with datasources of running applications is flaky on the JBossAS itself, so the question is if we should disable this for the RHQ server anyway
Comment 6 John Sanda 2010-06-16 08:58:39 EDT
I don't think that there was a particular change I was trying to make. If memory serves, I think I stumbled onto this while testing a group config update bug, and I was using a group of data sources.
Comment 7 Heiko W. Rupp 2010-06-21 06:16:01 EDT
1e195a7280079f300fba44f774de96dc42201e18 on release-3.0.0
Comment 8 Rajan Timaniya 2010-06-28 06:41:23 EDT
Tested on JON 2.4 GA_QA (tag-jon-release build #42)
http://hudson-qe.rhq.rdu.redhat.com:8080/view/JON/job/tag-jon-release/42/

Steps:
1. Log into your rhq server which should also be in inventory
2. Go the configuration tab for the RHQDS data source
3. Change the value of the prepared statement cache size
4. Refer server log

Observation:
Update 'prepared statement cache size' value but it gives error/exception in server log

2010-06-28 15:38:28,796 ERROR [org.quartz.core.ErrorLogger] An error occured while firing trigger 'org.rhq.enterprise.server.scheduler.jobs.CloudManagerJob.org.rhq.enterprise.server.scheduler.jobs.CloudManagerJob'
org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'NoTxRHQDS': java.sql.SQLException: Could not retrieve datasource via JNDI url 'java:/NoTxRHQDS' org.jboss.util.NestedSQLException: You are trying to use a connection factory that has been shut down: ManagedConnectionFactory is null.; - nested throwable: (javax.resource.ResourceException: You are trying to use a connection factory that has been shut down: ManagedConnectionFactory is null.) [See nested exception: java.sql.SQLException: Could not retrieve datasource via JNDI url 'java:/NoTxRHQDS' org.jboss.util.NestedSQLException: You are trying to use a connection factory that has been shut down: ManagedConnectionFactory is null.; - nested throwable: (javax.resource.ResourceException: You are trying to use a connection factory that has been shut down: ManagedConnectionFactory is null.)]
        at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:167)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3750)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.triggerFired(JobStoreSupport.java:2841)
        at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:342)
Caused by: java.sql.SQLException: Could not retrieve datasource via JNDI url 'java:/NoTxRHQDS' org.jboss.util.NestedSQLException: You are trying to use a connection factory that has been shut down: ManagedConnectionFactory is null.; - nested throwable: (javax.resource.ResourceException: You are trying to use a connection factory that has been shut down: ManagedConnectionFactory is null.)
        at org.quartz.utils.JNDIConnectionProvider.getConnection(JNDIConnectionProvider.java:166)
        at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:112)
        at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:164)
        ... 3 more
Comment 9 Rajan Timaniya 2010-06-28 06:43:14 EDT
Created attachment 427379 [details]
server log
Comment 10 Heiko W. Rupp 2010-06-28 07:09:50 EDT
Rajan,
can you please attach the created rhq-ds.xml ?
 Heiko
Comment 11 Rajan Timaniya 2010-06-28 07:21:44 EDT
Created attachment 427388 [details]
rhq-ds.xml
Comment 12 Heiko W. Rupp 2010-06-29 11:12:26 EDT
I was discussing this with Rajan yesterday .. the server spit out some exceptions, which is expected, as the datasource becomes unavailable for a moment when it gets (re)deployed by the update.

Rajan confirmed, that he was able to continue working on the server later on, so I see this as fixed.

Also the generated rhq-ds.xml is (except for whitespace) the same as the one we supply.

Assigning back to QA
Comment 13 Rajan Timaniya 2010-06-30 06:10:27 EDT
Verified on JON 2.4 GA_QA (tag-jon-release build #44)

As per discussed with Heiko - JON server gives exception in server log which is expected, as the datasource unavailable for a moment when it gets (re)deployed by the update.
Comment 14 Corey Welton 2010-08-12 12:52:16 EDT
Mass-closure of verified bugs against JON.

Note You need to log in before you can comment on or make changes to this bug.