Description of problem: Attempting to do a data configuration update on my rhq data source crashed my rhq server. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Log into your rhq server which should also be in inventory 2. Go the configuration tab for the RHQDS data source 3. Change the value of the prepared statement cache size Actual results: 2010-05-26 15:28:47,119 WARN [org.hibernate.util.JDBCExceptionReporter] SQL Error: 0, SQLState: null 2010-05-26 15:28:47,127 ERROR [org.hibernate.util.JDBCExceptionReporter] Could not create connection; - nested throwable: (org.postgresql.util.PSQLException: FATAL: database "null" does not exist); - nested throwable: (org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (org.postgresql.util.PSQLException: FATAL: database "null" does not exist)) 2010-05-26 15:28:47,127 WARN [org.rhq.enterprise.communications.command.server.CommandProcessor] {CommandProcessor.failed-authentication}Command failed to be authenticated! This command will be ignored and not processed: Command: type=[remotepojo]; cmd-in-response=[false]; config=[{rhq.agent-name=localhost, rhq.security-token=lQUqy56WLrLbxLPX0uAKjGzFOZaI3qryTRvTOTTg2Whrapn/oS0N8Ago3mkGPtZdCeA=, rhq.externalizable-strategy=AGENT, rhq.send-throttle=true, rhq.guaranteed-delivery=true}]; params=[{invocation=NameBasedInvocation[persistUpdatedResourceConfiguration], targetInterfaceName=org.rhq.core.clientapi.server.configuration.ConfigurationServerService}] 2010-05-26 15:28:47,138 WARN [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] Throwable while attempting to get a new connection: null org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (org.postgresql.util.PSQLException: FATAL: database "null" does not exist) at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.createManagedConnection(XAManagedConnectionFactory.java:155) at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.createConnectionEventListener(InternalManagedConnectionPool.java:619) at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:264) at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:613) at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:347) at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:330) at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:402) at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:849) at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89) at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:47) at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423) at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144) at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:140) at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1547) at org.hibernate.loader.Loader.doQuery(Loader.java:673) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236) at org.hibernate.loader.Loader.doList(Loader.java:2213) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2104) at org.hibernate.loader.Loader.list(Loader.java:2099) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:378) at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:338) at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:172) at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1121) at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79) at org.hibernate.ejb.QueryImpl.getSingleResult(QueryImpl.java:80) at org.rhq.enterprise.server.core.AgentManagerBean.getAgentByAgentToken(AgentManagerBean.java:306) at sun.reflect.GeneratedMethodAccessor801.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166) at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79) at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:95) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77) at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:240) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:210) at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:84) at $Proxy272.getAgentByAgentToken(Unknown Source) at org.rhq.enterprise.server.core.comm.SecurityTokenCommandAuthenticator.isAuthenticated(SecurityTokenCommandAuthenticator.java:96) at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:246) at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:184) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809) at org.jboss.remoting.transport.servlet.ServletServerInvoker.processRequest(ServletServerInvoker.java:232) at sun.reflect.GeneratedMethodAccessor211.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:288) at $Proxy431.processRequest(Unknown Source) at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:128) at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.doPost(ServerInvokerServlet.java:157) at javax.servlet.http.HttpServlet.service(HttpServlet.java:710) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:124) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446) at java.lang.Thread.run(Thread.java:619) Caused by: org.postgresql.util.PSQLException: FATAL: database "null" does not exist at org.postgresql.core.v3.ConnectionFactoryImpl.readStartupMessages(ConnectionFactoryImpl.java:464) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:112) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66) at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125) at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30) at org.postgresql.jdbc3.Jdbc3Connection.<init>(Jdbc3Connection.java:24) at org.postgresql.Driver.makeConnection(Driver.java:393) at org.postgresql.Driver.connect(Driver.java:267) at java.sql.DriverManager.getConnection(DriverManager.java:582) at java.sql.DriverManager.getConnection(DriverManager.java:185) at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:87) at org.postgresql.xa.PGXADataSource.getXAConnection(PGXADataSource.java:47) at org.postgresql.xa.PGXADataSource.getXAConnection(PGXADataSource.java:32) at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.createManagedConnection(XAManagedConnectionFactory.java:137) ... 94 more Expected results: Update should proceed without error Additional info: I initially ran into this while trying to perform a group config update on a group of data sources. Subsequent testing revealed that the problem happens for an individual resource update as well. The rhq server may still actually be running, but it is virtually inaccessible do to the above database exception that seems to reoccur on every subsequent http request I send to the server.
Further testing reveals that the rhq-ds.xml is getting screwed up. I realized this after bouncing both of my database and rhq server and still having the problem. It wasn't until I replaced rhq-ds.xml with a clean version that my server started up properly.
Created attachment 417252 [details] diff of rhq-ds.xml before and after the configuration update
Here is a diff ignoring whitespace, 1,2c1 < <?xml version="1.0"?> < --- > <?xml version="1.0" encoding="UTF-8"?> 4d2 < 6d3 < 15,18c12 < < <track-connection-by-tx/> < <user-name>${rhq.server.database.user-name}</user-name> < <password>${rhq.server.database.password}</password> --- > <track-connection-by-tx /> 25a20 > <xa-datasource-property name="URL">jdbc:postgresql://127.0.0.1:5432/rhqdev</xa-datasource-property> 27d21 < 44d37 < 45a39 >
So the problem is we're adding <xa-datasource-property name="URL"> and losing username/password properties
John: do you recall the change you wanted to make? In general messing with datasources of running applications is flaky on the JBossAS itself, so the question is if we should disable this for the RHQ server anyway
I don't think that there was a particular change I was trying to make. If memory serves, I think I stumbled onto this while testing a group config update bug, and I was using a group of data sources.
1e195a7280079f300fba44f774de96dc42201e18 on release-3.0.0
Tested on JON 2.4 GA_QA (tag-jon-release build #42) http://hudson-qe.rhq.rdu.redhat.com:8080/view/JON/job/tag-jon-release/42/ Steps: 1. Log into your rhq server which should also be in inventory 2. Go the configuration tab for the RHQDS data source 3. Change the value of the prepared statement cache size 4. Refer server log Observation: Update 'prepared statement cache size' value but it gives error/exception in server log 2010-06-28 15:38:28,796 ERROR [org.quartz.core.ErrorLogger] An error occured while firing trigger 'org.rhq.enterprise.server.scheduler.jobs.CloudManagerJob.org.rhq.enterprise.server.scheduler.jobs.CloudManagerJob' org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'NoTxRHQDS': java.sql.SQLException: Could not retrieve datasource via JNDI url 'java:/NoTxRHQDS' org.jboss.util.NestedSQLException: You are trying to use a connection factory that has been shut down: ManagedConnectionFactory is null.; - nested throwable: (javax.resource.ResourceException: You are trying to use a connection factory that has been shut down: ManagedConnectionFactory is null.) [See nested exception: java.sql.SQLException: Could not retrieve datasource via JNDI url 'java:/NoTxRHQDS' org.jboss.util.NestedSQLException: You are trying to use a connection factory that has been shut down: ManagedConnectionFactory is null.; - nested throwable: (javax.resource.ResourceException: You are trying to use a connection factory that has been shut down: ManagedConnectionFactory is null.)] at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:167) at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3750) at org.quartz.impl.jdbcjobstore.JobStoreSupport.triggerFired(JobStoreSupport.java:2841) at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:342) Caused by: java.sql.SQLException: Could not retrieve datasource via JNDI url 'java:/NoTxRHQDS' org.jboss.util.NestedSQLException: You are trying to use a connection factory that has been shut down: ManagedConnectionFactory is null.; - nested throwable: (javax.resource.ResourceException: You are trying to use a connection factory that has been shut down: ManagedConnectionFactory is null.) at org.quartz.utils.JNDIConnectionProvider.getConnection(JNDIConnectionProvider.java:166) at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:112) at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:164) ... 3 more
Created attachment 427379 [details] server log
Rajan, can you please attach the created rhq-ds.xml ? Heiko
Created attachment 427388 [details] rhq-ds.xml
I was discussing this with Rajan yesterday .. the server spit out some exceptions, which is expected, as the datasource becomes unavailable for a moment when it gets (re)deployed by the update. Rajan confirmed, that he was able to continue working on the server later on, so I see this as fixed. Also the generated rhq-ds.xml is (except for whitespace) the same as the one we supply. Assigning back to QA
Verified on JON 2.4 GA_QA (tag-jon-release build #44) As per discussed with Heiko - JON server gives exception in server log which is expected, as the datasource unavailable for a moment when it gets (re)deployed by the update.
Mass-closure of verified bugs against JON.