Created attachment 432330 [details] rhq-installer-dbupgrade log Description of problem: Upgrade JON2.3.1 to JON2.4GA_QA with postgres gives error - 2010-07-16 12:34:56,767 INFO [org.rhq.enterprise.installer.ServerInformation] Database is supported: postgresql8.4(postgresql:8.4) 2010-07-16 12:34:58,054 FATAL [org.rhq.enterprise.installer.ServerInformation] Cannot upgrade the database schema - RHQ Server will not run properly java.lang.RuntimeException: Cannot run ANT on script [/NotBackedUp/install/jontag65/jon-server-2.4.0.GA_QA/logs/db-upgrade.xml]. Cause: /NotBackedUp/install/jontag65/jon-server-2.4.0.GA_QA/logs/db-upgrade.xml:27: Failed to upgrade - error in spec version [2.70.1]. Cause: Error executing the task [org.rhq.core.db.ant.dbupgrade.SST_AlterColumn] in schema spec version [2.70.1]. Cause: Failed to alter column. Cause: org.postgresql.util.PSQLException: ERROR: column "sender_name" contains null values at org.rhq.enterprise.installer.ServerInformation.startAnt(ServerInformation.java:785) at org.rhq.enterprise.installer.ServerInformation.upgradeExistingDatabaseSchema(ServerInformation.java:276) at org.rhq.enterprise.installer.ConfigurationBean.save(ConfigurationBean.java:735) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.el.parser.AstValue.invoke(AstValue.java:131) at org.apache.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:276) at org.apache.jasper.el.JspMethodExpression.invoke(JspMethodExpression.java:68) at javax.faces.component.MethodBindingMethodExpressionAdapter.invoke(MethodBindingMethodExpressionAdapter.java:88) at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:102) at javax.faces.component.UICommand.broadcast(UICommand.java:387) at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:475) at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:756) at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:82) at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:100) at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:265) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:124) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446) at java.lang.Thread.run(Thread.java:619) Caused by: /NotBackedUp/install/jontag65/jon-server-2.4.0.GA_QA/logs/db-upgrade.xml:27: Failed to upgrade - error in spec version [2.70.1]. Cause: Error executing the task [org.rhq.core.db.ant.dbupgrade.SST_AlterColumn] in schema spec version [2.70.1]. Cause: Failed to alter column. Cause: org.postgresql.util.PSQLException: ERROR: column "sender_name" contains null values at org.rhq.core.db.ant.dbupgrade.DBUpgrader.execute(DBUpgrader.java:298) at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:275) at org.apache.tools.ant.Task.perform(Task.java:364) at org.apache.tools.ant.Target.execute(Target.java:341) at org.apache.tools.ant.Target.performTasks(Target.java:369) at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1216) at org.apache.tools.ant.Project.executeTarget(Project.java:1185) at org.rhq.enterprise.installer.ServerInformation.startAnt(ServerInformation.java:783) ... 39 more Caused by: Failed to upgrade - error in spec version [2.70.1]. Cause: Error executing the task [org.rhq.core.db.ant.dbupgrade.SST_AlterColumn] in schema spec version [2.70.1]. Cause: Failed to alter column. Cause: org.postgresql.util.PSQLException: ERROR: column "sender_name" contains null values at org.rhq.core.db.ant.dbupgrade.DBUpgrader.execute(DBUpgrader.java:284) ... 46 more Caused by: Error executing the task [org.rhq.core.db.ant.dbupgrade.SST_AlterColumn] in schema spec version [2.70.1]. Cause: Failed to alter column. Cause: org.postgresql.util.PSQLException: ERROR: column "sender_name" contains null values at org.rhq.core.db.ant.dbupgrade.SchemaSpec.execute(SchemaSpec.java:154) at org.rhq.core.db.ant.dbupgrade.DBUpgrader.execute(DBUpgrader.java:272) ... 46 more Caused by: Failed to alter column. Cause: org.postgresql.util.PSQLException: ERROR: column "sender_name" contains null values at org.rhq.core.db.ant.dbupgrade.SST_AlterColumn.execute(SST_AlterColumn.java:156) at org.rhq.core.db.ant.dbupgrade.SchemaSpec.execute(SchemaSpec.java:149) ... 47 more Caused by: org.postgresql.util.PSQLException: ERROR: column "sender_name" contains null values at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:299) at org.rhq.core.db.DatabaseType.executeSql(DatabaseType.java:410) at org.rhq.core.db.DatabaseType.executeSql(DatabaseType.java:428) at org.rhq.core.db.PostgresqlDatabaseType.alterColumn(PostgresqlDatabaseType.java:149) at org.rhq.core.db.ant.dbupgrade.SST_AlterColumn.execute(SST_AlterColumn.java:154) ... 48 more QE Environment: Platform : RHEL 5.5 Database: Postgres 8.4 Java: Sun JDK1.6 Version-Release number of selected component (if applicable): version: 2.4.0.GA_QA build number: 10836:1ba43ec JON tag-jon-release (build #65) How reproducible: Always Steps to Reproduce: 1) Install JON2.3.1 & agent with Postgres 8.4 2) Create alerts (i) Name: Platform_alert_test [Type: Linux (Platform)] If Condition: Free Memory > 0.1B Dampening Rule: Each time condition set is true Action Filters: Disable alert until re-enabled manually or by recovery alert : false Notify Other Recipients: Eamil - rtimaniy (ii)Name: JBOSSAS_JVM_ALERT [Type: JBoss AS JVM (Server)] If Condition: Availability goes DOWN Dampening Rule: Each time condition set is true Action Filters: Disable alert until re-enabled manually or by recovery alert : false Notify Other Recipients: Eamil - rtimaniy (iii)Name: RHQ_Agent_JVM_Active_Threads_Alert [Type: RHQ Agent (Server)] If Condition: JVM Active Threads > -1.0 Dampening Rule: Each time condition set is true Action Filters: Disable alert until re-enabled manually or by recovery alert : false Notify Other Recipients: Eamil - rtimaniy Notify Roles (All Resources Role, Super User Role) Notify JON Users (username - rhqadmin) 3) Stop JON2.3.1 server (don't stop agent) 4) Upgrad JON2.3.1 to JON2.4GA_QA Actual results: Upgrade JON2.3.1 to JON2.4GA_QA with postgres gives error - Failed to alter column. Cause: org.postgresql.util.PSQLException: ERROR: column "sender_name" contains null values Expected results: Upgrade JON2.3.1 to JON2.4GA_QA should not give error. Additional info: Please refer attached screenshot and server log.
Created attachment 432332 [details] screenshot
Created attachment 432335 [details] server log
The failure is during schemaSpec 2.70.1, which have the following statements: <schema-addColumn table="RHQ_ALERT_NOTIFICATION" column="SENDER_NAME" precision="100" columnType="VARCHAR2" /> <schema-alterColumn table="RHQ_ALERT_NOTIFICATION" column="SENDER_NAME" nullable="FALSE" /> This will fail because the column is changed to non-null before values are inserted. Alternately, a default value could have been provided to prevent the error, but there is no natural default value for this column. ----- Looking in the general vicinity, I see a similar error in schemaSpec 2.70.2: <schema-addColumn table="RHQ_ALERT_NOTIF_LOG" column="SENDER" columnType="VARCHAR2" precision="100" /> <schema-alterColumn table="RHQ_ALERT_NOTIF_LOG" column="SENDER" nullable="FALSE" /> ----- What we need to do in both scenarios is to move the alterColumn definitions to a later schemaSpec in the upgrade script. It needs to come after the CustomAlertSenderUpgradeTask in schemaSpec 2.70.4, which converts the first-class notification constructs to the plugin-based (a.k.a. custom senders) de-normalized solution.
commit 3e720b2bab8c4752c4b3585138b32a0396063edc Author: Joseph Marques <joseph> Date: Sat Jul 17 15:12:17 2010 -0400 BZ-615245: only set the sender columns to NON-NULL after the CustomAlertSenderUpgradeTask completes ----- FYI - this bug was introduced accidentally during some minor tweaks I make to the upgrade script back on June 30th. I see that this bug was filed on July 16th. How did this get overlooked for more than 2 weeks? Don't we have dbsetup/upgrade testing automated?
This bug is blocked by bug #615952
QA Verified. The upgrade now completes w/o problem here, 2.3.1->2.4
Oops, my test might have been faulty. Reopening to assure everything is clean...
QA Verified.
Mass-closure of verified bugs against JON.