Bug 615245 - Upgrade JON2.3.1 to JON2.4GA_QA with postgres gives error - Failed to alter column. Cause: org.postgresql.util.PSQLException: ERROR: column "sender_name" contains null values
Upgrade JON2.3.1 to JON2.4GA_QA with postgres gives error - Failed to alter c...
Status: CLOSED CURRENTRELEASE
Product: RHQ Project
Classification: Other
Component: Database (Show other bugs)
3.0.0
All Linux
urgent Severity urgent (vote)
: ---
: ---
Assigned To: Joseph Marques
Rajan Timaniya
:
Depends On:
Blocks: jon-sprint12-bugs 614432
  Show dependency treegraph
 
Reported: 2010-07-16 05:27 EDT by Rajan Timaniya
Modified: 2010-08-12 12:55 EDT (History)
2 users (show)

See Also:
Fixed In Version: 2.4
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-08-12 12:55:22 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
rhq-installer-dbupgrade log (55.68 KB, text/x-log)
2010-07-16 05:27 EDT, Rajan Timaniya
no flags Details
screenshot (67.00 KB, image/png)
2010-07-16 05:29 EDT, Rajan Timaniya
no flags Details
server log (43.17 KB, text/x-log)
2010-07-16 05:29 EDT, Rajan Timaniya
no flags Details

  None (edit)
Description Rajan Timaniya 2010-07-16 05:27:52 EDT
Created attachment 432330 [details]
rhq-installer-dbupgrade log

Description of problem:
Upgrade JON2.3.1 to JON2.4GA_QA with postgres gives error - 

2010-07-16 12:34:56,767 INFO  [org.rhq.enterprise.installer.ServerInformation] Database is supported: postgresql8.4(postgresql:8.4)
2010-07-16 12:34:58,054 FATAL [org.rhq.enterprise.installer.ServerInformation] Cannot upgrade the database schema - RHQ Server will not run properly
java.lang.RuntimeException: Cannot run ANT on script [/NotBackedUp/install/jontag65/jon-server-2.4.0.GA_QA/logs/db-upgrade.xml]. Cause: /NotBackedUp/install/jontag65/jon-server-2.4.0.GA_QA/logs/db-upgrade.xml:27: Failed to upgrade - error in spec version [2.70.1]. Cause: Error executing the task [org.rhq.core.db.ant.dbupgrade.SST_AlterColumn] in schema spec version [2.70.1]. Cause: Failed to alter column. Cause: org.postgresql.util.PSQLException: ERROR: column "sender_name" contains null values
        at org.rhq.enterprise.installer.ServerInformation.startAnt(ServerInformation.java:785)
        at org.rhq.enterprise.installer.ServerInformation.upgradeExistingDatabaseSchema(ServerInformation.java:276)
        at org.rhq.enterprise.installer.ConfigurationBean.save(ConfigurationBean.java:735)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.el.parser.AstValue.invoke(AstValue.java:131)
        at org.apache.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:276)
        at org.apache.jasper.el.JspMethodExpression.invoke(JspMethodExpression.java:68)
        at javax.faces.component.MethodBindingMethodExpressionAdapter.invoke(MethodBindingMethodExpressionAdapter.java:88)
        at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:102)
        at javax.faces.component.UICommand.broadcast(UICommand.java:387)
        at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:475)
        at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:756)
        at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:82)
        at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:100)
        at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118)
        at javax.faces.webapp.FacesServlet.service(FacesServlet.java:265)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:124)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
        at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
        at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
        at java.lang.Thread.run(Thread.java:619)
Caused by: /NotBackedUp/install/jontag65/jon-server-2.4.0.GA_QA/logs/db-upgrade.xml:27: Failed to upgrade - error in spec version [2.70.1]. Cause: Error executing the task [org.rhq.core.db.ant.dbupgrade.SST_AlterColumn] in schema spec version [2.70.1]. Cause: Failed to alter column. Cause: org.postgresql.util.PSQLException: ERROR: column "sender_name" contains null values
        at org.rhq.core.db.ant.dbupgrade.DBUpgrader.execute(DBUpgrader.java:298)
        at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:275)
        at org.apache.tools.ant.Task.perform(Task.java:364)
        at org.apache.tools.ant.Target.execute(Target.java:341)
        at org.apache.tools.ant.Target.performTasks(Target.java:369)
        at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1216)
        at org.apache.tools.ant.Project.executeTarget(Project.java:1185)
        at org.rhq.enterprise.installer.ServerInformation.startAnt(ServerInformation.java:783)
        ... 39 more
Caused by: Failed to upgrade - error in spec version [2.70.1]. Cause: Error executing the task [org.rhq.core.db.ant.dbupgrade.SST_AlterColumn] in schema spec version [2.70.1]. Cause: Failed to alter column. Cause: org.postgresql.util.PSQLException: ERROR: column "sender_name" contains null values
        at org.rhq.core.db.ant.dbupgrade.DBUpgrader.execute(DBUpgrader.java:284)
        ... 46 more
Caused by: Error executing the task [org.rhq.core.db.ant.dbupgrade.SST_AlterColumn] in schema spec version [2.70.1]. Cause: Failed to alter column. Cause: org.postgresql.util.PSQLException: ERROR: column "sender_name" contains null values
        at org.rhq.core.db.ant.dbupgrade.SchemaSpec.execute(SchemaSpec.java:154)
        at org.rhq.core.db.ant.dbupgrade.DBUpgrader.execute(DBUpgrader.java:272)
        ... 46 more
Caused by: Failed to alter column. Cause: org.postgresql.util.PSQLException: ERROR: column "sender_name" contains null values
        at org.rhq.core.db.ant.dbupgrade.SST_AlterColumn.execute(SST_AlterColumn.java:156)
        at org.rhq.core.db.ant.dbupgrade.SchemaSpec.execute(SchemaSpec.java:149)
        ... 47 more
Caused by: org.postgresql.util.PSQLException: ERROR: column "sender_name" contains null values
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1795)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:353)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:299)
        at org.rhq.core.db.DatabaseType.executeSql(DatabaseType.java:410)
        at org.rhq.core.db.DatabaseType.executeSql(DatabaseType.java:428)
        at org.rhq.core.db.PostgresqlDatabaseType.alterColumn(PostgresqlDatabaseType.java:149)
        at org.rhq.core.db.ant.dbupgrade.SST_AlterColumn.execute(SST_AlterColumn.java:154)
        ... 48 more

QE Environment:
Platform : RHEL 5.5
Database: Postgres 8.4
Java: Sun JDK1.6

Version-Release number of selected component (if applicable):
version: 2.4.0.GA_QA
build number: 10836:1ba43ec
JON tag-jon-release (build #65)

How reproducible:
Always

Steps to Reproduce:
1) Install JON2.3.1 & agent with Postgres 8.4

2) Create alerts
     (i) Name:  	Platform_alert_test    [Type: Linux (Platform)]
	If Condition:  	 Free Memory > 0.1B
	Dampening Rule: 	Each time condition set is true
	Action Filters: 	Disable alert until re-enabled manually or by recovery alert : false 
 	Notify Other Recipients: Eamil - rtimaniy@redhat.com

     (ii)Name:  	JBOSSAS_JVM_ALERT [Type: JBoss AS JVM (Server)]
	If Condition:  	 Availability goes DOWN
	Dampening Rule: 	Each time condition set is true
	Action Filters: 	Disable alert until re-enabled manually or by recovery alert : false 
 	Notify Other Recipients: Eamil - rtimaniy@redhat.com

     (iii)Name:  	RHQ_Agent_JVM_Active_Threads_Alert    [Type: RHQ Agent (Server)]
	If Condition:  	 JVM Active Threads > -1.0
	Dampening Rule: 	Each time condition set is true
	Action Filters: 	Disable alert until re-enabled manually or by recovery alert : false 
 	Notify Other Recipients: Eamil - rtimaniy@redhat.com
	Notify Roles (All Resources Role, Super User Role)
	Notify JON Users  (username - rhqadmin)

3) Stop JON2.3.1 server (don't stop agent)
4) Upgrad JON2.3.1 to JON2.4GA_QA
  
Actual results:
Upgrade JON2.3.1 to JON2.4GA_QA with postgres gives error - Failed to alter column. Cause: org.postgresql.util.PSQLException: ERROR: column "sender_name" contains null values

Expected results:
Upgrade JON2.3.1 to JON2.4GA_QA should not give error.

Additional info:
Please refer attached screenshot and server log.
Comment 1 Rajan Timaniya 2010-07-16 05:29:05 EDT
Created attachment 432332 [details]
screenshot
Comment 2 Rajan Timaniya 2010-07-16 05:29:37 EDT
Created attachment 432335 [details]
server log
Comment 3 Joseph Marques 2010-07-17 15:10:44 EDT
The failure is during schemaSpec 2.70.1, which have the following statements:

<schema-addColumn table="RHQ_ALERT_NOTIFICATION" column="SENDER_NAME" precision="100" columnType="VARCHAR2" />
<schema-alterColumn table="RHQ_ALERT_NOTIFICATION" column="SENDER_NAME" nullable="FALSE" />

This will fail because the column is changed to non-null before values are inserted.  Alternately, a default value could have been provided to prevent the error, but there is no natural default value for this column.

-----

Looking in the general vicinity, I see a similar error in schemaSpec 2.70.2:

<schema-addColumn table="RHQ_ALERT_NOTIF_LOG" column="SENDER" columnType="VARCHAR2" precision="100" />
<schema-alterColumn table="RHQ_ALERT_NOTIF_LOG" column="SENDER" nullable="FALSE" />

-----

What we need to do in both scenarios is to move the alterColumn definitions to a later schemaSpec in the upgrade script.  It needs to come after the CustomAlertSenderUpgradeTask in schemaSpec 2.70.4, which converts the first-class notification constructs to the plugin-based (a.k.a. custom senders) de-normalized solution.
Comment 4 Joseph Marques 2010-07-17 15:14:53 EDT
commit 3e720b2bab8c4752c4b3585138b32a0396063edc
Author: Joseph Marques <joseph@redhat.com>
Date:   Sat Jul 17 15:12:17 2010 -0400

BZ-615245: only set the sender columns to NON-NULL after the CustomAlertSenderUpgradeTask completes

-----

FYI - this bug was introduced accidentally during some minor tweaks I make to the upgrade script back on June 30th.  I see that this bug was filed on July 16th.  How did this get overlooked for more than 2 weeks?  Don't we have dbsetup/upgrade testing automated?
Comment 5 Rajan Timaniya 2010-07-19 07:41:34 EDT
This bug is blocked by bug #615952
Comment 6 Corey Welton 2010-07-21 00:05:19 EDT
QA Verified.  The upgrade now completes w/o problem here, 2.3.1->2.4
Comment 7 Corey Welton 2010-07-21 00:38:07 EDT
Oops, my test might have been faulty.  Reopening to assure everything is clean...
Comment 8 Corey Welton 2010-07-21 01:13:09 EDT
QA Verified.
Comment 9 Corey Welton 2010-08-12 12:55:22 EDT
Mass-closure of verified bugs against JON.

Note You need to log in before you can comment on or make changes to this bug.