Bug 588700 - Problem upgrading alerts from JON 2.3 to JON 2.4
Problem upgrading alerts from JON 2.3 to JON 2.4
Status: CLOSED CURRENTRELEASE
Product: RHQ Project
Classification: Other
Component: Database (Show other bugs)
3.0.0
All Linux
urgent Severity medium (vote)
: ---
: ---
Assigned To: John Sanda
Rajan Timaniya
:
Depends On:
Blocks: rhq_triage
  Show dependency treegraph
 
Reported: 2010-05-04 06:32 EDT by Rajan Timaniya
Modified: 2010-08-12 12:55 EDT (History)
2 users (show)

See Also:
Fixed In Version: 2.4
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-08-12 12:55:35 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
screenshot (56.76 KB, image/png)
2010-05-04 06:32 EDT, Rajan Timaniya
no flags Details
rhq-installer-dbupgrade log (130.33 KB, text/x-log)
2010-05-04 06:34 EDT, Rajan Timaniya
no flags Details
rhq-server-log4j log (23.46 KB, text/x-log)
2010-05-04 06:36 EDT, Rajan Timaniya
no flags Details
Dashboard without menu (99.11 KB, image/png)
2010-05-05 05:01 EDT, Rajan Timaniya
no flags Details
Error on click of platform (32.56 KB, image/png)
2010-05-05 05:02 EDT, Rajan Timaniya
no flags Details
Stack Trace on screen (156.59 KB, image/png)
2010-05-05 05:02 EDT, Rajan Timaniya
no flags Details
rhq-server-log4j log - after JON2.1 to JON2.4 (990.01 KB, text/x-log)
2010-05-05 05:03 EDT, Rajan Timaniya
no flags Details
UI Error (20.61 KB, image/png)
2010-05-18 05:40 EDT, Rajan Timaniya
no flags Details
UI error stack trace (232.70 KB, image/png)
2010-05-18 05:41 EDT, Rajan Timaniya
no flags Details

  None (edit)
Description Rajan Timaniya 2010-05-04 06:32:01 EDT
Created attachment 411243 [details]
screenshot

Description of problem:
Fail upgrade JON 2.1 to JON 2.4 on Oracle 10g.

Version-Release number of selected component (if applicable):
JON build #124  Revision: 10615 
http://hudson-qe.rhq.rdu.redhat.com:8080/job/jon/124/

How reproducible:
Always

Steps to Reproduce:
1) Install JON 2.1 on Oracle 10g
2) Create some alerts, schedule, etc..  (basic operations)
3) Upgrade JON 2.1 to JON 2.4 (same database Oracle 10g)
  
Actual results:
Upgrade JON 2.1 to JON 2.4 is fail with the error.

Expected results:
Upgrade JON 2.1 to JON 2.4 should work without any error.

Additional info:
2010-05-04 15:20:37,567 FATAL [org.rhq.enterprise.installer.ServerInformation] Cannot upgrade the database schema - RHQ Server will not run properly
java.lang.RuntimeException: Cannot run ANT on script [/NotBackedUp/install/jon124/jon-server-2.4.0-SNAPSHOT/logs/db-upgrade.xml]. Cause: /NotBackedUp/install/jon124/jon-server-2.4.0-SNAPSHOT/logs/db-upgrade.xml:27: Failed to upgrade - error in spec version [2.70.2]. Cause: Error executing the task [org.rhq.core.db.ant.dbupgrade.SST_JavaTask] in schema spec version [2.70.2]. Cause: Java Task Error: class=[java.lang.NullPointerException]
	at org.rhq.enterprise.installer.ServerInformation.startAnt(ServerInformation.java:781)
	at org.rhq.enterprise.installer.ServerInformation.upgradeExistingDatabaseSchema(ServerInformation.java:276)
	at org.rhq.enterprise.installer.ConfigurationBean.save(ConfigurationBean.java:723)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
Comment 1 Rajan Timaniya 2010-05-04 06:34:45 EDT
Created attachment 411244 [details]
rhq-installer-dbupgrade log
Comment 2 Rajan Timaniya 2010-05-04 06:36:47 EDT
Created attachment 411245 [details]
rhq-server-log4j log
Comment 3 Rajan Timaniya 2010-05-04 07:59:46 EDT
Fail upgrade JON 2.1 to JON 2.4 on Postgres(8.4.2) also.
Steps to Reproduce:
1) Install JON 2.1 on Postgres
2) Create some alerts, schedule, etc..  (basic operations)
3) Upgrade JON 2.1 to JON 2.4 (same database Postgres)
Comment 4 Charles Crouch 2010-05-04 08:51:00 EDT
Rajan, can you add some more details to this bug about exactly what alert you added to the 2.1 install? Especially alerts which triggered operations.

Joseph,
From  rhq-installer-dbupgrade log
Caused by: java.lang.NullPointerException
	at org.rhq.core.db.upgrade.CustomAlertSenderUpgradeTask.upgradeOperationNotifications(CustomAlertSenderUpgradeTask.java:142)
	at org.rhq.core.db.upgrade.CustomAlertSenderUpgradeTask.execute(CustomAlertSenderUpgradeTask.java:65)
	at org.rhq.core.db.ant.dbupgrade.SST_JavaTask.execute(SST_JavaTask.java:89)
Comment 5 Rajan Timaniya 2010-05-04 08:55:24 EDT
Created two alerts on JON 2.1
Alert -1: Created on platform, the alert condition is Free Memory > 0.1
Alert -2: Created on resource - Jboss, the alert condition is Free Memory > 0.1
Comment 6 Joseph Marques 2010-05-04 09:44:51 EDT
commit 474584f80d94f7ae934ed7515dccdd702bdb9384

fix issue when trying to upgrade alert notifications
    
* should only dbupgrade operation-based alert notifications for those alert definitions that previously had associated operations
Comment 7 Rajan Timaniya 2010-05-05 05:00:45 EDT
Description of problem:
Fail upgrade JON 2.1 to JON 2.4 on Oracle 10g.

Version-Release number of selected component (if applicable):
JON build #126  Revision: 10615 
http://hudson-qe.rhq.rdu.redhat.com:8080/job/jon/126/

How reproducible:
Always

Steps to Reproduce:
1) Install JON 2.1 on Oracle 10g
2) Create some alerts, schedule, etc..  (basic operations)
   (i) Alert:
       Alert -1: Created on platform, the alert condition is Free Memory > 0.1
       Alert -2: Created on JBoss AS server, the alert condition is Free Memory > 0.1
       Alert -3: Created on RHQ Agent, the alert condition is Free Memory > 0.1
   (ii) schedule:
        Created a View Process List schedule for platform
   (iii)Modified JBoss AS Server connection parameters like 'Principal' & 'Credentials'
3) Upgrade JON 2.1 to JON 2.4 (same database Oracle 10g)
4) Login to JON 2.4 and select platform or (any) resource


Actual results:
After upgrade of JON2.1 to JON2.4, there is no menu and it gives error on click of platform or resource.

Expected results:
Upgrade JON 2.1 to JON 2.4 should work without any error.

Additional info:
2010-05-05 12:48:47,748 INFO  [org.rhq.enterprise.server.core.plugin.ProductPluginDeployer] Deploying [17] new or updated agent plugins: [JBossAS5, RHQServer, JBossAS, IIS, Database, JBossESB5, Postgres, JBossCache3, Augeas, JMX, Platforms, JBossCache, Apache, Hibernate, Tomcat, JBossESB, RHQAgent]
2010-05-05 12:48:52,188 WARN  [org.hibernate.util.JDBCExceptionReporter] SQL Error: 60, SQLState: 61000
2010-05-05 12:48:52,214 ERROR [org.hibernate.util.JDBCExceptionReporter] ORA-00060: deadlock detected while waiting for resource

2010-05-05 12:48:52,214 WARN  [org.hibernate.util.JDBCExceptionReporter] SQL Error: 60, SQLState: 61000
2010-05-05 12:48:52,214 ERROR [org.hibernate.util.JDBCExceptionReporter] ORA-00060: deadlock detected while waiting for resource

2010-05-05 12:48:52,216 ERROR [org.hibernate.event.def.AbstractFlushingEventListener] Could not synchronize database state with session
org.hibernate.exception.LockAcquisitionException: Could not execute JDBC batch update
	at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:87)
	at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
	at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:254)
	at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:237)
	at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:146)
	at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
	at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
	at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
Comment 8 Rajan Timaniya 2010-05-05 05:01:58 EDT
Created attachment 411518 [details]
Dashboard without menu
Comment 9 Rajan Timaniya 2010-05-05 05:02:27 EDT
Created attachment 411519 [details]
Error on click of platform
Comment 10 Rajan Timaniya 2010-05-05 05:02:56 EDT
Created attachment 411520 [details]
Stack Trace on screen
Comment 11 Rajan Timaniya 2010-05-05 05:03:42 EDT
Created attachment 411521 [details]
rhq-server-log4j log - after JON2.1 to JON2.4
Comment 12 Rajan Timaniya 2010-05-05 10:39:54 EDT
Fail upgrade JON 2.1 to JON build # 126 on Postgres (8.4.2).

The steps to reproduce the bug is same as above and getting same error.

Actual results:
After upgrade of JON2.1 to JON build #126, there is no menu and it gives error on click of platform or resource.

Expected results:
Upgrade JON 2.1 to JON build #126 should work without any error.
Comment 13 John Mazzitelli 2010-05-05 10:42:06 EDT
I suspect this might have to do with some recent changes to that problem of needing to update/merging plugin configuration on plugin upgrade... since I see this in the root cause stack:

at org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerBean.updatePluginConfiguration(ResourceMetadataManagerBean.java:782)

I also see this, which is probably separate, but looks very wrong:

ERROR [org.rhq.enterprise.server.scheduler.jobs.AbstractStatefulJob] Failed to execute job [SavedSearchResultCountRecalculationJob] javax.ejb.EJBException: java.lang.IllegalArgumentException: Unknown name value for enum class org.rhq.core.domain.search.SearchSubsystem: Resource
Comment 14 Joseph Marques 2010-05-08 16:31:52 EDT
The following error:

ERROR [org.rhq.enterprise.server.scheduler.jobs.AbstractStatefulJob] Failed to
execute job [SavedSearchResultCountRecalculationJob] javax.ejb.EJBException:
java.lang.IllegalArgumentException: Unknown name value for enum class
org.rhq.core.domain.search.SearchSubsystem: Resource    

Tells me you're not using a current build.

-----

Presuming you're not using a current build, the error at:

org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerBean.updatePluginConfiguration(ResourceMetadataManagerBean.java:782

May already be fixed too.  I know jshaughn did some work in this area to fix some sporadic deadlocking issues on Oracle, so you should retest the upgrade with the latest available JON build and report back.
Comment 15 Rajan Timaniya 2010-05-18 05:39:44 EDT
Description of problem:
Upgrade JON 2.1 to JON 2.4 (JON build #152) gives error/exception, if user click on already created alert.

Version-Release number of selected component (if applicable):
JON build #152 - Revision: 10620
http://hudson-qe.rhq.rdu.redhat.com:8080/job/jon/152/

How reproducible:
Always

Steps to Reproduce:
1) Install JON 2.1 on Oracle 10g
2) Create some alerts, schedule, etc..  (basic operations)
   (i) Alert:
       Alert -1: Created on platform, the alert condition is Free Memory > 0.1
       Alert -2: Created on JBoss AS server, the alert condition is Free Memory
> 0.1
       Alert -3: Created on RHQ Agent, the alert condition is Free Memory > 0.1
   (ii) schedule:
        Created a View Process List schedule for platform
   (iii)Modified JBoss AS Server connection parameters like 'Principal' &
'Credentials'
3) Upgrade JON 2.1 to JON 2.4 (same database Oracle 10g)
4) Login to JON 2.4 and select platform or (any) resource
5) Click on already created alert

Actual results:
After upgrade of JON2.1 to JON2.4, click on already created alert give error/exception on UI.

javax.ejb.EJBException: org.hibernate.PropertyAccessException: Null value was assigned to a property of primitive type setter of org.rhq.core.domain.configuration.Configuration.version The page requested cannot be displayed due to some error.

Expected results:
Upgrade JON 2.1 to JON 2.4 should work without any error.

Additional info:
org.hibernate.PropertyAccessException: Null value was assigned to a property of primitive type setter of org.rhq.core.domain.configuration.Configuration.version
javax.ejb.EJBException: org.hibernate.PropertyAccessException: Null value was assigned to a property of primitive type setter of org.rhq.core.domain.configuration.Configuration.version at org.jboss.ejb3.tx.Ejb3TxPolicy.handleExceptionInOurTx(Ejb3TxPolicy.java:63) at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:83) at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:95) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77) at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:240) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:210) at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:84) at $Proxy323.getAlertDefinitionById(Unknown Source) at org.rhq.enterprise.gui.legacy.action.resource.common.monitor.alerts.AlertDefUtil.getAlertDefinition(AlertDefUtil.java:226) at
...



Please refer attached screenshot for error/excepton.
Comment 16 Rajan Timaniya 2010-05-18 05:40:39 EDT
Created attachment 414786 [details]
UI Error
Comment 17 Rajan Timaniya 2010-05-18 05:41:05 EDT
Created attachment 414787 [details]
UI error stack trace
Comment 18 Rajan Timaniya 2010-05-25 07:13:18 EDT
Fond the same issue as mentioned in comment-15 (https://bugzilla.redhat.com/show_bug.cgi?id=588700#c15) for upgrade JON2.3 to JON2.4  with Postgres 8.4 (JON build #166).
Comment 19 Charles Crouch 2010-05-25 10:03:36 EDT
Updated title based on Rajan's last comment
Comment 20 Joseph Marques 2010-05-25 10:33:01 EDT
Strange how I never saw this error during my upgrade testing.  Quick analysis shows that Configuration.version is mapped to a primitive 'long', which means that it must be non-null in the database otherwise you'll get loading errors.

The class that upgrades the alert notifications, turning them from first-class constructs into de-typed / string-i-fied configuration objects is CustomAlertSenderUpgradeTask in the dbutils module.  Whoever works on this will want to edit the private method getInsertConfigSQL(int) to also persist the version (with a default value of 0?) when inserting the configs.
Comment 21 Joseph Marques 2010-05-25 10:34:04 EDT
FYI, this error will occur when upgrade from any current version of JON 2.x to the latest version of JON 2.4
Comment 22 John Sanda 2010-05-25 21:47:34 EDT
Updated CustomAlertSenderUpgradeTask.getInsertConfigSQL as per suggested in comment 20 and retested to verify that I no longer get the PropertyAccessException.

commit hash: 406825bb2a16302184f78dc7d0c032d22728df9a
Comment 23 Rajan Timaniya 2010-05-31 05:28:03 EDT
Verified on JON build #177 - Revision:  10621

Upgrade JON 2.1 to JON 2.4 with Oracle 10g
Upgrade JON 2.3.1 to JON 2.4 with Oracle 10g
Upgrade JON 2.1 to JON 2.4 with Postgres 8.3
Upgrade JON 2.3.1 to JON 2.4 with Postgres 8.4

Observed that there isn't any error/exception like - 
javax.ejb.EJBException: org.hibernate.PropertyAccessException: Null value was
assigned to a property of primitive type setter of
org.rhq.core.domain.configuration.Configuration.version The page requested
cannot be displayed due to some error.
Comment 24 Corey Welton 2010-08-12 12:55:35 EDT
Mass-closure of verified bugs against JON.

Note You need to log in before you can comment on or make changes to this bug.