Bug 588700

Created attachment 411244 [details]
rhq-installer-dbupgrade log

Created attachment 411245 [details]
rhq-server-log4j log

Fail upgrade JON 2.1 to JON 2.4 on Postgres(8.4.2) also.
Steps to Reproduce:
1) Install JON 2.1 on Postgres
2) Create some alerts, schedule, etc..  (basic operations)
3) Upgrade JON 2.1 to JON 2.4 (same database Postgres)

Rajan, can you add some more details to this bug about exactly what alert you added to the 2.1 install? Especially alerts which triggered operations.

Joseph,
From  rhq-installer-dbupgrade log
Caused by: java.lang.NullPointerException
	at org.rhq.core.db.upgrade.CustomAlertSenderUpgradeTask.upgradeOperationNotifications(CustomAlertSenderUpgradeTask.java:142)
	at org.rhq.core.db.upgrade.CustomAlertSenderUpgradeTask.execute(CustomAlertSenderUpgradeTask.java:65)
	at org.rhq.core.db.ant.dbupgrade.SST_JavaTask.execute(SST_JavaTask.java:89)

Created two alerts on JON 2.1
Alert -1: Created on platform, the alert condition is Free Memory > 0.1
Alert -2: Created on resource - Jboss, the alert condition is Free Memory > 0.1

commit 474584f80d94f7ae934ed7515dccdd702bdb9384

fix issue when trying to upgrade alert notifications
    
* should only dbupgrade operation-based alert notifications for those alert definitions that previously had associated operations

Description of problem:
Fail upgrade JON 2.1 to JON 2.4 on Oracle 10g.

Version-Release number of selected component (if applicable):
JON build #126  Revision: 10615 
http://hudson-qe.rhq.rdu.redhat.com:8080/job/jon/126/

How reproducible:
Always

Steps to Reproduce:
1) Install JON 2.1 on Oracle 10g
2) Create some alerts, schedule, etc..  (basic operations)
   (i) Alert:
       Alert -1: Created on platform, the alert condition is Free Memory > 0.1
       Alert -2: Created on JBoss AS server, the alert condition is Free Memory > 0.1
       Alert -3: Created on RHQ Agent, the alert condition is Free Memory > 0.1
   (ii) schedule:
        Created a View Process List schedule for platform
   (iii)Modified JBoss AS Server connection parameters like 'Principal' & 'Credentials'
3) Upgrade JON 2.1 to JON 2.4 (same database Oracle 10g)
4) Login to JON 2.4 and select platform or (any) resource


Actual results:
After upgrade of JON2.1 to JON2.4, there is no menu and it gives error on click of platform or resource.

Expected results:
Upgrade JON 2.1 to JON 2.4 should work without any error.

Additional info:
2010-05-05 12:48:47,748 INFO  [org.rhq.enterprise.server.core.plugin.ProductPluginDeployer] Deploying [17] new or updated agent plugins: [JBossAS5, RHQServer, JBossAS, IIS, Database, JBossESB5, Postgres, JBossCache3, Augeas, JMX, Platforms, JBossCache, Apache, Hibernate, Tomcat, JBossESB, RHQAgent]
2010-05-05 12:48:52,188 WARN  [org.hibernate.util.JDBCExceptionReporter] SQL Error: 60, SQLState: 61000
2010-05-05 12:48:52,214 ERROR [org.hibernate.util.JDBCExceptionReporter] ORA-00060: deadlock detected while waiting for resource

2010-05-05 12:48:52,214 WARN  [org.hibernate.util.JDBCExceptionReporter] SQL Error: 60, SQLState: 61000
2010-05-05 12:48:52,214 ERROR [org.hibernate.util.JDBCExceptionReporter] ORA-00060: deadlock detected while waiting for resource

2010-05-05 12:48:52,216 ERROR [org.hibernate.event.def.AbstractFlushingEventListener] Could not synchronize database state with session
org.hibernate.exception.LockAcquisitionException: Could not execute JDBC batch update
	at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:87)
	at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
	at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:254)
	at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:237)
	at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:146)
	at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
	at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
	at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)

Created attachment 411518 [details]
Dashboard without menu

Created attachment 411519 [details]
Error on click of platform

Created attachment 411520 [details]
Stack Trace on screen

Created attachment 411521 [details]
rhq-server-log4j log - after JON2.1 to JON2.4

Fail upgrade JON 2.1 to JON build # 126 on Postgres (8.4.2).

The steps to reproduce the bug is same as above and getting same error.

Actual results:
After upgrade of JON2.1 to JON build #126, there is no menu and it gives error on click of platform or resource.

Expected results:
Upgrade JON 2.1 to JON build #126 should work without any error.

I suspect this might have to do with some recent changes to that problem of needing to update/merging plugin configuration on plugin upgrade... since I see this in the root cause stack:

at org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerBean.updatePluginConfiguration(ResourceMetadataManagerBean.java:782)

I also see this, which is probably separate, but looks very wrong:

ERROR [org.rhq.enterprise.server.scheduler.jobs.AbstractStatefulJob] Failed to execute job [SavedSearchResultCountRecalculationJob] javax.ejb.EJBException: java.lang.IllegalArgumentException: Unknown name value for enum class org.rhq.core.domain.search.SearchSubsystem: Resource

The following error:

ERROR [org.rhq.enterprise.server.scheduler.jobs.AbstractStatefulJob] Failed to
execute job [SavedSearchResultCountRecalculationJob] javax.ejb.EJBException:
java.lang.IllegalArgumentException: Unknown name value for enum class
org.rhq.core.domain.search.SearchSubsystem: Resource    

Tells me you're not using a current build.

-----

Presuming you're not using a current build, the error at:

org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerBean.updatePluginConfiguration(ResourceMetadataManagerBean.java:782

May already be fixed too.  I know jshaughn did some work in this area to fix some sporadic deadlocking issues on Oracle, so you should retest the upgrade with the latest available JON build and report back.

Description of problem:
Upgrade JON 2.1 to JON 2.4 (JON build #152) gives error/exception, if user click on already created alert.

Version-Release number of selected component (if applicable):
JON build #152 - Revision: 10620
http://hudson-qe.rhq.rdu.redhat.com:8080/job/jon/152/

How reproducible:
Always

Steps to Reproduce:
1) Install JON 2.1 on Oracle 10g
2) Create some alerts, schedule, etc..  (basic operations)
   (i) Alert:
       Alert -1: Created on platform, the alert condition is Free Memory > 0.1
       Alert -2: Created on JBoss AS server, the alert condition is Free Memory
> 0.1
       Alert -3: Created on RHQ Agent, the alert condition is Free Memory > 0.1
   (ii) schedule:
        Created a View Process List schedule for platform
   (iii)Modified JBoss AS Server connection parameters like 'Principal' &
'Credentials'
3) Upgrade JON 2.1 to JON 2.4 (same database Oracle 10g)
4) Login to JON 2.4 and select platform or (any) resource
5) Click on already created alert

Actual results:
After upgrade of JON2.1 to JON2.4, click on already created alert give error/exception on UI.

javax.ejb.EJBException: org.hibernate.PropertyAccessException: Null value was assigned to a property of primitive type setter of org.rhq.core.domain.configuration.Configuration.version The page requested cannot be displayed due to some error.

Expected results:
Upgrade JON 2.1 to JON 2.4 should work without any error.

Additional info:
org.hibernate.PropertyAccessException: Null value was assigned to a property of primitive type setter of org.rhq.core.domain.configuration.Configuration.version
javax.ejb.EJBException: org.hibernate.PropertyAccessException: Null value was assigned to a property of primitive type setter of org.rhq.core.domain.configuration.Configuration.version at org.jboss.ejb3.tx.Ejb3TxPolicy.handleExceptionInOurTx(Ejb3TxPolicy.java:63) at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:83) at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:191) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:95) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77) at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:240) at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:210) at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:84) at $Proxy323.getAlertDefinitionById(Unknown Source) at org.rhq.enterprise.gui.legacy.action.resource.common.monitor.alerts.AlertDefUtil.getAlertDefinition(AlertDefUtil.java:226) at
...



Please refer attached screenshot for error/excepton.

Created attachment 414786 [details]
UI Error

Created attachment 414787 [details]
UI error stack trace

Fond the same issue as mentioned in comment-15 (https://bugzilla.redhat.com/show_bug.cgi?id=588700#c15) for upgrade JON2.3 to JON2.4  with Postgres 8.4 (JON build #166).

Updated title based on Rajan's last comment

Strange how I never saw this error during my upgrade testing.  Quick analysis shows that Configuration.version is mapped to a primitive 'long', which means that it must be non-null in the database otherwise you'll get loading errors.

The class that upgrades the alert notifications, turning them from first-class constructs into de-typed / string-i-fied configuration objects is CustomAlertSenderUpgradeTask in the dbutils module.  Whoever works on this will want to edit the private method getInsertConfigSQL(int) to also persist the version (with a default value of 0?) when inserting the configs.

FYI, this error will occur when upgrade from any current version of JON 2.x to the latest version of JON 2.4

Updated CustomAlertSenderUpgradeTask.getInsertConfigSQL as per suggested in comment 20 and retested to verify that I no longer get the PropertyAccessException.

commit hash: 406825bb2a16302184f78dc7d0c032d22728df9a

Verified on JON build #177 - Revision:  10621

Upgrade JON 2.1 to JON 2.4 with Oracle 10g
Upgrade JON 2.3.1 to JON 2.4 with Oracle 10g
Upgrade JON 2.1 to JON 2.4 with Postgres 8.3
Upgrade JON 2.3.1 to JON 2.4 with Postgres 8.4

Observed that there isn't any error/exception like - 
javax.ejb.EJBException: org.hibernate.PropertyAccessException: Null value was
assigned to a property of primitive type setter of
org.rhq.core.domain.configuration.Configuration.version The page requested
cannot be displayed due to some error.

Mass-closure of verified bugs against JON.