Created attachment 433128 [details] server log Description of problem: Upgrade JON2.3.1 to JON2.4 with Oracle gives SQLGrammarException: Could not execute JDBC batch update 2010-07-20 15:58:20,339 INFO [org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerBean] Removing type [JBossAS5:Queue(id=10072)] from parent type [JBossAS5:JBossAS Server(id=10026)]... 2010-07-20 15:58:24,744 WARN [org.hibernate.util.JDBCExceptionReporter] SQL Error: 2049, SQLState: 42000 2010-07-20 15:58:24,744 ERROR [org.hibernate.util.JDBCExceptionReporter] ORA-02049: timeout: distributed transaction waiting for lock 2010-07-20 15:58:24,744 WARN [org.hibernate.util.JDBCExceptionReporter] SQL Error: 2049, SQLState: 42000 2010-07-20 15:58:24,744 ERROR [org.hibernate.util.JDBCExceptionReporter] ORA-02049: timeout: distributed transaction waiting for lock 2010-07-20 15:58:24,745 ERROR [org.hibernate.event.def.AbstractFlushingEventListener] Could not synchronize database state with session org.hibernate.exception.SQLGrammarException: Could not execute JDBC batch update at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:67) at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43) ... Version-Release number of selected component (if applicable): version: 2.4.0.GA_QA (build #70) build number: 10856:52c274a How reproducible: Always Steps to Reproduce: 1) Install JON 2.3.1 and agent with Oracle 10g 2) Perform operations on JON 2.3.1 (i) Create alerts: (a) Name: Platform_alert_test [Type: Linux (Platform)] If Condition: Free Memory > 0.1B Dampening Rule: Each time condition set is true Action Filters: Disable alert until re-enabled manually or by recovery alert: false Notify Other Recipients: Eamil - rtimaniy (b)Name: JBOSSAS_JVM_ALERT [Type: JBoss AS JVM (Server)] If Condition: Availability goes DOWN Dampening Rule: Each time condition set is true Action Filters: Disable alert until re-enabled manually or by recovery alert : false Notify Other Recipients: Eamil - rtimaniy (c)Name:RHQ_Agent_JVM_Active_Threads_Alert [Type: RHQ Agent(Server)] If Condition: JVM Active Threads > -1.0 Dampening Rule: Each time condition set is true Action Filters: Disable alert until re-enabled manually or by recovery alert: false Notify Other Recipients: Eamil - rtimaniy Notify Roles (All Resources Role, Super User Role) Notify JON Users (username - rhqadmin) (ii) Create groups (Mix, Compatible, Group defination) (iii)Content repository syncronization (for JBoss Patch) (iv) Create Roles and Users (v) Change INVENTORY -> CONNECTION of JBoss AS4 (vi)Change configuration values of 'RHQ Agent' and create 'History' (vii)Perform LDAP settings (viii)Create 'SCHEDULES' for 'View Process List' on platform (ix) Enabled JBossAS events 5) Stop/Down all JON2.3.1 servers 6) Keep all agents running 7) Upgrade one server from JON2.3.1 to JON 2.4 (build #70) (with same database - Oracle 10g) 8) Please refer server log Actual results: Upgrade JON2.3.1 to JON2.4 with Oracle gives SQLGrammarException: Could not execute JDBC batch update Expected results: Upgrade JON2.3.1 to JON2.4 should not give error. Additional info: Please refer log file
This has nothing to do with SQL Grammar itself, but comes from Tx timeouts, as seen in the log at 2010-07-20 15:58:24,744 WARN [org.hibernate.util.JDBCExceptionReporter] SQL Error: 2049, SQLState: 42000 2010-07-20 15:58:24,744 ERROR [org.hibernate.util.JDBCExceptionReporter] ORA-02049: timeout: distributed transaction waiting for lock The upgrade process tries to delete an obsolete resource type which is probably still locked for update by the plugin update Tx 2010-07-20 15:58:24,745 ERROR [org.hibernate.event.def.AbstractFlushingEventListener] Could not synchronize database state with session org.hibernate.exception.SQLGrammarException: Could not execute JDBC batch update at org.jboss.ejb3.entity.TransactionScopedEntityManager.flush(TransactionScopedEntityManager.java:211) at org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerBean.removeResourceType(ResourceMetadataManagerBean.java:605) at org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerBean.removeResourceTypes(ResourceMetadataManagerBean.java:526) at org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerBean.removeObsoleteTypesInNewTransaction(ResourceMetadataManagerBean.java:489) at $Proxy383.removeObsoleteTypesInNewTransaction(Unknown Source) at org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerBean.registerPlugin(ResourceMetadataManagerBean.java:301) at $Proxy383.registerPlugin(Unknown Source) at org.rhq.enterprise.server.core.plugin.ProductPluginDeployer.registerPluginJar(ProductPluginDeployer.java:504) at org.rhq.enterprise.server.core.plugin.ProductPluginDeployer.access$000(ProductPluginDeployer.java:56) at org.rhq.enterprise.server.core.plugin.ProductPluginDeployer$LatchedPluginDeploymentService.executeService(ProductPluginDeployer.java:613) at org.rhq.enterprise.server.core.concurrency.LatchedServiceController$LatchedService.run(LatchedServiceController.java:266)
Steps to Reproduce: 1) Install JON 2.3.1 and agent with Oracle 10g 2) Perform operations on JON 2.3.1 (i) Create alerts: (a) Name: Platform_alert_test [Type: Linux (Platform)] If Condition: Free Memory > 0.1B Dampening Rule: Each time condition set is true Action Filters: Disable alert until re-enabled manually or by recovery alert: false Notify Other Recipients: Eamil - rtimaniy (b)Name: JBOSSAS_JVM_ALERT [Type: JBoss AS JVM (Server)] If Condition: Availability goes DOWN Dampening Rule: Each time condition set is true Action Filters: Disable alert until re-enabled manually or by recovery alert : false Notify Other Recipients: Eamil - rtimaniy (c)Name:RHQ_Agent_JVM_Active_Threads_Alert [Type: RHQ Agent(Server)] If Condition: JVM Active Threads > -1.0 Dampening Rule: Each time condition set is true Action Filters: Disable alert until re-enabled manually or by recovery alert: false Notify Other Recipients: Eamil - rtimaniy Notify Roles (All Resources Role, Super User Role) Notify JON Users (username - rhqadmin) (ii) Create groups (Mix, Compatible, Group definition) (a) Compatible group CompatibleGroup Linux Platforms GroupTest CPU Platforms (b) Mix group Name: Mix_Group_Recursive - Type: Mixed Types CPU 3 rajantest Service CPU Intel Xeon CacheConsistencyManagerBean rajantest RHQ Server, JBoss AS 4.2.3.GA default (0.0.0.0:2099) Service EJB3 Session Bean An EJB3 Session Bean Compilation JBoss AS JVM Service VM Compilation System Compilation RHQ Agent JVM Service VM Compilation System (c) Group Definition Name: GroupDef Group Definition Conditions: resource.type.plugin = JBossAS resource.type.name = ConnectionFactory (iii)Content repository syncronization (for JBoss Patch) (iv) Create Roles and Users Created 'Role1' with - no Permissions - created and assigned user 'rajan1' Created 'Role2' with - Permissions for 'Create Children' & 'Measure' - created and assigned user 'rajan2' Created 'Role3' with - All Permissions - created and assigned user 'rajan3' Created user 'Rajan' and assigned all roles to it (v) Change INVENTORY -> CONNECTION of JBoss AS4 (vi)Change configuration values of 'RHQ Agent' and create 'History' (vii)Perform LDAP settings Configured LDAP Configuration Properties with SSL (viii)Create 'SCHEDULES' for 'View Process List' on platform (ix) Enabled JBossAS events 5) Stop/Down all JON2.3.1 servers 6) Keep all agents running 7) Upgrade one server from JON2.3.1 to JON 2.4 (build #70) (with same database - Oracle 10g) 8) Please refer server log
*** Bug 603903 has been marked as a duplicate of this bug. ***
Fix commit: d454338af878eb9490814548b665e567abcc0a21 Generated a 2.3.1 DB populated with a variety of resources including rhq server/agent, EAP5 server, Tomcat Server. Populated with alert templates, alerts, events, dyna group defs, recursive groups, group op history, group op schedules, and group res config updates. With this data was able to reproduce consistently (2 for 2) locking issues at plugin registration time, during the upgrade to rhq 3.0.0 (jon 2.4). It may be due (imo) to the fact that obsolete resource type removal invokes resource bulk delete, which hits a lot of tables, while other plugins are being registered. But, it may also just be the meta-data tables/indexes themselves. By reducing the plugin deployer threads to 1 (making plugin update serial) we avoid the problem, although server startup time suffers.
After Upgrade JON2.3.1 to JON2.4 with Oracle 10g there is no SQLGrammarException in server logs. There was only this error : [schemaSpec] Error executing the task [org.rhq.core.db.ant.dbupgrade.SST_AlterColumn] in schema spec version [2.89]. Cause: Failed to alter column. Cause: java.sql.SQLException: ORA-01442: column to be modified to NOT NULL is already NOT NULL which I expect to be harmless because the upgrade was finished successfully.
yes, those log messages are harmless. some tasks may optionally fail given the state of the db being upgraded. As long as it completes with SUCCESS it is fine.
Upgrade JON2.3.1 to JON2.4 Verified on Oracle 10g. No error messages.
Verified on JON 2.4 (build #73) version: 2.4.0.GA_QA build number: 10868:d9790b0 Upgrade JON2.3.1 to JON2.4 (build #73) on Oracle 10g with a variety of resources including rhq server/agent, EAP5 server, Tomcat Server, postgres, apache. Populated with alert templates,alerts, events, dyna group defs, recursive groups, group op history, group op schedules, and group res config updates. No error messages in upgrade JON2.3.1 to JON2.4 on Oracle 10g.
Mass-closure of verified bugs against JON.