Bug 600273

Summary: Upgrade JON2.3.1 to JON2.4Beta0 gives exception in server log- "rhq_resource_type" violates foreign key constraint "rhq_resource_group_resource_type_id_fkey" on table "rhq_resource_group"
Product: [Other] RHQ Project Reporter: Rajan Timaniya <rtimaniy>
Component: Core ServerAssignee: Ian Springer <ian.springer>
Status: CLOSED CURRENTRELEASE QA Contact: Rajan Timaniya <rtimaniy>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.0.0CC: ccrouch, ian.springer
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: 2.4 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-08-12 16:47:02 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 565628, 591531    
Attachments:
Description Flags
server log
none
mix group screenshot
none
group definiation screenshot none

Description Rajan Timaniya 2010-06-04 10:34:45 UTC
Created attachment 421191 [details]
server log

Description of problem:
Upgrade JON2.3.1 to JON2.4Beta0 gives exception in server log:
2010-06-04 14:43:30,316 INFO  [org.rhq.enterprise.server.resource.ResourceManagerBean] User [org.rhq.core.domain.auth.Subject[id=1,name=admin]] is marking resource [Resource[id=10127, type=RHQ Server Content Source Server Plugin Subsystem, key=rhq.serverplugin:service=ContentSourcePluginService, name=RHQ Server Content Source Server Plugin Subsystem, parent=rajantest RHQ Server,  JBoss AS 4.2.3.GA default (0.0.0.0:2099)]] for asynchronous deletion
2010-06-04 14:43:30,737 WARN  [org.hibernate.util.JDBCExceptionReporter] SQL Error: 0, SQLState: 23503
2010-06-04 14:43:30,737 ERROR [org.hibernate.util.JDBCExceptionReporter] Batch entry 0 delete from RHQ_RESOURCE_TYPE where ID='10028' was aborted.  Call getNextException to see the cause.
2010-06-04 14:43:30,737 WARN  [org.hibernate.util.JDBCExceptionReporter] SQL Error: 0, SQLState: 23503
2010-06-04 14:43:30,737 ERROR [org.hibernate.util.JDBCExceptionReporter] ERROR: update or delete on table "rhq_resource_type" violates foreign key constraint "rhq_resource_group_resource_type_id_fkey" on table "rhq_resource_group"
  Detail: Key (id)=(10028) is still referenced from table "rhq_resource_group".
2010-06-04 14:43:30,738 ERROR [org.hibernate.event.def.AbstractFlushingEventListener] Could not synchronize database state with session
org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
	at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:71)
	at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
	at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:254)
	at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:93)
	at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:88)
	at org.hibernate.jdbc.AbstractBatcher.prepareBatchStatement(AbstractBatcher.java:223)

Version-Release number of selected component (if applicable):
JON 2.4.0.Beta0  (http://jon01.qa.atl2.redhat.com:8042/dist/qa/2.4.0/)

Steps to Reproduce:
1) Install JON 2.3.1 with Postgres 8.3
2) Install one more JON 2.3.1 server with same database  (High Availability)
3) Install/configure 2 rhq-agests (one agent per server)
4) Perform operations on JON 2.3.1
 (i) Create alerts for platform, JBoss AS4, RHQ Agent, Compatible group (alert
condition is Free Memory > 0.1 OR Total Memory < 999999)
 (ii) Create groups (Mix, Compatible, Group defination)
 (iii)Content repository syncronization (for JBoss Patch)
 (iv) Create Roles and Users
 (v) Change INVENTORY -> CONNECTION of JBoss AS4 
 (vi)Change configuration values of 'RHQ Agent' and create 'History'
 (vii)Perform LDAP settings
 (viii)Create 'SCHEDULES' for 'View Process List' on platform
 (ix) Enabled JBossAS events
5) Stop/Down all JON2.3.1 servers
6) Keep all agents running
7) Upgrade one server from JON2.3.1 to JON 2.4 Beta0 (with same database - Postgrace 8.3)
8) Please refer server log
  
Actual results:
Upgrade JON2.3.1 to JON2.4Beta0 gives exception in server log- "rhq_resource_type" violates foreign key constraint "rhq_resource_group_resource_type_id_fkey" on table "rhq_resource_group"

Expected results:
There should upgrade JON2.3.1 to JON2.4 without any error/exception.

Additional info:
Please refer attached server log.

Comment 1 Rajan Timaniya 2010-06-04 10:54:47 UTC
Test environment:
OS - RHEL 5.4
JDK - SUN JDK 1.6.0_18-ea
Database - Postgres 8.3

Comment 2 Rajan Timaniya 2010-06-04 14:26:15 UTC
Groups details:
1) Created one compatible group  and added resource 'JBoss AS 4.2.3.GA'. 
	- Created alert for this group and alert condition is 'JVM Free Memory > 0.1B'

2) Created mix group and added resources to it, please refer attached screenshot (mix group screenshot) for added resources.

3) Created group definition and created dynagroup, please refer attached screeshot (group definition screenshot)

Comment 3 Rajan Timaniya 2010-06-04 14:27:23 UTC
Created attachment 421241 [details]
mix group screenshot

Comment 4 Rajan Timaniya 2010-06-04 14:28:03 UTC
Created attachment 421242 [details]
group definiation screenshot

Comment 5 Ian Springer 2010-06-04 23:38:33 UTC
Commit 676d4d3c37a104b42d1a9c150f08e1222c7a3cfb fixes this by making the following changes to ResourceMetadataManagerBean.removeObsoleteTypesInNewTransaction():

1) Obsolete Resource types are removed in reverse hierarchical order (that is, the leafs are removed first, followed by parents, followed by grandparents, etc.)

2) Any compatible groups with a member type of a type being removed are themselves removed.

3) If an exception occurs anywhere in the method, it is caught, an error is logged, and it is not rethrown. This prevents such an exception from causing the outer transaction (the call to registerPlugin()) from being rolled back, which would cause the entire metadata update to fail.

Pushed to master.

Comment 6 Rajan Timaniya 2010-06-07 12:03:19 UTC
Verified on JON 2.4 Beta1

Steps
1) Install JON 2.3.1 with Oracle 10g (High Availability - 2 servers and 3
agents)
2) Perform operations on JON 2.3.1
 (i) Create alerts for platform, JBoss AS4, RHQ Agent, Compatible group (alert
condition is Free Memory > 0.1 OR Total Memory > 0.1)
 (ii) Create groups (Mix, Compatible, Group defination)
 (iii)Content repository syncronization (for JBoss Patch)
 (iv) Create Roles and Users
 (v) Change INVENTORY -> CONNECTION of JBoss AS4 
 (vi)Change configuration values of 'RHQ Agent' and create 'History'
 (vii)Perform LDAP settings
 (viii)Create 'SCHEDULES' for 'View Process List' on platform
 (ix) Enabled JBossAS events
3) Stop/Down all JON2.3.1 servers
4) Keep all agents running
5) Upgrade one server from JON2.3.1 to JON 2.4 Beta1 (with same database -
Oracle 10g)
6) Please refer server log

There is no error/exception observed with Oracle 10g:
"rhq_resource_type" violates foreign key constraint
"rhq_resource_group_resource_type_id_fkey" on table "rhq_resource_group"


I will test this bug with Postgres database.

Comment 7 Rajan Timaniya 2010-06-08 10:42:02 UTC
Verified on JON 2.4 Beta1

Steps
1) Install JON 2.3.1 with Postgres 8.3/8.4 (High Availability - 2 servers and 2
agents)
2) Perform operations on JON 2.3.1
 (i) Create alerts for platform, JBoss AS4, RHQ Agent, Compatible group (alert condition is Free Memory > 0.1 OR Total Memory > 0.1)
 (ii) Create groups (Mix, Compatible, Group defination)
 (iii)Content repository syncronization (for JBoss Patch)
 (iv) Create Roles and Users
 (v) Change INVENTORY -> CONNECTION of JBoss AS4 
 (vi)Change configuration values of 'RHQ Agent' and create 'History'
 (vii)Perform LDAP settings
 (viii)Create 'SCHEDULES' for 'View Process List' on platform
 (ix) Enabled JBossAS events
3) Stop/Down all JON2.3.1 servers
4) Keep all agents running
5) Upgrade one server from JON2.3.1 to JON 2.4 Beta1 (with same database - Postgres 8.3/8.4)
6) Please refer server log

There is no error/exception observed with Postgres 8.3/8.4:
"rhq_resource_type" violates foreign key constraint
"rhq_resource_group_resource_type_id_fkey" on table "rhq_resource_group"

Comment 8 Corey Welton 2010-08-12 16:47:02 UTC
Mass-closure of verified bugs against JON.