Bug 1200579

Description of problem:
It is no longer possible to remove an agent plug-in from a JBoss ON system. Although no error is reported when the request is submitted to delete the plug-in, the PurgeResourceTypesJob fails to execute resulting in all the resource types defined by the plug-in to be deleted remaining around forever.

Every 5 minutes, the server logs the failure to server.log:

    INFO  [org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl] (RHQScheduler_Worker-2) HHH000010: On release of batch it still contained JDBC statements
    WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) SQL Error: 0, SQLState: 23503
    ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) Batch entry 0 delete from RHQ_RESOURCE_TYPE where ID=10097 was aborted.  Call getNextException to see the cause.
    WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) SQL Error: 0, SQLState: 23503
    ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) ERROR: update or delete on table "rhq_resource_type" violates foreign key constraint "rhq_bundle_type_target_map_resource_type_id_fkey" on table "rhq_bundle_type_target_map"
      Detail: Key (id)=(10097) is still referenced from table "rhq_bundle_type_target_map".
    ERROR [org.hibernate.engine.jdbc.batch.internal.BatchingBatch] (RHQScheduler_Worker-2) HHH000315: Exception executing batch [could not execute batch]
    ERROR [org.jboss.as.ejb3] (RHQScheduler_Worker-2) javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.ConstraintViolationException: could not execute batch
    ERROR [org.jboss.as.ejb3.invocation] (RHQScheduler_Worker-2) JBAS014134: EJB Invocation failed on component ResourceMetadataManagerBean for method public abstract void org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerLocal.completeRemoveResourceType(org.rhq.core.domain.auth.Subject,org.rhq.core.domain.resource.ResourceType): javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.ConstraintViolationException: could not execute batch
    ERROR [org.jboss.as.ejb3.invocation] (RHQScheduler_Worker-2) JBAS014134: EJB Invocation failed on component InventoryManagerBean for method public abstract void org.rhq.enterprise.server.inventory.InventoryManagerLocal.purgeDeletedResourceType(org.rhq.core.domain.resource.ResourceType): javax.ejb.EJBException: java.lang.RuntimeException: Failed to purge resource types
    ERROR [org.rhq.enterprise.server.scheduler.jobs.AbstractStatefulJob] (RHQScheduler_Worker-2) Failed to execute job [PurgeResourceTypesJob]: javax.ejb.EJBException: java.lang.RuntimeException: Failed to purge resource types
    Caused by: java.lang.RuntimeException: Failed to purge resource types
    Caused by: javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.ConstraintViolationException: could not execute batch
    Caused by: javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not execute batch
    Caused by: org.hibernate.exception.ConstraintViolationException: could not execute batch
    Caused by: java.sql.BatchUpdateException: Batch entry 0 delete from RHQ_RESOURCE_TYPE where ID=10097 was aborted.  Call getNextException to see the cause.

See attached log.

Version-Release number of selected component (if applicable):
3.3.0

How reproducible:
Always

Steps to Reproduce:
1.  Install JBoss ON 3.3 server and storage node but only install JBossAS7 plug-in from JBoss EAP plug-in pack and the default plug-ins.
2.  From _agent plug-ins configuration administration_ page, delete the plug-in _JBoss Application Server 7.x_ plug-in.
3.  Wait for at least 5 minutes.

Actual results:
Within 5 minutes of plug-in deletion, server.log will begin filling up with errors every 5 minutes:

    20:36:45,770 INFO  [org.rhq.coregui.server.gwt.PluginGWTServiceImpl] (http-/0.0.0.0:7080-1) Deleted agent plugins: [JBoss Application Server 7.x]
    20:36:45,779 INFO  [org.rhq.enterprise.server.core.plugin.AgentPluginScanner] (http-/0.0.0.0:7080-1) Plugin file [/opt/jboss/on/jon-server-3.3.0.GA/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-downloads/rhq-plugins/rhq-jboss-as-7-plugin-4.12.0.JON330GA.jar] has been deleted from the file system.

... wait up to 5 minutes ...

    20:39:30,938 INFO  [org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl] (RHQScheduler_Worker-2) HHH000010: On release of batch it still contained JDBC statements
    20:39:30,940 WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) SQL Error: 0, SQLState: 23503
    20:39:30,940 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) Batch entry 0 delete from RHQ_RESOURCE_TYPE where ID=10097 was aborted.  Call getNextException to see the cause.
    20:39:30,940 WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) SQL Error: 0, SQLState: 23503
    20:39:30,940 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) ERROR: update or delete on table "rhq_resource_type" violates foreign key constraint "rhq_bundle_type_target_map_resource_type_id_fkey" on table "rhq_bundle_type_target_map"
      Detail: Key (id)=(10097) is still referenced from table "rhq_bundle_type_target_map".
    20:39:30,941 ERROR [org.hibernate.engine.jdbc.batch.internal.BatchingBatch] (RHQScheduler_Worker-2) HHH000315: Exception executing batch [could not execute batch]
    20:39:30,943 ERROR [org.jboss.as.ejb3] (RHQScheduler_Worker-2) javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.ConstraintViolationException: could not execute batch
    20:39:30,943 ERROR [org.jboss.as.ejb3.invocation] (RHQScheduler_Worker-2) JBAS014134: EJB Invocation failed on component ResourceMetadataManagerBean for method public abstract void org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerLocal.completeRemoveResourceType(org.rhq.core.domain.auth.Subject,org.rhq.core.domain.resource.ResourceType): javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.ConstraintViolationException: could not execute batch

Plug-in remains in database in a deleted state along with all of its types also in a deleted state:

    SELECT      rhq_plugin.id, rhq_plugin.name, rhq_plugin.display_name, 
                rhq_resource_type.id, rhq_resource_type.name, 
                rhq_resource_type.deleted 
    FROM        rhq_plugin
    LEFT JOIN   rhq_resource_type 
    ON          rhq_resource_type.plugin = rhq_plugin.name
    WHERE       rhq_plugin.status = 'DELETED';

Expected results:
No errors after 5 minutes and plug-in and its resource types should no longer be in the database.

Additional info:
In this instance, the problem is the AS7 plug-in defines a bunlde type target map and holds a reference to the resource_type in rhq_bundle_type_target_map:

    SELECT  * FROM rhq_bundle_type_target_map
    WHERE   resource_type_id IN (
                SELECT      rhq_resource_type.id 
                FROM        rhq_plugin 
                LEFT JOIN   rhq_resource_type 
                ON          rhq_resource_type.plugin = rhq_plugin.name 
                WHERE       rhq_plugin.status = 'DELETED'
            );

The rhq_bunlde_type table also seems to be related to this data but I am not seeing a constraint on it. However, its contents should also be updated based on the necessary delete for rhq_bundle_type_target_map.