Description of problem: It is no longer possible to remove an agent plug-in from a JBoss ON system. Although no error is reported when the request is submitted to delete the plug-in, the PurgeResourceTypesJob fails to execute resulting in all the resource types defined by the plug-in to be deleted remaining around forever. Every 5 minutes, the server logs the failure to server.log: INFO [org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl] (RHQScheduler_Worker-2) HHH000010: On release of batch it still contained JDBC statements WARN [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) SQL Error: 0, SQLState: 23503 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) Batch entry 0 delete from RHQ_RESOURCE_TYPE where ID=10097 was aborted. Call getNextException to see the cause. WARN [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) SQL Error: 0, SQLState: 23503 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) ERROR: update or delete on table "rhq_resource_type" violates foreign key constraint "rhq_bundle_type_target_map_resource_type_id_fkey" on table "rhq_bundle_type_target_map" Detail: Key (id)=(10097) is still referenced from table "rhq_bundle_type_target_map". ERROR [org.hibernate.engine.jdbc.batch.internal.BatchingBatch] (RHQScheduler_Worker-2) HHH000315: Exception executing batch [could not execute batch] ERROR [org.jboss.as.ejb3] (RHQScheduler_Worker-2) javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.ConstraintViolationException: could not execute batch ERROR [org.jboss.as.ejb3.invocation] (RHQScheduler_Worker-2) JBAS014134: EJB Invocation failed on component ResourceMetadataManagerBean for method public abstract void org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerLocal.completeRemoveResourceType(org.rhq.core.domain.auth.Subject,org.rhq.core.domain.resource.ResourceType): javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.ConstraintViolationException: could not execute batch ERROR [org.jboss.as.ejb3.invocation] (RHQScheduler_Worker-2) JBAS014134: EJB Invocation failed on component InventoryManagerBean for method public abstract void org.rhq.enterprise.server.inventory.InventoryManagerLocal.purgeDeletedResourceType(org.rhq.core.domain.resource.ResourceType): javax.ejb.EJBException: java.lang.RuntimeException: Failed to purge resource types ERROR [org.rhq.enterprise.server.scheduler.jobs.AbstractStatefulJob] (RHQScheduler_Worker-2) Failed to execute job [PurgeResourceTypesJob]: javax.ejb.EJBException: java.lang.RuntimeException: Failed to purge resource types Caused by: java.lang.RuntimeException: Failed to purge resource types Caused by: javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.ConstraintViolationException: could not execute batch Caused by: javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not execute batch Caused by: org.hibernate.exception.ConstraintViolationException: could not execute batch Caused by: java.sql.BatchUpdateException: Batch entry 0 delete from RHQ_RESOURCE_TYPE where ID=10097 was aborted. Call getNextException to see the cause. See attached log. Version-Release number of selected component (if applicable): 3.3.0 How reproducible: Always Steps to Reproduce: 1. Install JBoss ON 3.3 server and storage node but only install JBossAS7 plug-in from JBoss EAP plug-in pack and the default plug-ins. 2. From _agent plug-ins configuration administration_ page, delete the plug-in _JBoss Application Server 7.x_ plug-in. 3. Wait for at least 5 minutes. Actual results: Within 5 minutes of plug-in deletion, server.log will begin filling up with errors every 5 minutes: 20:36:45,770 INFO [org.rhq.coregui.server.gwt.PluginGWTServiceImpl] (http-/0.0.0.0:7080-1) Deleted agent plugins: [JBoss Application Server 7.x] 20:36:45,779 INFO [org.rhq.enterprise.server.core.plugin.AgentPluginScanner] (http-/0.0.0.0:7080-1) Plugin file [/opt/jboss/on/jon-server-3.3.0.GA/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-downloads/rhq-plugins/rhq-jboss-as-7-plugin-4.12.0.JON330GA.jar] has been deleted from the file system. ... wait up to 5 minutes ... 20:39:30,938 INFO [org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl] (RHQScheduler_Worker-2) HHH000010: On release of batch it still contained JDBC statements 20:39:30,940 WARN [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) SQL Error: 0, SQLState: 23503 20:39:30,940 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) Batch entry 0 delete from RHQ_RESOURCE_TYPE where ID=10097 was aborted. Call getNextException to see the cause. 20:39:30,940 WARN [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) SQL Error: 0, SQLState: 23503 20:39:30,940 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) ERROR: update or delete on table "rhq_resource_type" violates foreign key constraint "rhq_bundle_type_target_map_resource_type_id_fkey" on table "rhq_bundle_type_target_map" Detail: Key (id)=(10097) is still referenced from table "rhq_bundle_type_target_map". 20:39:30,941 ERROR [org.hibernate.engine.jdbc.batch.internal.BatchingBatch] (RHQScheduler_Worker-2) HHH000315: Exception executing batch [could not execute batch] 20:39:30,943 ERROR [org.jboss.as.ejb3] (RHQScheduler_Worker-2) javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.ConstraintViolationException: could not execute batch 20:39:30,943 ERROR [org.jboss.as.ejb3.invocation] (RHQScheduler_Worker-2) JBAS014134: EJB Invocation failed on component ResourceMetadataManagerBean for method public abstract void org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerLocal.completeRemoveResourceType(org.rhq.core.domain.auth.Subject,org.rhq.core.domain.resource.ResourceType): javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.ConstraintViolationException: could not execute batch Plug-in remains in database in a deleted state along with all of its types also in a deleted state: SELECT rhq_plugin.id, rhq_plugin.name, rhq_plugin.display_name, rhq_resource_type.id, rhq_resource_type.name, rhq_resource_type.deleted FROM rhq_plugin LEFT JOIN rhq_resource_type ON rhq_resource_type.plugin = rhq_plugin.name WHERE rhq_plugin.status = 'DELETED'; Expected results: No errors after 5 minutes and plug-in and its resource types should no longer be in the database. Additional info: In this instance, the problem is the AS7 plug-in defines a bunlde type target map and holds a reference to the resource_type in rhq_bundle_type_target_map: SELECT * FROM rhq_bundle_type_target_map WHERE resource_type_id IN ( SELECT rhq_resource_type.id FROM rhq_plugin LEFT JOIN rhq_resource_type ON rhq_resource_type.plugin = rhq_plugin.name WHERE rhq_plugin.status = 'DELETED' ); The rhq_bunlde_type table also seems to be related to this data but I am not seeing a constraint on it. However, its contents should also be updated based on the necessary delete for rhq_bundle_type_target_map.
Created attachment 1000170 [details] Server log showing continuous resource type purge failures
branch: master link: https://github.com/rhq-project/rhq/commit/a98a734bc time: 2015-03-23 10:07:40 +0100 commit: a98a734bca2c86669259d07dd0c9b3899f899254 author: Libor Zoubek - lzoubek message: Bug 1200579 - Unable to delete/remove agent plug-ins that define a bundle type target When removing a resouceType make sure we also remove associated bundleTypes (which may include Bundles of that type)
Cherry-picked to jon3.3.x: commit f613f22d72dae74ed92f70a635653aa6895f758f Author: Libor Zoubek <lzoubek> Date: Mon Mar 23 10:07:40 2015 +0100 Bug 1200579 - Unable to delete/remove agent plug-ins that define a bundle type target When removing a resouceType make sure we also remove associated bundleTypes (which may include Bundles of that type) (cherry picked from commit a98a734bca2c86669259d07dd0c9b3899f899254)
Moving to ON_QA for testing with latest cumulative patch build: https://brewweb.devel.redhat.com//buildinfo?buildID=429507 Note: Build maps to JON 3.3.2 ER01 build.
Created attachment 1014472 [details] verification logs
Created attachment 1014473 [details] verification logs
Created attachment 1014475 [details] verification logs
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0920.html