Bug 1200579 - Unable to delete/remove agent plug-ins that define a bundle type target
Summary: Unable to delete/remove agent plug-ins that define a bundle type target
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: JBoss Operations Network
Classification: JBoss
Component: Inventory
Version: JON 3.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ER01
: JON 3.3.2
Assignee: Libor Zoubek
QA Contact: Mike Foley
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-03-10 21:25 UTC by Larry O'Leary
Modified: 2019-05-20 11:35 UTC (History)
5 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2015-04-30 16:11:07 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Server log showing continuous resource type purge failures (715.44 KB, text/plain)
2015-03-10 21:29 UTC, Larry O'Leary
no flags Details
verification logs (170 bytes, application/zip)
2015-04-14 20:15 UTC, Viet Nguyen
no flags Details
verification logs (3.43 KB, application/zip)
2015-04-14 20:17 UTC, Viet Nguyen
no flags Details
verification logs (3.77 KB, application/zip)
2015-04-14 20:19 UTC, Viet Nguyen
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1211407 0 unspecified CLOSED Agent plug-in deletion fails if resource group or auto-group operation was used 2021-02-22 00:41:40 UTC
Red Hat Knowledge Base (Solution) 1376543 0 None None None Never
Red Hat Product Errata RHSA-2015:0920 0 normal SHIPPED_LIVE Moderate: Red Hat JBoss Operations Network 3.3.2 update 2015-04-30 20:09:30 UTC

Internal Links: 1211407

Description Larry O'Leary 2015-03-10 21:25:46 UTC
Description of problem:
It is no longer possible to remove an agent plug-in from a JBoss ON system. Although no error is reported when the request is submitted to delete the plug-in, the PurgeResourceTypesJob fails to execute resulting in all the resource types defined by the plug-in to be deleted remaining around forever.

Every 5 minutes, the server logs the failure to server.log:

    INFO  [org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl] (RHQScheduler_Worker-2) HHH000010: On release of batch it still contained JDBC statements
    WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) SQL Error: 0, SQLState: 23503
    ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) Batch entry 0 delete from RHQ_RESOURCE_TYPE where ID=10097 was aborted.  Call getNextException to see the cause.
    WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) SQL Error: 0, SQLState: 23503
    ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) ERROR: update or delete on table "rhq_resource_type" violates foreign key constraint "rhq_bundle_type_target_map_resource_type_id_fkey" on table "rhq_bundle_type_target_map"
      Detail: Key (id)=(10097) is still referenced from table "rhq_bundle_type_target_map".
    ERROR [org.hibernate.engine.jdbc.batch.internal.BatchingBatch] (RHQScheduler_Worker-2) HHH000315: Exception executing batch [could not execute batch]
    ERROR [org.jboss.as.ejb3] (RHQScheduler_Worker-2) javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.ConstraintViolationException: could not execute batch
    ERROR [org.jboss.as.ejb3.invocation] (RHQScheduler_Worker-2) JBAS014134: EJB Invocation failed on component ResourceMetadataManagerBean for method public abstract void org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerLocal.completeRemoveResourceType(org.rhq.core.domain.auth.Subject,org.rhq.core.domain.resource.ResourceType): javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.ConstraintViolationException: could not execute batch
    ERROR [org.jboss.as.ejb3.invocation] (RHQScheduler_Worker-2) JBAS014134: EJB Invocation failed on component InventoryManagerBean for method public abstract void org.rhq.enterprise.server.inventory.InventoryManagerLocal.purgeDeletedResourceType(org.rhq.core.domain.resource.ResourceType): javax.ejb.EJBException: java.lang.RuntimeException: Failed to purge resource types
    ERROR [org.rhq.enterprise.server.scheduler.jobs.AbstractStatefulJob] (RHQScheduler_Worker-2) Failed to execute job [PurgeResourceTypesJob]: javax.ejb.EJBException: java.lang.RuntimeException: Failed to purge resource types
    Caused by: java.lang.RuntimeException: Failed to purge resource types
    Caused by: javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.ConstraintViolationException: could not execute batch
    Caused by: javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not execute batch
    Caused by: org.hibernate.exception.ConstraintViolationException: could not execute batch
    Caused by: java.sql.BatchUpdateException: Batch entry 0 delete from RHQ_RESOURCE_TYPE where ID=10097 was aborted.  Call getNextException to see the cause.

See attached log.

Version-Release number of selected component (if applicable):
3.3.0

How reproducible:
Always

Steps to Reproduce:
1.  Install JBoss ON 3.3 server and storage node but only install JBossAS7 plug-in from JBoss EAP plug-in pack and the default plug-ins.
2.  From _agent plug-ins configuration administration_ page, delete the plug-in _JBoss Application Server 7.x_ plug-in.
3.  Wait for at least 5 minutes.

Actual results:
Within 5 minutes of plug-in deletion, server.log will begin filling up with errors every 5 minutes:

    20:36:45,770 INFO  [org.rhq.coregui.server.gwt.PluginGWTServiceImpl] (http-/0.0.0.0:7080-1) Deleted agent plugins: [JBoss Application Server 7.x]
    20:36:45,779 INFO  [org.rhq.enterprise.server.core.plugin.AgentPluginScanner] (http-/0.0.0.0:7080-1) Plugin file [/opt/jboss/on/jon-server-3.3.0.GA/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-downloads/rhq-plugins/rhq-jboss-as-7-plugin-4.12.0.JON330GA.jar] has been deleted from the file system.

... wait up to 5 minutes ...

    20:39:30,938 INFO  [org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl] (RHQScheduler_Worker-2) HHH000010: On release of batch it still contained JDBC statements
    20:39:30,940 WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) SQL Error: 0, SQLState: 23503
    20:39:30,940 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) Batch entry 0 delete from RHQ_RESOURCE_TYPE where ID=10097 was aborted.  Call getNextException to see the cause.
    20:39:30,940 WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) SQL Error: 0, SQLState: 23503
    20:39:30,940 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) ERROR: update or delete on table "rhq_resource_type" violates foreign key constraint "rhq_bundle_type_target_map_resource_type_id_fkey" on table "rhq_bundle_type_target_map"
      Detail: Key (id)=(10097) is still referenced from table "rhq_bundle_type_target_map".
    20:39:30,941 ERROR [org.hibernate.engine.jdbc.batch.internal.BatchingBatch] (RHQScheduler_Worker-2) HHH000315: Exception executing batch [could not execute batch]
    20:39:30,943 ERROR [org.jboss.as.ejb3] (RHQScheduler_Worker-2) javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.ConstraintViolationException: could not execute batch
    20:39:30,943 ERROR [org.jboss.as.ejb3.invocation] (RHQScheduler_Worker-2) JBAS014134: EJB Invocation failed on component ResourceMetadataManagerBean for method public abstract void org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerLocal.completeRemoveResourceType(org.rhq.core.domain.auth.Subject,org.rhq.core.domain.resource.ResourceType): javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.ConstraintViolationException: could not execute batch

Plug-in remains in database in a deleted state along with all of its types also in a deleted state:

    SELECT      rhq_plugin.id, rhq_plugin.name, rhq_plugin.display_name, 
                rhq_resource_type.id, rhq_resource_type.name, 
                rhq_resource_type.deleted 
    FROM        rhq_plugin
    LEFT JOIN   rhq_resource_type 
    ON          rhq_resource_type.plugin = rhq_plugin.name
    WHERE       rhq_plugin.status = 'DELETED';

Expected results:
No errors after 5 minutes and plug-in and its resource types should no longer be in the database.

Additional info:
In this instance, the problem is the AS7 plug-in defines a bunlde type target map and holds a reference to the resource_type in rhq_bundle_type_target_map:

    SELECT  * FROM rhq_bundle_type_target_map
    WHERE   resource_type_id IN (
                SELECT      rhq_resource_type.id 
                FROM        rhq_plugin 
                LEFT JOIN   rhq_resource_type 
                ON          rhq_resource_type.plugin = rhq_plugin.name 
                WHERE       rhq_plugin.status = 'DELETED'
            );

The rhq_bunlde_type table also seems to be related to this data but I am not seeing a constraint on it. However, its contents should also be updated based on the necessary delete for rhq_bundle_type_target_map.

Comment 1 Larry O'Leary 2015-03-10 21:29:50 UTC
Created attachment 1000170 [details]
Server log showing continuous resource type purge failures

Comment 2 Libor Zoubek 2015-03-23 09:09:31 UTC
branch:  master
link:    https://github.com/rhq-project/rhq/commit/a98a734bc
time:    2015-03-23 10:07:40 +0100
commit:  a98a734bca2c86669259d07dd0c9b3899f899254
author:  Libor Zoubek - lzoubek
message: Bug 1200579 - Unable to delete/remove agent plug-ins that define a
         bundle type target
         When removing a resouceType make sure we also remove associated
         bundleTypes
         (which may include Bundles of that type)

Comment 3 Michael Burman 2015-04-02 11:52:58 UTC
Cherry-picked to jon3.3.x:

commit f613f22d72dae74ed92f70a635653aa6895f758f
Author: Libor Zoubek <lzoubek>
Date:   Mon Mar 23 10:07:40 2015 +0100

    Bug 1200579 - Unable to delete/remove agent plug-ins that define a bundle
    type target
    
    When removing a resouceType make sure we also remove associated bundleTypes
    (which may include Bundles of that type)
    
    (cherry picked from commit a98a734bca2c86669259d07dd0c9b3899f899254)

Comment 4 Simeon Pinder 2015-04-13 04:14:38 UTC
Moving to ON_QA for testing with latest cumulative patch build:
https://brewweb.devel.redhat.com//buildinfo?buildID=429507

Note: Build maps to JON 3.3.2 ER01 build.

Comment 5 Viet Nguyen 2015-04-14 20:15:47 UTC
Created attachment 1014472 [details]
verification logs

Comment 6 Viet Nguyen 2015-04-14 20:17:55 UTC
Created attachment 1014473 [details]
verification logs

Comment 7 Viet Nguyen 2015-04-14 20:19:56 UTC
Created attachment 1014475 [details]
verification logs

Comment 9 errata-xmlrpc 2015-04-30 16:11:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0920.html


Note You need to log in before you can comment on or make changes to this bug.