Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1095919

Summary: Database deadlock errors when DiscoveryBossBean attempts to merge inventory report
Product: [JBoss] JBoss Operations Network Reporter: Larry O'Leary <loleary>
Component: DatabaseAssignee: RHQ Project Maintainer <rhq-maint>
Status: CLOSED WORKSFORME QA Contact: Mike Foley <mfoley>
Severity: high Docs Contact:
Priority: unspecified    
Version: JON 3.1.2CC: hrupp, jshaughn, loleary, myarboro, rhq-maint
Target Milestone: ---Keywords: Triaged
Target Release: JON 3.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-09-30 23:17:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Log excerpt showing complete error and stack traces none

Description Larry O'Leary 2014-05-08 20:20:00 UTC 
Created attachment 893790 [details]
Log excerpt showing complete error and stack traces

Description of problem:
Dead lock errors are reported resulting in database failures being reported.

From the stack trace (in the log excerpt) it appears these deadlocks occurred when an inventory merge was being done.

ERROR [org.hibernate.event.def.AbstractFlushingEventListener] Could not synchronize database state with session
org.hibernate.exception.LockAcquisitionException: Could not execute JDBC batch update
	at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:87)
	at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
	at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:254)
	at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:237)
	at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:142)
    ...
	at sun.proxy.$Proxy332.updateImplicitGroupMembership(Unknown Source)
	at org.rhq.enterprise.server.discovery.DiscoveryBossBean.addResourceToInventory(DiscoveryBossBean.java:944)
	at org.rhq.enterprise.server.discovery.DiscoveryBossBean.mergeResource(DiscoveryBossBean.java:711)
	at org.rhq.enterprise.server.discovery.DiscoveryBossBean.updatePreviouslyInventoriedResource(DiscoveryBossBean.java:899)
	at org.rhq.enterprise.server.discovery.DiscoveryBossBean.mergeResource(DiscoveryBossBean.java:708)
	at org.rhq.enterprise.server.discovery.DiscoveryBossBean.mergeInventoryReport(DiscoveryBossBean.java:178)
    ...
	at sun.proxy.$Proxy505.mergeInventoryReport(Unknown Source)
	at org.rhq.enterprise.server.discovery.DiscoveryServerServiceImpl.mergeInventoryReport(DiscoveryServerServiceImpl.java:75)
    ...
	at org.rhq.enterprise.communications.command.impl.remotepojo.server.RemotePojoInvocationCommandService.execute(RemotePojoInvocationCommandService.java:184)
    ...
	at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:290)
	at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:184)
    ...
Caused by: java.sql.BatchUpdateException: ORA-00060: deadlock detected while waiting for resource

	at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:10345)
	at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:230)
	at org.jboss.resource.adapter.jdbc.CachedPreparedStatement.executeBatch(CachedPreparedStatement.java:476)
	at org.jboss.resource.adapter.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:774)
	at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:48)
	at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:247)
	... 189 more


Version-Release number of selected component (if applicable):
3.1.2.GA

Additional info:
This seems very similar to the deadlock issue raised in bug 1093265 but this appears to be driven by inventory sync/merge rather then user invoking a function of the UI.
Comment 1 Jay Shaughnessy 2014-05-16 16:50:10 UTC 
Given the large changes in sync in JON 3.2.0 I'd say the recommendation here should be an upgrade.
Comment 6 Jay Shaughnessy 2014-09-30 21:40:35 UTC 
I can tell you that the call chain in the log no longer exists.

We removed the umbrella Tx in mergeInventoryReport in commit  22053b3794a724a1fa997df07684ed14dd55b564 and further updated relevant Tx delimiters in commit b6e70d104756db633df7b4ff642c29e3fad81d2b.

I don't think this problem exists in the current release.  I recommend closing. Asking Larry to see if he concurs or otherwise can recommend some other course of action.
Comment 7 Larry O'Leary 2014-09-30 23:17:51 UTC 
I concur. As we have had no reports of this issue past JBoss ON 3.1.2 and the inventory sync has been reworked, this issue seems like it should no longer occur.

If we find that this issue happens on 3.2 or -- when its released -- 3.3, then a new bug report should be opened providing steps to reproduce along with relevant logs.

Marking as CLOSED WORKSFORME.