Bug 713744

this also happens on postgres when I tweeked the dbunit .xml files.

hangs here:

org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerBeanTest.upgradePlugin()

If I uncomment this in plugin_v1.xml:

<!-- THIS CAUSES A DEADLOCK DURING PLUGIN REGISTRATION!
        <bundle-target>
           <destination-base-dir name="bundleTarget1">
               <value-context>pluginConfiguration</value-context>
               <value-name>connectionPropertyY</value-name>
           </destination-base-dir>
        </bundle-target>
-->

ResourceMetadataManagerBeanTest hangs during plugin update.

raising to high priority - this may be a problem with plugin updates for plugins that define bundle-target elements.

There were a few XXXMetadatManagerBean methods that were missing the REQUIRES_NEW transaction attribute and were not performing their updates in their own, separate transactions. I was able to reproduce the deadlock described in comment 2, and putting in REQUIRES_NEW so that those method execute in their own transactions resolved the issue.

commit hash: 0ef2b576b995d39ecc31d6b84d52c9d57d9fa498

Looks like the code changes in 0ef2b576b995d39ecc31d6b84d52c9d57d9fa498 did not resolve the issue on oracle. Moving back to ON_DEV for further investigation.

My previous comment was based on the tests hanging in the oracle hudson job. I proceeded to test locally against oracle 10g. Without the fix, I reproduce the deadlock. Running with the fix, there is no deadlock and tests pass. It could be that the oracle instance used by hudson is not in a good state. Going to move this back to ON_QA as I have tested this against both oracle and postgresql.

Found another place through a test where ResourceMetadataManagerBean is deadlocking. The method, testAddDeleteTemplate, in the class UpdateConfigurationSubsystemTest triggers the deadlock which is happening in ResourceMetadataManagerBean.mergeExistingType at the following line (399ish):

resourceConfigMetadataMgr.updateResourceConfigurationDefinition(existingType, resourceType)

I do not think that the above method call is causing the deadlock. I think it has to do with transaction boundaries established prior to mergeExistingType being called. Moving back to ON_DEV for further investigation.

This deadlock (described in comment 7) was discovered and reproduced by running the test UpdateConfigurationSubsystemTest.testAddDeleteTemplate. The deadlock
occurs in ResourceMetadataManagerBean.mergeExistingType on line 397 which is,

  resourceConfigMetadataMgr.updateResourceConfigurationDefinition(existingType,
      resourceType);

The problem as with all of these deadlock issues involves the table rhq_config. In the test method, a plugin that declares a config template, is installed and then updated. In an initial transaction that is started in PluginManagerBean, a write lock is allocated to update rhq_plugin. updateResourceConfigurationDefinition executes in its own transaction, and because the test involves some config templates, a write lock is needed for rhq_config since there is a one-to-one assocation between rhq_config_template and rhq_config. That lock cannot be obtained because rhq_plugin has a FK with rhq_config for the ServerPlugin class, hence the deadlock.

With this commit, PluginManagerBean.registerPlugin has been refactored to avoid the deadlock. It first calls a new method, installPluginJar, which executes in its own transaction and handles updating the Plugin object. After that method is called, registerPluginTypes is called. That call starts a new transaction, and registerPluginTypes now only handles the meta data updates.

commit hash: 3cc1ede3a509763de99633876bf90891204890ee

I will wait to move this back to ON_QA until we get a passing hudson build on the oracle job.

documenting passing tests on Oracle

https://hudson.qa.jboss.com/hudson/view/RHQ%20Core/job/rhq-master/220/

additionally, i use oracle as my rhq repo on a daily basis ... including running the ui automation.

changing status of VERIFIED BZs for JON 2.4.2 and JON 3.0 to CLOSED/CURRENTRELEASE