Bug 784590

Summary:	[eap6] Unable to retrieve current configuration after updating Resource Configuration Timed out
Product:	[Other] RHQ Project	Reporter:	Libor Zoubek <lzoubek>
Component:	Agent, Core Server, Configuration, Plugins	Assignee:	Simeon Pinder <spinder>
Status:	CLOSED NOTABUG	QA Contact:	Mike Foley <mfoley>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.3	CC:	hrupp, spinder, theute
Target Milestone:	---	Flags:	lzoubek: needinfo+
Target Release:	---
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-02-28 16:07:11 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	707223

Description Libor Zoubek 2012-01-25 13:51:17 UTC

Description of problem: When verifying https://bugzilla.redhat.com/show_bug.cgi?id=708332 it happened to me, that Update Configuration job failed, because it timed out (do not know why it timed out). What is worse, since that I am not able to retrieve current configuration of affected resource.


Version-Release number of selected component (if applicable):
Version: 4.3.0-SNAPSHOT
Build Number: 8125902

EAP6 DR12

How reproducible: hard


Steps to Reproduce:
1. Invoke a resource configuraion change - Add system property to EAP6  DomainController
2. Let the operation timeout (not sure how to reproduce)
3. Wait for RHQ UI to recognize that configuration change failed
4. Check Configuration change history - last change should be marked as failed with timeout message - correct
5. Try to see 'Current' configuration of EAP6 DomainController
  
Actual results:

StackTrace of failed operation on agent:

2012-01-25 14:23:36,871 ERROR [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.UpdateResourceConfigurationRunner)- Plugin Error: Exception thrown while updating Resource configuration for JBossAS7 Host Controller Resource with id [10013].
org.rhq.core.pc.inventory.TimeoutException: Call to [org.rhq.modules.plugins.jbossas7.HostControllerComponent.updateResourceConfiguration()] with args [[org.rhq.core.pluginapi.configuration.ConfigurationUpdateReport@1b834b9]] timed out. Invocation thread will be interrupted
        at org.rhq.core.pc.inventory.ResourceContainer$ResourceComponentInvocationHandler.invokeInNewThreadWithLock(ResourceContainer.java:492)
        at org.rhq.core.pc.inventory.ResourceContainer$ResourceComponentInvocationHandler.invoke(ResourceContainer.java:461)
        at $Proxy86.updateResourceConfiguration(Unknown Source)
        at org.rhq.core.pc.configuration.LegacyConfigManagement.executeUpdate(LegacyConfigManagement.java:87)
        at org.rhq.core.pc.configuration.UpdateResourceConfigurationRunner.call(UpdateResourceConfigurationRunner.java:121)
        at org.rhq.core.pc.configuration.UpdateResourceConfigurationRunner.run(UpdateResourceConfigurationRunner.java:109)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)


After that, a warning in server log, when attempting to see resource's current configuration

WARN  [org.rhq.enterprise.server.configuration.ConfigurationManagerBean] Could not get live resource configuration for resource [Resource[id=10013, type=JBossAS7 Host Controller, key=DomainController, name=EAP DomainController, parent=dhcp-1-216.brq.redhat.com, version=EAP domain]]; will assume latest resource configuration update is the current resource configuration.


Expected results: Even when resource change 


Additional info:

Comment 1 Heiko W. Rupp 2012-01-30 11:06:33 UTC

I can't reproduce this.

Comment 2 Libor Zoubek 2012-01-30 11:53:50 UTC

Have you succeeded to make configuration update to timeout? I can't reproduce the timeout either. I'll write automated tests for https://bugzilla.redhat.com/show_bug.cgi?id=708332 soon, so they could help discover whether timeouts in this case are random or do not happen at all.

Comment 3 Heiko W. Rupp 2012-01-30 12:13:09 UTC

Well, not time out, but not succeed by stopping the DC

Comment 5 Mike Foley 2012-01-30 16:23:34 UTC

12/30/2012 BZ triage meeting mfoley, ccrouch, loleary, asantos

Comment 6 Heiko W. Rupp 2012-02-14 17:07:50 UTC

This has been identified by Libor as being a bug in as7 (he was able to hang it with the as7-cli as well).

To prevent hangs of the as7 plugin, a timeout has been added so that a hanging connection is terminated after 10 secs.

Libor wanted to report an as7 issue

Comment 7 Simeon Pinder 2012-02-23 09:40:12 UTC

I am also unable to reproduce this. I did modify the plugin to generate resource errors when the AS7 communication is down because of invalid credentials.  The resource error icon shows up in the ui when trying to browse a resource. 

I'm guessing that Libor is seeing this because he's switching in between HC and Standalone modes and perhaps the communication over resources is getting confused. Either way with the new timeout and more information about when plugin operations cannot happen because of plugin problems we may not see this anymore.

Comment 8 Simeon Pinder 2012-02-23 14:44:10 UTC

Hi Libor, Can you try to reproduce this again?  Not being able to replicate this is making it difficult to find and fix if it's still there.

Comment 9 Libor Zoubek 2012-02-28 16:07:11 UTC

I cannot reproduce it either.