Bug 1100300

Summary:	Update of a resource configuration sometimes takes more then 100 seconds
Product:	[Other] RHQ Project	Reporter:	Filip Brychta <fbrychta>
Component:	Configuration	Assignee:	Jay Shaughnessy <jshaughn>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Mike Foley <mfoley>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.11	CC:	hrupp, jshaughn, lzoubek
Target Milestone:	GA
Target Release:	RHQ 4.12
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-12-15 11:36:35 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Filip Brychta 2014-05-22 13:09:39 UTC

Description of problem:
This issue was reported by automation. It's not easy to reproduce it locally.
So far it happened during configuration update of following resource types:
[as7] Security Domain
[as7] Security
JBossAS7 Standalone Server
DeploymentScanner
JAXR
and some more, so it seems it's not tied with specific resources but it's general problem.

Version-Release number of selected component (if applicable):
b1eec74
4.12.0-SNAPSHOT

How reproducible:
Sometimes

Steps to Reproduce:
1. navigate to some resource listed above 
2. and try to update it's configuration

Actual results:
It takes up to 100 seconds to update the configuration (Configure_inprogress_16.png is visible).

Expected results:
It should take a few seconds.

Additional info:
This is not related to HA (it happens with single server as well)
No exceptions in logs.

Comment 1 Libor Zoubek 2014-05-22 20:03:38 UTC

I was hitting this issue as well. There is a scheduled job on agent ConfigurationCheckExecutor, that collects configurations for all resources. This job is implemented as Runnable. Then there is a ThreadPool with size 1 thread, that handles above job + whenever there comes request from server it's also executed in this threadpool. 

So .. when this job is running, it's actively blocking config update request from server to happen.

This issue is visible since 4.10, when there has been added waiting 750ms to scheduled job, whenever it loads config for 1 resource. This job can then run for example 10minutes if you have enough resources.

I talked to Heiko .. and there's an intention of having only 1 thread (to avoid concurrency issues .. interference of scheduled job and "manual" config updates.

Possible solution would be to change scheduled job not to be single runnable, but rather a queue of runnables, that would be executed one after each other, and have priority queue so we can insert config update request to the beginning of queue to make sure it gets executed almost immediatelly.

Comment 2 Jay Shaughnessy 2014-06-25 20:22:58 UTC

Update: Work is progressing in https://github.com/rhq-project/rhq/pull/62

It's hopefully close to a merge.

Comment 3 Jay Shaughnessy 2014-06-26 20:45:46 UTC

master commit 03c2e260be2d7e92c95f6b2302e6bcba163c0797
Merge: d2427f0 a0922b4
Author: jshaughn <jshaughn>
Date:   Thu Jun 26 16:38:53 2014 -0400

    Merge pull request #62 from rhq-project/jshaughn/1100300

Comment 4 Filip Brychta 2014-07-03 13:55:14 UTC

Verified on
Version :	
4.12.0-SNAPSHOT
Build Number :	
7bf3544

Comment 5 Heiko W. Rupp 2014-12-15 11:36:35 UTC

Bulk close of items fixed in RHQ 4.12

If you think this is not solved, then please open a *new* BZ and link to this one.