Bug 912525

Summary: need to add concurrency limit to configuration updates from agent
Product: [Other] RHQ Project Reporter: John Mazzitelli <mazz>
Component: Communications SubsystemAssignee: John Mazzitelli <mazz>
Status: ON_QA --- QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: 4.5CC: hrupp
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description John Mazzitelli 2013-02-18 21:36:45 UTC
Configuration updates from the agent can be expensive, and when you have lots of agents, it ends up clobbering the server and DB. We should add a new concurrency limit to:

org.rhq.core.clientapi.server.configuration.ConfigurationServerService.persistUpdatedResourceConfiguration(int, Configuration)

Comment 1 Heiko W. Rupp 2013-02-19 07:32:39 UTC
We may in the long run try to "unwind" the whole configuration structure on disk, as the recursive structure does a lot of joins on the property table. 
Also the endless nesting is a pita in so many places, so that we may want to limit the possible depth (and thus also the number of joins on the table itself).

( the next may not directly apply to our case, the article itself is a good reading anyway )
From http://www.jans-sajt.se/contents/ora2/plsql/Oracle_SQL_SELECT.htm

"queries on queries ("cascades") are often imperformant when not tuned for speed. Like view-on-view cascades, only the deepest query/view can use indexes, while the pendend super-selects always are full table scans (of the result set) inside the database cache! If You cannot escape this, than make sure that Your deepest result sets are as small as possible, and put all the WHERE filters there!"

Comment 2 John Mazzitelli 2013-02-19 16:53:13 UTC
adding a new concurrency limit requires changes to the docs:

https://docs.jboss.org/author/display/RHQ/Startup+Properties

and rhq-server.properties - something like:

rhq.server.concurrency-limit.configuration-update

Comment 3 John Mazzitelli 2013-02-19 20:14:30 UTC
master git commit: 3005a262a20961e6c8583e02953d9af1501d8c68

this is a code change - not much to do to test. If you want to see this, turn on debug messages in server. In rhq-server.properties, set "rhq.server.concurrency-limit.configuration-update" to something low, like 1. Then import alot of resources at the same time (those resources must have <resource-configuration> associated with them). If you hit concurrency limits, you'll see this log message:

"Command not permitted - server reached its limit of concurrent invocations ..."