Bug 902406

Summary: disable types on agent so avoid large number of services causing inventory report to not get processed
Product: [Other] RHQ Project Reporter: John Mazzitelli <mazz>
Component: Core ServerAssignee: John Mazzitelli <mazz>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 4.5CC: hrupp, loleary, vnguyen
Target Milestone: ---   
Target Release: RHQ 4.6   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 905651 906866 (view as bug list) Environment:
Last Closed: 2013-09-04 09:08:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 906866    

Description John Mazzitelli 2013-01-21 15:39:47 UTC
The scenario is:

1) agent detects a server, sends it up to server via inventory report
2) user commits that server to inventory
3) agent is told about this, commits the server and auto-detects children services of that newly committed server. The number of children services is huge - say, on the order of tens of thousands (e.g. 40,000 or 50,000)
4) The agents sends up this huge inventory report

At this point, the server will have a hard time processing this report. The server needs to handle this scenario better.

Comment 1 John Mazzitelli 2013-01-21 17:55:28 UTC
with the perfplugin deployed, this helps reproduce the issue:

-Drhq.perftest.scenario=configurable-1 -Drhq.perftest.server-a-count=1 -Drhq.perftest.service-a-count=25000

I saw the issue with 50000, jay saw it with 25000. Note that with it only 5000 (that is, server has 5,000 child services) I did not get any errors. So we can support a large number, just not a really, really large number.

Comment 2 John Mazzitelli 2013-01-22 01:03:59 UTC
(In reply to comment #1)
> Note that with it only
> 5000 (that is, server has 5,000 child services) I did not get any errors. So
> we can support a large number, just not a really, really large number.

I'll walk back from this statement - even though I had 5,000 peer services successfully in inventory, the UI was unable to render the left-hand tree. I also saw some OOMs that were probably related. In any event, 5000 direct children under a parent is not working well either, even though it avoided the issue documented in this BZ.

Comment 3 John Mazzitelli 2013-01-22 15:27:53 UTC
assigning to jay - he is currently refactoring the merge inventory report code.

Comment 4 John Mazzitelli 2013-01-26 04:25:11 UTC
A new feature is added to allow someone to disable one or more resource types at the agent level. There is an agent config setting that you can set to disable any resource types - these disable types won't be discovered and any resources that previously made it into inventory before this will now appear RED (down). You'd have to uninventory the parent and re-inventory it in order for the resources to go away. But any new resources of the given disabled types will not be discovered.

git commit to bug/902406 branch:

13cb988bb4835289393c8e3f736918651d6e2bae (the main commit)
085acde34623280a7720ee866f0efd4fc005d435 (one line fix for NPE)

Comment 5 Larry O'Leary 2013-02-01 17:33:02 UTC
*** Bug 905654 has been marked as a duplicate of this bug. ***

Comment 6 Larry O'Leary 2013-02-01 18:05:16 UTC
Bug 905654 is not a duplicate of this bug. I fixed that.

Comment 7 Jay Shaughnessy 2013-02-11 16:19:26 UTC
Note that master has been updated with scalability work for:

1) Agent-side type disable
2) Inventory report handling

The work for these 2 items was cherry-picked to the release/jon3.1.x branch.

3) Inventory sync handling
4) Availability report handling
5) plugin/type removal handling

Master only.

Comment 8 Heiko W. Rupp 2013-09-04 09:08:30 UTC
Closed as this is implemented in an old RHQ release already.