Bug 823564

Summary: [plugin-container] with two or more AS7 host controllers in inventory, a full discovery scan takes an unacceptably long time
Product: [Other] RHQ Project Reporter: Jay Shaughnessy <jshaughn>
Component: Plugin ContainerAssignee: Jay Shaughnessy <jshaughn>
Status: CLOSED CURRENTRELEASE QA Contact: Mike Foley <mfoley>
Severity: high Docs Contact:
Priority: urgent    
Version: 4.4CC: ahovsepy, ccrouch, hrupp, jsanda, jshaughn, lzoubek, mazz
Target Milestone: ---   
Target Release: JON 3.1.0   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 820709 Environment:
Last Closed: 2013-09-03 15:16:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 818678, 820709, 844647    
Bug Blocks: 707223, 782579    

Description Jay Shaughnessy 2012-05-21 14:34:54 UTC
+++ This bug was initially created as a clone of Bug #820709 +++

Description of problem: I am facing performance issue. Default timeout for Manual Autodiscovery is 10minutes. When I have 2 EAPs on same host (and one of them runs in Domain mode) detailed autodiscovery mostly fails because of timeout. If it does not fail it finishes close to 10 minutes. 


Version-Release number of selected component (if applicable):
JON 3.1.CR1
EAP6 ER6

How reproducible:almost always


Steps to Reproduce:
1. stand up at least 2 EAP6 instances on same host (more is better) I am working with 1 standalone and 1 domain instance
2. stand up agent, import whole platform with all children
3. wait 'till everyting is imported (this can take quite a long time 5-10minutes)
4. run manual autodiscovery on platform with 2 EAPs
  
Actual results: autodiscovery takes almost 10 minutes, sometimes it fails because of default timeout. This is too much. I was observing machine when discovery was running and there are no symptoms of overloading, average CPU usage was at 4%.


Expected results: Autodiscovery must be much faster. I suspect as7plugin. Without imported AS7 servers discovery runs pretty fast. Solution would be to implement discovery of each subsystem to run in parallel - but this might be very complicated and might introduce new issues.


Additional info: I was tcpdumping requests sent by agent to EAP6 and really, plugin was requesting both EAPs for all 10 minutes. But it is hard to determine whether requests came from discovery scan or usual availability. When observing requests when discovery is not running, plugin is sending them all the time anyway. This has become an issue when more AS7 subsystems have been implemented to as7plugin.

Comment 1 Jay Shaughnessy 2012-05-21 14:41:50 UTC
Cherry Pick Commit: 2bd956bb608a411327b5e69352b2c4436c4cdf05

Comment 2 Jay Shaughnessy 2012-05-21 15:28:35 UTC
Ignore above hash, it was local.

Correct release/jon3.1.x commit has for the cherry pick commit:
ac6a8cb6370a423e286f99b6bffda3b547a85615

Comment 3 Armine Hovsepyan 2012-09-07 13:28:04 UTC
verified.

Longest auto-discovery for 1 domain and 1 standalone (in full ha mode) was ~4 mins, 1 domain and 1 standalone (default mode) is ~1 min min, less than 2 EAPserver -  are ~1 or less.

Comment 4 Heiko W. Rupp 2013-09-03 15:16:00 UTC
Bulk closing of old issues in VERIFIED state.