Bug 1353707 - MiqVimBrokerWorker uses much more cpu during same workload in 5.6 vs 5.5 (VMware Capacity & Utilization)
Summary: MiqVimBrokerWorker uses much more cpu during same workload in 5.6 vs 5.5 (VMw...
Keywords:
Status: CLOSED DUPLICATE of bug 1378998
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Performance
Version: 5.6.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: GA
: cfme-future
Assignee: Adam Grare
QA Contact: Dave Johnson
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-07 19:13 UTC by Alex Krzos
Modified: 2017-12-20 15:00 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-11-29 18:32:47 UTC
Category: ---
Cloudforms Team: ---
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Two stacked graphs displaying worker cpu usage during 4 C&U workload scenarios (152.84 KB, image/png)
2016-07-07 19:13 UTC, Alex Krzos
no flags Details
VimBroker CPU Usage GC vs No-GC (140.51 KB, image/png)
2016-11-02 02:10 UTC, Adam Grare
no flags Details

Description Alex Krzos 2016-07-07 19:13:11 UTC
Created attachment 1177420 [details]
Two stacked graphs displaying worker cpu usage during 4 C&U workload scenarios

Description of problem:
I ran 4 sets of 4 hour long workloads where:
1. Provider is added
2. C&U is turned on
3. Metrics collection and processing is occurring transparently to the user

Each workload uses a different provider:
Small/Medium/Large/XLarge - all of VMware type

Over two different appliance versions:
5.6.0.13
5.5.5.0

I then compared system resource utilization and found that the MiqVimBrokerWorker is utilizing the CPU far more in 5.6.0.13 than it previously did in 5.5.5.0

Version-Release number of selected component (if applicable):
5.6.0.13

How reproducible:
When compared correctly it is reproduced

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
This appears to be exacerbating a bottleneck in that we have a single point to communicate to the provider to capture metrics and that single point is using far more CPU than it used to in older builds.  Additionally more CPU utilization is occurring on the 5.6 builds (We are using the cpu more)


Attached Graph:
Top is 5.6, bottom is 5.5, each 4 hour segment is the next workload(C&U) scenario(Different provider) in order of 4 hours of small, 4 hours of medium, 4 hours of large, and 4 hours of xlarge. Note the filling out of available CPU on the 5.6 towards the Large/Xlarge scenarios.

Comment 2 Adam Grare 2016-10-31 18:08:44 UTC
It is possible this is also caused by running garbage collection every API call, https://bugzilla.redhat.com/show_bug.cgi?id=1341402#c3

I'll try to reproduce the high cpu load then revert that commit and see if the cpu load returns to normal levels.

Comment 3 Adam Grare 2016-11-02 02:10:29 UTC
Created attachment 1216327 [details]
VimBroker CPU Usage GC vs No-GC

This appears to be the same issue as https://bugzilla.redhat.com/show_bug.cgi?id=1341402, I enabled metrics collection on a 4096 VM/16 Cluster/1352 Datastore VC simulator with and without https://github.com/ManageIQ/manageiq/commit/4bbad2141b4ec820b56885584abd67ba94cf5569 on a current 5.6 appliance and the attachment shows the CPU usage.

Comment 6 Adam Grare 2016-11-29 18:32:47 UTC

*** This bug has been marked as a duplicate of bug 1378998 ***


Note You need to log in before you can comment on or make changes to this bug.