Red Hat Bugzilla – Bug 1264188
[Scale] - VIMBroker spends ~28s hot on a vcpu while outputting status to vim.log every 15minutes on large scale vmware provider
Last modified: 2015-12-08 08:31:05 EST
Created attachment 1074561 [details]
vim.log output on info showing vim broker status taking ~28s and top output showing 99-100% cpu util on process during status output
Description of problem:
The VIMBroker spends ~28-30s to dump status to the vim.log however logging level is set to warn by default in advanced configuration thus the worker is busy without actually outputting its status to vim.log. On long running benchmarks this manifests itself by showing 30s of time where a vcpu is 100% utilized once every 15 minutes.
Version-Release number of selected component (if applicable):
Always on 188.8.131.52
Steps to Reproduce:
1. Enable VIMBroker for large scale provider and observe cpu usage
In an idle cfme environment attached to a large scale provider (10,000 VMs), VIMBroker will consume 100% of a single vCPU for 30s every 15 minutes without actually outputting anything to vim.log by default.
Only until you enable vim.log to level info should the actual usage of the cpu really occur.
Attached vim.log showing status taking ~28s and output from top batched at 1s intervals filtered on VIMBroker PID displaying CPU usage
New commit detected on ManageIQ/manageiq/master:
Author: Joe Rafaniello <email@example.com>
AuthorDate: Mon Oct 19 14:22:22 2015 -0400
Commit: Joe Rafaniello <firstname.lastname@example.org>
CommitDate: Tue Oct 20 09:58:54 2015 -0400
Sync on lockHash only if we'll log something
For the default logger level, warn, we'll acquire the lock, hold it,
traverse the in memory cache, never log anything, and finally release
Let's only do this when the log level is high enough to print the
status as it will slow other threads by holding the lock.
gems/pending/VMwareWebService/MiqVimBroker.rb | 2 ++
1 file changed, 2 insertions(+)
Detected commit referencing this ticket while ticket status is POST.
Detected commit referencing this ticket while ticket status is MODIFIED.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.