Created attachment 1074561 [details] vim.log output on info showing vim broker status taking ~28s and top output showing 99-100% cpu util on process during status output Description of problem: The VIMBroker spends ~28-30s to dump status to the vim.log however logging level is set to warn by default in advanced configuration thus the worker is busy without actually outputting its status to vim.log. On long running benchmarks this manifests itself by showing 30s of time where a vcpu is 100% utilized once every 15 minutes. Version-Release number of selected component (if applicable): 5.4.2.0 How reproducible: Always on 5.4.2.0 Steps to Reproduce: 1. Enable VIMBroker for large scale provider and observe cpu usage 2. 3. Actual results: In an idle cfme environment attached to a large scale provider (10,000 VMs), VIMBroker will consume 100% of a single vCPU for 30s every 15 minutes without actually outputting anything to vim.log by default. Expected results: Only until you enable vim.log to level info should the actual usage of the cpu really occur. Additional info: Attached vim.log showing status taking ~28s and output from top batched at 1s intervals filtered on VIMBroker PID displaying CPU usage
https://github.com/ManageIQ/manageiq/pull/4969
https://github.com/ManageIQ/manageiq/pull/4965
New commit detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/2516030b6747560140ff7da7bf007a8c31bf9825 commit 2516030b6747560140ff7da7bf007a8c31bf9825 Author: Joe Rafaniello <jrafanie> AuthorDate: Mon Oct 19 14:22:22 2015 -0400 Commit: Joe Rafaniello <jrafanie> CommitDate: Tue Oct 20 09:58:54 2015 -0400 Sync on lockHash only if we'll log something For the default logger level, warn, we'll acquire the lock, hold it, traverse the in memory cache, never log anything, and finally release the lock. Let's only do this when the log level is high enough to print the status as it will slow other threads by holding the lock. https://bugzilla.redhat.com/show_bug.cgi?id=1264188 gems/pending/VMwareWebService/MiqVimBroker.rb | 2 ++ 1 file changed, 2 insertions(+)
https://github.com/ManageIQ/manageiq-appliance-build/pull/67
Detected commit referencing this ticket while ticket status is POST.
https://github.com/ManageIQ/manageiq/pull/4936
Detected commit referencing this ticket while ticket status is MODIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2015:2551