Fedora Account System
Red Hat Associate
Red Hat Customer
Created attachment 1042519 [details] one pid log of the several ems refresh workers that have all encountered the error contained in this log Description of problem:ems refresh for provider encounters error while gathering inventory information leaving valididty of refresh in doubt. Version-Release number of selected component (if applicable): Version: 5.4.0.5 Build: 20150605150206_7daa1a8 How reproducible: Error gathered from customer logs reporting various problems with SCVMM provider. Steps to Reproduce: 1. 2. 3. Actual results: error log line is logged roughly 26 minutes after refresh begins and roughly 4 minutes before information captured by ems refresh begins to be logged. Expected results:expect ems refresh to complete without error Additional info: Duration for ems refresh from SCVMM provider with about 860 VMs is roughly 30 minutes and requires more than 3 gigabytes of memory which seems both unusually long for ems refresh and unusually memory hungry.
The error in the log seems to be legitimate. Is the *real* problem that SCVMM refresh takes too long and consumes too much memory? Or, that there's an error in the log? Or, is the suggestion that the error in the log is caused by the refresh taking too long and consuming too much memory? If it's a memory/runtime problem, let's ignore the log for now, and change the summary of this bug to reflect the real issue to be investigated. And we can focus on fixing up the resource consumption of the refresher. However, if the error in the log is the real problem, then I think the customer needs to examine the error and take the necessary action in their SCVMM environment. Namely, "Virtual Guest Services is not installed on the guest operating system".
Greg, there are two errors here: 1- a: the posting of an 'ERROR' type log line in the evm.log suggests that there is an issue to which the customer, GPS, a partner, or GSS might reasonable be expected to correct. However as the error is presented in the raw xml format as returned from the provider, it is, in my humble opinion, completely unintelible and unactionable. 1- b: in the research I ahve performance each and every testing occurance of ems refreshes for SCVMM providers generates such ERROR instances. Since they are unintelible to me it is not clear if the message is routine and ignoreable (in which case it should not be generated into the evm.log) or is exposing some condition which is material to the operation of CFME and needs to be corrected. 2- the memory consumed and retained by the ems refresh worker for the SCVMM is at least twice the size of any other refresh worker for any other provider. At this customer, the initial refresh for 860 VMs exceeded the default 2GB limit causing the ems refresh worker to fail repeatedly for 'memory limit exceeded' reason. Increasing the limit to 4GB allowed the initial ems refresh to complete, but the second ems refresh caused the memory consumed by the ems refresh worker for the same number of VMs to expand beyond 5GB. This is simply not sustainable. There is either a memory leak in this worker or some other condition which seems to make the continuous running of this worker impossible where the size of the entire appliance real memory is by default 8GB and we know from experience that forcing the memory used by other workers into swap creates a situation where the appliance becomes unacceptable unresponsive. If you prefer that an additional BZ be opened for one or the other of these, please advise. I hope this answeres your question.
This bug really represents two different bugs: * error in the logs: https://bugzilla.redhat.com/show_bug.cgi?id=1210657 * refresh takes too long: https://bugzilla.redhat.com/show_bug.cgi?id=1235384 I'm marking this as a dup of "refresh takes too long". *** This bug has been marked as a duplicate of bug 1235384 ***