Bug 1327563

Summary: Excessive swapping in 5.6 beta appliances
Product: Red Hat CloudForms Management Engine Reporter: Milan Falešník <mfalesni>
Component: ApplianceAssignee: Gregg Tanzillo <gtanzill>
Status: CLOSED WORKSFORME QA Contact: Milan Falešník <mfalesni>
Severity: high Docs Contact:
Priority: unspecified    
Version: 5.6.0CC: abellott, cpelland, jhardy, obarenbo, thenness
Target Milestone: Beta 2   
Target Release: 5.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: appliance:perf
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-04-18 10:47:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: Bug
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
screenshot of top on an appliance managing QE's vsphere55, ~10 minutes after adding it.
none
evm.log file up to the point where the evmserverd service is stopped
none
standard top reports showing pid count growth, memory and swap growth and cpu idle
none
top output for detailed process analysis
none
summary page showing counts of processes by worker type none

Description Milan Falešník 2016-04-15 11:55:25 UTC
Created attachment 1147617 [details]
screenshot of top on an appliance managing QE's vsphere55, ~10 minutes after adding it.

Description of problem:
In QE, we have discovered that the 5.6 appliances develop excessive swapping with the same load as eg. 5.5 takes without problems. This has bad effects on our QE infrastructure.


Version-Release number of selected component (if applicable):
5.6.0.0-beta2

How reproducible:
always

Steps to Reproduce:
1. Deploy the appliance
2. Add a provider (eg. QE's vsphere55)
3. Wait and watch eg. top

Actual results:
Some swapping (in hundreds of MB) develops quite fast (~5min). We have also seen appliances taking 6G of memory, those probably managed more providers

Expected results:
Nothing special happens.

Additional info:

Comment 2 Thomas Hennessy 2016-04-15 13:47:17 UTC
Created attachment 1147651 [details]
evm.log file up to the point where the evmserverd service is stopped

Comment 3 Thomas Hennessy 2016-04-15 13:48:36 UTC
Created attachment 1147652 [details]
standard top reports showing pid count growth, memory and swap growth and cpu idle

Comment 4 Thomas Hennessy 2016-04-15 13:49:37 UTC
Created attachment 1147653 [details]
top output for detailed process analysis

Comment 5 Thomas Hennessy 2016-04-15 13:51:33 UTC
Created attachment 1147655 [details]
summary page showing counts of processes by worker type

Comment 7 Dave Johnson 2016-04-18 10:47:27 UTC
So we believe this was all caused by appliances being sent to the wrong (smaller) osp flavor, appliance is running much better under 8GB of memory.