Bug 1526473

Summary:	Large MiqServer process leads to large generic workers that get killed
Product:	Red Hat CloudForms Management Engine	Reporter:	Satoe Imaishi <simaishi>
Component:	Performance	Assignee:	Keenan Brock <kbrock>
Status:	CLOSED ERRATA	QA Contact:	Tasos Papaioannou <tpapaioa>
Severity:	high	Docs Contact:
Priority:	high
Version:	5.8.0	CC:	abellott, arcsharm, bsorota, cchen, dajohnso, dmetzger, epacific, fsimonce, hroy, jhardy, jrafanie, kbrock, mburman, myoder, niroy, obarenbo, otsuman, pmcgowan, psuriset, simaishi, tpapaioa, yzamir
Target Milestone:	GA	Keywords:	ZStream
Target Release:	5.8.3
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	c&u:worker:perf
Fixed In Version:	5.8.3.1	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1479356	Environment:
Last Closed:	2018-02-28 13:11:30 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	CFME Core	Target Upstream Version:
Embargoed:
Bug Depends On:	1479356
Bug Blocks:

Comment 2 Joe Rafaniello 2017-12-15 15:54:31 UTC

Note, the changes to use unique set size (USS) instead of proportional set size (PSS) don't fix the server leaking.  We now will not penalize workers for inheriting a large amount of memory from a large miq server process when they're forked.  top will still show these workers as high memory usage (RSS) if they inherited memory from a large server.  You need to use tools such as smem, smem -P MIQ, to see the USS.  bin/rake evm:status will show the USS value now.

We will continue to track down and fix the server memory growth but at least now, we won't be prematurely killing workers.

Comment 3 CFME Bot 2018-01-03 16:34:50 UTC

New commit detected on ManageIQ/manageiq-gems-pending/fine:
https://github.com/ManageIQ/manageiq-gems-pending/commit/d51285ca19c96304c7e3b521ae16713d75cfcee1

commit d51285ca19c96304c7e3b521ae16713d75cfcee1
Author:     Joe Rafaniello <jrafanie>
AuthorDate: Mon Nov 13 16:26:26 2017 -0500
Commit:     Joe Rafaniello <jrafanie>
CommitDate: Fri Dec 15 11:20:13 2017 -0500

    Store unique set size (USS) in the PSS column
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1479356
    Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1526473
    
    Unique set size is a better way to detect workers that are growing
    unbounded since any memory/reference leaks would be shown in their
    uss.  If the server process is large when forking, new workers would
    inherit a big pss immediately.
    
    We should really rename the column/hash key to uss.

 lib/gems/pending/util/miq-process.rb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comment 4 Tasos Papaioannou 2018-01-24 17:58:47 UTC

Verified on 5.8.3.1.

Comment 7 errata-xmlrpc 2018-02-28 13:11:30 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0374