Bug 1526474

Summary: Large MiqServer process leads to large generic workers that get killed
Product: Red Hat CloudForms Management Engine Reporter: Satoe Imaishi <simaishi>
Component: PerformanceAssignee: Keenan Brock <kbrock>
Status: CLOSED NEXTRELEASE QA Contact: Dave Johnson <dajohnso>
Severity: high Docs Contact:
Priority: high    
Version: 5.8.0CC: abellott, bsorota, cchen, cpelland, dajohnso, dmetzger, epacific, fsimonce, hroy, jhardy, jrafanie, kbrock, mburman, myoder, niroy, obarenbo, pmcgowan, psuriset, simaishi, yzamir
Target Milestone: GAKeywords: ZStream
Target Release: 5.7.5   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: c&u:worker:perf
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1479356 Environment:
Last Closed: 2018-06-20 16:02:36 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: CFME Core Target Upstream Version:
Embargoed:
Bug Depends On: 1479356    
Bug Blocks:    

Comment 2 Joe Rafaniello 2017-12-15 15:54:34 UTC
Note, the changes to use unique set size (USS) instead of proportional set size (PSS) don't fix the server leaking.  We now will not penalize workers for inheriting a large amount of memory from a large miq server process when they're forked.  top will still show these workers as high memory usage (RSS) if they inherited memory from a large server.  You need to use tools such as smem, smem -P MIQ, to see the USS.  bin/rake evm:status will show the USS value now.

We will continue to track down and fix the server memory growth but at least now, we won't be prematurely killing workers.

Comment 3 CFME Bot 2017-12-15 17:06:34 UTC
New commit detected on ManageIQ/manageiq/euwe:
https://github.com/ManageIQ/manageiq/commit/06030f3826af407ef18ee54efcd0e0c5b48b8044

commit 06030f3826af407ef18ee54efcd0e0c5b48b8044
Author:     Joe Rafaniello <jrafanie>
AuthorDate: Mon Nov 13 16:26:26 2017 -0500
Commit:     Joe Rafaniello <jrafanie>
CommitDate: Fri Dec 15 11:18:36 2017 -0500

    Store unique set size (USS) in the PSS column
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1479356
    Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1526474
    
    Unique set size is a better way to detect workers that are growing
    unbounded since any memory/reference leaks would be shown in their
    uss.  If the server process is large when forking, new workers would
    inherit a big pss immediately.
    
    We should really rename the column/hash key to uss.

 gems/pending/util/miq-process.rb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)