Bug 1798668

Summary: Configure default MongoDB WiredTiger cache to be 20% of RAM in the Satellite server
Product: Red Hat Satellite Reporter: sthirugn <sthirugn>
Component: InstallationAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Devendra Singh <desingh>
Severity: medium Docs Contact:
Priority: high    
Version: 6.6.0CC: aperotti, desingh, egolov, ktordeur, mmccune, smajumda, spetrosi, wclark, wpinheir, zhunting
Target Milestone: 6.8.0Keywords: PrioBumpField, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: foreman-installer-2.1.0-0 Doc Type: Known Issue
Doc Text:
Cause: Mongod process with wiredTiger engine consumes up to half of RAM in Satellite6 Consequence: The aggressive default memory usage limits other processes or can cause OOM killer to kill the mongod process Workaround (if any): Override custom-hiera.yml configuring WiredTiger cache size to be 20% of the physical RAM in the server. More info in https://access.redhat.com/solutions/4505561 Result: The cache size limit lets mongo not consume more than 20% of the physical RAM in the server.
Story Points: ---
Clone Of:
: 1851158 (view as bug list) Environment:
Last Closed: 2020-10-27 13:00:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1399395    

Description sthirugn@redhat.com 2020-02-05 18:45:30 UTC
Description of problem:
Configure default MongoDB cache to be 20% of RAM in the Satellite server. See KCS [1] for more info.

Version-Release number of selected component (if applicable):
Satellite 6.6

How reproducible:
N/A

Steps to Reproduce:
1. Mongod process with the WiredTiger engine consumes up to 50% of RAM on Satellite. See https://docs.mongodb.com/manual/core/wiredtiger/#memory-use On larger Satellite installations, MongoDB ends up consuming a large amount of memory affecting other Satellite tasks.
2. With internal testing from the Performance Engineering team, it has been found that using 20% of RAM is ideal for MongoDB WiredTiger cache.

Actual results:
There is no tuning for WiredTiger cache size so it goes on to consume up to 50% of RAM in Satellite.

Expected results:
Configure default MongoDB WiredTiger cache size to be 20% of RAM in the Satellite server.

Additional info:
[1] https://access.redhat.com/solutions/4505561

Comment 7 sthirugn@redhat.com 2020-03-27 17:40:29 UTC
Created redmine issue https://projects.theforeman.org/issues/29425 from this bug

Comment 8 sthirugn@redhat.com 2020-03-27 17:53:44 UTC
Created redmine issue https://projects.theforeman.org/issues/29426 from this bug

Comment 9 Bryan Kearney 2020-03-30 16:02:18 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/29426 has been resolved.

Comment 10 Devendra Singh 2020-06-11 13:43:16 UTC
Build Version: 6.8 Snap3

Verification point:

* Changes reflected in /usr/share/foreman-installer/config/foreman.hiera/tuning/common.yaml

---
lookup_options:
  postgresql::server::config_entries:
    merge: hash
....
....
....
mongodb::server::config_data:
  storage.wiredTiger.engineConfig.cacheSizeGB: "%{facts.kafo.scenario.custom.mongo_cache_siz}"


* Satellite Server has actual Memory(In KiB): 19931328

  The default configured cache memory in mongod.conf(/etc/opt/rh/rh-mongodb34/mongod.conf):  storage.wiredTiger.engineConfig.cacheSizeGB: 3.8

*  db.serverStatus().mem
{
	"bits" : 64,
	"resident" : 2906,
	"virtual" : 4680,
	"supported" : true,
	"mapped" : 0,
	"mappedWithJournal" : 0
}


KiB Mem : 19931328 total,  1913628 free, 11558312 used,  6459388 buff/cache
KiB Swap:  1048572 total,   989180 free,    59392 used.  7845296 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                                                                                                         
28733 mongodb   20   0 4793152   2.8g   9268 S  0.0 14.9  48:31.17 mongod     


It seems implementation is working fine

@sthirugn,

Can you please confirm the above validation is sufficient?

Comment 11 Mike McCune 2020-06-22 14:31:23 UTC
The above validation is sufficient, looks good.

Comment 12 Devendra Singh 2020-06-22 15:34:01 UTC
Verified.

Comment 13 sthirugn@redhat.com 2020-07-02 23:04:04 UTC
(In reply to Devendra Singh from comment #10)
> Build Version: 6.8 Snap3
> 
> Verification point:
> 
> * Changes reflected in
> /usr/share/foreman-installer/config/foreman.hiera/tuning/common.yaml
> 
> ---
> lookup_options:
>   postgresql::server::config_entries:
>     merge: hash
> ....
> ....
> ....
> mongodb::server::config_data:
>   storage.wiredTiger.engineConfig.cacheSizeGB:
> "%{facts.kafo.scenario.custom.mongo_cache_siz}"

The parameter name above looks incorrect - it should be `facts.kafo.scenario.custom.mongo_cache_size` - `e` is missing in your paste, typo?

Upstream code is here for reference - https://github.com/theforeman/foreman-installer/blob/develop/config/foreman.hiera/tuning/common.yaml#L27

Comment 14 Devendra Singh 2020-07-06 04:54:18 UTC
(In reply to sthirugn from comment #13)
> (In reply to Devendra Singh from comment #10)
> > Build Version: 6.8 Snap3
> > 
> > Verification point:
> > 
> > * Changes reflected in
> > /usr/share/foreman-installer/config/foreman.hiera/tuning/common.yaml
> > 
> > ---
> > lookup_options:
> >   postgresql::server::config_entries:
> >     merge: hash
> > ....
> > ....
> > ....
> > mongodb::server::config_data:
> >   storage.wiredTiger.engineConfig.cacheSizeGB:
> > "%{facts.kafo.scenario.custom.mongo_cache_siz}"
> 
> The parameter name above looks incorrect - it should be
> `facts.kafo.scenario.custom.mongo_cache_size` - `e` is missing in your
> paste, typo?
> 
> Upstream code is here for reference -
> https://github.com/theforeman/foreman-installer/blob/develop/config/foreman.
> hiera/tuning/common.yaml#L27

yes, it was a typo in the paste from my-side

# cat /usr/share/foreman-installer/config/foreman.hiera/tuning/common.yaml
---
lookup_options:
  postgresql::server::config_entries:
    merge: hash
.......
.......

mongodb::server::config_data:
  storage.wiredTiger.engineConfig.cacheSizeGB: "%{facts.kafo.scenario.custom.mongo_cache_size}"

Comment 17 errata-xmlrpc 2020-10-27 13:00:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.8 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4366

Comment 18 Tanya Tereshchenko 2021-04-20 09:01:55 UTC
*** Bug 1791853 has been marked as a duplicate of this bug. ***