Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1798668 - Configure default MongoDB WiredTiger cache to be 20% of RAM in the Satellite server
Summary: Configure default MongoDB WiredTiger cache to be 20% of RAM in the Satellite ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Installation
Version: 6.6.0
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: 6.8.0
Assignee: satellite6-bugs
QA Contact: Devendra Singh
URL:
Whiteboard:
: 1791853 (view as bug list)
Depends On:
Blocks: 1399395
TreeView+ depends on / blocked
 
Reported: 2020-02-05 18:45 UTC by sthirugn@redhat.com
Modified: 2023-10-06 19:08 UTC (History)
10 users (show)

Fixed In Version: foreman-installer-2.1.0-0
Doc Type: Known Issue
Doc Text:
Cause: Mongod process with wiredTiger engine consumes up to half of RAM in Satellite6 Consequence: The aggressive default memory usage limits other processes or can cause OOM killer to kill the mongod process Workaround (if any): Override custom-hiera.yml configuring WiredTiger cache size to be 20% of the physical RAM in the server. More info in https://access.redhat.com/solutions/4505561 Result: The cache size limit lets mongo not consume more than 20% of the physical RAM in the server.
Clone Of:
: 1851158 (view as bug list)
Environment:
Last Closed: 2020-10-27 13:00:15 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 29426 0 High Closed Configure default MongoDB WiredTiger cache to be 20% of RAM 2021-02-16 19:11:29 UTC
Red Hat Knowledge Base (Solution) 4505561 0 None None None 2020-02-05 18:46:57 UTC
Red Hat Product Errata RHSA-2020:4366 0 None None None 2020-10-27 13:00:36 UTC

Description sthirugn@redhat.com 2020-02-05 18:45:30 UTC 
Description of problem:
Configure default MongoDB cache to be 20% of RAM in the Satellite server. See KCS [1] for more info.

Version-Release number of selected component (if applicable):
Satellite 6.6

How reproducible:
N/A

Steps to Reproduce:
1. Mongod process with the WiredTiger engine consumes up to 50% of RAM on Satellite. See https://docs.mongodb.com/manual/core/wiredtiger/#memory-use On larger Satellite installations, MongoDB ends up consuming a large amount of memory affecting other Satellite tasks.
2. With internal testing from the Performance Engineering team, it has been found that using 20% of RAM is ideal for MongoDB WiredTiger cache.

Actual results:
There is no tuning for WiredTiger cache size so it goes on to consume up to 50% of RAM in Satellite.

Expected results:
Configure default MongoDB WiredTiger cache size to be 20% of RAM in the Satellite server.

Additional info:
[1] https://access.redhat.com/solutions/4505561
Comment 7 sthirugn@redhat.com 2020-03-27 17:40:29 UTC 
Created redmine issue https://projects.theforeman.org/issues/29425 from this bug
Comment 8 sthirugn@redhat.com 2020-03-27 17:53:44 UTC 
Created redmine issue https://projects.theforeman.org/issues/29426 from this bug
Comment 9 Bryan Kearney 2020-03-30 16:02:18 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/29426 has been resolved.
Comment 10 Devendra Singh 2020-06-11 13:43:16 UTC 
Build Version: 6.8 Snap3

Verification point:

* Changes reflected in /usr/share/foreman-installer/config/foreman.hiera/tuning/common.yaml

---
lookup_options:
  postgresql::server::config_entries:
    merge: hash
....
....
....
mongodb::server::config_data:
  storage.wiredTiger.engineConfig.cacheSizeGB: "%{facts.kafo.scenario.custom.mongo_cache_siz}"


* Satellite Server has actual Memory(In KiB): 19931328

  The default configured cache memory in mongod.conf(/etc/opt/rh/rh-mongodb34/mongod.conf):  storage.wiredTiger.engineConfig.cacheSizeGB: 3.8

*  db.serverStatus().mem
{
	"bits" : 64,
	"resident" : 2906,
	"virtual" : 4680,
	"supported" : true,
	"mapped" : 0,
	"mappedWithJournal" : 0
}


KiB Mem : 19931328 total,  1913628 free, 11558312 used,  6459388 buff/cache
KiB Swap:  1048572 total,   989180 free,    59392 used.  7845296 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                                                                                                         
28733 mongodb   20   0 4793152   2.8g   9268 S  0.0 14.9  48:31.17 mongod     


It seems implementation is working fine

@sthirugn,

Can you please confirm the above validation is sufficient?
Comment 11 Mike McCune 2020-06-22 14:31:23 UTC 
The above validation is sufficient, looks good.
Comment 12 Devendra Singh 2020-06-22 15:34:01 UTC 
Verified.
Comment 13 sthirugn@redhat.com 2020-07-02 23:04:04 UTC 
(In reply to Devendra Singh from comment #10)
> Build Version: 6.8 Snap3
> 
> Verification point:
> 
> * Changes reflected in
> /usr/share/foreman-installer/config/foreman.hiera/tuning/common.yaml
> 
> ---
> lookup_options:
>   postgresql::server::config_entries:
>     merge: hash
> ....
> ....
> ....
> mongodb::server::config_data:
>   storage.wiredTiger.engineConfig.cacheSizeGB:
> "%{facts.kafo.scenario.custom.mongo_cache_siz}"

The parameter name above looks incorrect - it should be `facts.kafo.scenario.custom.mongo_cache_size` - `e` is missing in your paste, typo?

Upstream code is here for reference - https://github.com/theforeman/foreman-installer/blob/develop/config/foreman.hiera/tuning/common.yaml#L27
Comment 14 Devendra Singh 2020-07-06 04:54:18 UTC 
(In reply to sthirugn from comment #13)
> (In reply to Devendra Singh from comment #10)
> > Build Version: 6.8 Snap3
> > 
> > Verification point:
> > 
> > * Changes reflected in
> > /usr/share/foreman-installer/config/foreman.hiera/tuning/common.yaml
> > 
> > ---
> > lookup_options:
> >   postgresql::server::config_entries:
> >     merge: hash
> > ....
> > ....
> > ....
> > mongodb::server::config_data:
> >   storage.wiredTiger.engineConfig.cacheSizeGB:
> > "%{facts.kafo.scenario.custom.mongo_cache_siz}"
> 
> The parameter name above looks incorrect - it should be
> `facts.kafo.scenario.custom.mongo_cache_size` - `e` is missing in your
> paste, typo?
> 
> Upstream code is here for reference -
> https://github.com/theforeman/foreman-installer/blob/develop/config/foreman.
> hiera/tuning/common.yaml#L27

yes, it was a typo in the paste from my-side

# cat /usr/share/foreman-installer/config/foreman.hiera/tuning/common.yaml
---
lookup_options:
  postgresql::server::config_entries:
    merge: hash
.......
.......

mongodb::server::config_data:
  storage.wiredTiger.engineConfig.cacheSizeGB: "%{facts.kafo.scenario.custom.mongo_cache_size}"
Comment 17 errata-xmlrpc 2020-10-27 13:00:15 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.8 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4366
Comment 18 Tanya Tereshchenko 2021-04-20 09:01:55 UTC 
*** Bug 1791853 has been marked as a duplicate of this bug. ***
Note You need to log in before you can comment on or make changes to this bug.