Bug 1834377

Summary: Disable mongo FTDC
Product: Red Hat Satellite Reporter: Pavel Moravec <pmoravec>
Component: InstallationAssignee: Eric Helms <ehelms>
Status: CLOSED ERRATA QA Contact: Devendra Singh <desingh>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.7.0CC: zhunting
Target Milestone: 6.8.0Keywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: foreman-installer-2.1.2.4-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-27 13:02:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pavel Moravec 2020-05-11 15:43:53 UTC 
Description of problem:
By default, mongo collects FTDC (Full Time Diagnostic Data Capture). See https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/ and grep for FTDC. Checking thread CPU usage, I noticed that ftdc thread responsible for this consumes nontrivial amount of CPU. On idle-ish Satellite, it is the most busy thread, on a very busy Satellite, it consumes 5% of overall mongod CPU time.

We have not utilised the collected data under /var/lib/mongodb/diagnostic.data/ so we should disable collecting the FTDC by default and save 5%+ CPU time of mongod.

I understand mongod will be deprecated since Sat7, letting on PrdM and Eng to decide if it is worth disabling since then.

It is sufficient to change /etc/opt/rh/rh-mongodb34/sysconfig/mongod to:

OPTIONS="-f /etc/opt/rh/rh-mongodb34/mongod.conf --setParameter diagnosticDataCollectionEnabled=false"


Version-Release number of selected component (if applicable):
Sat 6.7


How reproducible:
100%


Steps to Reproduce:
1. Deploy a Sat, produce a load
2. /bin/ps -p $(pgrep mongod) -L -o pid,tid,psr,pcpu,comm= | sort -nk4


Actual results:
2. shows one of busiest threads is:
  PID   TID PSR %CPU 
..
15086 15106  11  1.4 ftdc


Expected results:
ftdc does not consume CPU


Additional info:
Comment 3 Eric Helms 2020-08-27 14:15:43 UTC 
Created redmine issue https://projects.theforeman.org/issues/30721 from this bug
Comment 4 Bryan Kearney 2020-08-27 16:04:15 UTC 
Upstream bug assigned to ehelms
Comment 5 Bryan Kearney 2020-08-27 16:04:16 UTC 
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/30721 has been resolved.
Comment 7 Devendra Singh 2020-09-07 13:09:18 UTC 
Verified on 6.8 Snap14

Verification points:

1- # rpm -qa|grep foreman-installer
foreman-installer-2.1.2.4-1.el7sat.noarch

2- Changes of "diagnosticDataCollectionEnabled" present in /etc/opt/rh/rh-mongodb34/mongod.conf.

cat /etc/opt/rh/rh-mongodb34/mongod.conf
#mongodb.conf - generated from Puppet

#System Log
...
...
setParameter: {diagnosticDataCollectionEnabled: false}

storage.wiredTiger.engineConfig.cacheSizeGB: 3.8

3. FTDC process consume 0% CPU.

# /bin/ps -p $(pgrep mongod) -L -o pid,tid,psr,pcpu,comm= | sort -nk4
PID   TID PSR %CPU 
 1690  1692   2  0.1 Backgro.kSource
 1690  5249   0  0.1 conn77
 1690  5318   1  0.2 conn79
 1690  5296   2  0.5 conn78
 ...
 1690  1708   2  0.0 ftdc
Comment 10 errata-xmlrpc 2020-10-27 13:02:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.8 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4366