Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.

Bug 1295741

Summary: [RFE] Update mongodb to 3.X
Product: Red Hat Satellite 6 Reporter: Pradeep Kumar Surisetty <psuriset>
Component: PerformanceAssignee: Chris Roberts <chrobert>
Status: VERIFIED --- QA Contact: Corey Welton <cwelton>
Severity: high Docs Contact:
Priority: medium    
Version: 6.1.4CC: ajambhul, aperotti, bkearney, cduryee, chrobert, dgross, ehelms, jpriddy, mmccune, peter.vreman, sreber, vanhoof
Target Milestone: UnspecifiedKeywords: FutureFeature, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
URL: https://projects.theforeman.org/issues/23030
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 1122832, 1545876, 1546813    

Description Pradeep Kumar Surisetty 2016-01-05 06:35:46 EST
Description of problem:

we use mongodb 2.4 in latest satelitte-6.1
mongodb-3.0 provides better scalability and performance improvements.
Its high time to update mongodb

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Chris Duryee 2016-03-17 12:47:42 EDT
Mongo 2.4.9 will have a CPU spike during syncs.

For example:

* create 10 products, each with 10 yum repos. However, make the repos identical. You can accomplish this by creating a zoo repo and then symlinking zoo1-10 to zoo, so the URLs are different. When testing this, I used 32 pkgs in the repo.

* sync each repo

result: high mongo CPU usage, and this warning from pulp:

pulp: pulp_rpm.plugins.importers.yum.purge:INFO: Purging duplicate NEVRA can take significantly longer in versions of mongodb lower than 2.6. Consider upgrading mongodb if cleaning duplicate packages takes an unreasonably long time.

Note that the sync time remains constant over time, and there's no add'l memory consumption.
Comment 2 Chris Duryee 2016-03-17 13:15:42 EDT
On 6.2, syncing a small repo that is defined 100x:

# time hammer --username admin --password changeme repository synchronize --organization perf-org-1 --product perf-product-10 --
name perf-repo-10
[..................................................................................................................................................] [100%]
No new packages.


real    0m49.286s


after upgrading to mongodb 2.6.11:

# time hammer --username admin --password changeme repository synchronize --organization perf-org-1 --product perf-product-10
 --name perf-repo-10
[..................................................................................................................................................] [100%]
No new packages.


real    0m12.354s


note: i re-ran createrepo on the repo between each run to make sure the repomd.xml timestamp was getting updated.
Comment 3 Pradeep Kumar Surisetty 2016-03-23 00:23:45 EDT
Chris

You tried concurrent sync? 

--Pradeep
Comment 4 Chris Duryee 2016-03-23 08:44:50 EDT
This was not concurrent, it was just one sync. I think there is a code change in Pulp 2.8 that makes sync significantly faster, but it uses newer (2.6+) versions of mongodb.

I did not try with Mongo 3, just 2.6.
Comment 7 Bryan Kearney 2016-07-08 16:48:46 EDT
Per 6.3 planning, moving out non acked bugs to the backlog
Comment 9 Bryan Kearney 2017-04-06 16:22:43 EDT
*** Bug 1439083 has been marked as a duplicate of this bug. ***
Comment 11 Peter Vreman 2017-08-07 09:18:12 EDT
As information of a satellite in operation how big mongodb filesystems can grow over time:

/dev/mapper/vg_li_lc_1017_app1-lv_mongodb   150G  130G   21G  87% /var/lib/mongodb
Comment 12 Peter Vreman 2017-12-18 11:14:31 EST
Current mongodb size on by Sat6 instance that handles frequent Content (View) changes:

[crash/LI] root@li-lc-1017:~# df -h /var/lib/mongodb
Filesystem                                 Size  Used Avail Use% Mounted on
/dev/mapper/vg_li_lc_1017_app1-lv_mongodb  200G  185G   16G  93% /var/lib/mongodb

That is about 10-15G per month growth after i posted comment 11 above.
Comment 16 Chris Brown 2018-08-23 14:55:23 EDT
Verified in Satellite 6.4

# mongo
MongoDB shell version v3.4.9
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.9
Server has startup warnings: 
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] 
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] ** WARNING: You are running on a NUMA machine.
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] **              numactl --interleave=all mongod [other options]
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] 
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] 
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] 
> db.version()
3.4.9