Bug 1295741

Mongo 2.4.9 will have a CPU spike during syncs.

For example:

* create 10 products, each with 10 yum repos. However, make the repos identical. You can accomplish this by creating a zoo repo and then symlinking zoo1-10 to zoo, so the URLs are different. When testing this, I used 32 pkgs in the repo.

* sync each repo

result: high mongo CPU usage, and this warning from pulp:

pulp: pulp_rpm.plugins.importers.yum.purge:INFO: Purging duplicate NEVRA can take significantly longer in versions of mongodb lower than 2.6. Consider upgrading mongodb if cleaning duplicate packages takes an unreasonably long time.

Note that the sync time remains constant over time, and there's no add'l memory consumption.

On 6.2, syncing a small repo that is defined 100x:

# time hammer --username admin --password changeme repository synchronize --organization perf-org-1 --product perf-product-10 --
name perf-repo-10
[..................................................................................................................................................] [100%]
No new packages.


real    0m49.286s


after upgrading to mongodb 2.6.11:

# time hammer --username admin --password changeme repository synchronize --organization perf-org-1 --product perf-product-10
 --name perf-repo-10
[..................................................................................................................................................] [100%]
No new packages.


real    0m12.354s


note: i re-ran createrepo on the repo between each run to make sure the repomd.xml timestamp was getting updated.

Chris

You tried concurrent sync? 

--Pradeep

This was not concurrent, it was just one sync. I think there is a code change in Pulp 2.8 that makes sync significantly faster, but it uses newer (2.6+) versions of mongodb.

I did not try with Mongo 3, just 2.6.

Per 6.3 planning, moving out non acked bugs to the backlog

*** Bug 1439083 has been marked as a duplicate of this bug. ***

As information of a satellite in operation how big mongodb filesystems can grow over time:

/dev/mapper/vg_li_lc_1017_app1-lv_mongodb   150G  130G   21G  87% /var/lib/mongodb

Current mongodb size on by Sat6 instance that handles frequent Content (View) changes:

[crash/LI] root@li-lc-1017:~# df -h /var/lib/mongodb
Filesystem                                 Size  Used Avail Use% Mounted on
/dev/mapper/vg_li_lc_1017_app1-lv_mongodb  200G  185G   16G  93% /var/lib/mongodb

That is about 10-15G per month growth after i posted comment 11 above.

Verified in Satellite 6.4

# mongo
MongoDB shell version v3.4.9
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.4.9
Server has startup warnings: 
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] 
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] ** WARNING: You are running on a NUMA machine.
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] **              numactl --interleave=all mongod [other options]
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] 
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] 
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-08-22T18:08:22.599+0200 I CONTROL  [initandlisten] 
> db.version()
3.4.9

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2927