Bug 1303347 - After upgrade mongodb from 2.4.9 to 2.6.9 our queries on mongo is very slow
After upgrade mongodb from 2.4.9 to 2.6.9 our queries on mongo is very slow
Product: Red Hat OpenStack
Classification: Red Hat
Component: mongodb (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
high Severity unspecified
: async
: 7.0 (Kilo)
Assigned To: Flavio Percoco
: Rebase, Triaged, ZStream
Depends On:
Blocks: 1308855
  Show dependency treegraph
Reported: 2016-01-31 04:40 EST by Eduard Barrera
Modified: 2018-02-08 06:11 EST (History)
9 users (show)

See Also:
Fixed In Version: mongodb-2.6.11-1.el7ost
Doc Type: Rebase: Bug Fixes Only
Doc Text:
This rebase introduces improvements on the performance for range queries. Specifically, queries that used the `$or` operator were affected with the 2.4 release. Those regressions have now been fixed in the 2.6 release.
Story Points: ---
Clone Of:
: 1308855 (view as bug list)
Last Closed: 2016-04-26 11:29:19 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Eduard Barrera 2016-01-31 04:40:18 EST
Description of problem:

If I execute the same query on mongo 2.4 the result is very fast, but on 2.6 is very slow, I found a bug [1] on mongo, but this bug was fixed on 2.6.11, I would like to know when Red Hat will be release this version (2.6.11) or how to fix that?

[1] https://jira.mongodb.org/browse/SERVER-18364	

We made tests with ceilometer API and using direct query in MongoDB and the results is almost 40% slower than the old Mongo Version (2.4.9). What's going on?

Last weekend we rebuilt all the index and we had some improvement, around 10% of performance improvements

Version-Release number of selected component (if applicable):
mongodb 2.6.9

How reproducible:

Steps to Reproduce:
1. Do a direct query to mongodb via Ceilometer API

Actual results:
The query is 40% slower

Expected results:
Similar performance results
Comment 4 Flavio Percoco 2016-02-08 08:16:28 EST
Upgrading to mongodb 2.6.11 sounds straightforward enough but, before we do that, I'd like to understand better what query is causing issues. By looking at the customer case, it is not clear what query is slower in 2.6. 

Can we get the exact query?

In addition to the above, I'd be interested to know how the upgrade to 2.6 was done.

If the query took longer than 100ms, it should have been logged to the mongodb server's log. Can we get the logs from the whole replica set?
Comment 5 Eduard Barrera 2016-02-15 06:59:49 EST
They have mongodb in a iscsi volume. They just attched the lun to another server with mongodb 2.6 and replicated all data to another mongos running also mongo 2.6 using ISCSI.

When they started mongo after all replication, we observe that all queries was very slow. During the weekend we rebuild all indexes and the performance go up in 10%, but still it's very slow than before

After search on the internet,  we tried the db.upgradeCheckAllDBs() command to check if the upgrade was possible and the command return Ok to upgrade.

Attaching files:
Comment 6 Flavio Percoco 2016-02-15 11:24:42 EST

Any chance you can help updating mongodb's package to 2.6.11 ?

Thank you
Comment 14 Mehdi ABAAKOUK 2016-04-07 12:23:31 EDT
I have made some tests of the new package on a fresh rhosp 7, the package itself works well, but I didn't any performance improvements with 'ceilometer sample-list' command. 

I'm not surprise to not trigger the mongodb bug with this command. I will try to find another command that can trigger it.

If you can provide a ceilometer command that you known is slower as before it will help my investigation.

Extract of my testing:

# rpm -q openstack-ceilometer-api

# rpm -q mongodb-server

# du -sh /var/lib/mongodb/
4.3G    /var/lib/mongodb/

# mongo ceilometer --eval 'db.meter.count()'  
MongoDB shell version: 2.6.9
connecting to: ceilometer

# time ceilometer --timeout 1000000000 --ceilometer-url http://localhost:8777 --os-auth-token foo sample-list  --limit 100000 >/dev/null
real    1m14.967s
user    0m30.342s
sys 0m0.282s

# ls -1 mongod*.rpm

# rpm -U mongod*.rpm
# systemctl restart mongod

# time ceilometer --timeout 1000000000 --ceilometer-url http://localhost:8777 --os-auth-token foo sample-list  --limit 100000 >/dev/null
real    1m14.579s
user    0m30.208s
sys 0m0.293s

# mongo ceilometer --eval 'db.meter.reIndex()'

# time ceilometer --timeout 1000000000 --ceilometer-url http://localhost:8777 --os-auth-token foo sample-list  --limit 100000 >/dev/null
real    1m14.659s
user    0m29.998s
sys 0m0.438s
Comment 15 Mehdi ABAAKOUK 2016-04-08 04:39:35 EDT
I have made some additional queries:

ceilometer --timeout 1000000000 --ceilometer-url http://localhost:8777 --os-auth-token foo query-samples --filter '{"and": [{">": {"timestamp": "2011-03-20T18:15:00"}}, {"<": {"timestamp": "2017-03-20T18:15:00"}} ]}' --limit 50000

ceilometer --timeout 1000000000 --ceilometer-url http://localhost:8777 --os-auth-token foo sample-list -q 'timestamp>=2011-03-20T18:15:00;timestamp<2017-03-20T18:15:00' --limit 50000

ceilometer --timeout 1000000000 --ceilometer-url http://localhost:8777 --os-auth-token foo statistics -m instance

And done them on mongo 2.4.9-3, 2.6.9-1, 2.6.11-1, on an standalone mongo and then on a master/slave mongos.

The package upgrade works well, but I can't reproduce the performance issue, I got same number with all mongo version.
Comment 16 Mehdi ABAAKOUK 2016-04-08 05:19:57 EDT
I have take a look into the Ceilometer code, and even I don't see the issue on my setup, Ceilometer can clearly easily trigger this mongo bug, when timestamp range are used.
Comment 25 nlevinki 2016-04-26 03:04:16 EDT
mongodb is using mongodb-2.6.11-1.el7ost.x86_64.rpm   
fix is working according to commet 22
Comment 27 errata-xmlrpc 2016-04-26 11:29:19 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.