Description of problem: If I execute the same query on mongo 2.4 the result is very fast, but on 2.6 is very slow, I found a bug [1] on mongo, but this bug was fixed on 2.6.11, I would like to know when Red Hat will be release this version (2.6.11) or how to fix that? [1] https://jira.mongodb.org/browse/SERVER-18364 We made tests with ceilometer API and using direct query in MongoDB and the results is almost 40% slower than the old Mongo Version (2.4.9). What's going on? Last weekend we rebuilt all the index and we had some improvement, around 10% of performance improvements Version-Release number of selected component (if applicable): mongodb 2.6.9 How reproducible: 100% Steps to Reproduce: 1. Do a direct query to mongodb via Ceilometer API 2. 3. Actual results: The query is 40% slower Expected results: Similar performance results
Upgrading to mongodb 2.6.11 sounds straightforward enough but, before we do that, I'd like to understand better what query is causing issues. By looking at the customer case, it is not clear what query is slower in 2.6. Can we get the exact query? In addition to the above, I'd be interested to know how the upgrade to 2.6 was done. If the query took longer than 100ms, it should have been logged to the mongodb server's log. Can we get the logs from the whole replica set?
They have mongodb in a iscsi volume. They just attched the lun to another server with mongodb 2.6 and replicated all data to another mongos running also mongo 2.6 using ISCSI. When they started mongo after all replication, we observe that all queries was very slow. During the weekend we rebuild all indexes and the performance go up in 10%, but still it's very slow than before After search on the internet, we tried the db.upgradeCheckAllDBs() command to check if the upgrade was possible and the command return Ok to upgrade. Attaching files: http://10.12.211.122/01575671
Haikel, Any chance you can help updating mongodb's package to 2.6.11 ? Thank you
I have made some tests of the new package on a fresh rhosp 7, the package itself works well, but I didn't any performance improvements with 'ceilometer sample-list' command. I'm not surprise to not trigger the mongodb bug with this command. I will try to find another command that can trigger it. If you can provide a ceilometer command that you known is slower as before it will help my investigation. Extract of my testing: # rpm -q openstack-ceilometer-api openstack-ceilometer-api-2015.1.2-1.el7ost.noarch # rpm -q mongodb-server mongodb-server-2.6.9-1.el7ost.x86_64 # du -sh /var/lib/mongodb/ 4.3G /var/lib/mongodb/ # mongo ceilometer --eval 'db.meter.count()' MongoDB shell version: 2.6.9 connecting to: ceilometer 435463 # time ceilometer --timeout 1000000000 --ceilometer-url http://localhost:8777 --os-auth-token foo sample-list --limit 100000 >/dev/null real 1m14.967s user 0m30.342s sys 0m0.282s # ls -1 mongod*.rpm mongodb-2.6.11-1.el7ost.x86_64.rpm mongodb-server-2.6.11-1.el7ost.x86_64.rpm # rpm -U mongod*.rpm # systemctl restart mongod # time ceilometer --timeout 1000000000 --ceilometer-url http://localhost:8777 --os-auth-token foo sample-list --limit 100000 >/dev/null real 1m14.579s user 0m30.208s sys 0m0.293s # mongo ceilometer --eval 'db.meter.reIndex()' # time ceilometer --timeout 1000000000 --ceilometer-url http://localhost:8777 --os-auth-token foo sample-list --limit 100000 >/dev/null real 1m14.659s user 0m29.998s sys 0m0.438s
I have made some additional queries: ceilometer --timeout 1000000000 --ceilometer-url http://localhost:8777 --os-auth-token foo query-samples --filter '{"and": [{">": {"timestamp": "2011-03-20T18:15:00"}}, {"<": {"timestamp": "2017-03-20T18:15:00"}} ]}' --limit 50000 ceilometer --timeout 1000000000 --ceilometer-url http://localhost:8777 --os-auth-token foo sample-list -q 'timestamp>=2011-03-20T18:15:00;timestamp<2017-03-20T18:15:00' --limit 50000 ceilometer --timeout 1000000000 --ceilometer-url http://localhost:8777 --os-auth-token foo statistics -m instance And done them on mongo 2.4.9-3, 2.6.9-1, 2.6.11-1, on an standalone mongo and then on a master/slave mongos. The package upgrade works well, but I can't reproduce the performance issue, I got same number with all mongo version.
I have take a look into the Ceilometer code, and even I don't see the issue on my setup, Ceilometer can clearly easily trigger this mongo bug, when timestamp range are used.
mongodb is using mongodb-2.6.11-1.el7ost.x86_64.rpm fix is working according to commet 22
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-0692.html