Bug 1303347

Upgrading to mongodb 2.6.11 sounds straightforward enough but, before we do that, I'd like to understand better what query is causing issues. By looking at the customer case, it is not clear what query is slower in 2.6. 

Can we get the exact query?

In addition to the above, I'd be interested to know how the upgrade to 2.6 was done.

If the query took longer than 100ms, it should have been logged to the mongodb server's log. Can we get the logs from the whole replica set?

They have mongodb in a iscsi volume. They just attched the lun to another server with mongodb 2.6 and replicated all data to another mongos running also mongo 2.6 using ISCSI.


When they started mongo after all replication, we observe that all queries was very slow. During the weekend we rebuild all indexes and the performance go up in 10%, but still it's very slow than before

After search on the internet,  we tried the db.upgradeCheckAllDBs() command to check if the upgrade was possible and the command return Ok to upgrade.

Attaching files:

http://10.12.211.122/01575671

Haikel,

Any chance you can help updating mongodb's package to 2.6.11 ?

Thank you

I have made some tests of the new package on a fresh rhosp 7, the package itself works well, but I didn't any performance improvements with 'ceilometer sample-list' command. 

I'm not surprise to not trigger the mongodb bug with this command. I will try to find another command that can trigger it.

If you can provide a ceilometer command that you known is slower as before it will help my investigation.


Extract of my testing:

# rpm -q openstack-ceilometer-api
openstack-ceilometer-api-2015.1.2-1.el7ost.noarch

# rpm -q mongodb-server
mongodb-server-2.6.9-1.el7ost.x86_64

# du -sh /var/lib/mongodb/
4.3G    /var/lib/mongodb/

# mongo ceilometer --eval 'db.meter.count()'  
MongoDB shell version: 2.6.9
connecting to: ceilometer
435463

# time ceilometer --timeout 1000000000 --ceilometer-url http://localhost:8777 --os-auth-token foo sample-list  --limit 100000 >/dev/null
real    1m14.967s
user    0m30.342s
sys 0m0.282s

# ls -1 mongod*.rpm
mongodb-2.6.11-1.el7ost.x86_64.rpm
mongodb-server-2.6.11-1.el7ost.x86_64.rpm

# rpm -U mongod*.rpm
# systemctl restart mongod

# time ceilometer --timeout 1000000000 --ceilometer-url http://localhost:8777 --os-auth-token foo sample-list  --limit 100000 >/dev/null
real    1m14.579s
user    0m30.208s
sys 0m0.293s

# mongo ceilometer --eval 'db.meter.reIndex()'

# time ceilometer --timeout 1000000000 --ceilometer-url http://localhost:8777 --os-auth-token foo sample-list  --limit 100000 >/dev/null
real    1m14.659s
user    0m29.998s
sys 0m0.438s

I have made some additional queries:

ceilometer --timeout 1000000000 --ceilometer-url http://localhost:8777 --os-auth-token foo query-samples --filter '{"and": [{">": {"timestamp": "2011-03-20T18:15:00"}}, {"<": {"timestamp": "2017-03-20T18:15:00"}} ]}' --limit 50000

ceilometer --timeout 1000000000 --ceilometer-url http://localhost:8777 --os-auth-token foo sample-list -q 'timestamp>=2011-03-20T18:15:00;timestamp<2017-03-20T18:15:00' --limit 50000

ceilometer --timeout 1000000000 --ceilometer-url http://localhost:8777 --os-auth-token foo statistics -m instance

And done them on mongo 2.4.9-3, 2.6.9-1, 2.6.11-1, on an standalone mongo and then on a master/slave mongos.

The package upgrade works well, but I can't reproduce the performance issue, I got same number with all mongo version.

I have take a look into the Ceilometer code, and even I don't see the issue on my setup, Ceilometer can clearly easily trigger this mongo bug, when timestamp range are used.

mongodb is using mongodb-2.6.11-1.el7ost.x86_64.rpm   
fix is working according to commet 22

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0692.html