Hide Forgot
Description of problem: oo-admin-check -l 1 fails when attempting to sort the data. This is causing the script to fail in master as the data set is large. ----------------------- I, [2013-08-19T10:51:38.891016 #8473] INFO -- : Started at: 2013-08-19 09:00:19 -0400 Time to fetch mongo data: 2647.256s Total gears found in mongo: 178805 Error: User with ID 520ea6c550044657c3000638 not found in mongo Error: User with ID 520ff6b05004466b23000019 not found in mongo Error: User with ID 51fb311f50044679c2000280 not found in mongo Error: User with ID 51fb99c15973caa1de00012b not found in mongo Error: User with ID 51fba5e25973cae115000169 not found in mongo Error: User with ID 51fbac3c4382ec033f000472 not found in mongo Error: User with ID 51fbc9335973ca8545000110 not found in mongo Time to get all gears from nodes: 631.102s Total gears found on the nodes: 176478 Total nodes that responded : 262 Time to get all sshkeys for all gears from nodes: 158.175s Total gears found on the nodes: 176490 Total nodes that responded : 262 /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/cursor.rb:140:in `next': too much data for sort() with no index. add an index or specify a smaller limit (Mongo::OperationFailure) from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/cursor.rb:285:in `each' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.9/lib/openshift/data_store.rb:24:in `block in find' from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/collection.rb:276:in `find' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.9/lib/openshift/data_store.rb:23:in `find' from /usr/sbin/oo-admin-chk:668:in `<main>' Version-Release number of selected component (if applicable): --------------- How reproducible: Very, this fails every run currently. Steps to Reproduce: 1. 2. 3. Actual results: Fails when trying to sort. Expected results: Should return the latest results from oo-admin-chk. Additional info:
Fixed in https://github.com/openshift/origin-server/pull/3459
Commits pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/b8976df4f13a868ad37a32cdaf0444fc7e80af70 Bug 999702 - oo-admin-chk: Don't sort usage records based on time, instead update app_name based on created_at field https://github.com/openshift/origin-server/commit/e2bed2995419dd1822c3becad54723c40a96aa1d Merge pull request #3459 from pravisankar/dev/ravi/bug999702 Merged by openshift-bot
I'm not sure how large the data was, but I'm using a stage mongodump to verifying this bug, my data volume is: libra_rs:PRIMARY> db.usage_records.count() 2974 libra_rs:PRIMARY> db.applications.count() 2382 Run rhc-admin-chk -l 1 ....................... User '51e4f17fdbd93cf5930007c6' has usage_account_id '2974881' in mongo but does not exist in Aria billing provider. User '51e600aadbd93c53f800008b' has usage_account_id '2975614' in mongo but does not exist in Aria billing provider. User '51e62083dbd93cbc6400024a' has usage_account_id '2975723' in mongo but does not exist in Aria billing provider. User '51e62d7fdbd93c23330003d3' has usage_account_id '2975703' in mongo but does not exist in Aria billing provider. User '51ee16072587c88ab900038c' has usage_account_id '2979572' in mongo but does not exist in Aria billing provider. User '51f8b7c7dbd93cb125000052' has usage_account_id '2990798' in mongo but does not exist in Aria billing provider. Please refer to the oo-admin-repair tool to resolve some of these inconsistencies. Total time: 311.866s Finished at: 2013-08-22 04:52:06 -0400 Didn't see the error, I'm moving to verified. But please let me know if I need to create larger data volume to verify this bug, thanks.