Bug 999702

Summary: [oo-admin-check] Fails when attempting to sort
Product: OpenShift Online Reporter: Kenny Woodson <kwoodson>
Component: PodAssignee: Ravi Sankar <rpenta>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 2.xCC: jhou
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-08-29 12:54:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kenny Woodson 2013-08-21 21:46:59 UTC
Description of problem:

oo-admin-check -l 1 fails when attempting to sort the data.  This is causing the script to fail in master as the data set is large.

-----------------------
I, [2013-08-19T10:51:38.891016 #8473]  INFO -- : Started at: 2013-08-19 09:00:19 -0400
Time to fetch mongo data: 2647.256s
Total gears found in mongo: 178805
Error: User with ID 520ea6c550044657c3000638 not found in mongo
Error: User with ID 520ff6b05004466b23000019 not found in mongo
Error: User with ID 51fb311f50044679c2000280 not found in mongo
Error: User with ID 51fb99c15973caa1de00012b not found in mongo
Error: User with ID 51fba5e25973cae115000169 not found in mongo
Error: User with ID 51fbac3c4382ec033f000472 not found in mongo
Error: User with ID 51fbc9335973ca8545000110 not found in mongo
Time to get all gears from nodes: 631.102s
Total gears found on the nodes: 176478
Total nodes that responded : 262
Time to get all sshkeys for all gears from nodes: 158.175s
Total gears found on the nodes: 176490
Total nodes that responded : 262
/opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/cursor.rb:140:in `next': too much data for sort() with no index.  add an index or specify a smaller limit (Mongo::OperationFailure)
        from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/cursor.rb:285:in `each'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.9/lib/openshift/data_store.rb:24:in `block in find'
        from /opt/rh/ruby193/root/usr/local/share/gems/gems/mongo-1.8.1/lib/mongo/collection.rb:276:in `find'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.12.9/lib/openshift/data_store.rb:23:in `find'
        from /usr/sbin/oo-admin-chk:668:in `<main>'
Version-Release number of selected component (if applicable):
---------------

How reproducible:
Very, this fails every run currently.

Steps to Reproduce:
1. 
2.
3.

Actual results:

Fails when trying to sort.

Expected results:
Should return the latest results from oo-admin-chk.

Additional info:

Comment 1 Ravi Sankar 2013-08-22 00:06:17 UTC
Fixed in https://github.com/openshift/origin-server/pull/3459

Comment 2 openshift-github-bot 2013-08-22 03:19:29 UTC
Commits pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/b8976df4f13a868ad37a32cdaf0444fc7e80af70
Bug 999702 - oo-admin-chk: Don't sort usage records based on time, instead update app_name based on created_at field

https://github.com/openshift/origin-server/commit/e2bed2995419dd1822c3becad54723c40a96aa1d
Merge pull request #3459 from pravisankar/dev/ravi/bug999702

Merged by openshift-bot

Comment 3 Jianwei Hou 2013-08-22 09:22:04 UTC
I'm not sure how large the data was, but I'm using a stage mongodump to verifying this bug, my data volume is:
libra_rs:PRIMARY> db.usage_records.count()
2974
libra_rs:PRIMARY> db.applications.count()
2382

Run rhc-admin-chk -l 1
.......................
User '51e4f17fdbd93cf5930007c6' has usage_account_id '2974881' in mongo but does not exist in Aria billing provider.
User '51e600aadbd93c53f800008b' has usage_account_id '2975614' in mongo but does not exist in Aria billing provider.
User '51e62083dbd93cbc6400024a' has usage_account_id '2975723' in mongo but does not exist in Aria billing provider.
User '51e62d7fdbd93c23330003d3' has usage_account_id '2975703' in mongo but does not exist in Aria billing provider.
User '51ee16072587c88ab900038c' has usage_account_id '2979572' in mongo but does not exist in Aria billing provider.
User '51f8b7c7dbd93cb125000052' has usage_account_id '2990798' in mongo but does not exist in Aria billing provider.
Please refer to the oo-admin-repair tool to resolve some of these inconsistencies.
Total time: 311.866s
Finished at: 2013-08-22 04:52:06 -0400

Didn't see the error, I'm moving to verified. But please let me know if I need to create larger data volume to verify this bug, thanks.