Bug 975713

Summary: oo-admin-chk reports "Success" when it has detected a mismatch in 'usage_records' and 'usage' collections
Product: OpenShift Online Reporter: Jianwei Hou <jhou>
Component: PodAssignee: Ravi Sankar <rpenta>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: low Docs Contact:
Priority: medium    
Version: 2.xCC: rpenta
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-06-24 14:58:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jianwei Hou 2013-06-19 08:13:49 UTC
Description of problem:
When there is a mismatch with 'usage_records' and 'usage' collection, 'oo-admin-chk' is able to detect it, but does not mark the result as 'failed'.

Version-Release number of selected component (if applicable):
On devenv_3382

How reproducible:
Always

Steps to Reproduce:
1. Make sure the 'usage' and 'usage_records' collections are clean, then create an application
2. Delete the data from 'usage' collection
> db.usage.remove()
3. On broker, run 'oo-admin-chk -l 1'

Actual results:
oo-admin-chk has detected the problem, but marked the final result as 'Success'

[root@ip-10-138-34-145 ~]# oo-admin-chk -l 1
Started at: 2013-06-19 03:49:19 -0400
Time to fetch mongo data: 0.038s
Total gears found in mongo: 1
Time to get all gears from nodes: 20.411s
Total gears found on the nodes: 1
Total nodes that responded : 1
Time to get all sshkeys for all gears from nodes: 20.06s
Total gears found on the nodes: 1
Total nodes that responded : 1
Checking un-ended records in usage_records collection but not in usage collection and viceversa: FAIL
Success
Total time: 44.318s
Finished at: 2013-06-19 03:50:03 -0400

Expected results:
Should mark result as 'Check failed.'

Additional info:

Comment 1 Ravi Sankar 2013-06-19 19:12:47 UTC
Fixed in https://github.com/openshift/origin-server/pull/2905

Comment 2 openshift-github-bot 2013-06-19 22:43:41 UTC
Commits pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/d1339a82cc7a064d73b5bb02a56f9eb9191ee1bb
Bug 975713 - oo-admin-chk fix

https://github.com/openshift/origin-server/commit/fb48526fe9d84de364306d970ec2874c673ed1b6
Merge pull request #2905 from pravisankar/dev/ravi/bug975713

Merged by openshift-bot

Comment 3 Jianwei Hou 2013-06-20 03:05:51 UTC
Verified on devenv_3389

Repeat steps in description, the check as marked as fail now

[root@ip-10-151-4-231 ~]# oo-admin-chk -l 1 -v
Started at: 2013-06-19 22:54:40 -0400
Time to fetch mongo data: 0.149s
Total gears found in mongo: 4
Checking consumed gear count for user user_with_multiple_gear_sizes...	OK
Checking consumed gear count for user user_with_extra_storage...	OK
Checking consumed gear count for user user_with_certificate_capabilities...	OK
Checking consumed gear count for user openshiftmnzq3d+1...	OK
Checking consumed gear count for user openshiftmnzq3d+3...	OK
Checking consumed gear count for user openshiftmnzq3d+5...	OK
Checking consumed gear count for user qcong...	OK
Checking consumed gear count for user qcong+1...	OK
Checking consumed gear count for user qcong+2...	OK
Checking consumed gear count for user jhou...	OK
Time to get all gears from nodes: 20.472s
Total gears found on the nodes: 4
Total nodes that responded : 1
Time to get all sshkeys for all gears from nodes: 20.09s
Total gears found on the nodes: 5
Total nodes that responded : 1
Checking application gears and ssh keys on corresponding nodes:
dfcb2852d95311e2a49f22000a9704e7 : String...	OK
Checking ssh keys for gear: dfcb2852d95311e2a49f22000a9704e7...	OK
f1eedfe2d95311e2a5a422000a9704e7 : String...	OK
Checking ssh keys for gear: f1eedfe2d95311e2a5a422000a9704e7...	OK
51c26dd9f4380a7dbb000007 : String...	OK
Checking ssh keys for gear: 51c26dd9f4380a7dbb000007...	OK
51c26e66f4380a6231000001 : String...	OK
Checking ssh keys for gear: 51c26e66f4380a6231000001...	OK
Checking node gears in application database:
f1eedfe2d95311e2a5a422000a9704e7...	OK
51c26dd9f4380a7dbb000007...	OK
dfcb2852d95311e2a49f22000a9704e7...	OK
51c26e66f4380a6231000001...	OK
Checking for application without any group instances in the database:
Checking for application without any gears in the group instances in the database:
Checking for users with nil or empty or missing login in the database:
Checking for unreserved UIDs in the district:
Checking for unused UIDs in the district:
Checking for gears with the same UID:
Checking gears available in applications collection but not in usage_records and viceversa: FAIL
Re-checking for gear '51c26f0cf4380a57ed00004d'...	OK
Checking gears with additional storage in applications collection but not in usage_records and viceversa: OK
Checking gears with premium cartridge in applications collection but not in usage_records and viceversa: OK
Checking un-ended records in usage_records collection but not in usage collection and viceversa: FAIL
Re-checking for gear '51c26d77f4380af70d00015b'...	FAIL
Re-checking for gear '51c26d95f4380af2000000f7'...	FAIL
Re-checking for gear '51c26dd9f4380aac02000064'...	FAIL
Re-checking for gear '51c26e66f4380a57ed000038'...	FAIL
Check failed.
Found 1 un-ended records in usage_records collection for gear Id '51c26d77f4380af70d00015b' but could not find corresponding records in usage.
Found 1 un-ended records in usage_records collection for gear Id '51c26d95f4380af2000000f7' but could not find corresponding records in usage.
Found 1 un-ended records in usage_records collection for gear Id '51c26dd9f4380aac02000064' but could not find corresponding records in usage.
Found 1 un-ended records in usage_records collection for gear Id '51c26e66f4380a57ed000038' but could not find corresponding records in usage.
Please refer to the oo-admin-repair tool to resolve some of these inconsistencies.
Total time: 40.889s
Finished at: 2013-06-19 22:55:21 -0400