Bug 1040314

Summary: oo-admin-chk is broken when an application document is missing
Product: OpenShift Online Reporter: Jianwei Hou <jhou>
Component: PodAssignee: Ravi Sankar <rpenta>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: dmcphers, rpenta, xtian
Target Milestone: ---Keywords: UpcomingRelease
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-02-26 19:08:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Jianwei Hou 2013-12-11 07:40:38 UTC
Description of problem:
If an application document is removed, run oo-admin-chk at level 1 and the program is broken

Version-Release number of selected component (if applicable):
fork_ami_origin_broker_admin-chk_960
devenv-stage_609

How reproducible:
Always

Steps to Reproduce:
1. Create a diy application
rhc create-app d1 diy-0.1
2. Remove the app document from mongo
> db.applications.remove({name:'d1'})
3. oo-admin-chk -l 1

Actual results:
[root@ip-10-164-67-148 ~]# oo-admin-chk -l 1
Started at: 2013-12-11 07:32:22 UTC

Total gears found in mongo: 0
Total gears found on the nodes: 1
Total nodes that responded: 1
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.18.2/app/helpers/admin_helper.rb:102:in `+': nil can't be coerced into Fixnum (TypeError)
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.18.2/app/helpers/admin_helper.rb:102:in `block in check_consumed_gears'
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/referenced/many.rb:167:in `block in each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/targets/enumerable.rb:169:in `block in each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/contextual/mongo.rb:577:in `yield_document'
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/contextual/mongo.rb:133:in `block (2 levels) in each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/moped-1.5.1/lib/moped/query.rb:77:in `block in each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/moped-1.5.1/lib/moped/cursor.rb:26:in `block in each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/moped-1.5.1/lib/moped/cursor.rb:26:in `each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/moped-1.5.1/lib/moped/cursor.rb:26:in `each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/moped-1.5.1/lib/moped/query.rb:76:in `each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/moped-1.5.1/lib/moped/query.rb:76:in `each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/contextual/mongo.rb:132:in `block in each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/contextual/mongo.rb:556:in `selecting'
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/contextual/mongo.rb:131:in `each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/contextual.rb:18:in `each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/targets/enumerable.rb:166:in `each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/referenced/many.rb:167:in `each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.18.2/app/helpers/admin_helper.rb:101:in `check_consumed_gears'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.18.2/app/helpers/admin_helper.rb:435:in `block in find_consumed_gears_inconsistencies'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.18.2/app/helpers/admin_helper.rb:430:in `each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.18.2/app/helpers/admin_helper.rb:430:in `find_consumed_gears_inconsistencies'
	from /usr/sbin/oo-admin-chk:167:in `<main>'


Expected results:
Should not be broken and report result as 'fail'

Additional info:

Comment 1 Ravi Sankar 2013-12-12 01:59:37 UTC
Fixed in fork_ami_origin_broker_admin-chk_962

Comment 2 Jianwei Hou 2013-12-12 07:41:30 UTC
Verified on fork_ami_origin_broker_admin-chk_963

[root@ip-10-137-69-126 ~]# oo-admin-chk -l 1
Started at: 2013-12-12 07:40:18 UTC

Total gears found in mongo: 0
Total gears found on the nodes: 1
Total nodes that responded: 1

Finished at: 2013-12-12 07:40:59 UTC
Total time: 40.856s
Gear 52a95cea09c0bb8051000004 exists on node ip-10-137-69-126 (uid: 1000) but does not exist in mongo database
User jhou+6 has a mismatch in consumed gears (1) and actual gears (0)
Found usage record for gear Id '52a95cea09c0bb8051000004' but could not find corresponding gear in the application.
FAILED
Please refer to the oo-admin-repair tool to resolve some of these inconsistencies.

Comment 3 Jianwei Hou 2013-12-16 07:09:20 UTC
This is reproduced in devenv_4137 as well as fork_ami_bug_1039151_1041628_967
I'm afraid this fix has not been merged into master. re-open

Comment 4 Ravi Sankar 2013-12-16 18:30:23 UTC
Changes in fork_ami_origin_broker_admin-chk_963 are not merged into master till now (I was waiting for the code review from my team).
Master merge PR => https://github.com/openshift/origin-server/pull/4022

Comment 5 zhaozhanqi 2013-12-17 10:09:15 UTC
Tested this issue on devenv_4144,  it has been merged. so mark this bug as 'verified'

libra_rs:PRIMARY> db.applications.remove({name:'zqphp'})
libra_rs:PRIMARY> exit
bye
[root@ip-10-9-136-120 ~]# oo-admin-chk -l1
Started at: 2013-12-17 10:06:19 UTC

Total gears found in mongo: 0
Total gears found on the nodes: 1
Total nodes that responded: 1

Finished at: 2013-12-17 10:07:00 UTC
Total time: 40.872s
Gear 52b02185a587c82cf9000001 exists on node ip-10-9-136-120 (uid: 1000) but does not exist in mongo database
User zzhao has a mismatch in consumed gears (1) and actual gears (0)
Found usage record for gear Id '52b02185a587c82cf9000001' but could not find corresponding gear in the application.
FAILED
Please refer to the oo-admin-repair tool to resolve some of these inconsistencies.