Bug 1039589

Summary: [new relic] EmbCartController#update raised NoMethodError: undefined method `external_port' for nil:NilClass
Product: OpenShift Online Reporter: Jessica Forrester <jforrest>
Component: PodAssignee: Rajat Chopra <rchopra>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 1.xCC: jhou, rpenta
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-30 00:52:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Jessica Forrester 2013-12-09 14:51:52 UTC
Description of problem:
In PROD, EmbCartController#update raised NoMethodError: undefined method `external_port' for nil:NilClass

Reference id: 073e5afabadd2580cab5f5da407fbe7f

Stacktrace:
…b/openshift/
mcollective_application_container_proxy.rb:1735:in `block in build_update_cluster_args'
…b/openshift/
mcollective_application_container_proxy.rb:1730:in `each'
…b/openshift/
mcollective_application_container_proxy.rb:1730:in `build_update_cluster_args'
…b/openshift/
mcollective_application_container_proxy.rb:1746:in `update_cluster'
…/openshift-origin-controller-1.17.6/app/models/
gear.rb: 276:in `method_missing'
…ift-origin-controller-1.17.6/app/models/
application.rb:1272:in `update_cluster'
…ems/gems/mongoid-3.0.21/lib/mongoid/relations/
proxy.rb: 143:in `method_missing'
…ler-1.17.6/app/models/pending_ops/
update_cluster_op.rb:   4:in `execute'
…n-controller-1.17.6/app/models/
pending_app_op_group.rb:  75:in `block in execute'
…n-controller-1.17.6/app/models/
pending_app_op_group.rb:  64:in `each'
…n-controller-1.17.6/app/models/
pending_app_op_group.rb:  64:in `execute'
…ift-origin-controller-1.17.6/app/models/
application.rb:1557:in `run_jobs'
…ift-origin-controller-1.17.6/app/models/
application.rb: 867:in `block in update_component_limits'
…ift-origin-controller-1.17.6/app/models/
application.rb:1623:in `run_in_application_lock'
…ift-origin-controller-1.17.6/app/models/
application.rb: 861:in `update_component_limits'
…ntroller-1.17.6/app/controllers/
emb_cart_controller.rb: 243:in `update'
…ntroller-1.17.6/lib/openshift/controller/
action_log.rb:  80:in `set_logged_request'
…sr/share/gems/gems/journey-1.0.4/lib/journey/
router.rb:  68:in `block in call'
…sr/share/gems/gems/journey-1.0.4/lib/journey/
router.rb:  56:in `each'
…sr/share/gems/gems/journey-1.0.4/lib/journey/
router.rb:  56:in `call'
…per-0.11.1/lib/mongo_mapper/middleware/
identity_map.rb:  10:in `call'
…oid-3.0.21/lib/rack/mongoid/middleware/
identity_map.rb:  34:in `block in call'
…e/gems/gems/mongoid-3.0.21/lib/mongoid/
unit_of_work.rb:  39:in `unit_of_work'
…oid-3.0.21/lib/rack/mongoid/middleware/
identity_map.rb:  34:in `call'
…3/root/usr/share/gems/gems/rack-1.4.1/lib/rack/
etag.rb:  23:in `call'
…/share/gems/gems/rack-1.4.1/lib/rack/
conditionalget.rb:  35:in `call'
…e/gems/gems/rack-1.4.1/lib/rack/session/abstract/
id.rb: 205:in `context'
…e/gems/gems/rack-1.4.1/lib/rack/session/abstract/
id.rb: 200:in `call'
…/share/gems/gems/rack-1.4.1/lib/rack/
methodoverride.rb:  21:in `call'
…oot/usr/share/gems/gems/rack-1.4.1/lib/rack/
runtime.rb:  17:in `call'
…3/root/usr/share/gems/gems/rack-1.4.1/lib/rack/
lock.rb:  15:in `call'
…are/gems/gems/rack-cache-1.2/lib/rack/cache/
context.rb: 136:in `forward'
…are/gems/gems/rack-cache-1.2/lib/rack/cache/
context.rb: 143:in `pass'
…are/gems/gems/rack-cache-1.2/lib/rack/cache/
context.rb: 155:in `invalidate'
…are/gems/gems/rack-cache-1.2/lib/rack/cache/
context.rb:  71:in `call!'
…are/gems/gems/rack-cache-1.2/lib/rack/cache/
context.rb:  51:in `call'
…r-3.0.21/lib/phusion_passenger/rack/
request_handler.rb:  97:in `process_request'
…0.21/lib/phusion_passenger/
abstract_request_handler.rb: 521:in `accept_and_process_next_request'
…0.21/lib/phusion_passenger/
abstract_request_handler.rb: 274:in `main_loop'
…0.21/lib/phusion_passenger/rack/
application_spawner.rb: 206:in `start_request_handler'
…0.21/lib/phusion_passenger/rack/
application_spawner.rb:  79:in `block in spawn_application'
…s/gems/passenger-3.0.21/lib/phusion_passenger/
utils.rb: 470:in `safe_fork'
…0.21/lib/phusion_passenger/rack/
application_spawner.rb:  64:in `spawn_application'
…assenger-3.0.21/lib/phusion_passenger/
spawn_manager.rb: 264:in `spawn_rack_application'
…assenger-3.0.21/lib/phusion_passenger/
spawn_manager.rb: 137:in `spawn_application'
…assenger-3.0.21/lib/phusion_passenger/
spawn_manager.rb: 275:in `handle_spawn_application'
…senger-3.0.21/lib/phusion_passenger/
abstract_server.rb: 357:in `server_main_loop'
…senger-3.0.21/lib/phusion_passenger/
abstract_server.rb: 206:in `start_synchronously'

Comment 3 Rajat Chopra 2013-12-09 20:44:20 UTC
Missing port interfaces of some gears will cause this. Should fix such gears by calling out expose ports for all components of the gears.

Comment 5 Rajat Chopra 2013-12-09 20:53:27 UTC
Note to QE : 
1. Create some scalable apps and manually remove the port_interfaces of some of them.
2. Then find out about the broken apps using the query in comment#4.
3. Run the attached script with the uuids from step 2
4. Check mongo to see that port_interfaces are restored.

Comment 6 Rajat Chopra 2013-12-09 21:14:17 UTC
Also to be tested is that the fix script can be re-run several times without issues.

Comment 7 Jianwei Hou 2013-12-10 13:06:56 UTC
There is an error executing the re-expose-ports.rb.
The test was performed against devenv-stage_570 on which the gears haven't been flattened(gears were embedded in group_instances)

1. Create one scalable php app with mysql 
2. Delete one port_interface using rockmongo interface
3. Execute the query to list the app uuid needing fix:
libra_rs:PRIMARY> db.applications.find( {"scalable" : true, $where :  "function() {  for(var j=0; j < this.group_instances.length; j++) {for(var i=0; i<this.group_instances[j].gears.length; i++) { if( !this.group_instances[j].gears[i].port_interfaces || this.group_instances[j].gears[i].port_interfaces.length == 0 ) { return true; }}} return false; } " } , { "_id" : 1, "uuid" : 1})
{ "_id" : ObjectId("52a7103109a9c12653000006"), "uuid" : "52a7103109a9c10ae5000001" }
4. Add the uuid to the script, and execute it, an error show up.

[root@ip-10-183-191-83 ~]# ruby re-expose-ports.rb
/opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/attributes.rb:249:in `method_missing': undefined method `has_component?' for #<Gear:0x000000073c8b18> (NoMethodError)
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.16.12/app/models/gear.rb:268:in `method_missing'
	from re-expose-ports.rb:21:in `block (4 levels) in <main>'
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/proxy.rb:143:in `each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/proxy.rb:143:in `method_missing'
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/embedded/many.rb:396:in `method_missing'
	from re-expose-ports.rb:20:in `block (3 levels) in <main>'
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/proxy.rb:143:in `each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/proxy.rb:143:in `method_missing'
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/embedded/many.rb:396:in `method_missing'
	from re-expose-ports.rb:19:in `block (2 levels) in <main>'
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/proxy.rb:143:in `each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/proxy.rb:143:in `method_missing'
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/embedded/many.rb:396:in `method_missing'
	from re-expose-ports.rb:18:in `block in <main>'
	from re-expose-ports.rb:11:in `each'
	from re-expose-ports.rb:11:in `<main>'

Comment 8 Rajat Chopra 2013-12-10 15:13:18 UTC
Not sure what code is carried in devenv_stage-570. has_component? method was introduced with 2.0.36 release so it will not be available on AMIs before that.

Could you test it on code from stage-2.0.36?

Comment 9 Jianwei Hou 2013-12-11 02:37:59 UTC
sorry, I missed the devenv-stage_601 which has stage-2.0.36 and does not have flattened gears.

1. Create one scalable php app with mysql 
2. Delete one port_interface using rockmongo interface
3. Execute the query to list the app uuid needing fix:
libra_rs:PRIMARY> db.applications.find( {"scalable" : true, $where :  "function() {  for(var j=0; j < this.group_instances.length; j++) {for(var i=0; i<this.group_instances[j].gears.length; i++) { if( !this.group_instances[j].gears[i].port_interfaces || this.group_instances[j].gears[i].port_interfaces.length == 0 ) { return true; }}} return false; } " } , { "_id" : 1, "uuid" : 1})

{ "_id" : ObjectId("52a7c897ac7b63c02c00004b"), "uuid" : "52a7c897ac7b630a58000005" }
{ "_id" : ObjectId("52a7cac3ac7b6370a200001a"), "uuid" : "52a7cac3ac7b63253c000003" }
4. Run re-expose-ports.rb 
[root@domU-12-31-39-07-79-26 ~]# ruby re-expose-ports.rb 
Completed exposing ports for 52a7c897ac7b630a58000005
Completed exposing ports for 52a7cac3ac7b63253c000003
5. Re-check mongo, and the applications are fixed.
6. Delete some other port_interfaces, re-execute the script
There are no errors