Hide Forgot
Description of problem: In PROD, EmbCartController#update raised NoMethodError: undefined method `external_port' for nil:NilClass Reference id: 073e5afabadd2580cab5f5da407fbe7f Stacktrace: …b/openshift/ mcollective_application_container_proxy.rb:1735:in `block in build_update_cluster_args' …b/openshift/ mcollective_application_container_proxy.rb:1730:in `each' …b/openshift/ mcollective_application_container_proxy.rb:1730:in `build_update_cluster_args' …b/openshift/ mcollective_application_container_proxy.rb:1746:in `update_cluster' …/openshift-origin-controller-1.17.6/app/models/ gear.rb: 276:in `method_missing' …ift-origin-controller-1.17.6/app/models/ application.rb:1272:in `update_cluster' …ems/gems/mongoid-3.0.21/lib/mongoid/relations/ proxy.rb: 143:in `method_missing' …ler-1.17.6/app/models/pending_ops/ update_cluster_op.rb: 4:in `execute' …n-controller-1.17.6/app/models/ pending_app_op_group.rb: 75:in `block in execute' …n-controller-1.17.6/app/models/ pending_app_op_group.rb: 64:in `each' …n-controller-1.17.6/app/models/ pending_app_op_group.rb: 64:in `execute' …ift-origin-controller-1.17.6/app/models/ application.rb:1557:in `run_jobs' …ift-origin-controller-1.17.6/app/models/ application.rb: 867:in `block in update_component_limits' …ift-origin-controller-1.17.6/app/models/ application.rb:1623:in `run_in_application_lock' …ift-origin-controller-1.17.6/app/models/ application.rb: 861:in `update_component_limits' …ntroller-1.17.6/app/controllers/ emb_cart_controller.rb: 243:in `update' …ntroller-1.17.6/lib/openshift/controller/ action_log.rb: 80:in `set_logged_request' …sr/share/gems/gems/journey-1.0.4/lib/journey/ router.rb: 68:in `block in call' …sr/share/gems/gems/journey-1.0.4/lib/journey/ router.rb: 56:in `each' …sr/share/gems/gems/journey-1.0.4/lib/journey/ router.rb: 56:in `call' …per-0.11.1/lib/mongo_mapper/middleware/ identity_map.rb: 10:in `call' …oid-3.0.21/lib/rack/mongoid/middleware/ identity_map.rb: 34:in `block in call' …e/gems/gems/mongoid-3.0.21/lib/mongoid/ unit_of_work.rb: 39:in `unit_of_work' …oid-3.0.21/lib/rack/mongoid/middleware/ identity_map.rb: 34:in `call' …3/root/usr/share/gems/gems/rack-1.4.1/lib/rack/ etag.rb: 23:in `call' …/share/gems/gems/rack-1.4.1/lib/rack/ conditionalget.rb: 35:in `call' …e/gems/gems/rack-1.4.1/lib/rack/session/abstract/ id.rb: 205:in `context' …e/gems/gems/rack-1.4.1/lib/rack/session/abstract/ id.rb: 200:in `call' …/share/gems/gems/rack-1.4.1/lib/rack/ methodoverride.rb: 21:in `call' …oot/usr/share/gems/gems/rack-1.4.1/lib/rack/ runtime.rb: 17:in `call' …3/root/usr/share/gems/gems/rack-1.4.1/lib/rack/ lock.rb: 15:in `call' …are/gems/gems/rack-cache-1.2/lib/rack/cache/ context.rb: 136:in `forward' …are/gems/gems/rack-cache-1.2/lib/rack/cache/ context.rb: 143:in `pass' …are/gems/gems/rack-cache-1.2/lib/rack/cache/ context.rb: 155:in `invalidate' …are/gems/gems/rack-cache-1.2/lib/rack/cache/ context.rb: 71:in `call!' …are/gems/gems/rack-cache-1.2/lib/rack/cache/ context.rb: 51:in `call' …r-3.0.21/lib/phusion_passenger/rack/ request_handler.rb: 97:in `process_request' …0.21/lib/phusion_passenger/ abstract_request_handler.rb: 521:in `accept_and_process_next_request' …0.21/lib/phusion_passenger/ abstract_request_handler.rb: 274:in `main_loop' …0.21/lib/phusion_passenger/rack/ application_spawner.rb: 206:in `start_request_handler' …0.21/lib/phusion_passenger/rack/ application_spawner.rb: 79:in `block in spawn_application' …s/gems/passenger-3.0.21/lib/phusion_passenger/ utils.rb: 470:in `safe_fork' …0.21/lib/phusion_passenger/rack/ application_spawner.rb: 64:in `spawn_application' …assenger-3.0.21/lib/phusion_passenger/ spawn_manager.rb: 264:in `spawn_rack_application' …assenger-3.0.21/lib/phusion_passenger/ spawn_manager.rb: 137:in `spawn_application' …assenger-3.0.21/lib/phusion_passenger/ spawn_manager.rb: 275:in `handle_spawn_application' …senger-3.0.21/lib/phusion_passenger/ abstract_server.rb: 357:in `server_main_loop' …senger-3.0.21/lib/phusion_passenger/ abstract_server.rb: 206:in `start_synchronously'
Missing port interfaces of some gears will cause this. Should fix such gears by calling out expose ports for all components of the gears.
Note to QE : 1. Create some scalable apps and manually remove the port_interfaces of some of them. 2. Then find out about the broken apps using the query in comment#4. 3. Run the attached script with the uuids from step 2 4. Check mongo to see that port_interfaces are restored.
Also to be tested is that the fix script can be re-run several times without issues.
There is an error executing the re-expose-ports.rb. The test was performed against devenv-stage_570 on which the gears haven't been flattened(gears were embedded in group_instances) 1. Create one scalable php app with mysql 2. Delete one port_interface using rockmongo interface 3. Execute the query to list the app uuid needing fix: libra_rs:PRIMARY> db.applications.find( {"scalable" : true, $where : "function() { for(var j=0; j < this.group_instances.length; j++) {for(var i=0; i<this.group_instances[j].gears.length; i++) { if( !this.group_instances[j].gears[i].port_interfaces || this.group_instances[j].gears[i].port_interfaces.length == 0 ) { return true; }}} return false; } " } , { "_id" : 1, "uuid" : 1}) { "_id" : ObjectId("52a7103109a9c12653000006"), "uuid" : "52a7103109a9c10ae5000001" } 4. Add the uuid to the script, and execute it, an error show up. [root@ip-10-183-191-83 ~]# ruby re-expose-ports.rb /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/attributes.rb:249:in `method_missing': undefined method `has_component?' for #<Gear:0x000000073c8b18> (NoMethodError) from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.16.12/app/models/gear.rb:268:in `method_missing' from re-expose-ports.rb:21:in `block (4 levels) in <main>' from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/proxy.rb:143:in `each' from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/proxy.rb:143:in `method_missing' from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/embedded/many.rb:396:in `method_missing' from re-expose-ports.rb:20:in `block (3 levels) in <main>' from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/proxy.rb:143:in `each' from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/proxy.rb:143:in `method_missing' from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/embedded/many.rb:396:in `method_missing' from re-expose-ports.rb:19:in `block (2 levels) in <main>' from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/proxy.rb:143:in `each' from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/proxy.rb:143:in `method_missing' from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/embedded/many.rb:396:in `method_missing' from re-expose-ports.rb:18:in `block in <main>' from re-expose-ports.rb:11:in `each' from re-expose-ports.rb:11:in `<main>'
Not sure what code is carried in devenv_stage-570. has_component? method was introduced with 2.0.36 release so it will not be available on AMIs before that. Could you test it on code from stage-2.0.36?
sorry, I missed the devenv-stage_601 which has stage-2.0.36 and does not have flattened gears. 1. Create one scalable php app with mysql 2. Delete one port_interface using rockmongo interface 3. Execute the query to list the app uuid needing fix: libra_rs:PRIMARY> db.applications.find( {"scalable" : true, $where : "function() { for(var j=0; j < this.group_instances.length; j++) {for(var i=0; i<this.group_instances[j].gears.length; i++) { if( !this.group_instances[j].gears[i].port_interfaces || this.group_instances[j].gears[i].port_interfaces.length == 0 ) { return true; }}} return false; } " } , { "_id" : 1, "uuid" : 1}) { "_id" : ObjectId("52a7c897ac7b63c02c00004b"), "uuid" : "52a7c897ac7b630a58000005" } { "_id" : ObjectId("52a7cac3ac7b6370a200001a"), "uuid" : "52a7cac3ac7b63253c000003" } 4. Run re-expose-ports.rb [root@domU-12-31-39-07-79-26 ~]# ruby re-expose-ports.rb Completed exposing ports for 52a7c897ac7b630a58000005 Completed exposing ports for 52a7cac3ac7b63253c000003 5. Re-check mongo, and the applications are fixed. 6. Delete some other port_interfaces, re-execute the script There are no errors