It is unclear how the application got in this state but as a result it cannot be deleted. The error in the log file: Started DELETE "/broker/rest/domains/foobar/applications/sample" for 66.187.233.202 at Wed Apr 04 12:59:41 -0400 2012 Processing by ApplicationsController#destroy as JSON Parameters: {"id"=>"sample", "domain_id"=>"foobar"} Response from Streamline took (/wapps/streamline/login.html): 478.72 ms POST Response code = 200 Current login = xcoulon+test / authenticated for xcoulon+test Adding user xcoulon+test...inside authenticate MongoDataStore.find(CloudUser, xcoulon+test, xcoulon+test) MongoDataStore.find(CloudUser, xcoulon+test, xcoulon+test) Deleting application sample DEBUG: DYNECT Login with path: /REST/Session/ DEBUG: DYNECT Response: {"status": "success", "data": {"token": "dvViHwHGnC4pn4Z/hqiUnk5yx5PLSSCbTrbwPWP7B9Jd1MY0nTzKfeK/22VZoIXrPIVeZXyPhLcGdJv5UklSg4VdYL91XmU2aAr9VL9kPd8lA7+FcBF1Afvl/kJpkls6g8wTmSyG/DGwNHdzcc8pVAmoux1NvozK", "version": "3.3.2"}, "job_id": 83285140, "msgs": [{"INFO": "login: Login successful", "SOURCE": "BLL", "ERR_CD": null, "LVL": "INFO"}]} DEBUG: DYNECT Response data: tokendvViHwHGnC4pn4Z/hqiUnk5yx5PLSSCbTrbwPWP7B9Jd1MY0nTzKfeK/22VZoIXrPIVeZXyPhLcGdJv5UklSg4VdYL91XmU2aAr9VL9kPd8lA7+FcBF1Afvl/kJpkls6g8wTmSyG/DGwNHdzcc8pVAmoux1NvozKversion3.3.2 DEBUG: DYNECT delete with path: /REST/CNAMERecord/rhcloud.com/sample-foobarbaz.stg.rhcloud.com/ and headers: {"Content-Type"=>"application/json", "Auth-Token"=>"dvViHwHGnC4pn4Z/hqiUnk5yx5PLSSCbTrbwPWP7B9Jd1MY0nTzKfeK/22VZoIXrPIVeZXyPhLcGdJv5UklSg4VdYL91XmU2aAr9VL9kPd8lA7+FcBF1Afvl/kJpkls6g8wTmSyG/DGwNHdzcc8pVAmoux1NvozK"} DEBUG: DYNECT: Could not find /REST/CNAMERecord/rhcloud.com/sample-foobarbaz.stg.rhcloud.com/ to delete DEBUG: DYNECT put/post with path: /REST/Zone/rhcloud.com/ json data: {"publish":"true"} and headers: {"Content-Type"=>"application/json", "Auth-Token"=>"dvViHwHGnC4pn4Z/hqiUnk5yx5PLSSCbTrbwPWP7B9Jd1MY0nTzKfeK/22VZoIXrPIVeZXyPhLcGdJv5UklSg4VdYL91XmU2aAr9VL9kPd8lA7+FcBF1Afvl/kJpkls6g8wTmSyG/DGwNHdzcc8pVAmoux1NvozK"} DEBUG: DYNECT Response: {"status": "success", "data": {"zone_type": "Primary", "serial_style": "increment", "serial": 547296, "zone": "rhcloud.com"}, "job_id": 83285148, "msgs": [{"INFO": "changeset: No changes to apply.", "SOURCE": "BLL", "ERR_CD": null, "LVL": "INFO"}, {"INFO": "publish: Could not publish rhcloud.com. No changes to apply.", "SOURCE": "BLL", "ERR_CD": null, "LVL": "INFO"}]} DEBUG: DYNECT Response data: serial547296zone_typePrimaryzonerhcloud.comserial_styleincrement DEBUG: DYNECT delete with path: /REST/Session/ and headers: {"Content-Type"=>"application/json", "Auth-Token"=>"dvViHwHGnC4pn4Z/hqiUnk5yx5PLSSCbTrbwPWP7B9Jd1MY0nTzKfeK/22VZoIXrPIVeZXyPhLcGdJv5UklSg4VdYL91XmU2aAr9VL9kPd8lA7+FcBF1Afvl/kJpkls6g8wTmSyG/DGwNHdzcc8pVAmoux1NvozK"} DEBUG: DYNECT Response: {"status": "success", "data": {}, "job_id": 83285155, "msgs": [{"INFO": "logout: Logout successful", "SOURCE": "BLL", "ERR_CD": null, "LVL": "INFO"}]} DEBUG: DYNECT Response data: DEBUG: Deconfiguring embedded application 'mongodb-2.0' in application 'sample' on node 'ex-std-node1.stg.rhcloud.com' DEBUG: Sending to Nurture:application: app_uuid='5229a2c0f1424aed90d4ca8f20c0f9a2' action='deconfigure' 0 DEBUG: Wed Apr 04 12:59:45 -0400 2012 Sending to Apptegic:application: user='xcoulon+test' app_uuid='5229a2c0f1424aed90d4ca8f20c0f9a2' action='deconfigure' DEBUG: Wed Apr 04 12:59:45 -0400 2012 Done sending to Apptegic DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fea088e7070> DEBUG: rpc_client.custom_request('cartridge_do', {:action=>"deconfigure", :args=>"'sample' 'foobarbaz' '5229a2c0f1424aed90d4ca8f20c0f9a2'", :cartridge=>"embedded/mongodb-2.0"}, @id, {'identity' => @id}) DEBUG: [#<MCollective::RPC::Result:0x7fea08c90de8 @agent="libra", @results={:sender=>"ex-std-node1.stg.rhcloud.com", :statusmsg=>"cartridge_do_action failed 125. Output sample. Application directory doesn't exist: /var/lib/stickshift/5229a2c0f1424aed90d4ca8f20c0f9a2/sample\n", :data=>{:exitcode=>125, :output=>"sample. Application directory doesn't exist: /var/lib/stickshift/5229a2c0f1424aed90d4ca8f20c0f9a2/sample\n"}, :statuscode=>1}, @action="cartridge_do">] DEBUG: server results: sample. Application directory doesn't exist: /var/lib/stickshift/5229a2c0f1424aed90d4ca8f20c0f9a2/sample DEBUG: rpc_exec: Filtering rpc_exec to server ex-std-node1.stg.rhcloud.com DEBUG: rpc_exec: rpc_client=#<MCollective::RPC::Client:0x7fea08c881c0> Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support. #<StickShift::NodeException: Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support.> ["/var/www/stickshift/broker/lib/express/broker/application_container_proxy.rb:890:in `run_cartridge_command'", "/var/www/stickshift/broker/lib/express/broker/application_container_proxy.rb:683:in `remove_component'", "/var/www/stickshift/broker/lib/express/broker/application_container_proxy.rb:202:in `deconfigure_cartridge'", "/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.7.8/lib/stickshift-controller/app/models/gear.rb:66:in `deconfigure'", "/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.7.8/lib/stickshift-controller/app/models/application.rb:560:in `deconfigure_dependencies'", "/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.7.8/lib/stickshift-controller/app/models/application.rb:1353:in `call'", "/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.7.8/lib/stickshift-controller/app/models/application.rb:1353:in `run_on_gears'", "/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.7.8/lib/stickshift-controller/app/models/application.rb:1351:in `each'", "/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.7.8/lib/stickshift-controller/app/models/application.rb:1351:in `run_on_gears'", "/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.7.8/lib/stickshift-controller/app/models/application.rb:559:in `deconfigure_dependencies'", "/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.7.8/lib/stickshift-controller/app/models/application.rb:554:in `each'", "/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.7.8/lib/stickshift-controller/app/models/application.rb:554:in `deconfigure_dependencies'", "/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.7.8/lib/stickshift-controller/app/models/application.rb:286:in `cleanup_and_delete'", "/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.7.8/lib/stickshift-controller/app/controllers/applications_controller.rb:228:in `destroy'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/implicit_render.rb:4:in `send_action'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/implicit_render.rb:4:in `send_action'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/base.rb:150:in `process_action'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rendering.rb:11:in `process_action'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/callbacks.rb:18:in `process_action'", "/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:436:in `_run__1535252375__process_action__851827830__callbacks'", "/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:410:in `send'", "/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:410:in `_run_process_action_callbacks'", "/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:94:in `send'", "/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:94:in `run_callbacks'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/callbacks.rb:17:in `process_action'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rescue.rb:17:in `process_action'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:30:in `process_action'", "/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52:in `instrument'", "/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications/instrumenter.rb:21:in `instrument'", "/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/notifications.rb:52:in `instrument'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/instrumentation.rb:29:in `process_action'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/base.rb:119:in `process'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/abstract_controller/rendering.rb:41:in `process'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal.rb:138:in `dispatch'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_controller/metal.rb:178:in `action'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:62:in `call'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:62:in `dispatch'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:27:in `call'", "/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/route_set.rb:150:in `call'", "/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/code_generation.rb:93:in `recognize'", "/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/code_generation.rb:75:in `optimized_each'", "/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/code_generation.rb:92:in `recognize'", "/usr/lib/ruby/gems/1.8/gems/rack-mount-0.7.1/lib/rack/mount/route_set.rb:141:in `call'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/routing/route_set.rb:493:in `call'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/head.rb:14:in `call'", "/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/methodoverride.rb:24:in `call'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/params_parser.rb:21:in `call'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/flash.rb:182:in `call'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/session/abstract_store.rb:149:in `call'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/cookies.rb:302:in `call'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/callbacks.rb:46:in `call'", "/usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/callbacks.rb:416:in `_run_call_callbacks'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/callbacks.rb:44:in `call'", "/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/sendfile.rb:102:in `call'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/remote_ip.rb:48:in `call'", "/usr/lib/ruby/gems/1.8/gems/actionpack-3.0.10/lib/action_dispatch/middleware/show_exceptions.rb:47:in `call'", "/usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/rack/logger.rb:13:in `call'", "/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/runtime.rb:17:in `call'", "/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/lock.rb:34:in `call'", "/usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:168:in `call'", "/usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:77:in `send'", "/usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:77:in `method_missing'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/rack/request_handler.rb:96:in `process_request'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/abstract_request_handler.rb:514:in `accept_and_process_next_request'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/classic_rails/application_spawner.rb:321:in `start_request_handler'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/classic_rails/application_spawner.rb:275:in `send'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/classic_rails/application_spawner.rb:275:in `handle_spawn_application'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/utils.rb:479:in `safe_fork'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/classic_rails/application_spawner.rb:270:in `handle_spawn_application'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/abstract_server.rb:357:in `__send__'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/abstract_server.rb:180:in `start'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/classic_rails/application_spawner.rb:149:in `start'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/spawn_manager.rb:219:in `spawn_rails_application'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/abstract_server_collection.rb:132:in `lookup_or_add'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/spawn_manager.rb:214:in `spawn_rails_application'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/abstract_server_collection.rb:82:in `synchronize'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/abstract_server_collection.rb:79:in `synchronize'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/spawn_manager.rb:213:in `spawn_rails_application'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/spawn_manager.rb:132:in `spawn_application'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/abstract_server.rb:357:in `__send__'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'", "/usr/lib/ruby/gems/1.8/gems/passenger-3.0.4/helper-scripts/passenger-spawn-server:102"] MongoDataStore.save(Application, xcoulon+test, sample, #hidden)
The app deletion code should deal with this scenario until the root cause can be determined and fixed.
This one may have the same root cause as #809230. Keeping both open until confirmed.
The problem has been fixed. But unfortunately, the apps that have already run into this problem, will stay in the bad shape.
Bug 809230 has been fixed, so mark this bug as "Verified".