The OPENSHIFT_<CART>_* variables got deleted, OPENSHIFT_<CART>_VERSION was removed.  Seems like that isn't required for deletion, and those values shouldn't be in the cart's env variables if they can be deleted (i.e. if they can be removed by the user in the gear).

Stack from node:

/var/log/openshift/node/platform.log:June 22 16:00:21 ERROR can't convert nil into String
/var/log/openshift/node/platform.log-June 22 16:00:21 ERROR /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.10.5/lib/openshift-origin-node/model/v2_cart_model.rb:163:in `exists?'
/var/log/openshift/node/platform.log-/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.10.5/lib/openshift-origin-node/model/v2_cart_model.rb:163:in `get_cartridge_from_directory'
/var/log/openshift/node/platform.log-/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.10.5/lib/openshift-origin-node/model/v2_cart_model.rb:142:in `get_cartridge'

It was OPENSHIFT_<cart>_IDENT that was in error - it had been deleted.  If the IDENT var has been removed we should still be able to delete the gear in the node and broker (since we know the name of the cart).

Node platform code should probably only be depending on ENV vars that are completely under their control - in V2 I'm not sure that anything other than the cart name from the broker is required to complete the deletion.  Would also recommend we move the *special* cart vars into .env, and not reference vars in $CART/env from runtime code.

This doesn't seem to be reproducible any more. Requesting QE to confirm.

I was able to reproduce with following steps according to Comment 3

Steps:
1. Create a scaled application with db cartridge
rhc app create php1s php-5.3 mysql-5.1 -s
2. On node, delete the OPENSHIFT_MYSQL_IDENT under /var/lib/openshift/$MYSQL_GEAR_UUID/mysql/env
3. Remove mysql cartridge from rhc
rhc cartridge remove mysql-5.1 -a php1s

Result:
Removing mysql-5.1 from 'php1s' ... 
Unable to complete the requested operation due to: Node execution failure (invalid exit code from node).  If the problem
persists please contact Red Hat support..
Reference ID: 8f8872ed2eb6131550448f60ac468a66

Fixed in stage https://github.com/openshift/origin-server/pull/2960

Created attachment 767243 [details]
remove_db

Test on devenv-stage_383, with the same steps as comment#5. Still get error when removing the cartridge.

# rhc cartridge remove mysql-5 -a php1s --confirm
Using mysql-5.1 (MySQL Database 5.1) for 'mysql-5'
Removing mysql-5.1 from 'php1s' ... 
The server did not respond correctly. This may be an issue with the server configuration or with your connection to the server (such as a Web proxy or firewall).
Please verify that you can access the OpenShift server https://localhost/broker/rest/domains/bmeng1/applications/php1s/cartridges/mysql-5.1


development log attached.

Commits pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/7d264345ab900ecb090741e05e099cdb0b3cdbc5
Bug 977034 - Removing IDENT breaks deconfigure

https://github.com/openshift/origin-server/commit/b077a583884f9590053578f9aa20bdd26ec4f84b
Bug 977034 - Removing IDENT breaks destroy

* introduced error when logging error message

Checked on devenv_3447, still cannot remove cartridge successfully.

# rhc cartridge remove mysql -a php1s --confirm
Password: ***

Using mysql-5.1 (MySQL Database 5.1) for 'mysql'
Removing mysql-5.1 from 'php1s' ... 
Unable to complete the requested operation due to: Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support..
Reference ID: af1d50cd3eb0af8185dcfcaac987e0fe


============
development.log:
# grep af1d50cd3eb0af8185dcfcaac987e0fe /var/log/openshift/broker/development.log 
2013-07-04 02:22:43.145 [DEBUG] DEBUG: rpc_client.custom_request('cartridge_do', {:cartridge=>"openshift-origin-node", :action=>"app-destroy", :args=>{"--with-app-uuid"=>"51d512e977b7a70a37000009", "--with-app-name"=>"php1s", "--with-container-uuid"=>"52450542e47111e29b6522000aa44e30", "--with-container-name"=>"52450542e47111e29b6522000aa44e30", "--with-namespace"=>"bmengdev", "--with-request-id"=>"af1d50cd3eb0af8185dcfcaac987e0fe", "--cart-name"=>"openshift-origin-node"}}, ip-10-164-78-48, {'identity' => ip-10-164-78-48}) (Request ID: af1d50cd3eb0af8185dcfcaac987e0fe) (pid:2615)
2013-07-04 02:22:43.233 [DEBUG] DEBUG: [#<MCollective::RPC::Result:0x000000066a85a0 @agent="openshift", @action="cartridge_do", @results={:sender=>"ip-10-164-78-48", :statuscode=>1, :statusmsg=>"cartridge_do_action failed -1. Output can't convert nil into String", :data=>{:time=>nil, :output=>"can't convert nil into String", :exitcode=>-1}}>] (Request ID: af1d50cd3eb0af8185dcfcaac987e0fe) (pid:2615)
2013-07-04 02:22:43.234 [DEBUG] DEBUG: MCollective Response Time (execute_direct: app-destroy): 0.092719574s  (Request ID: af1d50cd3eb0af8185dcfcaac987e0fe) (pid:2615)
2013-07-04 02:22:43.270 [ERROR] Reference ID: af1d50cd3eb0af8185dcfcaac987e0fe - Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.
Reference ID: af1d50cd3eb0af8185dcfcaac987e0fe (pid:2615)

When I attempt to reproduce I get the following on dev_3479

> ssh 471822159417319511556096.rhcloud.com
> rm -rf *
ERROR: Refusing unsafe deletion: app-root


471822159417319511556096 is the mysql gear.

Hi, Jhon

Comment 5 should be current consistent reproducing steps according to all the comments,  and step 2 should be done by root user ( login to the instance directly).

But we will try if we could still reproduce it on the latest devenv today before moving it back to assigned.

Checked on devenv_3482, the issue has still not been fixed.

# rhc cartridge remove mysql-5 -a php1s --confirm
Using mysql-5.1 (MySQL Database 5.1) for 'mysql-5'
Removing mysql-5.1 from 'php1s' ... 
Unable to complete the requested operation due to: Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support..
Reference ID: 83a463c234ff4956eb5ff9f184e1de01

No Online user can have root access, so this is not a blocker bug.