Bug 977034 - Unable to remove gear or delete app that has been broken on node
Unable to remove gear or delete app that has been broken on node
Status: CLOSED WONTFIX
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
medium Severity low
: ---
: ---
Assigned To: Jhon Honce
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-22 15:47 EDT by Clayton Coleman
Modified: 2015-05-14 19:22 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-20 12:46:30 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
remove_db (48.65 KB, text/plain)
2013-07-01 01:53 EDT, Meng Bo
no flags Details

  None (edit)
Description Clayton Coleman 2013-06-22 15:47:00 EDT
Code: latest master

Scenario:

1) Create a scaled app
2) Add a db cart to scaled gear
3) Accidentally run rm -rf * from the root of the scaled db gear
4) Attempt to remove scaled DB cartridge

Get broker error, cartridge is unremovable and application is undeletable

Failure from cartridge remove:

2013-06-22 15:43:40.928 [DEBUG] DEBUG: rpc_client.custom_request('cartridge_do', {:cartridge=>"openshift-origin-node", :action=>"app-destroy", :args=>{"--with-app-uuid"=>"456817058247940675469312", "--with-app-n
ame"=>"foo", "--with-container-uuid"=>"ad2fccfcdb6c11e2921d12313b0cb1be", "--with-container-name"=>"ad2fccfcdb6c11e2921d12313b0cb1be", "--with-namespace"=>"abc", "--with-request-id"=>"b88751a3fa9fa53e11616dd0be8
31496", "--cart-name"=>"openshift-origin-node"}}, ip-10-98-178-76, {'identity' => ip-10-98-178-76}) (Request ID: b88751a3fa9fa53e11616dd0be831496) (pid:2936)
2013-06-22 15:43:41.482 [DEBUG] DEBUG: [#<MCollective::RPC::Result:0x0000000662e200 @agent="openshift", @action="cartridge_do", @results={:sender=>"ip-10-98-178-76", :statuscode=>1, :statusmsg=>"cartridge_do_act
ion failed -1. Output can't convert nil into String", :data=>{:time=>nil, :output=>"can't convert nil into String", :exitcode=>-1}}>] (Request ID: b88751a3fa9fa53e11616dd0be831496) (pid:2936)
2013-06-22 15:43:41.488 [DEBUG] DEBUG: MCollective Response Time (execute_direct: app-destroy): 0.121825099s  (Request ID: b88751a3fa9fa53e11616dd0be831496) (pid:2936)
2013-06-22 15:43:41.489 [DEBUG] DEBUG: server results: can't convert nil into String (pid:2936)
2013-06-22 15:43:41.492 [ERROR] Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support. (pid:2936)
2013-06-22 15:43:41.495 [ERROR] ["/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.10.3/lib/openshift/mcollective_application_container_proxy.rb:2633:in `parse_result'", "/opt/r
h/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.10.3/lib/openshift/mcollective_application_container_proxy.rb:440:in `destroy'", "/var/www/openshift/broker/lib/online/broker/mcollect
ive_ext.rb:26:in `destroy'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.10.5/app/models/gear.rb:83:in `destroy_gear'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-cont
roller-1.10.5/app/models/pending_app_op_group.rb:225:in `block in execute'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.10.5/app/models/pending_app_op_group.rb:155:in `each'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.10.5/app/models/pending_app_op_group.rb:155:in `execute'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.10.5/app/models/application.rb:1256:in `run_jobs'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.10.5/app/models/application.rb:546:in `block in remove_features'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.10.5/app/models/application.rb:1298:in `run_in_application_lock'",
Comment 1 Clayton Coleman 2013-06-22 16:05:19 EDT
The OPENSHIFT_<CART>_* variables got deleted, OPENSHIFT_<CART>_VERSION was removed.  Seems like that isn't required for deletion, and those values shouldn't be in the cart's env variables if they can be deleted (i.e. if they can be removed by the user in the gear).
Comment 2 Clayton Coleman 2013-06-22 16:07:05 EDT
Stack from node:

/var/log/openshift/node/platform.log:June 22 16:00:21 ERROR can't convert nil into String
/var/log/openshift/node/platform.log-June 22 16:00:21 ERROR /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.10.5/lib/openshift-origin-node/model/v2_cart_model.rb:163:in `exists?'
/var/log/openshift/node/platform.log-/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.10.5/lib/openshift-origin-node/model/v2_cart_model.rb:163:in `get_cartridge_from_directory'
/var/log/openshift/node/platform.log-/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.10.5/lib/openshift-origin-node/model/v2_cart_model.rb:142:in `get_cartridge'
Comment 3 Clayton Coleman 2013-06-22 16:32:44 EDT
It was OPENSHIFT_<cart>_IDENT that was in error - it had been deleted.  If the IDENT var has been removed we should still be able to delete the gear in the node and broker (since we know the name of the cart).

Node platform code should probably only be depending on ENV vars that are completely under their control - in V2 I'm not sure that anything other than the cart name from the broker is required to complete the deletion.  Would also recommend we move the *special* cart vars into .env, and not reference vars in $CART/env from runtime code.
Comment 4 Abhishek Gupta 2013-06-26 13:37:24 EDT
This doesn't seem to be reproducible any more. Requesting QE to confirm.
Comment 5 Jianwei Hou 2013-06-26 22:15:40 EDT
I was able to reproduce with following steps according to Comment 3

Steps:
1. Create a scaled application with db cartridge
rhc app create php1s php-5.3 mysql-5.1 -s
2. On node, delete the OPENSHIFT_MYSQL_IDENT under /var/lib/openshift/$MYSQL_GEAR_UUID/mysql/env
3. Remove mysql cartridge from rhc
rhc cartridge remove mysql-5.1 -a php1s

Result:
Removing mysql-5.1 from 'php1s' ... 
Unable to complete the requested operation due to: Node execution failure (invalid exit code from node).  If the problem
persists please contact Red Hat support..
Reference ID: 8f8872ed2eb6131550448f60ac468a66
Comment 6 Jhon Honce 2013-06-27 18:50:31 EDT
Fixed in stage https://github.com/openshift/origin-server/pull/2960
Comment 7 Meng Bo 2013-07-01 01:53:07 EDT
Created attachment 767243 [details]
remove_db

Test on devenv-stage_383, with the same steps as comment#5. Still get error when removing the cartridge.

# rhc cartridge remove mysql-5 -a php1s --confirm
Using mysql-5.1 (MySQL Database 5.1) for 'mysql-5'
Removing mysql-5.1 from 'php1s' ... 
The server did not respond correctly. This may be an issue with the server configuration or with your connection to the server (such as a Web proxy or firewall).
Please verify that you can access the OpenShift server https://localhost/broker/rest/domains/bmeng1/applications/php1s/cartridges/mysql-5.1


development log attached.
Comment 8 openshift-github-bot 2013-07-01 13:41:28 EDT
Commits pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/7d264345ab900ecb090741e05e099cdb0b3cdbc5
Bug 977034 - Removing IDENT breaks deconfigure

https://github.com/openshift/origin-server/commit/b077a583884f9590053578f9aa20bdd26ec4f84b
Bug 977034 - Removing IDENT breaks destroy

* introduced error when logging error message
Comment 9 Meng Bo 2013-07-04 02:28:58 EDT
Checked on devenv_3447, still cannot remove cartridge successfully.

# rhc cartridge remove mysql -a php1s --confirm
Password: ***

Using mysql-5.1 (MySQL Database 5.1) for 'mysql'
Removing mysql-5.1 from 'php1s' ... 
Unable to complete the requested operation due to: Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support..
Reference ID: af1d50cd3eb0af8185dcfcaac987e0fe


============
development.log:
# grep af1d50cd3eb0af8185dcfcaac987e0fe /var/log/openshift/broker/development.log 
2013-07-04 02:22:43.145 [DEBUG] DEBUG: rpc_client.custom_request('cartridge_do', {:cartridge=>"openshift-origin-node", :action=>"app-destroy", :args=>{"--with-app-uuid"=>"51d512e977b7a70a37000009", "--with-app-name"=>"php1s", "--with-container-uuid"=>"52450542e47111e29b6522000aa44e30", "--with-container-name"=>"52450542e47111e29b6522000aa44e30", "--with-namespace"=>"bmengdev", "--with-request-id"=>"af1d50cd3eb0af8185dcfcaac987e0fe", "--cart-name"=>"openshift-origin-node"}}, ip-10-164-78-48, {'identity' => ip-10-164-78-48}) (Request ID: af1d50cd3eb0af8185dcfcaac987e0fe) (pid:2615)
2013-07-04 02:22:43.233 [DEBUG] DEBUG: [#<MCollective::RPC::Result:0x000000066a85a0 @agent="openshift", @action="cartridge_do", @results={:sender=>"ip-10-164-78-48", :statuscode=>1, :statusmsg=>"cartridge_do_action failed -1. Output can't convert nil into String", :data=>{:time=>nil, :output=>"can't convert nil into String", :exitcode=>-1}}>] (Request ID: af1d50cd3eb0af8185dcfcaac987e0fe) (pid:2615)
2013-07-04 02:22:43.234 [DEBUG] DEBUG: MCollective Response Time (execute_direct: app-destroy): 0.092719574s  (Request ID: af1d50cd3eb0af8185dcfcaac987e0fe) (pid:2615)
2013-07-04 02:22:43.270 [ERROR] Reference ID: af1d50cd3eb0af8185dcfcaac987e0fe - Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.
Reference ID: af1d50cd3eb0af8185dcfcaac987e0fe (pid:2615)
Comment 10 Jhon Honce 2013-07-10 17:37:06 EDT
When I attempt to reproduce I get the following on dev_3479

> ssh 471822159417319511556096@471822159417319511556096-honcenet.dev.rhcloud.com
> rm -rf *
ERROR: Refusing unsafe deletion: app-root


471822159417319511556096 is the mysql gear.
Comment 11 Xiaoli Tian 2013-07-10 22:05:09 EDT
Hi, Jhon

Comment 5 should be current consistent reproducing steps according to all the comments,  and step 2 should be done by root user ( login to the instance directly).

But we will try if we could still reproduce it on the latest devenv today before moving it back to assigned.
Comment 12 Meng Bo 2013-07-11 04:24:26 EDT
Checked on devenv_3482, the issue has still not been fixed.

# rhc cartridge remove mysql-5 -a php1s --confirm
Using mysql-5.1 (MySQL Database 5.1) for 'mysql-5'
Removing mysql-5.1 from 'php1s' ... 
Unable to complete the requested operation due to: Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support..
Reference ID: 83a463c234ff4956eb5ff9f184e1de01
Comment 13 Jhon Honce 2013-07-11 11:28:30 EDT
No Online user can have root access, so this is not a blocker bug.

Note You need to log in before you can comment on or make changes to this bug.