Bug 841407 - Got a partially destroyed app in my account after trying to destroy a scalable app
Got a partially destroyed app in my account after trying to destroy a scalabl...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Rajat Chopra
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-07-18 18:14 EDT by Nam Duong
Modified: 2015-05-14 18:56 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-08-07 16:42:52 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Nam Duong 2012-07-18 18:14:09 EDT
Description of problem:
Steps to reproduce:
1) Created a scalable app:
rhc app create -a kitchensink -t jbossas-7 -s

2) Embedded mysql:
rhc app cartridge add -a kitchensink -c mysql-5.1

3) Proceeded to onboard https://github.com/openshift/kitchensink-example as per README.md

4) ssh'd to my haproxy gear and disabled autoscaling
haproxy_ctld_daemon stop

5) Then manually scaled up with 
haproxy_ctld -u
This returned with an error, but I didn't think anything of it.  I got 2 nodes so I was happy.  Validated I got 2 nodes by viewing my gear registry.

6) I tried to destroy the app and I got this error:
---
rhc app destroy -a kitchensink -d
Password: ********

!!!! WARNING !!!! WARNING !!!! WARNING !!!!
You are about to destroy the kitchensink application.

This is NOT reversible, all remote data for this application will be removed.
Do you want to destroy this application (y/n): y
Contacting https://openshift.redhat.com
Problem reported from server. Response code was 500.

DEBUG:
ERROR: unable to destroy user account bd12377be92848f6bc32314131a07623ERROR: unable to destroy user account 32174eeb63e149af9c1c7211fd785daf

Exit Code: 1
api_c: placeholder
broker_c: namespacerhloginsshapp_uuiddebugaltercartridgecart_typeactionapp_nameapi
API version:    1.1.3

RESULT:
Could not destroy all gears of application.
---


I wouldn't have minded so much but it was also blocking me from removing an sshkey:
---
rhc sshkey remove -i cloud9 -d
Password: ********

Submitting form:
key_name: cloud9
action: remove-key
rhlogin: nduong+mac@redhat.com
Contacting https://openshift.redhat.com
Problem reported from server. Response code was 500.

DEBUG:


Exit Code: 1
api_c: placeholder
broker_c: namespacerhloginsshapp_uuiddebugaltercartridgecart_typeactionapp_nameapi
API version:    1.1.3

RESULT:
Error removing settings from gear: bd12377be92848f6bc32314131a07623 with status: -1 and output: 
---

Please let me know if I need to write another bug for the sshkey removal issue.
Comment 1 Dan McPherson 2012-07-18 18:44:15 EDT
When you retry to delete does it continue to fail with the same message about unable to destroy user account?
Comment 2 Thomas Wiest 2012-07-18 19:10:28 EDT
I've checked and the gears are gone from the nodes.

Here's what I get when I try to destroy it with the admin tools:

# rhc-admin-ctl-app -l nduong+mac@redhat.com -a kitchensink -c destroy
  !!!! WARNING !!!! WARNING !!!! WARNING !!!!
  You are about to destroy the kitchensink application.

  This is NOT reversible, all remote data for this application will be removed.
Do you want to destroy this application (y/n): y
/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.13.14/lib/stickshift-controller/app/models/application.rb:338:in `destroy': Could not destroy all gears of application. (StickShift::NodeException)
	from /usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.13.14/lib/stickshift-controller/app/models/application.rb:315:in `cleanup_and_delete'
	from /usr/bin/rhc-admin-ctl-app:103
Comment 3 Nam Duong 2012-07-18 21:07:49 EDT
(In reply to comment #1)
> When you retry to delete does it continue to fail with the same message
> about unable to destroy user account?

Yes.  I'm still getting this:
rhc app destroy -a kitchensink -d
Password: ********

!!!! WARNING !!!! WARNING !!!! WARNING !!!!
You are about to destroy the kitchensink application.

This is NOT reversible, all remote data for this application will be removed.
Do you want to destroy this application (y/n): y
Contacting https://openshift.redhat.com
Problem reported from server. Response code was 500.

DEBUG:
ERROR: unable to destroy user account bd12377be92848f6bc32314131a07623ERROR: unable to destroy user account 32174eeb63e149af9c1c7211fd785daf

Exit Code: 1
broker_c: namespacerhloginsshapp_uuiddebugaltercartridgecart_typeactionapp_nameapi
api_c: placeholder
API version:    1.1.3

RESULT:
Could not destroy all gears of application.
Comment 4 Nam Duong 2012-07-19 13:36:37 EDT
I posted this issue to the Forums here:
https://openshift.redhat.com/community/forums/openshift/cant-destroy-a-scalable-application

Since yesterday, another user in the community has run into it.
Comment 5 Adam Miller 2012-07-20 09:27:40 EDT
Rajat has put together a fix for this.

Pull request for master:
https://github.com/openshift/crankcase/pull/255

Pull request for stage:
https://github.com/openshift/crankcase/pull/254

Will update with build information.
Comment 6 Adam Miller 2012-07-20 15:49:37 EDT
Package built: rubygem-stickshift-node-0.13.7-1.el6_3
Package included in: devenv-stage_226
Comment 7 Peter Ruan 2012-07-20 17:53:41 EDT
verified on STG by following the steps described above.

RESULT:
Successfully destroyed application: kitchensink
Comment 8 Xiaoli Tian 2012-07-22 22:32:27 EDT
Is this  bug duplicate of bug 839886, does this mean bug 839886 is also fixed?
Comment 9 Meng Bo 2012-07-22 22:46:11 EDT
I have a partially scalable app on stage since Jul 13th, and the app can be destroyed successfully after the stage hot fix.

===================

[bmeng@localhost ~]$ rhc domain show

User Info
=========
Namespace: bmeng1stg1
  RHLogin: bmeng+1@redhat.com


Application Info
================
jbas1
    Framework: jbossas-7
     Creation: 2012-07-13T01:47:13-04:00
         UUID: 1e22aa66fcf94da7bb2d9ac5dee214fb
      Git URL: ssh://1e22aa66fcf94da7bb2d9ac5dee214fb@jbas1-bmeng1stg1.stg.rhcloud.com/~/git/jbas1.git/
   Public URL: http://jbas1-bmeng1stg1.stg.rhcloud.com/

 Embedded: 
      haproxy-1.4

php1
    Framework: php-5.3
     Creation: 2012-07-17T01:37:13-04:00
         UUID: e872eb8aa7c44f9fa2fcf88ff976cdb4
      Git URL: ssh://e872eb8aa7c44f9fa2fcf88ff976cdb4@php1-bmeng1stg1.stg.rhcloud.com/~/git/php1.git/
   Public URL: http://php1-bmeng1stg1.stg.rhcloud.com/

 Embedded: 
      None

[bmeng@localhost ~]$ rhc app destroy -a jbas1 -d
Contacting https://stg.openshift.redhat.com
Response from server:
DEBUG:


Exit Code: 0
api_c: placeholder
broker_c: namespacerhloginsshapp_uuiddebugaltercartridgecart_typeactionapp_nameapi
API version:    1.1.3

RESULT:
Successfully destroyed application: jbas1

[bmeng@localhost ~]$ rhc app destroy -a php1 -d
Contacting https://stg.openshift.redhat.com
Response from server:
DEBUG:


Exit Code: 0
api_c: placeholder
broker_c: namespacerhloginsshapp_uuiddebugaltercartridgecart_typeactionapp_nameapi
API version:    1.1.3

RESULT:
Successfully destroyed application: php1

[bmeng@localhost ~]$ rhc domain show

User Info
=========
Namespace: bmeng1stg1
  RHLogin: bmeng+1@redhat.com


Application Info
================
No applications found.  You can use 'rhc app create' to create new applications.
Comment 10 Adam Miller 2012-07-23 15:18:03 EDT
Fix deployed to PROD by Thomas.

Note You need to log in before you can comment on or make changes to this bug.