Bug 844490 - rhc-admin-ctl-app not removing apps
rhc-admin-ctl-app not removing apps
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Pod (Show other bugs)
2.x
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Rajat Chopra
libra bugs
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-07-30 17:57 EDT by Troy Dawson
Modified: 2015-05-14 22:02 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-08-07 16:42:18 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Troy Dawson 2012-07-30 17:57:23 EDT
Description of problem:
A user has three apps that they tried unsuccesfully to remove.

The dns name is gone, and as far as I can tell, they are removed from the the nodes they are supposed to be on. I have checked on all the other nodes as well and cannot find these apps.  As far as I can tell, they are on mongo only.

When I run 
rhc-admin-ctl-app -l kevin.j.hooke@accenture.com -a joomlatest2 -c destroy

  !!!! WARNING !!!! WARNING !!!! WARNING !!!!
  You are about to destroy the joomlatest2 application.

  This is NOT reversible, all remote data for this application will be removed.
Do you want to destroy this application (y/n): y
/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.13.14/lib/stickshift-controller/app/models/application.rb:338:in `destroy': Could not destroy all gears of application. (StickShift::NodeException)
	from /usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.13.14/lib/stickshift-controller/app/models/application.rb:315:in `cleanup_and_delete'
	from /usr/bin/rhc-admin-ctl-app:103

Version-Release number of selected component (if applicable):
rhc-broker-0.95.22-1.el6_3

How reproducible:
For this users apps, 100%

Steps to Reproduce:
1. Have an app that is only in mongo
2. Try to remove it with rhc-admin-ctl-app
3.
  
Actual results:
/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.13.14/lib/stickshift-controller/app/models/application.rb:338:in `destroy': Could not destroy all gears of application. (StickShift::NodeException)
	from /usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.13.14/lib/stickshift-controller/app/models/application.rb:315:in `cleanup_and_delete'
	from /usr/bin/rhc-admin-ctl-app:103


Expected results:
The app to be removed from mongo

Additional info:
Comment 1 Troy Dawson 2012-07-30 18:02:22 EDT
I found the problem, when I looked at how many gears the user had.

User kevin.j.hooke@accenture.com:
      consumed gears: 5
           max gears: 3
          vip status: false

I'm not sure how it got into that state, but when I raised his gears to 6, then I was able to delete his applications.
Comment 2 Thomas Wiest 2012-07-30 18:11:39 EDT
I think that this is still a bug as the admin tools should always be able to remove an app.
Comment 3 Rajat Chopra 2012-07-30 21:02:32 EDT
Its failing the consistency check - (consumed_gears + ngears <= max_gears)
where, ngears is '-1' for a gear delete. The appropriate fix is coming with US2550, where gear and app deletion is separated. That also prevents the known cases where consumed_gears gets out of sync with actual gears consumed.

While the admin way should be to fix up 'consumed_gears' and then delete the app, because consumed_gears need to be fixed up anyway, should there be a method to 'force delete' the app? 


The 'force-delete' feature is implemented with pull request#147 in li.repo


Be aware that such a 'force delete' may result with gears being on the disk but not on mongo - the feature does list out the warnings indicating such slips, though.



How to test the feature?
1. Create some apps in a user account (on devenv)

2. Then force set the consumed gears count to reflect the situation in this bug. Use rhc-admin-ctl-user --setconsumedgears for that. Set it to a number much more than the max_gear limit.

3. Check mismatch between consumed_gears and max_gear limit by running rhc-admin-ctl-user -l <login>

4. Then try deleting the app, either as user, or by rhc-admin-ctl-app -a <app-name> -c destroy -l <login>

5. The deletion should fail.

6. Now try with the new feature rhc-admin-ctl-app -a <app-name> -c force-destroy -l <login>. The operation should succeed with some warnings on the screen.
Comment 4 John Poelstra 2012-07-31 13:14:31 EDT
expect to have fix today
Comment 5 Rajat Chopra 2012-07-31 15:09:47 EDT
Fix available with rev#2a968bbdfd024d7d3029ead211c6f15f4b90c4bc in li.repo and rev#1035699e2bf9d66255ad230f781552f21bfe36b6 in crankcase.repo

The consumed_gears count is not checked at gear deletion time now. With the new fix step#4 in comment#3 will succeed also.
Comment 6 Johnny Liu 2012-08-01 06:49:54 EDT
Verififid this bug on devenv_1931, and PASSED.


1. Create some dummy data into mongodb, like this:
PRIMARY> db.user.find()
{ "_id" : "jialiu@redhat.com", "apps" : [ 	{ 	"categories" : [ ], 	"suggests_feature" : [ ], 	"creation_time" : "2012-08-01T05:32:13-04:00", 	"cart_data_def" : { 	 }, 	"help_topics" : { 	 }, 	"license" : "unknown", 	"name" : "phptest", 	"start_order" : [ 	"@@app/cart-php-5.3", 	"@@app" ], 	"conflicts_feature" : [ ], 	"conn_endpoints_list" : [ ], 	"vendor" : "unknown", 	"website" : "", 	"default_profile" : "default", 	"requires" : [ ], 	"group_instances" : [ 	{ 	"component_instances" : [ 	"@@app", 	"@@app/cart-php-5.3" ], 	"profile_name" : "default", 	"name" : "@@app/cart-php-5.3", 	"gears" : [ 	{ 	"configured_components" : [ 	"@@app/cart-php-5.3" ], 	"name" : "phptest", 	"group_instance_name" : "@@app/cart-php-5.3", 	"server_identity" : "ip-10-190-25-42", 	"uuid" : "5999fb93ea7743528f53036c43102acc", 	"uid" : null, 	"node_profile" : "small" } ], 	"group_name" : "default", 	"cart_name" : "php-5.3", 	"reused_by" : [ 	"@@app/cart-php-5.3", 	"@@app" ], 	"min" : 1, 	"max" : -1, 	"node_profile" : "small" } ], 	"requires_feature" : [ 	"php-5.3" ], 	"domain" : { 	"namespace" : "jialiu", 	"uuid" : "6eda7bcc625d43428c3411b4a47fcef5" }, 	"profiles" : [ 	{ 	"stop_order" : [ ], 	"name" : "default", 	"start_order" : [ ], 	"components" : [ 	{ 	"subscribes" : [ ], 	"depends_service" : [ ], "depends" : [ ], 	"name" : "default", 	"publishes" : [ ], 	"generated" : true } ], 	"group_overrides" : [ ], 	"groups" : [ 	{ 	"component_refs" : [ 	{ 	"name" : "default", 	"component" : "default" } ], 	"scaling" : { 	"min" : 1, 	"max" : -1 }, 	"name" : "default", 	"generated" : true } ], 	"connections" : [ ], 	"configure_order" : [ ], 	"generated" : true, 	"provides" : [ ] } ], 	"comp_instances" : [ 	{ 	"exec_order" : [ ], 	"parent_cart_group" : "default", 	"name" : "@@app/cart-php-5.3", 	"parent_component_name" : "default", 	"cart_properties" : { 	 }, 	"group_instance_name" : "@@app/cart-php-5.3", 	"parent_cart_profile" : "default", 	"cart_data" : [ 	null ], 	"parent_cart_name" : "php-5.3", 	"dependencies" : [ ] }, 	{ 	"exec_order" : [ 	"@@app/cart-php-5.3" ], 	"parent_cart_group" : "default", 	"name" : "@@app", 	"parent_component_name" : "default", 	"cart_properties" : { 	 }, 	"group_instance_name" : "@@app", 	"parent_cart_profile" : "default", 	"cart_data" : [ ], 	"parent_cart_name" : "phptest", 	"dependencies" : [ 	"@@app/cart-php-5.3" ] } ], 	"description" : "", 	"uuid" : "5999fb93ea7743528f53036c43102acc", 	"license_url" : "", 	"scalable" : false, 	"display_name" : "phptest-0.0-noarch", 	"provides_feature" : [ ], 	"configure_order" : [ 	"@@app/cart-php-5.3", 	"@@app" ], 	"node_profile" : "small", 	"architecture" : "noarch", 	"version" : "0.0" } ], "capabilities" : { "gear_sizes" : [ "small" ] }, "consumed_gears" : 5, "domains" : [ { "namespace" : "jialiu", "uuid" : "6eda7bcc625d43428c3411b4a47fcef5" } ], "env_vars" : null, "login" : "jialiu@redhat.com", "max_gears" : 3, "parent_user_login" : null, "ssh_keys" : { "default" : { "key" : "AAAAB3NzaC1yc2EAAAADAQABAAABAQDSv44aEPcObZkAN5VI8XHW23b7JL0wBftkPwtXwHF6ppxnvpIhQNyBy5crHWGrigEOGLsJWH7hmo/rfkELuhfpdaGIB582AAJ5Eeug+Fv7yQFQodCafALhh/piXXnJ7xsnFpy6Pz5OVuxC2nRoew8oqSIjKaHTdjzuSPNRviEKLTypcREtnQp7nCCTDm3NFjaM40tDA3/i9m708qViQHv5tqkdyrfLMu5Lq+oJMrzP911aCn3F0GTc+T/cUC/R2ay5wLhv9FT+eTrDSOsMFt9BZYFT+mfSyIJhaGuxB7OUQ3qf4RRMo+0hGINRSLldFtRaZiQUyqw6nKPR460MfgMN", "type" : "ssh-rsa" } }, "system_ssh_keys" : null, "usage_records" : [ 	{ 	"gear_size" : "small", 	"sync_time" : null, 	"usage_type" : "GEAR_USAGE", 	"gear_uuid" : "5999fb93ea7743528f53036c43102acc", 	"uuid" : "8d38a66f9d264762bcc327786312d34b", 	"time" : ISODate("2012-08-01T09:32:23.808Z"), 	"event" : "begin" } ], "uuid" : "0fd23545dce548529239c647df510e50" }


2. Check mismatch between consumed_gears and max_gear limit
# rhc-admin-ctl-user -l jialiu@redhat.com


User jialiu@redhat.com:
      consumed gears: 5
           max gears: 3
          gear sizes: small


3. Run rhc-admin-ctl-app with destroy option successfully, and check mongo to verify the dummp app data is also removed from mongo.
# rhc-admin-ctl-app -a phptest -c destroy -l jialiu@redhat.com 
  !!!! WARNING !!!! WARNING !!!! WARNING !!!!
  You are about to destroy the phptest application.

  This is NOT reversible, all remote data for this application will be removed.
Do you want to destroy this application (y/n): y
Successfully destroyed application: phptest

Check mongo, find the dummp app data is also removed from mongo.

4. Run rhc-admin-ctl-app with force-destroy option successfully, and check mongo to verify the dummp app data is also removed from mongo.
# rhc-admin-ctl-app -a phptest -c force-destroy -l jialiu@redhat.com
  !!!! WARNING !!!! WARNING !!!! WARNING !!!!
  You are about to destroy the phptest application.

  This is NOT reversible, all remote data for this application will be forcefully removed.
Do you want to forcefully destroy this application (y/n): y
Successfully destroyed application: phptest

Note You need to log in before you can comment on or make changes to this bug.