Bug 807638

Summary: Unable to destroy an failure app succussfully
Product: OKD Reporter: Xiaoli Tian <xtian>
Component: PodAssignee: Rajat Chopra <rchopra>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 2.xCC: dmcphers, rchopra, rmillner, twiest
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-04-13 18:32:33 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Xiaoli Tian 2012-03-28 11:48:33 UTC 
Description of problem:
If meeting an error while creating an app, like node execution failure, will find error log in development.log like following:
DEBUG: server results: ERROR: unable to destroy user account 766b449b5c49461995672a103a542f8e

For detail see the additional information 


Version-Release number of selected component (if applicable):
devenv_1677

How reproducible:
Always

Steps to Reproduce:
1.Make a failure while creating app like : create an scalable jboss app with CLI like bug 806834
2.If failed , check development.log

  
Actual results:
The app can not be totally destroyed, the symlink in /var/lib/stickshift/ is not totally cleared.

Expected results:
Once a failure happen, the deconfigure steps should run smoothly,the failure app can be cleared as well.

Additional info:
DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fea3aa77840>
DEBUG: rpc_client.custom_request('cartridge_do', mc_args.inspect, @id, {'identity' => @id})
DEBUG: [#<MCollective::RPC::Result:0x7fea3a9d34e8 @results={:sender=>"ip-10-84-229-121", :data=>{:exitcode=>1, :output=>"id: 766b449b5c49461995672a103a542f8e: No such user\n"}, :statusmsg=>"cartridge_do_action failed 1.  Output id: 766b449b5c49461995672a103a542f8e: No such user\n", :statuscode=>1}, @action="cartridge_do", @agent="libra">]
DEBUG: server results: id: 766b449b5c49461995672a103a542f8e: No such user
DEBUG: Failed to embed 'haproxy-1.4' in 'scalejboss1' for user 'xtian+t51'
DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fea3a9c8d18>
DEBUG: rpc_client.custom_request('cartridge_do', mc_args.inspect, @id, {'identity' => @id})
DEBUG: [#<MCollective::RPC::Result:0x7fea3a98a950 @results={:sender=>"ip-10-84-229-121", :data=>{:exitcode=>125, :output=>"dd6e28a37e.  Application directory doesn't exist:  /var/lib/stickshift/766b449b5c49461995672a103a542f8e/dd6e28a37e\n"}, :statusmsg=>"cartridge_do_action failed 125.  Output dd6e28a37e.  Application directory doesn't exist:  /var/lib/stickshift/766b449b5c49461995672a103a542f8e/dd6e28a37e\n", :statuscode=>1}, @action="cartridge_do", @agent="libra">]
DEBUG: server results: dd6e28a37e.  Application directory doesn't exist:  /var/lib/stickshift/766b449b5c49461995672a103a542f8e/dd6e28a37e
DEBUG: rpc_exec: Filtering rpc_exec to server ip-10-84-229-121
DEBUG: rpc_exec: rpc_client=#<MCollective::RPC::Client:0x7fea3a977a80>
DEBUG: Component 'embedded/haproxy-1.4' in application 'scalejboss1' not found on node 'ip-10-84-229-121'.  Continuing with deconfigure.
Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support.
EBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fea3ac557e8>
DEBUG: rpc_client.custom_request('cartridge_do', mc_args.inspect, @id, {'identity' => @id})
DEBUG: [#<MCollective::RPC::Result:0x7fea3abc5c38 @results={:sender=>"ip-10-84-229-121", :data=>{:exitcode=>255, :output=>"ERROR: unable to destroy user account 766b449b5c49461995672a103a542f8e\n"}, :statusmsg=>"cartridge_do_action failed 255.  Output ERROR: unable to destroy user account 766b449b5c49461995672a103a542f8e\n", :statuscode=>1}, @action="cartridge_do", @agent="libra">]
DEBUG: server results: ERROR: unable to destroy user account 766b449b5c49461995672a103a542f8e
Comment 1 Dan McPherson 2012-03-28 15:51:09 UTC 
This is happening because app-destroy is being called twice but deconfigure isn't being called at all.  So the second app destroy gives the user can't delete error and the symlink is left because deconfigure is never called.  To recreate this I put an exit 1 at the bottom of php configure.  Here is the mcollective output (abbreviated):

D, [2012-03-28T11:43:41.265950 #17733] DEBUG -- : libra.rb:60:in `cartridge_do_action' cartridge_do_action call / request = #<MCollective::RPC::Request:0x7f66395f3ba0
 @action="cartridge_do",
 @agent="libra",
 @caller="cert=mcollective-public",
 @data=
  {:process_results=>true,
   :args=>"'danmcp414a' 'danmcp414' 'b132937b95f447e5aee48f239e0060cf'",
   :cartridge=>"php-5.3",
   :action=>"configure"},
 @sender="ip-10-224-110-193",
 @time=1332949421,
 @uniqid="5b7ec5269e2d75115f13357b1006da01">

D, [2012-03-28T11:43:41.266191 #17733] DEBUG -- : libra.rb:61:in `cartridge_do_action' cartridge_do_action validation = php-5.3 configure 'danmcp414a' 'danmcp414' 'b132937b95f447e5aee48f239e0060cf'
D, [2012-03-28T11:43:43.514634 #17733] DEBUG -- : libra.rb:104:in `cartridge_do_action' cartridge_do_action ERROR (1)
------
Initialized empty Git repository in /var/lib/stickshift/b132937b95f447e5aee48f239e0060cf/git/danmcp414a.git/
/var/lib/stickshift/b132937b95f447e5aee48f239e0060cf/git/danmcp414a.git /tmp
/tmp

------)

D, [2012-03-28T11:43:43.824601 #17733] DEBUG -- : libra.rb:60:in `cartridge_do_action' cartridge_do_action call / request = #<MCollective::RPC::Request:0x7f66395ba378
 @action="cartridge_do",
 @agent="libra",
 @caller="cert=mcollective-public",
 @data=
  {:process_results=>true,
   :args=>
    "--with-app-uuid 'b132937b95f447e5aee48f239e0060cf' --with-container-uuid 'b132937b95f447e5aee48f239e0060cf'",
   :cartridge=>"stickshift-node",
   :action=>"app-destroy"},
 @sender="ip-10-224-110-193",
 @time=1332949423,
 @uniqid="7453d80fbe838d7837db39fd2756d58c">

D, [2012-03-28T11:43:43.826665 #17733] DEBUG -- : libra.rb:61:in `cartridge_do_action' cartridge_do_action validation = stickshift-node app-destroy --with-app-uuid 'b132937b95f447e5aee48f239e0060cf' --with-container-uuid 'b132937b95f447e5aee48f239e0060cf'
D, [2012-03-28T11:43:45.605425 #17733] DEBUG -- : libra.rb:102:in `cartridge_do_action' cartridge_do_action (0)
------

------)

D, [2012-03-28T11:43:45.988379 #17733] DEBUG -- : libra.rb:60:in `cartridge_do_action' cartridge_do_action call / request = #<MCollective::RPC::Request:0x7f663958ef20
 @action="cartridge_do",
 @agent="libra",
 @caller="cert=mcollective-public",
 @data=
  {:process_results=>true,
   :args=>
    "--with-app-uuid 'b132937b95f447e5aee48f239e0060cf' --with-container-uuid 'b132937b95f447e5aee48f239e0060cf'",
   :cartridge=>"stickshift-node",
   :action=>"app-destroy"},
 @sender="ip-10-224-110-193",
 @time=1332949425,
 @uniqid="b7d3e16ac48e7b0bf60269ea57e14288">

D, [2012-03-28T11:43:45.988579 #17733] DEBUG -- : libra.rb:61:in `cartridge_do_action' cartridge_do_action validation = stickshift-node app-destroy --with-app-uuid 'b132937b95f447e5aee48f239e0060cf' --with-container-uuid 'b132937b95f447e5aee48f239e0060cf'
D, [2012-03-28T11:43:46.725089 #17733] DEBUG -- : libra.rb:104:in `cartridge_do_action' cartridge_do_action ERROR (255)
------
ERROR: unable to destroy user account b132937b95f447e5aee48f239e0060cf

------)
Comment 2 Dan McPherson 2012-03-28 16:26:33 UTC 
FYI: I am guessing this is what's causing the dangling .conf file failures from jenkins.
Comment 3 Rajat Chopra 2012-03-28 20:53:21 UTC 
Fixed with rev#dbe8ad2f7cf5b5869871feb58228a84e2d42b563
Comment 4 Xiaoli Tian 2012-03-29 03:52:53 UTC 
Tested this on devenv_1679 by make an creation failure intentionally, it's fixed now.