Bug 1015866

Summary: [app-destroy] Exception when calling app-destroy
Product: OpenShift Online Reporter: Kenny Woodson <kwoodson>
Component: PodAssignee: Abhishek Gupta <abhgupta>
Status: CLOSED UPSTREAM QA Contact: libra bugs <libra-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 2.xCC: mmahut, rchopra
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-10-07 19:14:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kenny Woodson 2013-10-06 11:16:36 UTC
Description of problem:

Our automation creates and then destroys applications to test the broker and verify that it is running properly.  One of the applications was stuck and not able to be destroyed.

Here is the mcollective.log from the server.


I, [2013-10-06T07:11:08.675968 #1305]  INFO -- : openshift.rb:180:in `execute_parallel_action' execute_parallel_action call / action: execute_parallel, agent=openshift, data={"ex-std-node234.prod.rhcloud.com"=>
  [{:tag=>"",
    :gear=>"524dee49e0b8cdf32e000001",
    :job=>
     {:cartridge=>"openshift-origin-node",
      :action=>"app-state-show",
      :args=>
       {"--with-app-uuid"=>"524dee49e0b8cdf32e000001",
        "--with-app-name"=>"chkexsrv1",
        "--with-container-uuid"=>"524dee49e0b8cdf32e000001",
        "--with-container-name"=>"chkexsrv1",
        "--with-namespace"=>"openshiftnagios",
        "--with-uid"=>6479,
        "--with-request-id"=>"969099581a9a5708db6d39f4e71712af"}},
    :result_stdout=>"",
    :result_stderr=>"",
    :result_exit_code=>""}],
 :process_results=>true}

I, [2013-10-06T07:11:08.676286 #1305]  INFO -- : openshift.rb:131:in `execute_action' Executing action [app-state-show] using method oo_app_state_show with args [{"--with-app-uuid"=>"524dee49e0b8cdf32e000001", "--with-app-name"=>"chkexsrv1", "--with-container-uuid"=>"524dee49e0b8cdf32e000001", "--with-container-name"=>"chkexsrv1", "--with-namespace"=>"openshiftnagios", "--with-uid"=>6479, "--with-request-id"=>"969099581a9a5708db6d39f4e71712af"}]
I, [2013-10-06T07:11:08.688036 #1305]  INFO -- : openshift.rb:146:in `execute_action' Finished executing action [app-state-show] (0)
I, [2013-10-06T07:11:08.720790 #1305]  INFO -- : openshift.rb:201:in `execute_parallel_action' execute_parallel_action call - [{:tag=>"", :gear=>"524dee49e0b8cdf32e000001", :job=>{:cartridge=>"openshift-origin-node", :action=>"app-state-show", :args=>{"--with-app-uuid"=>"524dee49e0b8cdf32e000001", "--with-app-name"=>"chkexsrv1", "--with-container-uuid"=>"524dee49e0b8cdf32e000001", "--with-container-name"=>"chkexsrv1", "--with-namespace"=>"openshiftnagios", "--with-uid"=>6479, "--with-request-id"=>"969099581a9a5708db6d39f4e71712af"}}, :result_stdout=>"started", :result_stderr=>"", :result_exit_code=>0}]
I, [2013-10-06T07:11:36.537698 #1305]  INFO -- : openshift.rb:91:in `cartridge_do_action' cartridge_do_action call / action: cartridge_do, agent=openshift, data={:cartridge=>"openshift-origin-node",
 :action=>"app-destroy",
 :args=>
  {"--with-app-uuid"=>"524dee49e0b8cdf32e000001",
   "--with-app-name"=>"chkexsrv1",
   "--with-container-uuid"=>"524dee49e0b8cdf32e000001",
   "--with-container-name"=>"chkexsrv1",
   "--with-namespace"=>"openshiftnagios",
   "--with-uid"=>6479,
   "--with-request-id"=>"62a618a77651bbbb2d37f52e63c84b71",
   "--cart-name"=>"openshift-origin-node"},
 :process_results=>true}

I, [2013-10-06T07:11:36.537931 #1305]  INFO -- : openshift.rb:92:in `cartridge_do_action' cartridge_do_action validation = openshift-origin-node app-destroy {"--with-app-uuid"=>"524dee49e0b8cdf32e000001", "--with-app-name"=>"chkexsrv1", "--with-container-uuid"=>"524dee49e0b8cdf32e000001", "--with-container-name"=>"chkexsrv1", "--with-namespace"=>"openshiftnagios", "--with-uid"=>6479, "--with-request-id"=>"62a618a77651bbbb2d37f52e63c84b71", "--cart-name"=>"openshift-origin-node"}
I, [2013-10-06T07:11:36.538262 #1305]  INFO -- : openshift.rb:131:in `execute_action' Executing action [app-destroy] using method oo_app_destroy with args [{"--with-app-uuid"=>"524dee49e0b8cdf32e000001", "--with-app-name"=>"chkexsrv1", "--with-container-uuid"=>"524dee49e0b8cdf32e000001", "--with-container-name"=>"chkexsrv1", "--with-namespace"=>"openshiftnagios", "--with-uid"=>6479, "--with-request-id"=>"62a618a77651bbbb2d37f52e63c84b71", "--cart-name"=>"openshift-origin-node"}]
I, [2013-10-06T07:11:36.558267 #1305]  INFO -- : openshift.rb:330:in `rescue in oo_app_destroy' '' is not a legal cartridge identifier
I, [2013-10-06T07:11:36.558551 #1305]  INFO -- : openshift.rb:331:in `rescue in oo_app_destroy' ["/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.14.1/lib/openshift-origin-common/models/manifest.rb:322:in `parse_ident'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.14.7/lib/openshift-origin-node/model/v2_cart_model.rb:164:in `get_cartridge_from_directory'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.14.7/lib/openshift-origin-node/model/v2_cart_model.rb:77:in `block in each_cartridge'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.14.7/lib/openshift-origin-node/model/v2_cart_model.rb:975:in `block in process_cartridges'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.14.7/lib/openshift-origin-node/model/v2_cart_model.rb:973:in `each'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.14.7/lib/openshift-origin-node/model/v2_cart_model.rb:973:in `process_cartridges'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.14.7/lib/openshift-origin-node/model/v2_cart_model.rb:76:in `each_cartridge'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.14.7/lib/openshift-origin-node/model/application_container.rb:237:in `block in destroy'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.14.1/lib/openshift-origin-common/utils/file_needs_sync.rb:38:in `block in open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.14.1/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.14.1/lib/openshift-origin-common/utils/file_needs_sync.rb:36:in `open'", "/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.14.7/lib/openshift-origin-node/model/application_container.rb:233:in `destroy'", "/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:328:in `oo_app_destroy'", "/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:136:in `execute_action'", "/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:103:in `cartridge_do_action'", "/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'", "/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'", "/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'", "/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'"]
I, [2013-10-06T07:11:36.558681 #1305]  INFO -- : openshift.rb:146:in `execute_action' Finished executing action [app-destroy] (-1)
I, [2013-10-06T07:11:36.581485 #1305]  INFO -- : openshift.rb:112:in `cartridge_do_action' cartridge_do_action failed (-1)
------
'' is not a legal cartridge identifier
------)


Version-Release number of selected component (if applicable):
Current

How reproducible:
I don't know.

Steps to Reproduce:
1.
2.
3.

Actual results:
Calling destroy fails.

Expected results:
Should succeed and remove the app.

Additional info:

Comment 1 Rajat Chopra 2013-10-07 19:14:34 UTC
'mcollective' was restarted in the middle of configure.
Strange part is that the new process sent the response back to what was initiated by the previous process. Half of what was logged in the active queue, but with exitcode '0'.

This is a behaviour issue with mcollective.

Comment 2 Rajat Chopra 2013-10-08 18:19:55 UTC
*** Bug 1015960 has been marked as a duplicate of this bug. ***