Description of problem: A gear destroy call may fail from the broker's point of view because of several reasons (mcoll timeout/activemq stoppage/network failure etc). In such circumstances, a repeat call for gear destroy always returns an error from the node because the node apparently has no trace of the said gear anymore (it succeeded destroying the gear the previous time). This means that such applications always get stuck in mongo although the gear has already been destroyed cleanly. The node should respond with a 'success' if an app-destroy call is made for a gear that does not exist. Version-Release number of selected component (if applicable): How reproducible: Difficult to reproduce on demand, but can be simulated. Steps to Reproduce: 1. Temporarily introduce a non-zero exitcode with a gear-destroy call on the broker. This is to simulate a network error that fails to receive a response from the node after the app-destroy call has been made. 2. Try to delete the app again, using oo-admin-ctl-app -c destroy or otherwise. Actual results: Subsequent app/gear-destroy calls fail because the gear has already been deleted from the node. Expected results: Node should report success on app-destroy calls for gears that do not exist. Additional info:
Its debatable whether the 'node' should return a zero exitcode on 'not found' situations. For our case here, though, its probably ok to have an warning message with a zero exitcode.
The node already behaves as desired; turns out there was a regression corrected by: https://github.com/openshift/origin-server/commit/1adf4900493e378f2df567762cb11bc1e9f1fd4e Should be fixed in master. If it crops up again, be sure to attach logs. Thanks!
Checked on devenv_3762, 1.Create app 2.Add "return 1" to the middle of "def oo-app-destroy" in mcollective/agent/openshift.rb 3.Restart mcollective and broker 4.Delete the app from rhc client Will meet Node execution failure here. 5.Check the app on node Should be deleted 6.Check the app on broker Should be still existed 7.Remove the change in openshift.rb 8.Restart mcollective and broker 9.Try to destroy the app again via oo-admin-ctl-app App is deleted successfully. Move bug to verified.