Bug 987218 - oo-admin-move fails ungracefully with nodes in v2 cartridge mode
Summary: oo-admin-move fails ungracefully with nodes in v2 cartridge mode
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OKD
Classification: Red Hat
Component: Pod
Version: 2.x
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Rajat Chopra
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-07-23 02:19 UTC by Miciah Dashiel Butler Masters
Modified: 2015-05-15 02:20 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-08-07 22:53:05 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Miciah Dashiel Butler Masters 2013-07-23 02:19:04 UTC
Description of problem:
When oo-admin-move fails on nodes that are configured for v2 cartridges, it fails with the following error message and a backtrace in mcollective.log:

undefined method `remove_httpd_proxy' for #<OpenShift::V2CartridgeModel:0x000000038105d0>

This error and the backtrace make the problem look like a bug in OpenShift even when OpenShift is failing correctly (what an odd phrase! but the Steps to Reproduce demonstrate what I mean).


Version-Release number of selected component (if applicable):
The problematic code (as described under Additional info below) is in Origin HEAD and in OSE 1.2.


How reproducible:
Consistently.


Steps to Reproduce:

1. Install an OSE 1.2 PaaS with a broker and two nodes both with district 'NONE' and configured to use v2 cartridges.

2. Create an application and find its uuid.

3. Try to move the application from the node where it resides to the other node:

# oo-admin-move --gear_uuid <application-uuid> -i <other-node>

(The above command must be run on the broker.)


Actual results:
Step 3 fails, as should be expected, because the two nodes are not in the same district (neither one is in a district at all).  After Step 3, the following appears in /var/log/mcollective.log on the node where the application was initially created:

I, [2013-07-22T19:42:18.731198 #23951]  INFO -- : openshift.rb:51:in `cartridge_do_action' cartridge_do_action call / action: cartridge_do, agent=openshift, data={:cartridge=>"diy-0.1",
 :action=>"remove-httpd-proxy",
 :args=>
  {"--with-app-uuid"=>"51e55b0b6892df01bf000031",
   "--with-app-name"=>"testapp3",
   "--with-container-uuid"=>"51e55b0b6892df01bf000031",
   "--with-container-name"=>"testapp3",
   "--with-namespace"=>"ose",
   "--with-request-id"=>nil,
   "--cart-name"=>"diy-0.1",
   "--component-name"=>"diy-0.1",
   "--with-software-version"=>"0.1",
   "--cartridge-vendor"=>"redhat"},
 :process_results=>true}

I, [2013-07-22T19:42:18.731320 #23951]  INFO -- : openshift.rb:52:in `cartridge_do_action' cartridge_do_action validation = diy-0.1 remove-httpd-proxy {"--with-app-uuid"=>"51e55b0b6892df01bf000031", "--with-app-name"=>"testapp3", "--with-container-uuid"=>"51e55b0b6892df01bf000031", "--with-container-name"=>"testapp3"
, "--with-namespace"=>"ose", "--with-request-id"=>nil, "--cart-name"=>"diy-0.1", "--component-name"=>"diy-0.1", "--with-software-version"=>"0.1", "--cartridge-vendor"=>"redhat"}
I, [2013-07-22T19:42:18.731499 #23951]  INFO -- : openshift.rb:91:in `execute_action' Executing action [remove-httpd-proxy] using method oo_remove_httpd_proxy with args [{"--with-app-uuid"=>"51e55b0b6892df01bf000031", "--with-app-name"=>"testapp3", "--with-container-uuid"=>"51e55b0b6892df01bf000031", "--with-containe
r-name"=>"testapp3", "--with-namespace"=>"ose", "--with-request-id"=>nil, "--cart-name"=>"diy-0.1", "--component-name"=>"diy-0.1", "--with-software-version"=>"0.1", "--cartridge-vendor"=>"redhat"}]
E, [2013-07-22T19:42:18.735679 #23951] ERROR -- : openshift.rb:170:in `rescue in with_container_from_args' undefined method `remove_httpd_proxy' for #<OpenShift::V2CartridgeModel:0x000000038105d0>
E, [2013-07-22T19:42:18.735742 #23951] ERROR -- : openshift.rb:171:in `rescue in with_container_from_args' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.13/lib/openshift-origin-node/model/application_container.rb:427:in `remove_httpd_proxy'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:628:in `block in oo_remove_httpd_proxy'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:166:in `with_container_from_args'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:627:in `oo_remove_httpd_proxy'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:93:in `execute_action'
/opt/rh/ruby193/root/usr/libexec/mcollective/mcollective/agent/openshift.rb:65:in `cartridge_do_action'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/agent.rb:86:in `handlemsg'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'
/opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout'
/opt/rh/ruby193/root/usr/share/ruby/mcollective/agents.rb:125:in `block in dispatch'
I, [2013-07-22T19:42:18.735936 #23951]  INFO -- : openshift.rb:100:in `execute_action' Finished executing action [remove-httpd-proxy] (-1)
I, [2013-07-22T19:42:18.736060 #23951]  INFO -- : openshift.rb:73:in `cartridge_do_action' cartridge_do_action failed (-1)
------
undefined method `remove_httpd_proxy' for #<OpenShift::V2CartridgeModel:0x000000038105d0>
------)


Expected results:
Step 3 should fail, but oo-admin-move should fail gracefully without generating an "undefined method" error or backtrace in the node's mcollective.log.


Additional info:
oo-admin-move itself gives reasonable output and does not mention any undefined methods or give any backtraces.

The "remove-httpd-proxy" hook was removed from the v2 cartridge model, so calling the hook results in calling an undefined method in the cartridge model when v2 cartridges are used.  The platform now configures the port proxy for both v1 and v2 cartridges, so it is no longer necessary for move_gear to invoke "remove-httpd-proxy."  Commit 7d458f76116a0b72b651155d1178ad3d70398104 already dropped the call in move_gear to the "move" hook, which invoked the "deploy-httpd-proxy" hook.  All that is left is to remove the call to "remove-httpd-proxy" in the rollback code.

Comment 1 Miciah Dashiel Butler Masters 2013-07-23 02:20:20 UTC
This pull request fixes the bug:   https://github.com/openshift/origin-server/pull/3140

Comment 2 Abhishek Gupta 2013-07-23 19:27:41 UTC
Rajat. please review the pull request.

Comment 3 Rajat Chopra 2013-07-29 19:12:38 UTC
Merge pull request : https://github.com/openshift/origin-server/pull/3140

Comment 4 openshift-github-bot 2013-07-29 19:54:54 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/5e8976026d7533ce798088399fd3c318492bfae5
move_gear: drop outdated rollback code

Don't call the remove-httpd-proxy hook when rolling back a failed
move_gear operation in MCollectiveApplicationContainerProxy.

The remove-httpd-proxy hook was removed from the v2 cartridge model, so
calling the hook results in calling an undefined method in the cartridge
model when v2 cartridges are used.  Furthermore, the platform now takes
care of configuring the port proxy, so it is no longer necessary for
move_gear to invoke remove-httpd-proxy.  Commit
7d458f76116a0b72b651155d1178ad3d70398104 already dropped the call in
move_gear to the "move" hook, which invoked the deploy-httpd-proxy hook,
which is also handled by the platform now.

This commit fixes Bug 987218.

Comment 5 zhaozhanqi 2013-07-30 07:28:07 UTC
Tested this bug on devenv_3580


[root@ip-10-167-10-72 lib]# oo-admin-move --gear_uuid 51f74f5405ad100a65000001 -i ip-10-167-10-72 
URL: http://zqphp-zqd.dev.rhcloud.com
Login: zzhao
App UUID: 51f74f5405ad10cb0b00002b
Gear UUID: 51f74f5405ad10cb0b00002b
DEBUG: Source district uuid: 70f40c2af8d711e2a7fb22000aa70a48
DEBUG: Destination district uuid: NONE
DEBUG: Getting existing app 'zqphp' status before moving
DEBUG: Gear component 'php-5.3' was running
DEBUG: Stopping existing app cartridge 'php-5.3' before moving
DEBUG: Force stopping existing app cartridge 'php-5.3' before moving
DEBUG: Stopping existing app cartridge 'mongodb-2.2' before moving
DEBUG: Stopping existing app cartridge 'rockmongo-1.1' before moving
DEBUG: Reserved uid '4914' on district: 'NONE'
DEBUG: Creating new account for gear 'zqphp' on ip-10-167-10-72
DEBUG: Moving failed.  Rolling back gear 'zqphp' in 'zqphp' with delete on 'ip-10-167-10-72'
Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support

 move field and rollback, check the mcollective.log, won't found 'remove-http-proxy'


[root@ip-10-167-10-72 log]# grep remove-httpd-proxy mcollective.log 
[root@ip-10-167-10-72 log]#


Note You need to log in before you can comment on or make changes to this bug.