Bug 967706

Summary: Met "undefined method `cartridge_name'" error when moving gear to another node
Product: OpenShift Online Reporter: zhaozhanqi <zzhao>
Component: PodAssignee: Rajat Chopra <rchopra>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 2.xCC: rchopra, xtian
Target Milestone: ---Keywords: Regression, TestBlocker
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-06-11 04:12:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description zhaozhanqi 2013-05-28 05:50:25 UTC
Description of problem:
Met script errorl when moving gear to another node

Version-Release number of selected component (if applicable):
devenv_3277

How reproducible:
always

Steps to Reproduce:
1. set up a multi-node env
2. create one app
3. move the gear to another node


Actual results:
[root@ip-10-147-217-66 ~]# oo-admin-move --gear_uuid 12e822eac74211e28cb422000a93d942 -i ip-10-147-217-66
URL: http://zqeap-zqd.dev.rhcloud.com
Login: zzhao
App UUID: 51a41ca9f3ac152798000174
Gear UUID: 51a41ca9f3ac152798000174
DEBUG: Source district uuid: a2d8c848c73b11e29db722000a93d942
DEBUG: Destination district uuid: a2d8c848c73b11e29db722000a93d942
DEBUG: Getting existing app 'zqeap' status before moving
DEBUG: Error performing status on existing app on try 1: undefined method `cartridge_name' for "jbosseap-6.0":String
DEBUG: Error performing status on existing app on try 2: undefined method `cartridge_name' for "jbosseap-6.0":String
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.5/lib/openshift/mcollective_application_container_proxy.rb:1079:in `status': undefined method `cartridge_name' for "jbosseap-6.0":String (NoMethodError)
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.5/lib/openshift/mcollective_application_container_proxy.rb:2064:in `block in get_cart_status'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.5/lib/openshift/mcollective_application_container_proxy.rb:2209:in `block in do_with_retry'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.5/lib/openshift/mcollective_application_container_proxy.rb:2207:in `each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.5/lib/openshift/mcollective_application_container_proxy.rb:2207:in `do_with_retry'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.5/lib/openshift/mcollective_application_container_proxy.rb:2063:in `get_cart_status'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.5/lib/openshift/mcollective_application_container_proxy.rb:2039:in `get_app_status'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.5/lib/openshift/mcollective_application_container_proxy.rb:1765:in `move_gear'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.5/lib/openshift/mcollective_application_container_proxy.rb:1737:in `block in move_gear_secure'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.7/app/models/application.rb:1269:in `run_in_application_lock'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.5/lib/openshift/mcollective_application_container_proxy.rb:1736:in `move_gear_secure'
	from /usr/sbin/oo-admin-move:112:in `<main>'


Expected results:

should move successfully.

Additional info:

Comment 1 Rajat Chopra 2013-05-28 16:28:41 UTC
Fixed with pull request#2654

Comment 2 zhaozhanqi 2013-05-29 02:05:30 UTC
Test this issue on devenv_3281, it still can be reproduced.


[root@ip-10-154-158-161 ~]# oo-admin-move --gear_uuid 51a56063eeaffe0a22000001 -i ip-10-154-158-161
URL: http://zqphp-zqd.dev.rhcloud.com
Login: zzhao
App UUID: 51a56063eeaffeb51e000006
Gear UUID: 51a56063eeaffeb51e000006
DEBUG: Source district uuid: b24bc95cc80211e280d022000a9a9ea1
DEBUG: Destination district uuid: b24bc95cc80211e280d022000a9a9ea1
DEBUG: Getting existing app 'zqphp' status before moving
DEBUG: Error performing status on existing app on try 1: undefined method `cartridge_name' for "php-5.3":String
DEBUG: Error performing status on existing app on try 2: undefined method `cartridge_name' for "php-5.3":String
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:1079:in `status': undefined method `cartridge_name' for "php-5.3":String (NoMethodError)
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:2064:in `block in get_cart_status'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:2209:in `block in do_with_retry'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:2207:in `each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:2207:in `do_with_retry'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:2063:in `get_cart_status'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:2039:in `get_app_status'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:1765:in `move_gear'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:1737:in `block in move_gear_secure'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.8/app/models/application.rb:1269:in `run_in_application_lock'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:1736:in `move_gear_secure'
	from /usr/sbin/oo-admin-move:112:in `<main>'

Comment 3 zhaozhanqi 2013-05-29 05:45:19 UTC
Test this issue on devenv_3282, it also can be reproduced

[root@ip-10-165-28-115 ~]# oo-admin-move --gear_uuid 51a593ef28ce9f0bf9000001 -i ip-10-165-28-115
URL: http://zqpy26-zqd.dev.rhcloud.com
Login: zzhao
App UUID: 51a593ef28ce9ff626000006
Gear UUID: 51a593ef28ce9ff626000006
DEBUG: Source district uuid: 51a5937b28ce9f10aa000001
DEBUG: Destination district uuid: 51a5937b28ce9f10aa000001
DEBUG: Getting existing app 'zqpy26' status before moving
DEBUG: Error performing status on existing app on try 1: undefined method `cartridge_name' for "python-2.6":String
DEBUG: Error performing status on existing app on try 2: undefined method `cartridge_name' for "python-2.6":String
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:1079:in `status': undefined method `cartridge_name' for "python-2.6":String (NoMethodError)
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:2064:in `block in get_cart_status'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:2209:in `block in do_with_retry'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:2207:in `each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:2207:in `do_with_retry'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:2063:in `get_cart_status'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:2039:in `get_app_status'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:1765:in `move_gear'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:1737:in `block in move_gear_secure'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.8/app/models/application.rb:1269:in `run_in_application_lock'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:1736:in `move_gear_secure'
	from /usr/sbin/oo-admin-move:112:in `<main>'

Comment 4 Xiaoli Tian 2013-05-29 05:51:58 UTC
Seems /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb +2065
is still calling cart_name to the get the status.

def get_cart_status(gear, cart_name)

  result = source_container.status(gear, cart_name)

Comment 5 Rajat Chopra 2013-05-29 06:57:14 UTC
Yes. The gear model got fixed, but the mcollective container didnt. Thanks for the pointer. 

Fix with pull request#2670 !

Comment 6 zhaozhanqi 2013-05-29 12:58:51 UTC
Test it on devenv_3284, it still can be reproduced.


[root@ip-10-145-149-212 ~]# oo-admin-move --gear_uuid 669699248860149502181376 -i ip-10-145-149-212
URL: http://zqphp-zqd.dev.rhcloud.com
Login: zzhao
App UUID: 51a5fadda32c89da10000006
Gear UUID: 51a5fadda32c89da10000006
DEBUG: Source district uuid: 438489750166208367296512
DEBUG: Destination district uuid: 438489750166208367296512
DEBUG: Getting existing app 'zqphp' status before moving
DEBUG: Gear component 'php-5.3' was running
DEBUG: Stopping existing app cartridge 'php-5.3' before moving
DEBUG: Error performing stop on existing app on try 1: undefined method `cartridge_name' for "php-5.3":String
DEBUG: Error performing stop on existing app on try 2: undefined method `cartridge_name' for "php-5.3":String
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:987:in `stop': undefined method `cartridge_name' for "php-5.3":String (NoMethodError)
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:1697:in `block (2 levels) in move_gear_pre'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:2209:in `block in do_with_retry'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:2207:in `each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:2207:in `do_with_retry'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:1696:in `block in move_gear_pre'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:1688:in `each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:1688:in `move_gear_pre'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:1774:in `move_gear'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:1737:in `block in move_gear_secure'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.9.8/app/models/application.rb:1269:in `run_in_application_lock'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.9.6/lib/openshift/mcollective_application_container_proxy.rb:1736:in `move_gear_secure'
	from /usr/sbin/oo-admin-move:112:in `<main>'

Comment 7 Rajat Chopra 2013-05-29 17:44:48 UTC
refixed with pull request#2675
This time it was stop. Fixed expose-port also.

Comment 8 zhaozhanqi 2013-05-30 02:02:51 UTC
Test this issue on devenv_3288, it has been fixed.


[root@ip-10-145-184-56 ~]# oo-admin-move --gear_uuid 7190e288c8cc11e2801722000a91b838 -i ip-10-152-149-45
URL: http://zqphp-zqd.dev.rhcloud.com
Login: zzhao
App UUID: 51a6b24d8a9955ab3e000006
Gear UUID: 51a6b24d8a9955ab3e000006
DEBUG: Source district uuid: 323967506680864137281536
DEBUG: Destination district uuid: 323967506680864137281536
DEBUG: Getting existing app 'zqphp' status before moving
DEBUG: Gear component 'php-5.3' was running
DEBUG: Stopping existing app cartridge 'php-5.3' before moving
DEBUG: Force stopping existing app cartridge 'php-5.3' before moving
DEBUG: Creating new account for gear 'zqphp' on ip-10-152-149-45
DEBUG: Moving content for app 'zqphp', gear 'zqphp' to ip-10-152-149-45
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Warning: Permanently added '10.145.184.56' (RSA) to the list of known hosts.
Warning: Permanently added '10.152.149.45' (RSA) to the list of known hosts.
Agent pid 7766
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 7766 killed;
DEBUG: Moving system components for app 'zqphp', gear 'zqphp' to ip-10-152-149-45
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 7827
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 7827 killed;
DEBUG: Starting cartridge 'php-5.3' in 'zqphp' after move on ip-10-152-149-45
DEBUG: Fixing DNS and mongo for gear 'zqphp' after move
DEBUG: Changing server identity of 'zqphp' from 'ip-10-145-184-56' to 'ip-10-152-149-45'
DEBUG: Deconfiguring old app 'zqphp' on ip-10-145-184-56 after move
Successfully moved gear with uuid '7190e288c8cc11e2801722000a91b838' of app 'zqphp' from 'ip-10-145-184-56' to 'ip-10-152-149-45'