Bug 1028364

Summary:	Failed to move non-scale app across district with different gear size
Product:	OpenShift Online	Reporter:	zhaozhanqi <zzhao>
Component:	Pod	Assignee:	Rajat Chopra <rchopra>
Status:	CLOSED CURRENTRELEASE	QA Contact:	libra bugs <libra-bugs>
Severity:	high	Docs Contact:
Priority:	high
Version:	2.x	CC:	abhgupta, rchopra, xtian
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-02-26 19:08:12 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description zhaozhanqi 2013-11-08 09:55:45 UTC

Description of problem:
Given a non-scale small gear app created. and then move this gear to another node across district which is medium gear size. will move failed with error:
DEBUG: Moving failed.  Rolling back gear 'app2' in 'app2' with delete on 'ip-10-151-3-4'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.17.0/app/models/group_instance.rb:143:in `[]=': string not matched (IndexError)
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.17.0/app/models/group_instance.rb:143:in `set_group_override'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.17.0/app/models/group_instance.rb:66:in `gear_size='
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/proxy.rb:143:in `method_missing'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.17.0/lib/openshift/mcollective_application_container_proxy.rb:1807:in `move_gear_post'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.17.0/lib/openshift/mcollective_application_container_proxy.rb:1972:in `move_gear'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.17.0/lib/openshift/mcollective_application_container_proxy.rb:1899:in `block in move_gear_secure'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.17.0/app/models/application.rb:1622:in `run_in_application_lock'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.17.0/lib/openshift/mcollective_application_container_proxy.rb:1898:in `move_gear_secure'
	from /usr/sbin/oo-admin-move:112:in `<main>'



Version-Release number of selected component (if applicable):
devenv_4003

How reproducible:
always

Steps to Reproduce:
1. setup multi-node env ( 2 districts and 2 nodes: small and medium)
2. create one non scale small gear app
3. move this gear to medium node across district

Actual results:

oo-admin-move --gear_uuid  527ca25b6092959a02000391  -i ip-10-151-3-4 --change_district
URL: http://app2-zqd.dev.rhcloud.com
Login: zzhao
App UUID: 527ca25b6092959a02000391
Gear UUID: 527ca25b6092959a02000391
DEBUG: Source district uuid: 269218789935952510844928
DEBUG: Destination district uuid: 847631806651380447313920
DEBUG: Getting existing app 'app2' status before moving
DEBUG: Gear component 'php-5.3' was running
DEBUG: Stopping existing app cartridge 'php-5.3' before moving
DEBUG: Force stopping existing app cartridge 'php-5.3' before moving
DEBUG: Reserved uid '1577' on district: '847631806651380447313920'
DEBUG: Creating new account for gear 'app2' on ip-10-151-3-4
DEBUG: Moving content for app 'app2', gear 'app2' to ip-10-151-3-4
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 15379
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 15379 killed;
DEBUG: Moving system components for app 'app2', gear 'app2' to ip-10-151-3-4
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 15417
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 15417 killed;
DEBUG: Starting cartridge 'php-5.3' in 'app2' after move on ip-10-151-3-4
DEBUG: Fixing DNS and mongo for gear 'app2' after move
DEBUG: Changing server identity of 'app2' from 'ip-10-239-9-117' to 'ip-10-151-3-4'
DEBUG: Moving failed.  Rolling back gear 'app2' in 'app2' with delete on 'ip-10-151-3-4'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.17.0/app/models/group_instance.rb:143:in `[]=': string not matched (IndexError)
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.17.0/app/models/group_instance.rb:143:in `set_group_override'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.17.0/app/models/group_instance.rb:66:in `gear_size='
	from /opt/rh/ruby193/root/usr/share/gems/gems/mongoid-3.0.21/lib/mongoid/relations/proxy.rb:143:in `method_missing'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.17.0/lib/openshift/mcollective_application_container_proxy.rb:1807:in `move_gear_post'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.17.0/lib/openshift/mcollective_application_container_proxy.rb:1972:in `move_gear'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.17.0/lib/openshift/mcollective_application_container_proxy.rb:1899:in `block in move_gear_secure'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.17.0/app/models/application.rb:1622:in `run_in_application_lock'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.17.0/lib/openshift/mcollective_application_container_proxy.rb:1898:in `move_gear_secure'
	from /usr/sbin/oo-admin-move:112:in `<main>'


Expected results:

should move successfully.

Additional info:

Comment 1 Rajat Chopra 2013-11-08 22:54:43 UTC

Fix with https://github.com/openshift/origin-server/pull/4140

Comment 2 zhaozhanqi 2013-11-11 02:43:15 UTC

Tested this bug on devenv_4016, it has been fixed.

 oo-admin-move --gear_uuid 5280433925da37bd57000007 -i ip-10-204-102-77 --change_district
URL: http://zqphp-zqd.dev.rhcloud.com
Login: zzhao
App UUID: 5280433925da37bd57000007
Gear UUID: 5280433925da37bd57000007
DEBUG: Source district uuid: c8f40ee64a7811e3a34112313b08227a
DEBUG: Destination district uuid: cd00a2a04a7911e3b1a712313b08227a
DEBUG: Getting existing app 'zqphp' status before moving
DEBUG: Gear component 'php-5.3' was running
DEBUG: Stopping existing app cartridge 'php-5.3' before moving
DEBUG: Force stopping existing app cartridge 'php-5.3' before moving
DEBUG: Reserved uid '3725' on district: 'cd00a2a04a7911e3b1a712313b08227a'
DEBUG: Creating new account for gear 'zqphp' on ip-10-204-102-77
DEBUG: Moving content for app 'zqphp', gear 'zqphp' to ip-10-204-102-77
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Warning: Permanently added '10.196.33.132' (RSA) to the list of known hosts.
Warning: Permanently added '10.204.102.77' (RSA) to the list of known hosts.
Agent pid 17078
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 17078 killed;
DEBUG: Moving system components for app 'zqphp', gear 'zqphp' to ip-10-204-102-77
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 17144
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 17144 killed;
DEBUG: Starting cartridge 'php-5.3' in 'zqphp' after move on ip-10-204-102-77
DEBUG: Fixing DNS and mongo for gear 'zqphp' after move
DEBUG: Changing server identity of 'zqphp' from 'ip-10-196-33-132' to 'ip-10-204-102-77'
DEBUG: Deconfiguring old app 'zqphp' on ip-10-196-33-132 after move
Successfully moved gear with uuid '5280433925da37bd57000007' of app 'zqphp' from 'ip-10-196-33-132' to 'ip-10-204-102-77'

 so mark it to verified.