Bug 1171002

Summary: Moving gear from non-district node to district node will fail when set USE_PREDICTABLE_GEAR_UUIDS=true
Product: OpenShift Container Platform Reporter: Yanping Zhang <yanpzhan>
Component: ContainersAssignee: Luke Meyer <lmeyer>
Status: CLOSED ERRATA QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.2.0CC: bleanhar, jokerman, libra-onpremise-devel, lmeyer, mmccomas
Target Milestone: ---Keywords: Upstream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openshift-origin-msg-common-1.22.0.1-1 Doc Type: Bug Fix
Doc Text:
Cause: A bug in UUID validation prevented gears using predictable UUIDs from being relocated from non districted nodes to districted nodes. Consequence: Gears using predictable UUIDs would not properly relocate. Fix: UUID validation has been updated to account for predictable UUID logic. Result: Gears using predictable UUIDs can now be relocated from un-districted nodes to districted nodes.
Story Points: ---
Clone Of:
: 1180004 (view as bug list) Environment:
Last Closed: 2015-02-12 13:09:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1180004    
Bug Blocks:    

Description Yanping Zhang 2014-12-05 09:22:01 UTC
Description of problem:
Moving gear from non-district node to district node will fail when set USE_PREDICTABLE_GEAR_UUIDS=true in broker.conf.

Version-Release number of selected component (if applicable):
ose2.2/2014-12-02.3

How reproducible:
Always

Steps to Reproduce:
1.Prepare env with 2 small gear size nodes, node1 in district, node2 not in district. Set USE_PREDICTABLE_GEAR_UUIDS=true in broker.conf.
2.Create app on node2(not in district). 
# rhc app create pl1 perl-5.10
Make sure gear's uid available in district.
# id -u hanli1dom-pl1-1
1001
3.Move app gear from node2 to node1.
# oo-admin-move --gear_uuid hanli1dom-pl1-1 -i node1.ose22-move.com.cn


Actual results:
3.Moving failed.
# oo-admin-move --gear_uuid hanli1dom-pl1-1 -i node1.ose22-move.com.cn
URL: http://pl1-hanli1dom.ose22-20141201.com.cn
Login: hanli1
App UUID: 548166218e45f07eb8000016
Gear UUID: 548166218e45f07eb8000016
/opt/rh/ruby193/root/usr/share/ruby/mcollective/ddl/base.rb:145:in `rescue in validate_input_argument': Cannot validate input gear_uuid: value should match ^[a-zA-Z0-9]+$ (MCollective::DDLValidationError)
	from /opt/rh/ruby193/root/usr/share/ruby/mcollective/ddl/base.rb:126:in `validate_input_argument'
	from /opt/rh/ruby193/root/usr/share/ruby/mcollective/ddl/agentddl.rb:194:in `block in validate_rpc_request'
	from /opt/rh/ruby193/root/usr/share/ruby/mcollective/ddl/agentddl.rb:186:in `each'
	from /opt/rh/ruby193/root/usr/share/ruby/mcollective/ddl/agentddl.rb:186:in `validate_rpc_request'
	from /opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/client.rb:169:in `validate_request'
	from /opt/rh/ruby193/root/usr/share/ruby/mcollective/rpc/client.rb:240:in `method_missing'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.31.1.1/lib/openshift/mcollective_application_container_proxy.rb:2873:in `block in get_gear_uid'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.31.1.1/lib/openshift/mcollective_application_container_proxy.rb:2440:in `rpc_exec'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.31.1.1/lib/openshift/mcollective_application_container_proxy.rb:2872:in `get_gear_uid'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.31.1.1/lib/openshift/mcollective_application_container_proxy.rb:1947:in `move_gear'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.31.1.1/lib/openshift/mcollective_application_container_proxy.rb:1937:in `block in move_gear_secure'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.32.3.2/app/models/lock.rb:62:in `run_in_app_lock'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.31.1.1/lib/openshift/mcollective_application_container_proxy.rb:1934:in `move_gear_secure'
	from /usr/sbin/oo-admin-move:114:in `<main>'


Expected results:
3.Should move gear successfully.

Additional info:
Moving gear between districts or moving gear between nodes in the same district work well when USE_PREDICTABLE_GEAR_UUIDS=true.
If set USE_PREDICTABLE_GEAR_UUIDS=false in broker.conf, repeat the step 2 and 3, move will succeed.

Comment 1 Luke Meyer 2014-12-05 13:28:52 UTC
Another validation to correct. Thank you for catching this.

Comment 3 openshift-github-bot 2014-12-05 20:55:32 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/544c81e62f9b1e2eb9cb06cb064a53844a7c7c34
openshift.ddl: fix gear UUID regexes

Bug 1171002 - Moving gear from non-district node to district node will
fail when set USE_PREDICTABLE_GEAR_UUIDS=true
https://bugzilla.redhat.com/show_bug.cgi?id=1171002

Comment 6 Yanping Zhang 2015-01-26 02:35:09 UTC
Verified on puddle-2-2-2015-01-23
Steps to verify:
1.Prepare env with 2 small gear size nodes, node1 in district, node2 not in district. Set USE_PREDICTABLE_GEAR_UUIDS=true in broker.conf.
2.Create app on node2(not in district). 
#rhc app create php1 php-5.4 -n domtest
3.Move app gear from node2 to node1.
#oo-admin-move --gear_uuid domtest-php1-1 -i node1.hosts.ose22.com
Actual results:
3.Move successfully.
# oo-admin-move --gear_uuid domtest-php1-1 -i node1.hosts.ose22.com
URL: http://php1-domtest.ose22.com
Login: test
App UUID: 54af86bb11490db285000001
Gear UUID: 54af86bb11490db285000001
DEBUG: Source district uuid: NONE
DEBUG: Destination district uuid: 54a8be0e11490dbc37000001
DEBUG: Getting existing app 'php1' status before moving
DEBUG: Gear component 'php-5.4' was running
DEBUG: Stopping existing app cartridge 'php-5.4' before moving
DEBUG: Force stopping existing app before moving
DEBUG: Reserved uid '1000' on district: '54a8be0e11490dbc37000001'
DEBUG: Gear platform is 'linux'
DEBUG: Creating new account for gear 'domtest-php1-1' on node1.hosts.ose22.com
DEBUG: Moving content for app 'php1', gear 'domtest-php1-1' to node1.hosts.ose22.com
Identity added: /etc/openshift/rsync_id_rsa (/etc/openshift/rsync_id_rsa)
Warning: Permanently added '192.168.1.54' (RSA) to the list of known hosts.
Warning: Permanently added '192.168.1.55' (RSA) to the list of known hosts.
Agent pid 24871
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 24871 killed;
DEBUG: Moving system components for app 'php1', gear 'domtest-php1-1' to node1.hosts.ose22.com
Identity added: /etc/openshift/rsync_id_rsa (/etc/openshift/rsync_id_rsa)
Agent pid 24903
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 24903 killed;
DEBUG: Starting cartridge 'php-5.4' in 'php1' after move on node1.hosts.ose22.com
DEBUG: Fixing DNS and mongo for gear 'domtest-php1-1' after move
DEBUG: Changing server identity of 'domtest-php1-1' from 'node2.hosts.ose22.com' to 'node1.hosts.ose22.com'
DEBUG: Deconfiguring old app 'php1' on node2.hosts.ose22.com after move
Successfully moved gear with uuid 'domtest-php1-1' of app 'php1' from 'node2.hosts.ose22.com' to 'node1.hosts.ose22.com'

Comment 8 errata-xmlrpc 2015-02-12 13:09:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0220.html