Bug 824375

Summary: Fail to move haproxy gear with jenkins client embedded
Product: OKD Reporter: Johnny Liu <jialiu>
Component: PodAssignee: Rajat Chopra <rchopra>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: qgong, rchopra, rmillner
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-06-08 17:59:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
broker log
none
taget node mcollective log none

Description Johnny Liu 2012-05-23 11:21:37 UTC
Description of problem:


Version-Release number of selected component (if applicable):
devenv_1786

How reproducible:
Always

Steps to Reproduce:
1. Create a scalable app with jenkins embeded.
2. Log into broker, move haproxy gears
3.
  
Actual results:
Fail to move haproxy gear.

# rhc-admin-move --gear_uuid 875500b7aa754887b804c52d688ceb5e -i ip-10-85-3-183
/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.11.3/lib/stickshift-controller/app/models/cloud_user.rb:8: warning: already initialized constant DEFAULT_SSH_KEY_NAME
/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.11.3/lib/stickshift-controller/app/models/validators/key_validator.rb:2: warning: already initialized constant KEY_NAME_MAX_LENGTH
/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.11.3/lib/stickshift-controller/app/models/validators/key_validator.rb:3: warning: already initialized constant KEY_NAME_MIN_LENGTH
/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.11.3/lib/stickshift-controller/app/models/validators/namespace_validator.rb:2: warning: already initialized constant NAMESPACE_MAX_LENGTH
/usr/lib/ruby/gems/1.8/gems/stickshift-controller-0.11.3/lib/stickshift-controller/app/models/validators/namespace_validator.rb:3: warning: already initialized constant NAMESPACE_MIN_LENGTH
URL: http://myapp2-jialiu.dev.rhcloud.com
Login: jialiu
App UUID: 875500b7aa754887b804c52d688ceb5e
Gear UUID: 875500b7aa754887b804c52d688ceb5e
DEBUG: Source district uuid: 4956ebcde88d4123b4a6ba566dbbde06
DEBUG: Destination district uuid: 4956ebcde88d4123b4a6ba566dbbde06
DEBUG: District unchanged keeping uid
DEBUG: Getting existing app 'myapp2' status before moving
DEBUG: App 'myapp2' was running
DEBUG: Getting existing app 'myapp2' status before moving
DEBUG: App 'myapp2' was running
DEBUG: Getting existing app 'myapp2' status before moving
DEBUG: App 'myapp2' was running
DEBUG: Stopping existing app cartridge 'jenkins-client-1.4' before moving
DEBUG: Performing cartridge level pre-move for embedded jenkins-client-1.4 for 'myapp2' on ip-10-98-86-232
DEBUG: Stopping existing app cartridge 'haproxy-1.4' before moving
DEBUG: Stopping existing app cartridge 'perl-5.10' before moving
DEBUG: Creating new account for gear 'myapp2' on ip-10-85-3-183
DEBUG: Moving content for app 'myapp2', gear 'myapp2' to ip-10-85-3-183
Identity added: /var/www/stickshift/broker/config/keys/rsync_id_rsa (/var/www/stickshift/broker/config/keys/rsync_id_rsa)
Warning: Permanently added '10.85.3.183' (RSA) to the list of known hosts.
Agent pid 2462
DEBUG: Performing cartridge level move for 'perl-5.10' on ip-10-85-3-183
DEBUG: Performing cartridge level move for 'haproxy-1.4' on ip-10-85-3-183
DEBUG: Performing cartridge level move for embedded jenkins-client-1.4 for 'myapp2' on ip-10-85-3-183
DEBUG: Moving failed.  Rolling back gear 'myapp2' 'myapp2' with remove-httpd-proxy on 'ip-10-85-3-183'
DEBUG: Moving failed.  Rolling back gear 'myapp2' in 'myapp2' with destroy on 'ip-10-85-3-183'
/usr/lib/ruby/gems/1.8/gems/gearchanger-m-collective-plugin-0.10.2/lib/gearchanger-m-collective-plugin/gearchanger/mcollective_application_container_proxy.rb:1627:in `run_cartridge_command': Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support. (StickShift::NodeException)
	from /usr/lib/ruby/gems/1.8/gems/gearchanger-m-collective-plugin-0.10.2/lib/gearchanger-m-collective-plugin/gearchanger/mcollective_application_container_proxy.rb:366:in `expose_port'
	from /usr/lib/ruby/gems/1.8/gems/gearchanger-m-collective-plugin-0.10.2/lib/gearchanger-m-collective-plugin/gearchanger/mcollective_application_container_proxy.rb:613:in `move_gear'
	from /usr/lib/ruby/gems/1.8/gems/gearchanger-m-collective-plugin-0.10.2/lib/gearchanger-m-collective-plugin/gearchanger/mcollective_application_container_proxy.rb:585:in `each'
	from /usr/lib/ruby/gems/1.8/gems/gearchanger-m-collective-plugin-0.10.2/lib/gearchanger-m-collective-plugin/gearchanger/mcollective_application_container_proxy.rb:585:in `move_gear'
	from /usr/bin/rhc-admin-move:129


Expected results:
Should move haproxy gear with jenkins client embeded

Additional info:

Comment 1 Johnny Liu 2012-05-23 11:22:15 UTC
Created attachment 586309 [details]
broker log

Comment 2 Johnny Liu 2012-05-23 11:22:54 UTC
Created attachment 586314 [details]
taget node mcollective log

Comment 3 Rajat Chopra 2012-05-25 18:13:49 UTC
Fixed with rev#bbb24229ff6962a89f9dbb0566d91a05b1d19afd in li repo

Comment 4 Johnny Liu 2012-05-28 07:25:59 UTC
Retest this bug with devenv_1806, and failed.


Broker log:

DEBUG: Performing cartridge level move for embedded jenkins-client-1.4 for 'pytest' on ip-10-99-38-17
DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fba312f65f0>
DEBUG: rpc_client.custom_request('cartridge_do', {:action=>"move", :cartridge=>"embedded/jenkins-client-1.4", :args=>"'pytest' 'jialiu' '2967a176b67140ecbad7bd9b328010de'"}, @id, {'identity' => @id})
DEBUG: [#<MCollective::RPC::Result:0x7fba3129bdd0 @results={:data=>{:output=>"", :exitcode=>0}, :statusmsg=>"OK", :statuscode=>0, :sender=>"ip-10-99-38-17"}, @action="cartridge_do", @agent="libra">]
DEBUG: Cartridge command embedded/jenkins-client-1.4::move exitcode = 0
DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fba31299378>
DEBUG: rpc_client.custom_request('cartridge_do', {:action=>"expose-port", :cartridge=>"jenkins-client-1.4", :args=>"'pytest' 'jialiu' '2967a176b67140ecbad7bd9b328010de'"}, @id, {'identity' => @id})
DEBUG: [#<MCollective::RPC::Result:0x7fba31232f38 @results={:data=>{:exitcode=>127}, :statusmsg=>"cartridge_do_action ERROR action 'expose-port' not found.", :statuscode=>1, :sender=>"ip-10-99-38-17"}, @action="cartridge_do", @agent="libra">]
DEBUG: Cartridge command jenkins-client-1.4::expose-port exitcode = 127
DEBUG: Performing cartridge level move for 'haproxy-1.4' on ip-10-99-38-17
DEBUG: rpc_exec_direct: rpc_client=#<MCollective::RPC::Client:0x7fba3122ef78>
DEBUG: rpc_client.custom_request('cartridge_do', {:action=>"move", :cartridge=>"haproxy-1.4", :args=>"'pytest' 'jialiu' '2967a176b67140ecbad7bd9b328010de'"}, @id, {'identity' => @id})
DEBUG: [#<MCollective::RPC::Result:0x7fba311a5818 @results={:data=>{:output=>"pytest. Cartridge type '' not set.\nchcon: missing operand\nTry `chcon --help' for more information.\n", :exitcode=>1}, :statusmsg=>"cartridge_do_action failed 1.  Output pytest. Cartridge type '' not set.\nchcon: missing operand\nTry `chcon --help' for more information.\n", :statuscode=>1, :sender=>"ip-10-99-38-17"}, @action="cartridge_do", @agent="libra">]
DEBUG: server results: pytest. Cartridge type '' not set.
DEBUG: server results: chcon: missing operand
DEBUG: server results: Try `chcon --help' for more information.
DEBUG: Cartridge command haproxy-1.4::move exitcode = 1


Looks like "expose-port" action for jenkins-client-1.4 does not break move operation now, but move action for haproxy-1.4 failed, this issue is introduced by the fix for #824423, so keep this bug in "ON_QA" status, once 824423 is fixed, will re-test this bug.

Comment 5 Rony Gong 🔥 2012-05-30 06:33:56 UTC
verified on devenv_1808