Bug 874445

Summary: Can't move idled scalable app
Product: OKD Reporter: Rony Gong 🔥 <qgong>
Component: ContainersAssignee: Jhon Honce <jhonce>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 2.x   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-12-19 19:27:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Rony Gong 🔥 2012-11-08 08:31:58 UTC 
Description of problem:
Move the haproxy gear uuid that has been idle status, but failed.
Same error like:
https://bugzilla.redhat.com/show_bug.cgi?id=874311

Version-Release number of selected component (if applicable):
devenv-stag_246

How reproducible:
always

Steps to Reproduce:
Setup multi-node env
1.Create scalable app like php
2.Idle the haproxy gear uuid by
oo-idler -u 96c460aaac0f43878ee5adb2faf3cf98
3.Move this idle gear

  
Actual results:
root@ip-10-120-206-173 openshift]# oo-admin-move --gear_uuid 96c460aaac0f43878ee5adb2faf3cf98 -i ip-10-120-206-173
URL: http://qsphp-qgong9.dev.rhcloud.com
Login: qgong
App UUID: 96c460aaac0f43878ee5adb2faf3cf98
Gear UUID: 96c460aaac0f43878ee5adb2faf3cf98
DEBUG: Source district uuid: 065cfa08e44c48729c67f97fd99db49b
DEBUG: Destination district uuid: 065cfa08e44c48729c67f97fd99db49b
DEBUG: District unchanged keeping uid
DEBUG: Getting existing app 'qsphp' status before moving
DEBUG: Gear component 'php-5.3' was idle
DEBUG: Creating new account for gear 'qsphp' on ip-10-120-206-173
DEBUG: Moving content for app 'qsphp', gear 'qsphp' to ip-10-120-206-173
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 4449
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 4449 killed;
DEBUG: Performing cartridge level move for 'php-5.3' on ip-10-120-206-173
DEBUG: Performing cartridge level move for 'haproxy-1.4' on ip-10-120-206-173
DEBUG: Moving failed.  Rolling back gear 'qsphp' 'qsphp' with remove-httpd-proxy on 'ip-10-120-206-173'
DEBUG: Moving failed.  Rolling back gear 'qsphp' in 'qsphp' with destroy on 'ip-10-120-206-173'
/usr/lib/ruby/gems/1.8/gems/openshift-origin-msg-broker-mcollective-1.0.1/lib/openshift-origin-msg-broker-mcollective/lib/openshift/mcollective_application_container_proxy.rb:1265:in `run_cartridge_command_old': Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support. (OpenShift::NodeException)
	from /var/www/openshift/broker/lib/express/broker/mcollective_ext.rb:12:in `run_cartridge_command'
	from /usr/lib/ruby/gems/1.8/gems/openshift-origin-msg-broker-mcollective-1.0.1/lib/openshift-origin-msg-broker-mcollective/lib/openshift/mcollective_application_container_proxy.rb:785:in `send'
	from /usr/lib/ruby/gems/1.8/gems/openshift-origin-msg-broker-mcollective-1.0.1/lib/openshift-origin-msg-broker-mcollective/lib/openshift/mcollective_application_container_proxy.rb:785:in `move_gear'
	from /usr/lib/ruby/gems/1.8/gems/openshift-origin-msg-broker-mcollective-1.0.1/lib/openshift-origin-msg-broker-mcollective/lib/openshift/mcollective_application_container_proxy.rb:776:in `each'
	from /usr/lib/ruby/gems/1.8/gems/openshift-origin-msg-broker-mcollective-1.0.1/lib/openshift-origin-msg-broker-mcollective/lib/openshift/mcollective_application_container_proxy.rb:776:in `move_gear'
	from /usr/sbin/oo-admin-move:111


Expected results:
Move successfully.


Additional info:
I Suggest check all "rhc-idler" in below list and upate to "oo-idler"
[root@ip-10-120-199-69 hooks]# grep -R "rhc-id" /usr/libexec/openshift/cartridges
/usr/libexec/openshift/cartridges/embedded/mysql-5.1/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/embedded/haproxy-1.4/info/hooks/move:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/nodejs-0.6/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/nodejs-0.6/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/mysql-5.1/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/zend-5.6/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
grep: /usr/libexec/openshift/cartridges/zend-5.6/files/shared-files/usr/local/zend/bin/apachectl: No such file or directory
/usr/libexec/openshift/cartridges/ruby-1.9/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/ruby-1.9/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/abstract/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/abstract/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/jbossas-7/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/jbossas-7/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/haproxy-1.4/info/hooks/move:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/python-2.6/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/python-2.6/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/jenkins-1.4/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/jenkins-1.4/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/ruby-1.8/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/ruby-1.8/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/jbosseap-6.0/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/jbosseap-6.0/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/diy-0.1/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/diy-0.1/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/jbossews-1.0/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/jbossews-1.0/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/php-5.3/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/php-5.3/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/perl-5.10/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/perl-5.10/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
[root@ip-10-120-199-69 hooks]# ls -al /usr/libexec/openshift/cartridges/jenkins-1.4/info/hooks/move
Comment 1 Jhon Honce 2012-11-09 16:02:21 UTC 
Fix on master https://github.com/openshift/origin-server/pull/862
Comment 2 Jhon Honce 2012-11-09 23:15:22 UTC 
Fixed on master https://github.com/openshift/origin-server/pull/865
Comment 3 Rony Gong 🔥 2012-11-12 05:34:12 UTC 
verified on devenv_2453
[root@ip-10-122-130-194 openshift]# oo-admin-move --gear_uuid b16c2ecfa68b4dc8ae1eafaf059e1e0b -i ip-10-122-130-194
URL: http://qsphp-qgong15.dev.rhcloud.com
Login: qgong
App UUID: b16c2ecfa68b4dc8ae1eafaf059e1e0b
Gear UUID: b16c2ecfa68b4dc8ae1eafaf059e1e0b
DEBUG: Source district uuid: ca5713f5574a4eb3a2bd4d9a7af1db31
DEBUG: Destination district uuid: ca5713f5574a4eb3a2bd4d9a7af1db31
DEBUG: District unchanged keeping uid
DEBUG: Getting existing app 'qsphp' status before moving
DEBUG: Gear component 'php-5.3' was idle
DEBUG: Creating new account for gear 'qsphp' on ip-10-122-130-194
DEBUG: Moving content for app 'qsphp', gear 'qsphp' to ip-10-122-130-194
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Warning: Permanently added '10.122.130.194' (RSA) to the list of known hosts.
Agent pid 8225
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 8225 killed;
DEBUG: Performing cartridge level move for 'php-5.3' on ip-10-122-130-194
DEBUG: Performing cartridge level move for 'haproxy-1.4' on ip-10-122-130-194
DEBUG: Fixing DNS and mongo for gear 'qsphp' after move
DEBUG: Changing server identity of 'qsphp' from 'ip-10-127-158-168' to 'ip-10-122-130-194'
DEBUG: Explicitly stopping cartridge 'haproxy-1.4' in 'qsphp' after move on ip-10-122-130-194
DEBUG: Deconfiguring old app 'qsphp' on ip-10-127-158-168 after move
Successfully moved 'qsphp' with gear uuid 'b16c2ecfa68b4dc8ae1eafaf059e1e0b' from 'ip-10-127-158-168' to 'ip-10-122-130-194'

[root@ip-10-122-130-194 openshift]# oo-idler -l
b16c2ecfa68b4dc8ae1eafaf059e1e0b is idled