874445 – Can't move idled scalable app

Bug 874445 - Can't move idled scalable app

Summary: Can't move idled scalable app

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	OKD
Classification:	Red Hat
Component:	Containers
Sub Component:
Version:	2.x
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	medium
Target Milestone:	---
Target Release:	---
Assignee:	Jhon Honce
QA Contact:	libra bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2012-11-08 08:31 UTC by Rony Gong 🔥
Modified:	2015-05-14 23:01 UTC (History)
CC List:	0 users
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2012-12-19 19:27:10 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Bugzilla	876103	0	high	CLOSED	Haproxy process is dead for scalable jboss apps after server upgrade and migrate	2021-02-22 00:41:40 UTC

Internal Links: 876103

Description Rony Gong 🔥 2012-11-08 08:31:58 UTC

Description of problem:
Move the haproxy gear uuid that has been idle status, but failed.
Same error like:
https://bugzilla.redhat.com/show_bug.cgi?id=874311

Version-Release number of selected component (if applicable):
devenv-stag_246

How reproducible:
always

Steps to Reproduce:
Setup multi-node env
1.Create scalable app like php
2.Idle the haproxy gear uuid by
oo-idler -u 96c460aaac0f43878ee5adb2faf3cf98
3.Move this idle gear

  
Actual results:
root@ip-10-120-206-173 openshift]# oo-admin-move --gear_uuid 96c460aaac0f43878ee5adb2faf3cf98 -i ip-10-120-206-173
URL: http://qsphp-qgong9.dev.rhcloud.com
Login: qgong
App UUID: 96c460aaac0f43878ee5adb2faf3cf98
Gear UUID: 96c460aaac0f43878ee5adb2faf3cf98
DEBUG: Source district uuid: 065cfa08e44c48729c67f97fd99db49b
DEBUG: Destination district uuid: 065cfa08e44c48729c67f97fd99db49b
DEBUG: District unchanged keeping uid
DEBUG: Getting existing app 'qsphp' status before moving
DEBUG: Gear component 'php-5.3' was idle
DEBUG: Creating new account for gear 'qsphp' on ip-10-120-206-173
DEBUG: Moving content for app 'qsphp', gear 'qsphp' to ip-10-120-206-173
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 4449
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 4449 killed;
DEBUG: Performing cartridge level move for 'php-5.3' on ip-10-120-206-173
DEBUG: Performing cartridge level move for 'haproxy-1.4' on ip-10-120-206-173
DEBUG: Moving failed.  Rolling back gear 'qsphp' 'qsphp' with remove-httpd-proxy on 'ip-10-120-206-173'
DEBUG: Moving failed.  Rolling back gear 'qsphp' in 'qsphp' with destroy on 'ip-10-120-206-173'
/usr/lib/ruby/gems/1.8/gems/openshift-origin-msg-broker-mcollective-1.0.1/lib/openshift-origin-msg-broker-mcollective/lib/openshift/mcollective_application_container_proxy.rb:1265:in `run_cartridge_command_old': Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support. (OpenShift::NodeException)
	from /var/www/openshift/broker/lib/express/broker/mcollective_ext.rb:12:in `run_cartridge_command'
	from /usr/lib/ruby/gems/1.8/gems/openshift-origin-msg-broker-mcollective-1.0.1/lib/openshift-origin-msg-broker-mcollective/lib/openshift/mcollective_application_container_proxy.rb:785:in `send'
	from /usr/lib/ruby/gems/1.8/gems/openshift-origin-msg-broker-mcollective-1.0.1/lib/openshift-origin-msg-broker-mcollective/lib/openshift/mcollective_application_container_proxy.rb:785:in `move_gear'
	from /usr/lib/ruby/gems/1.8/gems/openshift-origin-msg-broker-mcollective-1.0.1/lib/openshift-origin-msg-broker-mcollective/lib/openshift/mcollective_application_container_proxy.rb:776:in `each'
	from /usr/lib/ruby/gems/1.8/gems/openshift-origin-msg-broker-mcollective-1.0.1/lib/openshift-origin-msg-broker-mcollective/lib/openshift/mcollective_application_container_proxy.rb:776:in `move_gear'
	from /usr/sbin/oo-admin-move:111


Expected results:
Move successfully.


Additional info:
I Suggest check all "rhc-idler" in below list and upate to "oo-idler"
[root@ip-10-120-199-69 hooks]# grep -R "rhc-id" /usr/libexec/openshift/cartridges
/usr/libexec/openshift/cartridges/embedded/mysql-5.1/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/embedded/haproxy-1.4/info/hooks/move:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/nodejs-0.6/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/nodejs-0.6/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/mysql-5.1/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/zend-5.6/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
grep: /usr/libexec/openshift/cartridges/zend-5.6/files/shared-files/usr/local/zend/bin/apachectl: No such file or directory
/usr/libexec/openshift/cartridges/ruby-1.9/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/ruby-1.9/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/abstract/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/abstract/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/jbossas-7/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/jbossas-7/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/haproxy-1.4/info/hooks/move:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/python-2.6/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/python-2.6/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/jenkins-1.4/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/jenkins-1.4/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/ruby-1.8/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/ruby-1.8/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/jbosseap-6.0/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/jbosseap-6.0/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/diy-0.1/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/diy-0.1/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/jbossews-1.0/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/jbossews-1.0/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/php-5.3/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/php-5.3/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
/usr/libexec/openshift/cartridges/perl-5.10/info/hooks/deploy-httpd-proxy:  rhc-idler -u $uuid
/usr/libexec/openshift/cartridges/perl-5.10/info/hooks/move:  nohup rhc-idler -u $uuid &> /dev/null &
[root@ip-10-120-199-69 hooks]# ls -al /usr/libexec/openshift/cartridges/jenkins-1.4/info/hooks/move

Comment 1 Jhon Honce 2012-11-09 16:02:21 UTC

Fix on master https://github.com/openshift/origin-server/pull/862

Comment 2 Jhon Honce 2012-11-09 23:15:22 UTC

Fixed on master https://github.com/openshift/origin-server/pull/865

Comment 3 Rony Gong 🔥 2012-11-12 05:34:12 UTC

verified on devenv_2453
[root@ip-10-122-130-194 openshift]# oo-admin-move --gear_uuid b16c2ecfa68b4dc8ae1eafaf059e1e0b -i ip-10-122-130-194
URL: http://qsphp-qgong15.dev.rhcloud.com
Login: qgong
App UUID: b16c2ecfa68b4dc8ae1eafaf059e1e0b
Gear UUID: b16c2ecfa68b4dc8ae1eafaf059e1e0b
DEBUG: Source district uuid: ca5713f5574a4eb3a2bd4d9a7af1db31
DEBUG: Destination district uuid: ca5713f5574a4eb3a2bd4d9a7af1db31
DEBUG: District unchanged keeping uid
DEBUG: Getting existing app 'qsphp' status before moving
DEBUG: Gear component 'php-5.3' was idle
DEBUG: Creating new account for gear 'qsphp' on ip-10-122-130-194
DEBUG: Moving content for app 'qsphp', gear 'qsphp' to ip-10-122-130-194
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Warning: Permanently added '10.122.130.194' (RSA) to the list of known hosts.
Agent pid 8225
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 8225 killed;
DEBUG: Performing cartridge level move for 'php-5.3' on ip-10-122-130-194
DEBUG: Performing cartridge level move for 'haproxy-1.4' on ip-10-122-130-194
DEBUG: Fixing DNS and mongo for gear 'qsphp' after move
DEBUG: Changing server identity of 'qsphp' from 'ip-10-127-158-168' to 'ip-10-122-130-194'
DEBUG: Explicitly stopping cartridge 'haproxy-1.4' in 'qsphp' after move on ip-10-122-130-194
DEBUG: Deconfiguring old app 'qsphp' on ip-10-127-158-168 after move
Successfully moved 'qsphp' with gear uuid 'b16c2ecfa68b4dc8ae1eafaf059e1e0b' from 'ip-10-127-158-168' to 'ip-10-122-130-194'

[root@ip-10-122-130-194 openshift]# oo-idler -l
b16c2ecfa68b4dc8ae1eafaf059e1e0b is idled

Note You need to log in before you can comment on or make changes to this bug.