Bug 1012377 - Fail to move a HA gear
Summary: Fail to move a HA gear
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Pod
Version: 2.x
Hardware: All
OS: All
high
high
Target Milestone: ---
: ---
Assignee: Rajat Chopra
QA Contact: libra bugs
URL:
Whiteboard:
: 1012403 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-09-26 11:22 UTC by zhaozhanqi
Modified: 2015-05-15 00:21 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-10-17 13:31:54 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description zhaozhanqi 2013-09-26 11:22:56 UTC
Description of problem:
Given a scalable app enable HA and scale-up, then move the scale-up gear with HA, will be failed.

Version-Release number of selected component (if applicable):
devenv_3829

How reproducible:
always

Steps to Reproduce:
1. ALLOW_HA_APPLICATIONS="true" and  oo-admin-ctl-user --allowha true
2. create one scalable app and enable HA
3. scale-up app
    rhc cartridge scale -a app -c php-5.3 --min 3
4. rhc app show app -g
5. move the scale-up gear with HA

Actual results:

step 4:
[zqzhao@dhcp-13-222 log]$ rhc ssh  php3s --gear ls
=== 9def4750269811e3b83f12313b09194e php-5.3+haproxy-1.4
app-root
git
php
=== 52440f714977d10e2d00001f php-5.3+haproxy-1.4
app-root
git
haproxy
php
=== 52440ee84977d163550001ce php-5.3+haproxy-1.4
app-root
git
haproxy
php

step 5:

[root@ip-10-203-26-188 broker]# oo-admin-move --gear_uuid 52440f714977d10e2d00001f -i ip-10-203-26-188
URL: http://php3s-zqd.dev.rhcloud.com
Login: zzhao
App UUID: 52440ee84977d163550001ce
Gear UUID: 52440f714977d16355000216
DEBUG: Source district uuid: 835207688557239203790848
DEBUG: Destination district uuid: 835207688557239203790848
DEBUG: Getting existing app 'php3s' status before moving
DEBUG: Gear component 'php-5.3' was running
DEBUG: Stopping existing app cartridge 'php-5.3' before moving
DEBUG: Force stopping existing app cartridge 'php-5.3' before moving
DEBUG: Stopping existing app cartridge 'haproxy-1.4' before moving
DEBUG: Creating new account for gear '52440f714977d10e2d00001f' on ip-10-203-26-188
DEBUG: Moving content for app 'php3s', gear '52440f714977d10e2d00001f' to ip-10-203-26-188
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 16404
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 16404 killed;
DEBUG: Moving system components for app 'php3s', gear '52440f714977d10e2d00001f' to ip-10-203-26-188
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 16476
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 16476 killed;
DEBUG: Starting cartridge 'php-5.3' in 'php3s' after move on ip-10-203-26-188
DEBUG: Starting cartridge 'haproxy-1.4' in 'php3s' after move on ip-10-203-26-188
DEBUG: Fixing DNS and mongo for gear '52440f714977d10e2d00001f' after move
DEBUG: Changing server identity of '52440f714977d10e2d00001f' from 'ip-10-114-25-51' to 'ip-10-203-26-188'
DEBUG: Moving failed.  Rolling back gear '52440f714977d10e2d00001f' in 'php3s' with delete on 'ip-10-203-26-188'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.15.3/lib/openshift/mcollective_application_container_proxy.rb:1702:in `ensure in move_gear_post': undefined method `close' for nil:NilClass (NoMethodError)
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.15.3/lib/openshift/mcollective_application_container_proxy.rb:1702:in `move_gear_post'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.15.3/lib/openshift/mcollective_application_container_proxy.rb:1857:in `move_gear'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.15.3/lib/openshift/mcollective_application_container_proxy.rb:1784:in `block in move_gear_secure'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.15.3/app/models/application.rb:1506:in `run_in_application_lock'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.15.3/lib/openshift/mcollective_application_container_proxy.rb:1783:in `move_gear_secure'
	from /usr/sbin/oo-admin-move:112:in `<main>'

Expected results:

should move successfully.

Additional info:

Comment 1 zhaozhanqi 2013-09-26 12:17:30 UTC
*** Bug 1012403 has been marked as a duplicate of this bug. ***

Comment 2 Rajat Chopra 2013-09-27 20:36:57 UTC
This seems to have failed because of DNS issues. The move is expected to fail then. Could you repeat the test and verify its not because of temporary DNS failure?

Comment 3 zhaozhanqi 2013-09-29 09:35:50 UTC
Tested this issue on devenv_3844, it has been fixed.


[root@ip-10-204-113-68 node]# oo-admin-move --gear_uuid 5247f1f42d19cf900200057d -i ip-10-100-211-42
URL: http://php4s-zqd.dev.rhcloud.com
Login: zzhao
App UUID: 5247f1f42d19cf900200057d
Gear UUID: 5247f1f42d19cf900200057d
DEBUG: Source district uuid: 37441bf628b011e392f112313b0a72ba
DEBUG: Destination district uuid: 37441bf628b011e392f112313b0a72ba
DEBUG: Getting existing app 'php4s' status before moving
DEBUG: Gear component 'php-5.3' was running
DEBUG: Stopping existing app cartridge 'php-5.3' before moving
DEBUG: Force stopping existing app cartridge 'php-5.3' before moving
DEBUG: Stopping existing app cartridge 'haproxy-1.4' before moving
DEBUG: Creating new account for gear 'php4s' on ip-10-100-211-42
DEBUG: Moving content for app 'php4s', gear 'php4s' to ip-10-100-211-42
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Warning: Permanently added '10.195.219.47' (RSA) to the list of known hosts.
Warning: Permanently added '10.100.211.42' (RSA) to the list of known hosts.
Agent pid 7696
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 7696 killed;
DEBUG: Moving system components for app 'php4s', gear 'php4s' to ip-10-100-211-42
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 7732
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 7732 killed;
DEBUG: Starting cartridge 'php-5.3' in 'php4s' after move on ip-10-100-211-42
DEBUG: Starting cartridge 'haproxy-1.4' in 'php4s' after move on ip-10-100-211-42
DEBUG: Fixing DNS and mongo for gear 'php4s' after move
DEBUG: Changing server identity of 'php4s' from 'ip-10-195-219-47' to 'ip-10-100-211-42'
DEBUG: Deconfiguring old app 'php4s' on ip-10-195-219-47 after move
Successfully moved gear with uuid '5247f1f42d19cf900200057d' of app 'php4s' from 'ip-10-195-219-47' to 'ip-10-100-211-42'


Note You need to log in before you can comment on or make changes to this bug.