Bug 1012377 - Fail to move a HA gear
Fail to move a HA gear
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Pod (Show other bugs)
2.x
All All
high Severity high
: ---
: ---
Assigned To: Rajat Chopra
libra bugs
:
: 1012403 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-26 07:22 EDT by zhaozhanqi
Modified: 2015-05-14 20:21 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-17 09:31:54 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description zhaozhanqi 2013-09-26 07:22:56 EDT
Description of problem:
Given a scalable app enable HA and scale-up, then move the scale-up gear with HA, will be failed.

Version-Release number of selected component (if applicable):
devenv_3829

How reproducible:
always

Steps to Reproduce:
1. ALLOW_HA_APPLICATIONS="true" and  oo-admin-ctl-user --allowha true
2. create one scalable app and enable HA
3. scale-up app
    rhc cartridge scale -a app -c php-5.3 --min 3
4. rhc app show app -g
5. move the scale-up gear with HA

Actual results:

step 4:
[zqzhao@dhcp-13-222 log]$ rhc ssh  php3s --gear ls
=== 9def4750269811e3b83f12313b09194e php-5.3+haproxy-1.4
app-root
git
php
=== 52440f714977d10e2d00001f php-5.3+haproxy-1.4
app-root
git
haproxy
php
=== 52440ee84977d163550001ce php-5.3+haproxy-1.4
app-root
git
haproxy
php

step 5:

[root@ip-10-203-26-188 broker]# oo-admin-move --gear_uuid 52440f714977d10e2d00001f -i ip-10-203-26-188
URL: http://php3s-zqd.dev.rhcloud.com
Login: zzhao@redhat.com
App UUID: 52440ee84977d163550001ce
Gear UUID: 52440f714977d16355000216
DEBUG: Source district uuid: 835207688557239203790848
DEBUG: Destination district uuid: 835207688557239203790848
DEBUG: Getting existing app 'php3s' status before moving
DEBUG: Gear component 'php-5.3' was running
DEBUG: Stopping existing app cartridge 'php-5.3' before moving
DEBUG: Force stopping existing app cartridge 'php-5.3' before moving
DEBUG: Stopping existing app cartridge 'haproxy-1.4' before moving
DEBUG: Creating new account for gear '52440f714977d10e2d00001f' on ip-10-203-26-188
DEBUG: Moving content for app 'php3s', gear '52440f714977d10e2d00001f' to ip-10-203-26-188
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 16404
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 16404 killed;
DEBUG: Moving system components for app 'php3s', gear '52440f714977d10e2d00001f' to ip-10-203-26-188
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 16476
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 16476 killed;
DEBUG: Starting cartridge 'php-5.3' in 'php3s' after move on ip-10-203-26-188
DEBUG: Starting cartridge 'haproxy-1.4' in 'php3s' after move on ip-10-203-26-188
DEBUG: Fixing DNS and mongo for gear '52440f714977d10e2d00001f' after move
DEBUG: Changing server identity of '52440f714977d10e2d00001f' from 'ip-10-114-25-51' to 'ip-10-203-26-188'
DEBUG: Moving failed.  Rolling back gear '52440f714977d10e2d00001f' in 'php3s' with delete on 'ip-10-203-26-188'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.15.3/lib/openshift/mcollective_application_container_proxy.rb:1702:in `ensure in move_gear_post': undefined method `close' for nil:NilClass (NoMethodError)
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.15.3/lib/openshift/mcollective_application_container_proxy.rb:1702:in `move_gear_post'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.15.3/lib/openshift/mcollective_application_container_proxy.rb:1857:in `move_gear'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.15.3/lib/openshift/mcollective_application_container_proxy.rb:1784:in `block in move_gear_secure'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.15.3/app/models/application.rb:1506:in `run_in_application_lock'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.15.3/lib/openshift/mcollective_application_container_proxy.rb:1783:in `move_gear_secure'
	from /usr/sbin/oo-admin-move:112:in `<main>'

Expected results:

should move successfully.

Additional info:
Comment 1 zhaozhanqi 2013-09-26 08:17:30 EDT
*** Bug 1012403 has been marked as a duplicate of this bug. ***
Comment 2 Rajat Chopra 2013-09-27 16:36:57 EDT
This seems to have failed because of DNS issues. The move is expected to fail then. Could you repeat the test and verify its not because of temporary DNS failure?
Comment 3 zhaozhanqi 2013-09-29 05:35:50 EDT
Tested this issue on devenv_3844, it has been fixed.


[root@ip-10-204-113-68 node]# oo-admin-move --gear_uuid 5247f1f42d19cf900200057d -i ip-10-100-211-42
URL: http://php4s-zqd.dev.rhcloud.com
Login: zzhao@redhat.com
App UUID: 5247f1f42d19cf900200057d
Gear UUID: 5247f1f42d19cf900200057d
DEBUG: Source district uuid: 37441bf628b011e392f112313b0a72ba
DEBUG: Destination district uuid: 37441bf628b011e392f112313b0a72ba
DEBUG: Getting existing app 'php4s' status before moving
DEBUG: Gear component 'php-5.3' was running
DEBUG: Stopping existing app cartridge 'php-5.3' before moving
DEBUG: Force stopping existing app cartridge 'php-5.3' before moving
DEBUG: Stopping existing app cartridge 'haproxy-1.4' before moving
DEBUG: Creating new account for gear 'php4s' on ip-10-100-211-42
DEBUG: Moving content for app 'php4s', gear 'php4s' to ip-10-100-211-42
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Warning: Permanently added '10.195.219.47' (RSA) to the list of known hosts.
Warning: Permanently added '10.100.211.42' (RSA) to the list of known hosts.
Agent pid 7696
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 7696 killed;
DEBUG: Moving system components for app 'php4s', gear 'php4s' to ip-10-100-211-42
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 7732
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 7732 killed;
DEBUG: Starting cartridge 'php-5.3' in 'php4s' after move on ip-10-100-211-42
DEBUG: Starting cartridge 'haproxy-1.4' in 'php4s' after move on ip-10-100-211-42
DEBUG: Fixing DNS and mongo for gear 'php4s' after move
DEBUG: Changing server identity of 'php4s' from 'ip-10-195-219-47' to 'ip-10-100-211-42'
DEBUG: Deconfiguring old app 'php4s' on ip-10-195-219-47 after move
Successfully moved gear with uuid '5247f1f42d19cf900200057d' of app 'php4s' from 'ip-10-195-219-47' to 'ip-10-100-211-42'

Note You need to log in before you can comment on or make changes to this bug.