Bug 863998

Summary: Can't move the app-gear for scalable applications
Product: OKD Reporter: Rony Gong 🔥 <qgong>
Component: ContainersAssignee: Mrunal Patel <mpatel>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: jhou, xtian
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-11-06 18:50:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
development.log none

Description Rony Gong 🔥 2012-10-08 09:55:47 UTC 
Created attachment 623369 [details]
development.log

Description of problem:
Can't move the app-gear within district for scalable applications, could move other gear like haproxy-gear, mysql-5.1-gear, mongodb-gear, postgresql-gear


Version-Release number of selected component (if applicable):
devenv_2296

How reproducible:
always


Steps to Reproduce:
1.Create scalable app like jbosseap
2.Get the app gear uuid by rest api
curl -k -X GET -H 'Accept: application/xml' --user qgong:111111 https://ec2-23-22-228-78.compute-1.amazonaws.com/broker/rest/domains/qgong14/applications/qsjbosseap/gears
    <gear>
      <components>
        <component>
          <proxy-host nil="true"></proxy-host>
          <internal-port nil="true"></internal-port>
          <name>jbosseap-6.0</name>
          <proxy-port nil="true"></proxy-port>
        </component>
      </components>
      <uuid>dc62e1deeb1942dbb79a19d4c8c62a85</uuid>
    </gear>
    <gear>
3.Move the app-gear by the uuid within district
oo-admin-move  --gear_uuid b098ee02eb894085ad6c364f70fa6855 -i ip-10-120-203-251

  
Actual results:
[root@ip-10-120-203-251 openshift]# oo-admin-move  --gear_uuid b098ee02eb894085ad6c364f70fa6855 -i ip-10-120-203-251
URL: http://qsjbossas-qgong14.dev.rhcloud.com
Login: qgong
App UUID: 68fad773938d43c6bb73f5775f653639
Gear UUID: b098ee02eb894085ad6c364f70fa6855
DEBUG: Source district uuid: e0d75a19a52040349033acb53b45efef
DEBUG: Destination district uuid: e0d75a19a52040349033acb53b45efef
DEBUG: District unchanged keeping uid
DEBUG: Getting existing app 'qsjbossas' status before moving
DEBUG: Gear component 'jbossas-7' was running
DEBUG: Stopping existing app cartridge 'jbossas-7' before moving
DEBUG: Force stopping existing app cartridge 'jbossas-7' before moving
DEBUG: Creating new account for gear 'b098ee02eb' on ip-10-120-203-251
DEBUG: Moving content for app 'qsjbossas', gear 'b098ee02eb' to ip-10-120-203-251
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 2333
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 2333 killed;
DEBUG: Performing cartridge level move for 'jbossas-7' on ip-10-120-203-251
DEBUG: Moving failed.  Rolling back gear 'b098ee02eb' 'qsjbossas' with remove-httpd-proxy on 'ip-10-120-203-251'
DEBUG: Moving failed.  Rolling back gear 'b098ee02eb' in 'qsjbossas' with destroy on 'ip-10-120-203-251'
/usr/lib/ruby/gems/1.8/gems/openshift-origin-msg-broker-mcollective-0.4.3/lib/openshift-origin-msg-broker-mcollective/openshift/mcollective_application_container_proxy.rb:1324:in `run_cartridge_command_old': Node execution failure (invalid exit code from node).  If the problem persists please contact Red Hat support. (OpenShift::NodeException)
	from /var/www/openshift/broker/lib/express/broker/mcollective_ext.rb:11:in `run_cartridge_command'
	from /usr/lib/ruby/gems/1.8/gems/openshift-origin-msg-broker-mcollective-0.4.3/lib/openshift-origin-msg-broker-mcollective/openshift/mcollective_application_container_proxy.rb:795:in `send'
	from /usr/lib/ruby/gems/1.8/gems/openshift-origin-msg-broker-mcollective-0.4.3/lib/openshift-origin-msg-broker-mcollective/openshift/mcollective_application_container_proxy.rb:795:in `move_gear'
	from /usr/lib/ruby/gems/1.8/gems/openshift-origin-msg-broker-mcollective-0.4.3/lib/openshift-origin-msg-broker-mcollective/openshift/mcollective_application_container_proxy.rb:769:in `each'
	from /usr/lib/ruby/gems/1.8/gems/openshift-origin-msg-broker-mcollective-0.4.3/lib/openshift-origin-msg-broker-mcollective/openshift/mcollective_application_container_proxy.rb:769:in `move_gear'
	from /usr/bin/oo-admin-move:111

Expected results:
move success.


Additional info:
This error happened for all application type. This works for non-scalable app.
Comment 1 Rony Gong 🔥 2012-10-08 10:26:53 UTC 
This kind error also happened for move across district
Comment 2 Dan McPherson 2012-10-08 14:30:54 UTC 
It seems the repo dir on the scaled app has changed from appname to gear name.  Shouldn't it still be appname?

Ex:

This is the error I got:

/usr/libexec/openshift/cartridges/abstract/info/bin/redeploy_config_dir.sh: line 11: pushd: /var/lib/openshift/27b0d30d18b044f8acae6522cf9cc165/git/danmcp990.git: No such file or directory
fatal: Not a git repository (or any of the parent directories): .git
Nothing found in .openshift/config/* to redeploy
/usr/libexec/openshift/cartridges/abstract/info/bin/redeploy_config_dir.sh: line 16: popd: directory stack empty



And this was the dir on that gear:

/var/lib/openshift/27b0d30d18b044f8acae6522cf9cc165/git/27b0d30d18.git/
Comment 3 Mrunal Patel 2012-10-08 23:50:15 UTC 
Fixed with https://github.com/openshift/origin-server/pull/623.
Comment 4 Xiaoli Tian 2012-10-10 01:30:10 UTC 
(In reply to comment #3)
> Fixed with https://github.com/openshift/origin-server/pull/623.

Above pull request is merged since devenv_2302, move it to ON_QA to verify.
Comment 5 Jianwei Hou 2012-10-10 04:32:24 UTC 
Verified on devenv_2304

Steps:
1. Setup multi-node env
2. Create a scalable app, call rest api to get app gear uuid
<gear-group>
      <uuid>ef63c94d618447559da7e108644bfb39</uuid>
      <gear-profile>small</gear-profile>
      <gears>
        <gear>
          <state>started</state>
          <id>08eaaee26f584484b0f804065778032d</id>
        </gear>
      </gears>
      <cartridges>
        <cartridge>
          <name>php-5.3</name>
        </cartridge>
      </cartridges>

3. Move this gear within district
[root@ip-10-110-222-171 openshift]# oo-admin-move --gear_uuid 08eaaee26f584484b0f804065778032d -i ip-10-110-222-171
Move was successfull
4. Setup another district, move gear across districts
[root@ip-10-110-222-171 openshift]# oo-admin-move --gear_uuid a1209f6ee1004f44b58c86503f75df04 -i ip-10-110-222-171 --allow_change_district

Result:
Both haproxy gear and cartridge gear are moved within/across district successfully.