Bug 1012779

Summary: Fail to move jenkins server/zend-5.6 app
Product: OpenShift Online Reporter: zhaozhanqi <zzhao>
Component: PodAssignee: Abhishek Gupta <abhgupta>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: xtian
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-10-17 13:32:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description zhaozhanqi 2013-09-27 06:34:52 UTC
Description of problem:
Given a jenkins server/zend-5.6 app, move this app to another node, will be failed

Version-Release number of selected component (if applicable):
devenv_3836

How reproducible:
always

Steps to Reproduce:
1.create one jenkins server app
2.move this app to another node


Actual results:
[root@ip-10-184-42-34 ~]# oo-admin-move --gear_uuid 5244f19e1cccf8c16000010a -i ip-10-167-9-172
URL: http://jenkins-zqd.dev.rhcloud.com
Login: zzhao
App UUID: 5244f19e1cccf8c16000010a
Gear UUID: 5244f19e1cccf8c16000010a
DEBUG: Source district uuid: 5244ef701cccf85cc5000001
DEBUG: Destination district uuid: 5244ef701cccf85cc5000001
DEBUG: Getting existing app 'jenkins' status before moving
DEBUG: Gear component 'jenkins-1' was running
DEBUG: Stopping existing app cartridge 'jenkins-1' before moving
DEBUG: Force stopping existing app cartridge 'jenkins-1' before moving
DEBUG: Creating new account for gear 'jenkins' on ip-10-167-9-172
DEBUG: Moving content for app 'jenkins', gear 'jenkins' to ip-10-167-9-172
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 22298
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 22298 killed;
DEBUG: Moving system components for app 'jenkins', gear 'jenkins' to ip-10-167-9-172
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 22485
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 22485 killed;
DEBUG: Moving failed.  Rolling back gear 'jenkins' in 'jenkins' with delete on 'ip-10-167-9-172'
Node execution failure (invalid exit code from node).

Expected results:
should move successfully.

Additional info:

development.log:

DEBUG: rpc_client.custom_request('cartridge_do', {:cartridge=>"openshift-origin-node", :action=>"frontend-backup", :args=>{"--with-container-uuid"=>"5244f19e1cccf8c16000010a", "--with-container-name"=>"jenkins", "--with-namespace"=>"zqd", "--cart-name"=>"openshift-origin-node"}}, ip-10-184-42-34, {'identity' => ip-10-184-42-34}) (Request ID: ) (pid:21358)
2013-09-27 02:06:41.861 [DEBUG] DEBUG: [#<MCollective::RPC::Result:0x00000006a58998 @agent="openshift", @action="cartridge_do", @results={:sender=>"ip-10-184-42-34", :statuscode=>1, :statusmsg=>"cartridge_do_action failed -1. Output undefined method `[]' for nil:NilClass", :data=>{:time=>nil, :output=>"undefined method `[]' for nil:NilClass", :exitcode=>-1}}>] (Request ID: ) (pid:21358)
2013-09-27 02:06:41.863 [DEBUG] DEBUG: MCollective Response Time (execute_direct: frontend-backup): 0.206759256s  (Request ID: ) (pid:21358)
2013-09-27 02:06:41.863 [DEBUG] DEBUG: server results: undefined method `[]' for nil:NilClass (pid:21358)
2013-09-27 02:06:41.944 [DEBUG] DEBUG: Moving failed.  Rolling back gear 'jenkins' in 'jenkins' with delete on 'ip-10-167-9-172' (pid:21358)
2013-09-27 02:06:41.947 [DEBUG] DEBUG: rpc_client.custom_request('cartridge_do', {:cartridge=>"openshift-origin-node", :action=>"app-destroy", :args=>{"--with-app-uuid"=>"5244f19e1cccf8c16000010a", "--with-app-name"=>"jenkins", "--with-container-uuid"=>"5244f19e1cccf8c16000010a", "--with-container-name"=>"jenkins", "--with-namespace"=>"zqd", "--with-uid"=>5747, "--with-request-id"=>nil, "--skip-hooks"=>true, "--cart-name"=>"openshift-origin-node"}}, ip-10-167-9-172, {'identity' => ip-10-167-9-172}) (Request ID: ) (pid:21358)
2013-09-27 02:06:44.678 [DEBUG] DEBUG: [#<MCollective::RPC::Result:0x000000069c4ab8 @agent="openshift", @action="cartridge_do", @results={:sender=>"ip-10-167-9-172", :statuscode=>0, :statusmsg=>"OK", :data=>{:time=>nil, :output=>"NOTIFY_ENDPOINT_DELETE: 54.242.118.13 \n", :exitcode=>0}}>] (Request ID: ) (pid:21358)
2013-09-27 02:06:44.697 [DEBUG] DEBUG: MCollective Response Time (execute_direct: app-destroy): 2.124673601s  (Request ID: ) (pid:21358)
2013-09-27 02:06:44.747 [DEBUG] DEBUG: rpc_client.custom_request('cartridge_do', {:cartridge=>"jenkins-1", :action=>"start", :args=>{"--with-app-uuid"=>"5244f19e1cccf8c16000010a", "--with-app-name"=>"jenkins", "--with-container-uuid"=>"5244f19e1cccf8c16000010a", "--with-container-name"=>"jenkins", "--with-namespace"=>"zqd", "--with-uid"=>5747, "--with-request-id"=>nil, "--cart-name"=>"jenkins-1", "--component-name"=>"jenkins-1", "--with-software-version"=>"1", "--cartridge-vendor"=>"redhat"}}, ip-10-184-42-34, {'identity' => ip-10-184-42-34}) (Request ID: ) (pid:21358)
2013-09-27 02:07:00.546 [DEBUG] DEBUG: [#<MCollective::RPC::Result:0x00000006aba3c8 @agent="openshift", @action="cartridge_do", @results={:sender=>"ip-10-184-42-34", :statuscode=>0, :statusmsg=>"OK", :data=>{:time=>nil, :output=>"Starting jenkins cartridge\nWaiting ...\nWaiting ...\nWaiting ...\nStill waiting ...\n", :exitcode=>0}}>] (Request ID: ) (pid:21358)
2013-09-27 02:07:00.548 [DEBUG] DEBUG: MCollective Response Time (execute_direct: start): 16.474694234s  (Request ID: ) (pid:21358)
2013-09-27 02:07:16.225 [INFO ] Started GET "/broker/rest/api" for 66.187.233.202 at 2013-09-27 02:07:16 -0400 (pid:3196)
2013-09-27 02:07:16.228 [INFO ] Processing by ApiController#show as JSON (pid:3196)
2013-09-27 02:07:16.229 [DEBUG] API version 1.3 (pid:3196)
2013-09-27 02:07:16.240 [INFO ] Completed 200 OK in 12ms (Views: 6.1ms) (pid:3196)
2013-09-27 02:07:16.599 [INFO ] Started GET "/broker/rest/api" for 66.187.233.202 at 2013-09-27 02:07:16 -0400 (pid:3196)
2013-09-27 02:07:16.601 [INFO ] Processing by ApiController#show as JSON (pid:3196)
2013-09-27 02:07:16.604 [DEBUG] API version 1.5 (pid:3196)
2013-09-27 02:07:16.616 [INFO ] Completed 200 OK in 12ms (Views: 6.8ms) (pid:3196)

Comment 1 Xiaoli Tian 2013-09-27 09:41:07 UTC
Seems it has been fixed in devenv_3838 now, back to ON_QA to re-test.

Comment 2 zhaozhanqi 2013-09-27 09:46:54 UTC
tested this issue on devenv_3838, it has been fixed.

jenkins:

[root@ip-10-147-189-146 ~]# oo-admin-move --gear_uuid 52454cb89553e9ec730000f7 -i ip-10-147-189-146
URL: http://jenkins-zqd.dev.rhcloud.com
Login: zzhao
App UUID: 52454cb89553e9ec730000f7
Gear UUID: 52454cb89553e9ec730000f7
DEBUG: Source district uuid: 215149309514994275057664
DEBUG: Destination district uuid: 215149309514994275057664
DEBUG: Getting existing app 'jenkins' status before moving
DEBUG: Gear component 'jenkins-1' was running
DEBUG: Stopping existing app cartridge 'jenkins-1' before moving
DEBUG: Force stopping existing app cartridge 'jenkins-1' before moving
DEBUG: Creating new account for gear 'jenkins' on ip-10-147-189-146
DEBUG: Moving content for app 'jenkins', gear 'jenkins' to ip-10-147-189-146
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 13972
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 13972 killed;
DEBUG: Moving system components for app 'jenkins', gear 'jenkins' to ip-10-147-189-146
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 14054
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 14054 killed;
DEBUG: Starting cartridge 'jenkins-1' in 'jenkins' after move on ip-10-147-189-146
DEBUG: Fixing DNS and mongo for gear 'jenkins' after move
DEBUG: Changing server identity of 'jenkins' from 'ip-10-185-23-32' to 'ip-10-147-189-146'
DEBUG: Deconfiguring old app 'jenkins' on ip-10-185-23-32 after move
Successfully moved gear with uuid '52454cb89553e9ec730000f7' of app 'jenkins' from 'ip-10-185-23-32' to 'ip-10-147-189-146'


zend app:

[root@ip-10-147-189-146 ~]# oo-admin-move --gear_uuid 5245536b9553e9ec7300018e -i ip-10-147-189-146
URL: http://zqzend-zqd.dev.rhcloud.com
Login: zzhao
App UUID: 5245536b9553e9ec7300018e
Gear UUID: 5245536b9553e9ec7300018e
DEBUG: Source district uuid: 215149309514994275057664
DEBUG: Destination district uuid: 215149309514994275057664
DEBUG: Getting existing app 'zqzend' status before moving
DEBUG: Gear component 'zend-5.6' was running
DEBUG: Stopping existing app cartridge 'zend-5.6' before moving
DEBUG: Force stopping existing app cartridge 'zend-5.6' before moving
DEBUG: Creating new account for gear 'zqzend' on ip-10-147-189-146
DEBUG: Moving content for app 'zqzend', gear 'zqzend' to ip-10-147-189-146
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 29338
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 29338 killed;
DEBUG: Moving system components for app 'zqzend', gear 'zqzend' to ip-10-147-189-146
Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa)
Agent pid 29407
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 29407 killed;
DEBUG: Starting cartridge 'zend-5.6' in 'zqzend' after move on ip-10-147-189-146
DEBUG: Fixing DNS and mongo for gear 'zqzend' after move
DEBUG: Changing server identity of 'zqzend' from 'ip-10-185-23-32' to 'ip-10-147-189-146'
DEBUG: Deconfiguring old app 'zqzend' on ip-10-185-23-32 after move
Successfully moved gear with uuid '5245536b9553e9ec7300018e' of app 'zqzend' from 'ip-10-185-23-32' to 'ip-10-147-189-146'