Created attachment 672820 [details] development.log Description of problem: For scalable-app, scale-up this app by REST API, then move the scaled-up gear Failed. And after this move, there always show error message in development.log about can't find Document of ... Version-Release number of selected component (if applicable): fork_ami_refctr1_381 How reproducible: always Steps to Reproduce: 1.Setup multi node env. 2.Create scalable app then scale up by RESTAPI, disable auto-scaling 3.Move the scaled-up gear within district. Actual results: [root@ip-10-38-21-76 ~]# oo-admin-move --gear_uuid 50e7f0c0776856a80800053c -i ip-10-38-21-76 URL: http://qs2php-qgong8.dev.rhcloud.com Login: qgong App UUID: 50e7ef79776856a8080004f6 Gear UUID: 50e7f0c0776856a80800053c DEBUG: Source district uuid: 50e79b17776856de32000001 DEBUG: Destination district uuid: 50e79b17776856de32000001 DEBUG: District unchanged keeping uid DEBUG: Getting existing app 'qs2php' status before moving DEBUG: Gear component 'php-5.3' was running DEBUG: Stopping existing app cartridge 'haproxy-1.4' before moving DEBUG: Error performing stop on existing app on try 1: Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support. DEBUG: Error performing stop on existing app on try 2: Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support. /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.3.1/lib/openshift/mcollective_application_container_proxy.rb:1131:in `parse_result': Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support. (OpenShift::NodeException) from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.3.1/lib/openshift/mcollective_application_container_proxy.rb:1204:in `run_cartridge_command' from /var/www/openshift/broker/lib/express/broker/mcollective_ext.rb:12:in `run_cartridge_command' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.3.1/lib/openshift/mcollective_application_container_proxy.rb:1063:in `stop_component' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.3.1/lib/openshift/mcollective_application_container_proxy.rb:378:in `stop' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.3.1/lib/openshift/mcollective_application_container_proxy.rb:648:in `block (2 levels) in move_gear_pre' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.3.1/lib/openshift/mcollective_application_container_proxy.rb:1014:in `block in do_with_retry' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.3.1/lib/openshift/mcollective_application_container_proxy.rb:1012:in `each' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.3.1/lib/openshift/mcollective_application_container_proxy.rb:1012:in `do_with_retry' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.3.1/lib/openshift/mcollective_application_container_proxy.rb:647:in `block in move_gear_pre' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.3.1/lib/openshift/mcollective_application_container_proxy.rb:640:in `each' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.3.1/lib/openshift/mcollective_application_container_proxy.rb:640:in `move_gear_pre' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.3.1/lib/openshift/mcollective_application_container_proxy.rb:701:in `move_gear' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.3.1/lib/openshift/mcollective_application_container_proxy.rb:670:in `block in move_gear_secure' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-controller-1.3.2/app/models/application.rb:947:in `run_in_application_lock' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.3.1/lib/openshift/mcollective_application_container_proxy.rb:669:in `move_gear_secure' from /usr/sbin/oo-admin-move:110:in `<main>' Expected results: Could move this gear successfully. Additional info:
Fixed with rev#658b4ccd220d02d074b5e1d087b552a11940f3d9
Verified on fork_ami_refctr1_395 1.Create scalable app, then scale-up 2.Move this scaled-up gear [root@ip-10-212-70-181 openshift]# oo-admin-move --gear_uuid 50ee76058d51f0f70b000049 -i ip-10-212-70-181 URL: http://qshp-qgong8.dev.rhcloud.com Login: qgong App UUID: 50ee75378d51f0f70b000023 Gear UUID: 50ee76058d51f0f70b000049 DEBUG: Source district uuid: 50ee70128d51f0cb0b000001 DEBUG: Destination district uuid: 50ee70128d51f0cb0b000001 DEBUG: District unchanged keeping uid DEBUG: Getting existing app 'qshp' status before moving DEBUG: Gear component 'php-5.3' was running DEBUG: Stopping existing app cartridge 'php-5.3' before moving DEBUG: Force stopping existing app cartridge 'php-5.3' before moving DEBUG: Creating new account for gear '50ee76058d' on ip-10-212-70-181 DEBUG: Moving content for app 'qshp', gear '50ee76058d' to ip-10-212-70-181 Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa) Agent pid 20033 unset SSH_AUTH_SOCK; unset SSH_AGENT_PID; echo Agent pid 20033 killed; DEBUG: Moving system components for app 'qshp', gear '50ee76058d' to ip-10-212-70-181 Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa) Agent pid 20095 unset SSH_AUTH_SOCK; unset SSH_AGENT_PID; echo Agent pid 20095 killed; DEBUG: Restarting httpd proxy for 'php-5.3' on ip-10-212-70-181 DEBUG: Starting cartridge 'php-5.3' in 'qshp' after move on ip-10-212-70-181 DEBUG: Fixing DNS and mongo for gear '50ee76058d' after move DEBUG: Changing server identity of '50ee76058d' from 'ip-10-64-62-60' to 'ip-10-212-70-181' DEBUG: Deconfiguring old app 'qshp' on ip-10-64-62-60 after move Successfully moved 'qshp' with gear uuid '50ee76058d51f0f70b000049' from 'ip-10-64-62-60' to 'ip-10-212-70-181'