Created attachment 658675 [details] move jbossas development.log Description of problem: Can't move non_scalable jbossas/jbossews accross dictrict successfully. but could move jbossas/jbossews within dictrict successfully. and could move scalable jbossas/jbossews accross dictrict successfully. move jbossas and jbossews app accross dictrict show different error in development.log, please see the attachement. Version-Release number of selected component (if applicable): devenv_2561 How reproducible: always Steps to Reproduce: 1.Setup multi node env, create 2 dists 2.Create a jbossas app 3.Move this app accross district Actual results: execute command: oo-admin-move --gear_uuid 9651239ae4044d8598b886b8ffcde982 -i ip-10-114-242-73 URL: http://qjbosass-qgong11.dev.rhcloud.com Login: qgong App UUID: 9651239ae4044d8598b886b8ffcde982 Gear UUID: 9651239ae4044d8598b886b8ffcde982 DEBUG: Source district uuid: 1b81999e98cf475eabeafbb954b97433 DEBUG: Destination district uuid: 64c0d04f22da481b9b5a31ee7bc6dc85 DEBUG: Getting existing app 'qjbosass' status before moving DEBUG: Gear component 'jbossas-7' was running DEBUG: Stopping existing app cartridge 'jbossas-7' before moving DEBUG: Force stopping existing app cartridge 'jbossas-7' before moving DEBUG: Reserved uid '1004' on district: '64c0d04f22da481b9b5a31ee7bc6dc85' DEBUG: Creating new account for gear 'qjbosass' on ip-10-114-242-73 DEBUG: Moving content for app 'qjbosass', gear 'qjbosass' to ip-10-114-242-73 Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa) Agent pid 16689 unset SSH_AUTH_SOCK; unset SSH_AGENT_PID; echo Agent pid 16689 killed; DEBUG: Performing cartridge level move for 'jbossas-7' on ip-10-114-242-73 DEBUG: Starting cartridge 'jbossas-7' in 'qjbosass' after move on ip-10-114-242-73 DEBUG: Fixing DNS and mongo for gear 'qjbosass' after move DEBUG: Changing server identity of 'qjbosass' from 'ip-10-114-219-87' to 'ip-10-114-242-73' DEBUG: Moving failed. Rolling back gear 'qjbosass' 'qjbosass' with remove-httpd-proxy on 'ip-10-114-242-73' DEBUG: Moving failed. Rolling back gear 'qjbosass' in 'qjbosass' with destroy on 'ip-10-114-242-73' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.2.3/lib/openshift/mcollective_application_container_proxy.rb:1194:in `run_cartridge_command': Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support. (OpenShift::NodeException) from /var/www/openshift/broker/lib/express/broker/mcollective_ext.rb:12:in `run_cartridge_command' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.2.3/lib/openshift/mcollective_application_container_proxy.rb:612:in `block in move_gear_post' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.2.3/lib/openshift/mcollective_application_container_proxy.rb:611:in `each' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.2.3/lib/openshift/mcollective_application_container_proxy.rb:611:in `move_gear_post' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.2.3/lib/openshift/mcollective_application_container_proxy.rb:741:in `move_gear' from /usr/sbin/oo-admin-move:110:in `<main>' Expected results: Move successfully Additional info:
Created attachment 658676 [details] move jbossews development.log
NOTE: This bug affects all carts, not just jboss. Resolved by: https://github.com/openshift/origin-server/pull/1033
Fixed in devenv_2570+.
Checked on devenv_2577, move gears across district still has problem for non-scalable app with db carts embedded and embedded db gears for scalable app. # oo-admin-move --gear_uuid a3c31f4b0ace4aa89d34efdca82e07d5 -i ip-10-64-61-114 --allow_change_district URL: http://mediawiki-bmeng1dev.dev.rhcloud.com Login: bmeng+1 App UUID: a3c31f4b0ace4aa89d34efdca82e07d5 Gear UUID: a3c31f4b0ace4aa89d34efdca82e07d5 DEBUG: Source district uuid: 20575b1bc9184290913fabb1ba108b2c DEBUG: Destination district uuid: NONE DEBUG: Getting existing app 'mediawiki' status before moving DEBUG: Gear component 'php-5.3' was running DEBUG: Stopping existing app cartridge 'cron-1.4' before moving DEBUG: Performing cartridge level pre-move for embedded cron-1.4 for 'mediawiki' on ip-10-10-145-24 DEBUG: Stopping existing app cartridge 'phpmyadmin-3.4' before moving DEBUG: Performing cartridge level pre-move for embedded phpmyadmin-3.4 for 'mediawiki' on ip-10-10-145-24 DEBUG: Stopping existing app cartridge 'mysql-5.1' before moving DEBUG: Performing cartridge level pre-move for embedded mysql-5.1 for 'mediawiki' on ip-10-10-145-24 DEBUG: Stopping existing app cartridge 'php-5.3' before moving DEBUG: Force stopping existing app cartridge 'php-5.3' before moving DEBUG: Reserved uid '' on district: 'NONE' DEBUG: Creating new account for gear 'mediawiki' on ip-10-64-61-114 DEBUG: Moving content for app 'mediawiki', gear 'mediawiki' to ip-10-64-61-114 Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa) Agent pid 11599 unset SSH_AUTH_SOCK; unset SSH_AGENT_PID; echo Agent pid 11599 killed; DEBUG: Performing cartridge level move for 'php-5.3' on ip-10-64-61-114 DEBUG: Performing cartridge level move for embedded mysql-5.1 for 'mediawiki' on ip-10-64-61-114 DEBUG: Moving failed. Rolling back gear 'mediawiki' 'mediawiki' with remove-httpd-proxy on 'ip-10-64-61-114' DEBUG: Moving failed. Rolling back gear 'mediawiki' in 'mediawiki' with destroy on 'ip-10-64-61-114' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.2.6/lib/openshift/mcollective_application_container_proxy.rb:1116:in `parse_result': Node execution failure (invalid exit code from node). If the problem persists please contact Red Hat support. (OpenShift::NodeException) from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.2.6/lib/openshift/mcollective_application_container_proxy.rb:1188:in `run_cartridge_command' from /var/www/openshift/broker/lib/express/broker/mcollective_ext.rb:12:in `run_cartridge_command' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.2.6/lib/openshift/mcollective_application_container_proxy.rb:716:in `block in move_gear' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.2.6/lib/openshift/mcollective_application_container_proxy.rb:698:in `each' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-msg-broker-mcollective-1.2.6/lib/openshift/mcollective_application_container_proxy.rb:698:in `move_gear' from /usr/sbin/oo-admin-move:110:in `<main>'
*** Bug 885840 has been marked as a duplicate of this bug. ***
New fix: https://github.com/openshift/origin-server/pull/1049
1.Move non_scalable php app with mongodb embed, accross district [root@ip-10-141-133-99 origin-server]# oo-admin-move --gear_uuid 295aad761c84450c9d75d9e4f9e9d850 -i ip-10-190-221-174 --allow_change_district URL: http://qphp-qgong1.dev.rhcloud.com Login: qgong App UUID: 295aad761c84450c9d75d9e4f9e9d850 Gear UUID: 295aad761c84450c9d75d9e4f9e9d850 DEBUG: Source district uuid: c64ff35155cf44f8bb7cdcb45bd87082 DEBUG: Destination district uuid: 422198d93d864ff29d821889088342f8 DEBUG: Getting existing app 'qphp' status before moving DEBUG: Gear component 'php-5.3' was running DEBUG: Stopping existing app cartridge 'mongodb-2.2' before moving DEBUG: Performing cartridge level pre-move for embedded mongodb-2.2 for 'qphp' on ip-10-141-133-99 DEBUG: Stopping existing app cartridge 'php-5.3' before moving DEBUG: Force stopping existing app cartridge 'php-5.3' before moving DEBUG: Reserved uid '1000' on district: '422198d93d864ff29d821889088342f8' DEBUG: Creating new account for gear 'qphp' on ip-10-190-221-174 DEBUG: Moving content for app 'qphp', gear 'qphp' to ip-10-190-221-174 Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa) Warning: Permanently added '10.141.133.99' (RSA) to the list of known hosts. Warning: Permanently added '10.190.221.174' (RSA) to the list of known hosts. Agent pid 1743 unset SSH_AUTH_SOCK; unset SSH_AGENT_PID; echo Agent pid 1743 killed; DEBUG: Performing cartridge level move for 'php-5.3' on ip-10-190-221-174 DEBUG: Performing cartridge level move for embedded mongodb-2.2 for 'qphp' on ip-10-190-221-174 DEBUG: Performing cartridge level post-move for embedded mongodb-2.2 for 'qphp' on ip-10-190-221-174 DEBUG: Starting cartridge 'php-5.3' in 'qphp' after move on ip-10-190-221-174 DEBUG: Starting cartridge 'mongodb-2.2' in 'qphp' after move on ip-10-190-221-174 DEBUG: Fixing DNS and mongo for gear 'qphp' after move DEBUG: Changing server identity of 'qphp' from 'ip-10-141-133-99' to 'ip-10-190-221-174' DEBUG: Deconfiguring old app 'qphp' on ip-10-141-133-99 after move Successfully moved 'qphp' with gear uuid '295aad761c84450c9d75d9e4f9e9d850' from 'ip-10-141-133-99' to 'ip-10-190-221-174' 1.Move scalable jbossews app with postgresql embed, accross district [root@ip-10-141-133-99 origin-server]# oo-admin-move --gear_uuid 60353d83ff8847dc98d50b5c0e65685a -i ip-10-190-221-174 --allow_change_district URL: http://qsjbossews-qgong1.dev.rhcloud.com Login: qgong App UUID: 60353d83ff8847dc98d50b5c0e65685a Gear UUID: 60353d83ff8847dc98d50b5c0e65685a DEBUG: Source district uuid: c64ff35155cf44f8bb7cdcb45bd87082 DEBUG: Destination district uuid: 422198d93d864ff29d821889088342f8 DEBUG: Getting existing app 'qsjbossews' status before moving DEBUG: Gear component 'jbossews-1.0' was running DEBUG: Stopping existing app cartridge 'haproxy-1.4' before moving DEBUG: Stopping existing app cartridge 'jbossews-1.0' before moving DEBUG: Force stopping existing app cartridge 'jbossews-1.0' before moving DEBUG: Reserved uid '1002' on district: '422198d93d864ff29d821889088342f8' DEBUG: Creating new account for gear 'qsjbossews' on ip-10-190-221-174 DEBUG: Moving content for app 'qsjbossews', gear 'qsjbossews' to ip-10-190-221-174 Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa) Agent pid 21526 unset SSH_AUTH_SOCK; unset SSH_AGENT_PID; echo Agent pid 21526 killed; DEBUG: Performing cartridge level move for 'jbossews-1.0' on ip-10-190-221-174 DEBUG: Performing cartridge level move for 'haproxy-1.4' on ip-10-190-221-174 DEBUG: Starting cartridge 'jbossews-1.0' in 'qsjbossews' after move on ip-10-190-221-174 DEBUG: Starting cartridge 'haproxy-1.4' in 'qsjbossews' after move on ip-10-190-221-174 DEBUG: Fixing DNS and mongo for gear 'qsjbossews' after move DEBUG: Changing server identity of 'qsjbossews' from 'ip-10-141-133-99' to 'ip-10-190-221-174' DEBUG: Deconfiguring old app 'qsjbossews' on ip-10-141-133-99 after move Successfully moved 'qsjbossews' with gear uuid '60353d83ff8847dc98d50b5c0e65685a' from 'ip-10-141-133-99' to 'ip-10-190-221-174' [root@ip-10-141-133-99 origin-server]# oo-admin-move --gear_uuid 297a8d6d995d4e1fba8a72b16d9fd5d2 -i ip-10-190-221-174 --allow_change_district URL: http://qsjbossews-qgong1.dev.rhcloud.com Login: qgong App UUID: 60353d83ff8847dc98d50b5c0e65685a Gear UUID: 297a8d6d995d4e1fba8a72b16d9fd5d2 DEBUG: Source district uuid: c64ff35155cf44f8bb7cdcb45bd87082 DEBUG: Destination district uuid: 422198d93d864ff29d821889088342f8 DEBUG: Getting existing app 'qsjbossews' status before moving DEBUG: Gear component 'jbossews-1.0' was running DEBUG: Stopping existing app cartridge 'postgresql-8.4' before moving DEBUG: Performing cartridge level pre-move for embedded postgresql-8.4 for 'qsjbossews' on ip-10-141-133-99 DEBUG: Reserved uid '1003' on district: '422198d93d864ff29d821889088342f8' DEBUG: Creating new account for gear '297a8d6d99' on ip-10-190-221-174 DEBUG: Moving content for app 'qsjbossews', gear '297a8d6d99' to ip-10-190-221-174 Identity added: /var/www/openshift/broker/config/keys/rsync_id_rsa (/var/www/openshift/broker/config/keys/rsync_id_rsa) Agent pid 23424 unset SSH_AUTH_SOCK; unset SSH_AGENT_PID; echo Agent pid 23424 killed; DEBUG: Performing cartridge level move for embedded postgresql-8.4 for 'qsjbossews' on ip-10-190-221-174 DEBUG: Performing cartridge level post-move for embedded postgresql-8.4 for 'qsjbossews' on ip-10-190-221-174 DEBUG: Starting cartridge 'postgresql-8.4' in 'qsjbossews' after move on ip-10-190-221-174 DEBUG: Fixing DNS and mongo for gear '297a8d6d99' after move DEBUG: Changing server identity of '297a8d6d99' from 'ip-10-141-133-99' to 'ip-10-190-221-174' DEBUG: Deconfiguring old app 'qsjbossews' on ip-10-141-133-99 after move Successfully moved 'qsjbossews' with gear uuid '297a8d6d995d4e1fba8a72b16d9fd5d2' from 'ip-10-141-133-99' to 'ip-10-190-221-174'
Verified this on devenv_2581
It's fixed based on comment 7 and comment 8, since its original state is not ON_QA, let me move it back to Assigned, and please move it to ON_QA if it's ready for QE to tesst
Sorry for the confusion: It's ready to test in devenv_2581+. Moving to ON_QA!
Verified on devenv_2589