Description of problem: Embedding mysql-5.1 to a scalable app,save snapshot after force stopping the app, will meet "OpenShift::Runtime::Utils::ShellExecutionException" errors. Version-Release number of selected component (if applicable): devenv_3449 How reproducible: always Steps to Reproduce: 1. Create a scalable app with mysql-5.1 rhc app create cphp0s0 php-5.3 mysql-5.1 -s --no-git 2. Force stopping this app rhc app force-stop cphp0s0 3. Save snapshot for this app rhc snapshot save cphp0s0 Actual results: Pulling down a snapshot to cpy26sdb.tar.gz... The authenticity of host 'cpy26sdb-cdm.dev.rhcloud.com (67.202.27.111)' can't be established. RSA key fingerprint is a0:4d:a9:29:ee:69:7b:6f:c8:70:ea:cc:a8:7a:3f:3c. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'cpy26sdb-cdm.dev.rhcloud.com' (RSA) to the list of known hosts. Saving snapshot for secondary mysql-5.1 gear mysqldump: Got error: 2003: Can't connect to MySQL server on '127.1.27.129' (111) when trying to connect kill 3942: No such process Creating and sending tar.gz /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/utils/shell_exec.rb:131:in `block (2 levels) in oo_spawn': Shell command ' (OpenShift::Runtime::Utils::ShellExecutionException) /bin/tar --ignore-failed-read -czf - --exclude=./$OPENSHIFT_GEAR_UUID/.tmp --exclude=./$OPENSHIFT_GEAR_UUID/.ssh --exclude=./$OPENSHIFT_GEAR_UUID/.sandbox --exclude=./$OPENSHIFT_GEAR_UUID/*/conf.d/openshift.conf --exclude=./$OPENSHIFT_GEAR_UUID/*/run/httpd.pid --exclude=./$OPENSHIFT_GEAR_UUID/haproxy-*/run/stats --exclude=./$OPENSHIFT_GEAR_UUID/app-root/runtime/.state --exclude=./$OPENSHIFT_DATA_DIR/.bash_history ./$OPENSHIFT_GEAR_UUID ' returned an error. rc=1 from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/utils/shell_exec.rb:94:in `pipe' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/utils/shell_exec.rb:94:in `block in oo_spawn' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/utils/shell_exec.rb:93:in `pipe' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/utils/shell_exec.rb:93:in `oo_spawn' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-container-selinux-0.0.3/lib/openshift/runtime/containerization/selinux_container.rb:284:in `run_in_container_context' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/model/application_container.rb:611:in `run_in_container_context' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:25:in `write_snapshot_archive' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:70:in `snapshot' from /usr/bin/gear:269:in `block (2 levels) in <main>' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:155:in `run' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:385:in `run_active_command' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:62:in `run!' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/delegates.rb:11:in `run!' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/import.rb:10:in `block in <top (required)>' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/utils/shell_exec.rb:131:in `block (2 levels) in oo_spawn': Shell command '/usr/bin/ssh -q -o 'BatchMode=yes' -o 'StrictHostKeyChecking=no' -i $OPENSHIFT_APP_SSH_KEY 51d69285b675b471cb000008.rhcloud.com 'snapshot' > mysql-5.1.tar.gz' returned an error. rc=1 (OpenShift::Runtime::Utils::ShellExecutionException) from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/utils/shell_exec.rb:94:in `pipe' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/utils/shell_exec.rb:94:in `block in oo_spawn' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/utils/shell_exec.rb:93:in `pipe' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/utils/shell_exec.rb:93:in `oo_spawn' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-container-selinux-0.0.3/lib/openshift/runtime/containerization/selinux_container.rb:284:in `run_in_container_context' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/model/application_container.rb:611:in `run_in_container_context' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:92:in `block in handle_scalable_snapshot' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:88:in `each' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:88:in `handle_scalable_snapshot' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:46:in `snapshot' from /usr/bin/gear:269:in `block (2 levels) in <main>' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:155:in `run' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:385:in `run_active_command' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:62:in `run!' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/delegates.rb:11:in `run!' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/import.rb:10:in `block in <top (required)>' Error in trying to save snapshot. You can try to save manually by running: ssh 51d69284b675b471cb000007.rhcloud.com 'snapshot' > cpy26sdb.tar.gz Expected results: Save snapshot should be successfully in this situation Additional info:
PR submitted
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/fded3856d9ea2afec40f8c60afc1ba6d9c73d99a Fix bug 981622
verified on devenv_3481 rhc-1.11.4 1.rhc app create cphp0s0 php-5.3 mysql-5.1 -s --no-git 2.rhc app force-stop cphp0s0 3.rhc snapshot save cphp0s0 [root@localhost ~]# rhc app force-stop cphp0s0 RESULT: cphp0s0 force stopped [root@localhost ~]# rhc snapshot save cphp0s0 Pulling down a snapshot to cphp0s0.tar.gz... Warning: Permanently added 'cphp0s0-aaa.dev.rhcloud.com' (RSA) to the list of known hosts. Saving snapshot for secondary mysql-5.1 gear -- Warning: Skipping the data of table mysql.event. Specify the --events option explicitly. Creating and sending tar.gz Creating and sending tar.gz /bin/tar: ./472161596716741138841600/haproxy/run/stats: socket ignored RESULT: Success move to verified