Description of problem: When verifying bz#981622, I found this issue - after force-stop of the app, I'm not able to restore app with rhc client command. The suggested manual way seems to work. Version-Release number of selected component (if applicable): AMI: devenv_3881 (ami-d75909be) rhc-client 1.15.6 How reproducible: always Steps to Reproduce: 1. rhc app create cphp0s0 php-5.3 mysql-5.1 -s --no-git 2. rhc app force-stop cphp0s0 3. Save snapshot for this app 4. rhc app start cphp0s0 5. SSH into this app and write some data in mysql DB 6. rhc app force-stop cphp0s0 FAILS>7. rhc snapshot restore cphp0s0 When running: cat '/home/mzimen/workdir/2013/10/10/11:06:14/Bug981622_snapshot_force-stopped_scalable_app_with_mysql/fxtgrxc1.tar.gz' | ssh 52566e1f06edc12af500001f.rhcloud.com 'restore INCLUDE_GIT' -- no errors Actual results: Expected results: Additional info: [11:09:13] INFO> Shell Command: rhc snapshot restore fxtgrxc1 -f /home/mzimen/workdir/2013/10/10/11:06:14/Bug9816 22_snapshot_force-stopped_scalable_app_with_mysql/fxtgrxc1.tar.gz -l mzimen -p 'redhat' --insecure --server ec2-54-205-138-116.compute-1.amazonaws.com Restoring from snapshot /home/mzimen/workdir/2013/10/10/11:06:14/Bug981622_snapshot_force-stopped_scalable_app_wi th_mysql/fxtgrxc1.tar.gz... Removing old git repo: ~/git/fxtgrxc1.git/ Removing old data dir: ~/app-root/data/* Restoring ~/git/fxtgrxc1.git and ~/app-root/data ERROR 2003 (HY000): Can't connect to MySQL server on '127.1.244.2' (111) Could not drop existing database /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:1169:in `block in do_control_with_directory': Failed to execute: 'control post-restore' for /var/lib/openshift/52566e1f06edc12af500001f/mysql (OpenShift::Runtime::Utils::ShellExecutionException) from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:970:in `process_cartridges' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:1132:in `do_control_with_directory' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:992:in `do_control' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:155:in `block in restore' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:80:in `block in each_cartridge' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:976:in `block in process_cartridges' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:974:in `each' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:974:in `process_cartridges' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:78:in `each_cartridge' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:154:in `restore' from /usr/bin/gear:555:in `block (2 levels) in <main>' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:155:in `run' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:385:in `run_active_command' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:62:in `run!' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/delegates.rb:11:in `run!' from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/import.rb:10:in `block in <top (required)>' Error in trying to restore snapshot. You can try to restore manually by running: cat '/home/mzimen/workdir/2013/10/10/11:06:14/Bug981622_snapshot_force-stopped_scalable_app_with_mysql/fxtgrxc1.tar.gz' | ssh 52566e1f06edc12af500001f.rhcloud.com 'restore INCLUDE_GIT' [11:09:23] INFO> Exit Status: 130
I can confirm the same issue occurs against AMI: devenv-stage_491 (ami-ff3b6b96)
https://github.com/openshift/origin-server/pull/3833 https://github.com/openshift/origin-server/pull/3834
Note to QE: You can reproduce this issue with a simplified set of steps: 1. Create the scalable php/mysql app 2. Take a snapshot 3. Force-stop the app 4. Restore the snapshot Thanks!
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/ae07360e8bcd033174cfaef23824ab4c7b77c9fc Bug 1017642: Wait for mysql to start before accesing during restore
Tested on devenv_3886. Retested the problem with the following steps: 1. rhc app create php1 php-5.3 mysql-5.1 2. rhc snapshot save php1 3. rhc app force-stop php1 4. rhc snapshot restore -a php1 -f php1.tar.gz The snapshot restore ends up successfully. The app is accessible after all the above steps. Mark the bug as verified.