Bug 1017642 - Unable to restore app after force-stop
Summary: Unable to restore app after force-stop
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Containers
Version: 2.x
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: ---
Assignee: Dan Mace
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-10-10 09:31 UTC by mzimen
Modified: 2015-05-14 23:29 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-10-17 13:34:45 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description mzimen 2013-10-10 09:31:30 UTC
Description of problem:

When verifying bz#981622, I found this issue - after force-stop of the app, I'm not able to restore app with rhc client command.
The suggested manual way seems to work.


Version-Release number of selected component (if applicable):
AMI: devenv_3881 (ami-d75909be)
rhc-client 1.15.6

How reproducible:
always

Steps to Reproduce:
1. rhc app create cphp0s0 php-5.3 mysql-5.1 -s --no-git
2. rhc app force-stop cphp0s0
3. Save snapshot for this app
4. rhc app start cphp0s0
5. SSH into this app and write some data in mysql DB
6. rhc app force-stop cphp0s0
FAILS>7. rhc snapshot restore cphp0s0

When running:
cat '/home/mzimen/workdir/2013/10/10/11:06:14/Bug981622_snapshot_force-stopped_scalable_app_with_mysql/fxtgrxc1.tar.gz' | ssh 52566e1f06edc12af500001f.rhcloud.com 'restore INCLUDE_GIT'
  -- no errors

Actual results:


Expected results:


Additional info:
[11:09:13] INFO> Shell Command: rhc snapshot restore fxtgrxc1 -f /home/mzimen/workdir/2013/10/10/11:06:14/Bug9816
22_snapshot_force-stopped_scalable_app_with_mysql/fxtgrxc1.tar.gz -l mzimen -p 'redhat' --insecure --server
ec2-54-205-138-116.compute-1.amazonaws.com
      Restoring from snapshot /home/mzimen/workdir/2013/10/10/11:06:14/Bug981622_snapshot_force-stopped_scalable_app_wi
th_mysql/fxtgrxc1.tar.gz...
      Removing old git repo: ~/git/fxtgrxc1.git/
      Removing old data dir: ~/app-root/data/*
      Restoring ~/git/fxtgrxc1.git and ~/app-root/data
      ERROR 2003 (HY000): Can't connect to MySQL server on '127.1.244.2' (111)
      Could not drop existing database
      /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:1169:in `block in do_control_with_directory': Failed to execute: 'control post-restore' for /var/lib/openshift/52566e1f06edc12af500001f/mysql (OpenShift::Runtime::Utils::ShellExecutionException)
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:970:in `process_cartridges'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:1132:in `do_control_with_directory'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:992:in `do_control'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:155:in `block in restore'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:80:in `block in each_cartridge'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:976:in `block in process_cartridges'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:974:in `each'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:974:in `process_cartridges'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:78:in `each_cartridge'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:154:in `restore'
        from /usr/bin/gear:555:in `block (2 levels) in <main>'
        from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
        from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
  from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:155:in `run'
        from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:385:in `run_active_command'
        from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:62:in `run!'
        from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/delegates.rb:11:in `run!'
        from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/import.rb:10:in `block in <top (required)>'
      Error in trying to restore snapshot. You can try to restore manually by running:
      cat '/home/mzimen/workdir/2013/10/10/11:06:14/Bug981622_snapshot_force-stopped_scalable_app_with_mysql/fxtgrxc1.tar.gz' | ssh 52566e1f06edc12af500001f.rhcloud.com 'restore INCLUDE_GIT'
      [11:09:23] INFO> Exit Status: 130

Comment 1 mzimen 2013-10-10 12:57:19 UTC
I can confirm the same issue occurs against AMI: devenv-stage_491 (ami-ff3b6b96)

Comment 3 Dan Mace 2013-10-10 15:09:12 UTC
Note to QE: You can reproduce this issue with a simplified set of steps:

1. Create the scalable php/mysql app
2. Take a snapshot
3. Force-stop the app
4. Restore the snapshot

Thanks!

Comment 4 openshift-github-bot 2013-10-10 16:47:10 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/ae07360e8bcd033174cfaef23824ab4c7b77c9fc
Bug 1017642: Wait for mysql to start before accesing during restore

Comment 5 Qiushui Zhang 2013-10-11 09:39:39 UTC
Tested on devenv_3886.

Retested the problem with the following steps:
1. rhc app create php1 php-5.3 mysql-5.1
2. rhc snapshot save php1
3. rhc app force-stop php1
4. rhc snapshot restore -a php1 -f php1.tar.gz

The snapshot restore ends up successfully. The app is accessible after all the above steps.

Mark the bug as verified.


Note You need to log in before you can comment on or make changes to this bug.