Bug 1017642 - Unable to restore app after force-stop
Unable to restore app after force-stop
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
unspecified Severity medium
: ---
: ---
Assigned To: Dan Mace
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-10 05:31 EDT by mzimen
Modified: 2015-05-14 19:29 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-17 09:34:45 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description mzimen 2013-10-10 05:31:30 EDT
Description of problem:

When verifying bz#981622, I found this issue - after force-stop of the app, I'm not able to restore app with rhc client command.
The suggested manual way seems to work.


Version-Release number of selected component (if applicable):
AMI: devenv_3881 (ami-d75909be)
rhc-client 1.15.6

How reproducible:
always

Steps to Reproduce:
1. rhc app create cphp0s0 php-5.3 mysql-5.1 -s --no-git
2. rhc app force-stop cphp0s0
3. Save snapshot for this app
4. rhc app start cphp0s0
5. SSH into this app and write some data in mysql DB
6. rhc app force-stop cphp0s0
FAILS>7. rhc snapshot restore cphp0s0

When running:
cat '/home/mzimen/workdir/2013/10/10/11:06:14/Bug981622_snapshot_force-stopped_scalable_app_with_mysql/fxtgrxc1.tar.gz' | ssh 52566e1f06edc12af500001f@fxtgrxc1-apzcum.dev.rhcloud.com 'restore INCLUDE_GIT'
  -- no errors

Actual results:


Expected results:


Additional info:
[11:09:13] INFO> Shell Command: rhc snapshot restore fxtgrxc1 -f /home/mzimen/workdir/2013/10/10/11:06:14/Bug9816
22_snapshot_force-stopped_scalable_app_with_mysql/fxtgrxc1.tar.gz -l mzimen@redhat.com -p 'redhat' --insecure --server
ec2-54-205-138-116.compute-1.amazonaws.com
      Restoring from snapshot /home/mzimen/workdir/2013/10/10/11:06:14/Bug981622_snapshot_force-stopped_scalable_app_wi
th_mysql/fxtgrxc1.tar.gz...
      Removing old git repo: ~/git/fxtgrxc1.git/
      Removing old data dir: ~/app-root/data/*
      Restoring ~/git/fxtgrxc1.git and ~/app-root/data
      ERROR 2003 (HY000): Can't connect to MySQL server on '127.1.244.2' (111)
      Could not drop existing database
      /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:1169:in `block in do_control_with_directory': Failed to execute: 'control post-restore' for /var/lib/openshift/52566e1f06edc12af500001f/mysql (OpenShift::Runtime::Utils::ShellExecutionException)
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:970:in `process_cartridges'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:1132:in `do_control_with_directory'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:992:in `do_control'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:155:in `block in restore'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:80:in `block in each_cartridge'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:976:in `block in process_cartridges'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:974:in `each'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:974:in `process_cartridges'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:78:in `each_cartridge'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:154:in `restore'
        from /usr/bin/gear:555:in `block (2 levels) in <main>'
        from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
        from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
  from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:155:in `run'
        from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:385:in `run_active_command'
        from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:62:in `run!'
        from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/delegates.rb:11:in `run!'
        from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/import.rb:10:in `block in <top (required)>'
      Error in trying to restore snapshot. You can try to restore manually by running:
      cat '/home/mzimen/workdir/2013/10/10/11:06:14/Bug981622_snapshot_force-stopped_scalable_app_with_mysql/fxtgrxc1.tar.gz' | ssh 52566e1f06edc12af500001f@fxtgrxc1-apzcum.dev.rhcloud.com 'restore INCLUDE_GIT'
      [11:09:23] INFO> Exit Status: 130
Comment 1 mzimen 2013-10-10 08:57:19 EDT
I can confirm the same issue occurs against AMI: devenv-stage_491 (ami-ff3b6b96)
Comment 3 Dan Mace 2013-10-10 11:09:12 EDT
Note to QE: You can reproduce this issue with a simplified set of steps:

1. Create the scalable php/mysql app
2. Take a snapshot
3. Force-stop the app
4. Restore the snapshot

Thanks!
Comment 4 openshift-github-bot 2013-10-10 12:47:10 EDT
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/ae07360e8bcd033174cfaef23824ab4c7b77c9fc
Bug 1017642: Wait for mysql to start before accesing during restore
Comment 5 Qiushui Zhang 2013-10-11 05:39:39 EDT
Tested on devenv_3886.

Retested the problem with the following steps:
1. rhc app create php1 php-5.3 mysql-5.1
2. rhc snapshot save php1
3. rhc app force-stop php1
4. rhc snapshot restore -a php1 -f php1.tar.gz

The snapshot restore ends up successfully. The app is accessible after all the above steps.

Mark the bug as verified.

Note You need to log in before you can comment on or make changes to this bug.