Bug 1017642

Summary:	Unable to restore app after force-stop
Product:	OpenShift Online	Reporter:	mzimen
Component:	Containers	Assignee:	Dan Mace <dmace>
Status:	CLOSED CURRENTRELEASE	QA Contact:	libra bugs <libra-bugs>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	2.x	CC:	dmace, qiuzhang, szhou
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-10-17 13:34:45 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description mzimen 2013-10-10 09:31:30 UTC

Description of problem:

When verifying bz#981622, I found this issue - after force-stop of the app, I'm not able to restore app with rhc client command.
The suggested manual way seems to work.


Version-Release number of selected component (if applicable):
AMI: devenv_3881 (ami-d75909be)
rhc-client 1.15.6

How reproducible:
always

Steps to Reproduce:
1. rhc app create cphp0s0 php-5.3 mysql-5.1 -s --no-git
2. rhc app force-stop cphp0s0
3. Save snapshot for this app
4. rhc app start cphp0s0
5. SSH into this app and write some data in mysql DB
6. rhc app force-stop cphp0s0
FAILS>7. rhc snapshot restore cphp0s0

When running:
cat '/home/mzimen/workdir/2013/10/10/11:06:14/Bug981622_snapshot_force-stopped_scalable_app_with_mysql/fxtgrxc1.tar.gz' | ssh 52566e1f06edc12af500001f.rhcloud.com 'restore INCLUDE_GIT'
  -- no errors

Actual results:


Expected results:


Additional info:
[11:09:13] INFO> Shell Command: rhc snapshot restore fxtgrxc1 -f /home/mzimen/workdir/2013/10/10/11:06:14/Bug9816
22_snapshot_force-stopped_scalable_app_with_mysql/fxtgrxc1.tar.gz -l mzimen -p 'redhat' --insecure --server
ec2-54-205-138-116.compute-1.amazonaws.com
      Restoring from snapshot /home/mzimen/workdir/2013/10/10/11:06:14/Bug981622_snapshot_force-stopped_scalable_app_wi
th_mysql/fxtgrxc1.tar.gz...
      Removing old git repo: ~/git/fxtgrxc1.git/
      Removing old data dir: ~/app-root/data/*
      Restoring ~/git/fxtgrxc1.git and ~/app-root/data
      ERROR 2003 (HY000): Can't connect to MySQL server on '127.1.244.2' (111)
      Could not drop existing database
      /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:1169:in `block in do_control_with_directory': Failed to execute: 'control post-restore' for /var/lib/openshift/52566e1f06edc12af500001f/mysql (OpenShift::Runtime::Utils::ShellExecutionException)
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:970:in `process_cartridges'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:1132:in `do_control_with_directory'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:992:in `do_control'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:155:in `block in restore'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:80:in `block in each_cartridge'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:976:in `block in process_cartridges'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:974:in `each'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:974:in `process_cartridges'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/v2_cart_model.rb:78:in `each_cartridge'
        from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.16.0/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:154:in `restore'
        from /usr/bin/gear:555:in `block (2 levels) in <main>'
        from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
        from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
  from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:155:in `run'
        from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:385:in `run_active_command'
        from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:62:in `run!'
        from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/delegates.rb:11:in `run!'
        from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/import.rb:10:in `block in <top (required)>'
      Error in trying to restore snapshot. You can try to restore manually by running:
      cat '/home/mzimen/workdir/2013/10/10/11:06:14/Bug981622_snapshot_force-stopped_scalable_app_with_mysql/fxtgrxc1.tar.gz' | ssh 52566e1f06edc12af500001f.rhcloud.com 'restore INCLUDE_GIT'
      [11:09:23] INFO> Exit Status: 130

Comment 1 mzimen 2013-10-10 12:57:19 UTC

I can confirm the same issue occurs against AMI: devenv-stage_491 (ami-ff3b6b96)

Comment 2 Dan Mace 2013-10-10 15:07:57 UTC

https://github.com/openshift/origin-server/pull/3833
https://github.com/openshift/origin-server/pull/3834

Comment 3 Dan Mace 2013-10-10 15:09:12 UTC

Note to QE: You can reproduce this issue with a simplified set of steps:

1. Create the scalable php/mysql app
2. Take a snapshot
3. Force-stop the app
4. Restore the snapshot

Thanks!

Comment 4 openshift-github-bot 2013-10-10 16:47:10 UTC

Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/ae07360e8bcd033174cfaef23824ab4c7b77c9fc
Bug 1017642: Wait for mysql to start before accesing during restore

Comment 5 Qiushui Zhang 2013-10-11 09:39:39 UTC

Tested on devenv_3886.

Retested the problem with the following steps:
1. rhc app create php1 php-5.3 mysql-5.1
2. rhc snapshot save php1
3. rhc app force-stop php1
4. rhc snapshot restore -a php1 -f php1.tar.gz

The snapshot restore ends up successfully. The app is accessible after all the above steps.

Mark the bug as verified.