Bug 1080381

Created attachment 878378 [details]
development log

Created attachment 878379 [details]
platform log

Created attachment 878380 [details]
mcollective log

Test on devenv_4570, 

Failed to restore snapshot after force-stop app

Step：
1. force-stop app
2. do restore snapshot

[lijun@ray Work]$ rhc snapshot restore -a myphp53 -f myphp53.tar.gz 
Restoring from snapshot myphp53.tar.gz...
Removing old git repo: ~/git/myphp53.git/
Removing old data dir: ~/app-root/data/*
Restoring ~/git/myphp53.git and ~/app-root/data
Error in trying to restore snapshot. You can try to restore manually by running:
cat 'myphp53.tar.gz' | ssh 53315d8e87b85301db0004d1.rhcloud.com 'restore INCLUDE_GIT'

(In reply to Lei Zhang from comment #4)
> Test on devenv_4570, 
> 
> Failed to restore snapshot after force-stop app
> 
> Step：
> 1. force-stop app
> 2. do restore snapshot
> 
> [lijun@ray Work]$ rhc snapshot restore -a myphp53 -f myphp53.tar.gz 
> Restoring from snapshot myphp53.tar.gz...
> Removing old git repo: ~/git/myphp53.git/
> Removing old data dir: ~/app-root/data/*
> Restoring ~/git/myphp53.git and ~/app-root/data
> Error in trying to restore snapshot. You can try to restore manually by
> running:
> cat 'myphp53.tar.gz' | ssh
> 53315d8e87b85301db0004d1.rhcloud.com 'restore INCLUDE_GIT'

Can you please open a separe bug for this issue?

The issue also can be reproduced on INT(devenv_4574).

The fix for original problem is here:

https://github.com/openshift/origin-server/pull/5073

(should be merged in a moment. i'm looking on the snapshot issue now).

Lei Zhang:

The snapshot/restore is unrelated to this bug. I tried to debug it a bit and I found that when you do restore via 'rhc', it fails with return code 255:

DEBUG:    code 200  210 ms
Restoring from snapshot myphp53.tar.gz...
DEBUG: cat 'myphp53.tar.gz' | ssh 5332a91a9cd2dc6dba000001.rhcloud.com 'restore INCLUDE_GIT'
Removing old git repo: ~/git/myphp53.git/
Removing old data dir: ~/app-root/data/*
Restoring ~/git/myphp53.git and ~/app-root/data
DEBUG: Activating deployment
Starting PHP 5.3 cartridge (Apache+mod_php)
Application directory "/" selected as DocumentRoot
return: 0 # <- this is from the end of control#start
Error in trying to restore snapshot. You can try to restore manually by running:
cat 'myphp53.tar.gz' | ssh 5332a91a9cd2dc6dba000001.rhcloud.com 'restore INCLUDE_GIT'

If you do:

cat 'myphp53.tar.gz' | ssh 5332a91a9cd2dc6dba000001.rhcloud.com 'restore INCLUDE_GIT'

then the app is restored and started, however the return code in shell is 255 for some reason... So far I don't have any good answer for this. Also, the rhc restore should preserve the state of the gear (force-stopped) and do not attempt to start it... in any case, we should move this to a new bug.

Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/d379990619948b102facd9c518b5ca738953cc2e
Bug 1080381 - Fixed problem with httpd based carts restart after force-stop

Another interesting bit about the restore:

 ~/tmp → cat 'myphp53.tar.gz' | ssh 5332a91a9cd2dc6dba000001.rhcloud.com 'restore INCLUDE_GIT'
Removing old git repo: ~/git/myphp53.git/
Removing old data dir: ~/app-root/data/*
Restoring ~/git/myphp53.git and ~/app-root/data
Activating deployment
Starting PHP 5.3 cartridge (Apache+mod_php)
Application directory "/" selected as DocumentRoot
 ~/tmp → echo $?
255

I think this is the primary cause of the error message rhc is giving us.

The restore bug was moved here:

https://bugzilla.redhat.com/show_bug.cgi?id=1081020

Test on devenv_4586:

Verified on python, ruby, perl(non-scale or scale), it works, thanks.

[root@qwang 20140327]# rhc app force-stop perl510
RESULT:
perl510 force stopped
[root@qwang 20140327]# rhc app restart perl510
RESULT:
perl510 restarted


[root@qwang 20140327]# rhc app force-stop perl510s
RESULT:
perl510s force stopped
[root@qwang 20140327]# rhc app restart perl510s
RESULT:
perl510s restarted