Bug 981622 - Failed to save snapshot after force stopping scalable app with mysql-5.1
Failed to save snapshot after force stopping scalable app with mysql-5.1
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Paul Morie
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-05 05:58 EDT by chunchen
Modified: 2016-09-29 22:15 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-07-22 11:21:47 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description chunchen 2013-07-05 05:58:49 EDT
Description of problem:
Embedding mysql-5.1 to a scalable app,save snapshot after force stopping the app, will meet "OpenShift::Runtime::Utils::ShellExecutionException" errors.

Version-Release number of selected component (if applicable):
devenv_3449

How reproducible:
always

Steps to Reproduce:
1. Create a scalable app with mysql-5.1
rhc app create cphp0s0 php-5.3 mysql-5.1 -s --no-git
2. Force stopping this app
rhc app force-stop cphp0s0
3. Save snapshot for this app
rhc snapshot save cphp0s0

Actual results:
Pulling down a snapshot to cpy26sdb.tar.gz...
The authenticity of host 'cpy26sdb-cdm.dev.rhcloud.com (67.202.27.111)' can't be established.
RSA key fingerprint is a0:4d:a9:29:ee:69:7b:6f:c8:70:ea:cc:a8:7a:3f:3c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'cpy26sdb-cdm.dev.rhcloud.com' (RSA) to the list of known hosts.
Saving snapshot for secondary mysql-5.1 gear
mysqldump: Got error: 2003: Can't connect to MySQL server on '127.1.27.129' (111) when trying to connect
kill 3942: No such process
Creating and sending tar.gz
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/utils/shell_exec.rb:131:in `block (2 levels) in oo_spawn': Shell command ' (OpenShift::Runtime::Utils::ShellExecutionException)
/bin/tar --ignore-failed-read -czf - --exclude=./$OPENSHIFT_GEAR_UUID/.tmp --exclude=./$OPENSHIFT_GEAR_UUID/.ssh --exclude=./$OPENSHIFT_GEAR_UUID/.sandbox --exclude=./$OPENSHIFT_GEAR_UUID/*/conf.d/openshift.conf --exclude=./$OPENSHIFT_GEAR_UUID/*/run/httpd.pid --exclude=./$OPENSHIFT_GEAR_UUID/haproxy-*/run/stats --exclude=./$OPENSHIFT_GEAR_UUID/app-root/runtime/.state --exclude=./$OPENSHIFT_DATA_DIR/.bash_history            ./$OPENSHIFT_GEAR_UUID
' returned an error. rc=1
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/utils/shell_exec.rb:94:in `pipe'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/utils/shell_exec.rb:94:in `block in oo_spawn'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/utils/shell_exec.rb:93:in `pipe'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/utils/shell_exec.rb:93:in `oo_spawn'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-container-selinux-0.0.3/lib/openshift/runtime/containerization/selinux_container.rb:284:in `run_in_container_context'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/model/application_container.rb:611:in `run_in_container_context'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:25:in `write_snapshot_archive'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:70:in `snapshot'
	from /usr/bin/gear:269:in `block (2 levels) in <main>'
	from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
	from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
	from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:155:in `run'
	from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:385:in `run_active_command'
	from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:62:in `run!'
	from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/delegates.rb:11:in `run!'
	from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/import.rb:10:in `block in <top (required)>'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/utils/shell_exec.rb:131:in `block (2 levels) in oo_spawn': Shell command '/usr/bin/ssh -q -o 'BatchMode=yes' -o 'StrictHostKeyChecking=no' -i $OPENSHIFT_APP_SSH_KEY  51d69285b675b471cb000008@51d69285b675b471cb000008-cdm.dev.rhcloud.com 'snapshot' > mysql-5.1.tar.gz' returned an error. rc=1 (OpenShift::Runtime::Utils::ShellExecutionException)
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/utils/shell_exec.rb:94:in `pipe'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/utils/shell_exec.rb:94:in `block in oo_spawn'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/utils/shell_exec.rb:93:in `pipe'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/utils/shell_exec.rb:93:in `oo_spawn'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-container-selinux-0.0.3/lib/openshift/runtime/containerization/selinux_container.rb:284:in `run_in_container_context'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/model/application_container.rb:611:in `run_in_container_context'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:92:in `block in handle_scalable_snapshot'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:88:in `each'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:88:in `handle_scalable_snapshot'
	from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.11.4/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:46:in `snapshot'
	from /usr/bin/gear:269:in `block (2 levels) in <main>'
	from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
	from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
	from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:155:in `run'
	from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:385:in `run_active_command'
	from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:62:in `run!'
	from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/delegates.rb:11:in `run!'
	from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/import.rb:10:in `block in <top (required)>'
Error in trying to save snapshot. You can try to save manually by running:
ssh 51d69284b675b471cb000007@cpy26sdb-cdm.dev.rhcloud.com 'snapshot' > cpy26sdb.tar.gz


Expected results:
Save snapshot should be successfully in this situation

Additional info:
Comment 1 Paul Morie 2013-07-10 15:33:16 EDT
PR submitted
Comment 3 Yan Du 2013-07-11 01:11:22 EDT
verified on devenv_3481
rhc-1.11.4

1.rhc app create cphp0s0 php-5.3 mysql-5.1 -s --no-git
2.rhc app force-stop cphp0s0
3.rhc snapshot save cphp0s0

[root@localhost ~]# rhc app force-stop cphp0s0
RESULT:
cphp0s0 force stopped
[root@localhost ~]# rhc snapshot save cphp0s0
Pulling down a snapshot to cphp0s0.tar.gz...
Warning: Permanently added 'cphp0s0-aaa.dev.rhcloud.com' (RSA) to the list of known hosts.
Saving snapshot for secondary mysql-5.1 gear
-- Warning: Skipping the data of table mysql.event. Specify the --events option explicitly.
Creating and sending tar.gz
Creating and sending tar.gz
/bin/tar: ./472161596716741138841600/haproxy/run/stats: socket ignored

RESULT:
Success


move to verified

Note You need to log in before you can comment on or make changes to this bug.