Bug 986808

Summary: Failed to do save snapshot for scalable app with db cartridge after scale up
Product: OpenShift Online Reporter: Lei Zhang <lzhang>
Component: ContainersAssignee: Paul Morie <pmorie>
Status: CLOSED WORKSFORME QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: medium    
Version: 2.xCC: xtian, zhewang
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-26 17:58:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lei Zhang 2013-07-22 07:13:42 UTC
Description of problem:
Given a scalable app embedded with one db cartridge(e.g myruby18), do scale up via REST API, do save snapshot. Failed to save snapshot:

[rayzhang@rayzhang Work]$ rhc snapshot save  myruby18
Pulling down a snapshot to myruby18.tar.gz...
Saving snapshot for secondary mysql-5.1 gear
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.12.1/lib/openshift-origin-node/utils/shell_exec.rb:131:in `block (2 levels) in oo_spawn': Shell command '/usr/bin/ssh -q -o 'BatchMode=yes' -o 'StrictHostKeyChecking=no' -i $OPENSHIFT_APP_SSH_KEY  51ecc07f5b684e66cb000025.rhcloud.com 'snapshot' > mysql-5.1.tar.gz' returned an error. rc=255 (OpenShift::Runtime::Utils::ShellExecutionException)
    from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.12.1/lib/openshift-origin-node/utils/shell_exec.rb:94:in `pipe'
    from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.12.1/lib/openshift-origin-node/utils/shell_exec.rb:94:in `block in oo_spawn'
    from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.12.1/lib/openshift-origin-node/utils/shell_exec.rb:93:in `pipe'
    from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.12.1/lib/openshift-origin-node/utils/shell_exec.rb:93:in `oo_spawn'
    from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-container-selinux-0.1.1/lib/openshift/runtime/containerization/selinux_container.rb:288:in `run_in_container_context'
    from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.12.1/lib/openshift-origin-node/model/application_container.rb:595:in `run_in_container_context'
    from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.12.1/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:93:in `block in handle_scalable_snapshot'
    from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.12.1/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:89:in `each'
    from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.12.1/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:89:in `handle_scalable_snapshot'
    from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.12.1/lib/openshift-origin-node/model/application_container_ext/snapshots.rb:47:in `snapshot'
    from /usr/bin/gear:269:in `block (2 levels) in <main>'
    from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
    from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
    from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:155:in `run'
    from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:385:in `run_active_command'
    from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:62:in `run!'
    from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/delegates.rb:11:in `run!'
    from /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/import.rb:10:in `block in <top (required)>'
Error in trying to save snapshot. You can try to save manually by running:
ssh 91dc541cf28b11e28d7a22000aa440c5.rhcloud.com 'snapshot' > myruby18.tar.gz

Version-Release number of selected component (if applicable):
devenv-3535
rhc-1.12.1

How reproducible:
always

Steps to Reproduce:
1. create a scalable app embedded with one db cartridge.
rhc app create myruby18 ruby-1.8 -s mysql-5.1
2. do scale up
curl -k -H "Accept:application/xml" --user "lzhang:rayzhang13" https://ec2-54-234-68-73.compute-1.amazonaws.com/broker/rest/domains/ray/applications/myruby18/events -d event=scale-up -X POST
3. do save snapshot 
rhc snapshot save myruby18

Actual results:
Failed to do save snapshot for scalable app  with db cartridge after scale up.

Expected results:
No error happens when do save snapshot.

Additional info:

Comment 1 Paul Morie 2013-07-26 17:58:07 UTC
I was unable to reproduce this issue.