Bug 1020148 - Sometimes failed to save/restore snapshot to php-5.3 app with postgresql-9.2/postgresql-8.4
Sometimes failed to save/restore snapshot to php-5.3 app with postgresql-9.2/...
Status: CLOSED CANTFIX
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Linux
medium Severity medium
: ---
: ---
Assigned To: Rob Millner
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-17 03:15 EDT by Lei Zhang
Modified: 2015-05-14 19:30 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-24 13:24:37 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
broker development.log (209.00 KB, application/octet-stream)
2013-10-17 03:16 EDT, Lei Zhang
no flags Details
Script to create and snapshot in a loop (390 bytes, application/octet-stream)
2013-10-17 17:20 EDT, Rob Millner
no flags Details
try harder to break the snapshot command (811 bytes, application/octet-stream)
2013-10-17 19:07 EDT, Rob Millner
no flags Details
devenv_3912 broker development.log (59.06 KB, text/x-log)
2013-10-18 02:54 EDT, Lei Zhang
no flags Details
devenv_3912 node platform.log (53.84 KB, text/x-log)
2013-10-18 02:55 EDT, Lei Zhang
no flags Details

  None (edit)
Description Lei Zhang 2013-10-17 03:15:34 EDT
Description of problem:
Given a normal app embedded with postgresql-9.2 created(e.g.,myphp53), do save snapshot, failed to save snapshot. after saving failed, can not access via rhcsh.

[rayzhang@ray Work]$ rhc snapshot save myphp53
Pulling down a snapshot to myphp53.tar.gz...
Creating and sending tar.gz
Write failed: Broken pipe
Error in trying to save snapshot. You can try to save manually by running:
ssh 525f76c897b18b043a000328@myphp53-chunchen.dev.rhcloud.com 'snapshot' > myphp53.tar.gz

[rayzhang@ray Work]$ rhc app ssh myphp53
Connecting to 525f76c897b18b043a000328@myphp53-chunchen.dev.rhcloud.com ...
Write failed: Connection reset by peer

Note:
1. Met this issue in nodejs-0.10/diy-0.1/php-5.3 non-scalable and 
jbossas-7/php-5.3/nodejs-0.10/ruby-1.8 scalable app.
2. After creating app with postgresql-9.2 or mondify the data of psql,  IMMEDIATELY do save or restore app, it's easy to reproduce this issue.

Version-Release number of selected component (if applicable):
devenv_3907

How reproducible:
sometimes

Steps to Reproduce:
1.create one app embedded with postgresql-9.2
#rhc app create myphp53 php-5.3 postgresql-9.2
2.do save snapshot 
#rhc snapshot save myphp53
3.

Actual results:
Failed to save snapshot for app with postgresql-9.2


Expected results:
It should succeed to save snapshot for app with postgresql-9.2 

Additional info:
Comment 1 Lei Zhang 2013-10-17 03:16:29 EDT
Created attachment 813205 [details]
broker development.log
Comment 2 Rob Millner 2013-10-17 15:11:51 EDT
It seems like the underlying issue is that a snapshot is requested while another cartridge operation is running and the two conflict; however, I'm not able to reproduce the issue.

Here's the command I'm using:
rhc app-delete --confirm myphp53; rm -f myphp53.tar.gz ; rhc app create myphp53 php-5.3 postgresql-9.2 --no-git --no-dns; rhc snapshot save myphp53

...and re-running it over and over again.

I'm going to put that in a loop for an hour and see if we can make it happen.
Comment 3 Rob Millner 2013-10-17 17:16:44 EDT
Two hours in a loop and I can't reproduce it.  Attaching the script I was using.
Comment 4 Rob Millner 2013-10-17 17:20:29 EDT
Created attachment 813556 [details]
Script to create and snapshot in a loop

Touch the file "KEEPGOING" and run the script like this:
touch KEEPGOING
nohup create_and_snapshot.sh &

To monitor the output:
tail -f nohup.out

To stop the script gently, remove the KEEPGOING file.
rm -f KEEPGOING
...it will finish up the current loop and exit.

If the snapshot file is not created, then the script will exit after packaging up the entire contents of /var/log to /var/lib/openshift/logs.tgz.
Comment 5 Rob Millner 2013-10-17 19:07:31 EDT
Created attachment 813566 [details]
try harder to break the snapshot command

Similar to the last attachment with a few tweaks. I added idle and simultaneous unidle and snapshot to see if I can force a failure by overlapping gear operations.
Comment 6 Rob Millner 2013-10-17 19:09:33 EDT
Updated the script to try and force a conflict and still no luck breaking snapshots.

Back to Q/E to see if you can recreate the problem.
Comment 7 Lei Zhang 2013-10-18 02:07:35 EDT
devenv_3912 still can reproduce this issue. after create php with postgresql-9.2, IMMEDIATELY do save snapshot, it's easy to reproduce.


[rayzhang@ray Work]$ rhc app create myphp php-5.3 postgresql-9.2 
Application Options
-------------------
  Namespace:  chunchen
  Cartridges: php-5.3, postgresql-9.2
  Gear Size:  default
  Scaling:    no

Creating application 'myphp' ... done

  PostgreSQL 9.2 database added.  Please make note of these credentials:

   Root User: admindbh5ay9
   Root Password: 1axqLG4gRr9h
   Database Name: myphp

Connection URL: postgresql://$OPENSHIFT_POSTGRESQL_DB_HOST:$OPENSHIFT_POSTGRESQL_DB_PORT

Waiting for your DNS name to be available ... done

Cloning into 'myphp'...
The authenticity of host 'myphp-chunchen.dev.rhcloud.com (50.19.76.135)' can't be established.
RSA key fingerprint is b4:85:aa:b6:1d:26:0d:3f:9c:39:54:8b:63:98:3e:09.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'myphp-chunchen.dev.rhcloud.com' (RSA) to the list of known hosts.
rhc     
Your application 'myphp' is now available.

  URL:        http://myphp-chunchen.dev.rhcloud.com/
  SSH to:     5260cef5a51462851e0001d0@myphp-chunchen.dev.rhcloud.com
  Git remote: ssh://5260cef5a51462851e0001d0@myphp-chunchen.dev.rhcloud.com/~/git/myphp.git/
  Cloned to:  /home/rayzhang/Work/myphp

Run 'rhc show-app myphp' for more details about your app.
[rayzhang@ray Work]$ rhc snapshot save myphp
Pulling down a snapshot to myphp.tar.gz...
Creating and sending tar.gz
Write failed: Broken pipe
Error in trying to save snapshot. You can try to save manually by running:
ssh 5260cef5a51462851e0001d0@myphp-chunchen.dev.rhcloud.com 'snapshot' > myphp.tar.gz
Comment 8 Lei Zhang 2013-10-18 02:54:02 EDT
Created attachment 813632 [details]
devenv_3912 broker development.log
Comment 9 Lei Zhang 2013-10-18 02:55:05 EDT
Created attachment 813633 [details]
devenv_3912 node platform.log
Comment 10 Rob Millner 2013-10-23 14:52:55 EDT
I can't get this problem to reproduce on any of the following networks from my laptop:
1. Locally on a devenv (loopback)
2. Mountain View Office, flaky network (>200ms latency, packet loss)
3. Mountain View Office, normal network
4. At home on cable-modem

I'm chaining the commands together so that the snapshot executes immediately after the app create call returns.  Also tried it entering the commands one after the other.

devenv_3935, running rhc tools directly from the rhc repo.

$ rhc app create myphp php-5.3 postgresql-9.2 ; rhc snapshot save myphp
Application Options
-------------------
  Domain:     rmillner0160
  Cartridges: php-5.3, postgresql-9.2
  Gear Size:  default
  Scaling:    no

Creating application 'myphp' ... done

  PostgreSQL 9.2 database added.  Please make note of these credentials:

   Root User: admin4jrssl6
   Root Password: iTc4_SqJ1Z2Q
   Database Name: myphp

Connection URL: postgresql://$OPENSHIFT_POSTGRESQL_DB_HOST:$OPENSHIFT_POSTGRESQL_DB_PORT

Waiting for your DNS name to be available ... done

Cloning into 'myphp'...

Your application 'myphp' is now available.

  URL:        http://myphp-rmillner0160.dev.rhcloud.com/
  SSH to:     526816d766e140c78e000039@myphp-rmillner0160.dev.rhcloud.com
  Git remote: ssh://526816d766e140c78e000039@myphp-rmillner0160.dev.rhcloud.com/~/git/myphp.git/
  Cloned to:  /extra/tmp/myphp

Run 'rhc show-app myphp' for more details about your app.
Pulling down a snapshot to myphp.tar.gz...
Creating and sending tar.gz

RESULT:
Success
Comment 11 Rob Millner 2013-10-23 14:59:45 EDT
Lets try this to debug the problem.

1. Create a script to run ssh in verbose mode.

cat > ssh.sh <<EOF
#!/bin/bash
ssh -vvv "\$@"
EOF

chmod +x ssh.sh


2. Use the script above as the ssh command for the snapshot.

rhc snapshot save myphp --ssh ./ssh.sh &> debug.txt


3. Send or attach debug.txt
Comment 12 Lei Zhang 2013-10-24 04:06:37 EDT
After myphp53 created, run 'rhc snapshot save myphp --ssh ./ssh.sh &> debug.txt'

[rayzhang@ray Work]$ cat debug.txt 
Pulling down a snapshot to myphp53.tar.gz...
OpenSSH_5.3p1, OpenSSL 1.0.0-fips 29 Mar 2010
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
ssh: Could not resolve hostname : Name or service not known
Error in trying to save snapshot. You can try to save manually by running:
./ssh.sh 5268d453fa7bab8715000183@myphp53-chunchen.dev.rhcloud.com 'snapshot' > myphp53.tar.gz
Comment 13 Rob Millner 2013-10-24 13:24:37 EDT
This is a local machine or site issue.  Is nscd running?  If so, that's likely the root of the problem.

The rhc tool does a direct DNS lookup on the application name and should not return until the name is available; but ssh is using the "gethostbyname" call which uses the local name service configuration.

Nothing on the node or broker will fix this issue.
Comment 14 N Bhat 2014-06-07 06:36:32 EDT
I encountered error while using 'rhc snapshot save' and forum searches did not help. Though this is old & Status is CLOSED, here is what I found and it worked after a 'rhc setup'!


E:\winapps\XYZ>rhc snapshot save XYZ
Password: *********

Pulling down a snapshot to XYZ.tar.gz...
Error in trying to save snapshot. You can try to save manually by running:
ssh 5243c3c25973caeb5a0000f7@XYZ-iop.rhcloud.com 'snapshot' > XYZ.tar.gz


RUN THE SETUP AGAIN:-

E:\winapps\XYZ>rhc setup
OpenShift Client Tools (RHC) Setup Wizard
:
:
Login to openshift.redhat.com: <account-email>
Password: *********
:
Your public ssh key must be uploaded to the OpenShift server.  Would you like us to upload it for you? (yes/no) yes

You can enter a name for your key, or leave it blank to use the default name. Using the same name as an existing key
will overwrite the old key.

Current Keys:
:
Checking for applications ... found
    * XYZ - http://XYZ.rhcloud.com/
    * xyz - http://xyz.rhcloud.com/

Thank you for setting up your system.  You can rerun this at any time by calling 'rhc setup'.


E:\winapps\XYZ>
E:\winapps\XYZ>rhc snapshot save XYZ
Password: *********

Pulling down a snapshot to XYZ.tar.gz...

RESULT:
Success

Note You need to log in before you can comment on or make changes to this bug.