Bug 1020148

Created attachment 813205 [details]
broker development.log

It seems like the underlying issue is that a snapshot is requested while another cartridge operation is running and the two conflict; however, I'm not able to reproduce the issue.

Here's the command I'm using:
rhc app-delete --confirm myphp53; rm -f myphp53.tar.gz ; rhc app create myphp53 php-5.3 postgresql-9.2 --no-git --no-dns; rhc snapshot save myphp53

...and re-running it over and over again.

I'm going to put that in a loop for an hour and see if we can make it happen.

Two hours in a loop and I can't reproduce it.  Attaching the script I was using.

Created attachment 813556 [details]
Script to create and snapshot in a loop

Touch the file "KEEPGOING" and run the script like this:
touch KEEPGOING
nohup create_and_snapshot.sh &

To monitor the output:
tail -f nohup.out

To stop the script gently, remove the KEEPGOING file.
rm -f KEEPGOING
...it will finish up the current loop and exit.

If the snapshot file is not created, then the script will exit after packaging up the entire contents of /var/log to /var/lib/openshift/logs.tgz.

Created attachment 813566 [details]
try harder to break the snapshot command

Similar to the last attachment with a few tweaks. I added idle and simultaneous unidle and snapshot to see if I can force a failure by overlapping gear operations.

Updated the script to try and force a conflict and still no luck breaking snapshots.

Back to Q/E to see if you can recreate the problem.

devenv_3912 still can reproduce this issue. after create php with postgresql-9.2, IMMEDIATELY do save snapshot, it's easy to reproduce.


[rayzhang@ray Work]$ rhc app create myphp php-5.3 postgresql-9.2 
Application Options
-------------------
  Namespace:  chunchen
  Cartridges: php-5.3, postgresql-9.2
  Gear Size:  default
  Scaling:    no

Creating application 'myphp' ... done

  PostgreSQL 9.2 database added.  Please make note of these credentials:

   Root User: admindbh5ay9
   Root Password: 1axqLG4gRr9h
   Database Name: myphp

Connection URL: postgresql://$OPENSHIFT_POSTGRESQL_DB_HOST:$OPENSHIFT_POSTGRESQL_DB_PORT

Waiting for your DNS name to be available ... done

Cloning into 'myphp'...
The authenticity of host 'myphp-chunchen.dev.rhcloud.com (50.19.76.135)' can't be established.
RSA key fingerprint is b4:85:aa:b6:1d:26:0d:3f:9c:39:54:8b:63:98:3e:09.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'myphp-chunchen.dev.rhcloud.com' (RSA) to the list of known hosts.
rhc     
Your application 'myphp' is now available.

  URL:        http://myphp-chunchen.dev.rhcloud.com/
  SSH to:     5260cef5a51462851e0001d0.rhcloud.com
  Git remote: ssh://5260cef5a51462851e0001d0.rhcloud.com/~/git/myphp.git/
  Cloned to:  /home/rayzhang/Work/myphp

Run 'rhc show-app myphp' for more details about your app.
[rayzhang@ray Work]$ rhc snapshot save myphp
Pulling down a snapshot to myphp.tar.gz...
Creating and sending tar.gz
Write failed: Broken pipe
Error in trying to save snapshot. You can try to save manually by running:
ssh 5260cef5a51462851e0001d0.rhcloud.com 'snapshot' > myphp.tar.gz

Created attachment 813632 [details]
devenv_3912 broker development.log

Created attachment 813633 [details]
devenv_3912 node platform.log

I can't get this problem to reproduce on any of the following networks from my laptop:
1. Locally on a devenv (loopback)
2. Mountain View Office, flaky network (>200ms latency, packet loss)
3. Mountain View Office, normal network
4. At home on cable-modem

I'm chaining the commands together so that the snapshot executes immediately after the app create call returns.  Also tried it entering the commands one after the other.

devenv_3935, running rhc tools directly from the rhc repo.

$ rhc app create myphp php-5.3 postgresql-9.2 ; rhc snapshot save myphp
Application Options
-------------------
  Domain:     rmillner0160
  Cartridges: php-5.3, postgresql-9.2
  Gear Size:  default
  Scaling:    no

Creating application 'myphp' ... done

  PostgreSQL 9.2 database added.  Please make note of these credentials:

   Root User: admin4jrssl6
   Root Password: iTc4_SqJ1Z2Q
   Database Name: myphp

Connection URL: postgresql://$OPENSHIFT_POSTGRESQL_DB_HOST:$OPENSHIFT_POSTGRESQL_DB_PORT

Waiting for your DNS name to be available ... done

Cloning into 'myphp'...

Your application 'myphp' is now available.

  URL:        http://myphp-rmillner0160.dev.rhcloud.com/
  SSH to:     526816d766e140c78e000039.rhcloud.com
  Git remote: ssh://526816d766e140c78e000039.rhcloud.com/~/git/myphp.git/
  Cloned to:  /extra/tmp/myphp

Run 'rhc show-app myphp' for more details about your app.
Pulling down a snapshot to myphp.tar.gz...
Creating and sending tar.gz

RESULT:
Success

Lets try this to debug the problem.

1. Create a script to run ssh in verbose mode.

cat > ssh.sh <<EOF
#!/bin/bash
ssh -vvv "\$@"
EOF

chmod +x ssh.sh


2. Use the script above as the ssh command for the snapshot.

rhc snapshot save myphp --ssh ./ssh.sh &> debug.txt


3. Send or attach debug.txt

After myphp53 created, run 'rhc snapshot save myphp --ssh ./ssh.sh &> debug.txt'

[rayzhang@ray Work]$ cat debug.txt 
Pulling down a snapshot to myphp53.tar.gz...
OpenSSH_5.3p1, OpenSSL 1.0.0-fips 29 Mar 2010
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
ssh: Could not resolve hostname : Name or service not known
Error in trying to save snapshot. You can try to save manually by running:
./ssh.sh 5268d453fa7bab8715000183.rhcloud.com 'snapshot' > myphp53.tar.gz

This is a local machine or site issue.  Is nscd running?  If so, that's likely the root of the problem.

The rhc tool does a direct DNS lookup on the application name and should not return until the name is available; but ssh is using the "gethostbyname" call which uses the local name service configuration.

Nothing on the node or broker will fix this issue.

I encountered error while using 'rhc snapshot save' and forum searches did not help. Though this is old & Status is CLOSED, here is what I found and it worked after a 'rhc setup'!


E:\winapps\XYZ>rhc snapshot save XYZ
Password: *********

Pulling down a snapshot to XYZ.tar.gz...
Error in trying to save snapshot. You can try to save manually by running:
ssh 5243c3c25973caeb5a0000f7.com 'snapshot' > XYZ.tar.gz


RUN THE SETUP AGAIN:-

E:\winapps\XYZ>rhc setup
OpenShift Client Tools (RHC) Setup Wizard
:
:
Login to openshift.redhat.com: <account-email>
Password: *********
:
Your public ssh key must be uploaded to the OpenShift server.  Would you like us to upload it for you? (yes/no) yes

You can enter a name for your key, or leave it blank to use the default name. Using the same name as an existing key
will overwrite the old key.

Current Keys:
:
Checking for applications ... found
    * XYZ - http://XYZ.rhcloud.com/
    * xyz - http://xyz.rhcloud.com/

Thank you for setting up your system.  You can rerun this at any time by calling 'rhc setup'.


E:\winapps\XYZ>
E:\winapps\XYZ>rhc snapshot save XYZ
Password: *********

Pulling down a snapshot to XYZ.tar.gz...

RESULT:
Success