Description of problem: Given a normal app embedded with postgresql-9.2 created(e.g.,myphp53), do save snapshot, failed to save snapshot. after saving failed, can not access via rhcsh. [rayzhang@ray Work]$ rhc snapshot save myphp53 Pulling down a snapshot to myphp53.tar.gz... Creating and sending tar.gz Write failed: Broken pipe Error in trying to save snapshot. You can try to save manually by running: ssh 525f76c897b18b043a000328.rhcloud.com 'snapshot' > myphp53.tar.gz [rayzhang@ray Work]$ rhc app ssh myphp53 Connecting to 525f76c897b18b043a000328.rhcloud.com ... Write failed: Connection reset by peer Note: 1. Met this issue in nodejs-0.10/diy-0.1/php-5.3 non-scalable and jbossas-7/php-5.3/nodejs-0.10/ruby-1.8 scalable app. 2. After creating app with postgresql-9.2 or mondify the data of psql, IMMEDIATELY do save or restore app, it's easy to reproduce this issue. Version-Release number of selected component (if applicable): devenv_3907 How reproducible: sometimes Steps to Reproduce: 1.create one app embedded with postgresql-9.2 #rhc app create myphp53 php-5.3 postgresql-9.2 2.do save snapshot #rhc snapshot save myphp53 3. Actual results: Failed to save snapshot for app with postgresql-9.2 Expected results: It should succeed to save snapshot for app with postgresql-9.2 Additional info:
Created attachment 813205 [details] broker development.log
It seems like the underlying issue is that a snapshot is requested while another cartridge operation is running and the two conflict; however, I'm not able to reproduce the issue. Here's the command I'm using: rhc app-delete --confirm myphp53; rm -f myphp53.tar.gz ; rhc app create myphp53 php-5.3 postgresql-9.2 --no-git --no-dns; rhc snapshot save myphp53 ...and re-running it over and over again. I'm going to put that in a loop for an hour and see if we can make it happen.
Two hours in a loop and I can't reproduce it. Attaching the script I was using.
Created attachment 813556 [details] Script to create and snapshot in a loop Touch the file "KEEPGOING" and run the script like this: touch KEEPGOING nohup create_and_snapshot.sh & To monitor the output: tail -f nohup.out To stop the script gently, remove the KEEPGOING file. rm -f KEEPGOING ...it will finish up the current loop and exit. If the snapshot file is not created, then the script will exit after packaging up the entire contents of /var/log to /var/lib/openshift/logs.tgz.
Created attachment 813566 [details] try harder to break the snapshot command Similar to the last attachment with a few tweaks. I added idle and simultaneous unidle and snapshot to see if I can force a failure by overlapping gear operations.
Updated the script to try and force a conflict and still no luck breaking snapshots. Back to Q/E to see if you can recreate the problem.
devenv_3912 still can reproduce this issue. after create php with postgresql-9.2, IMMEDIATELY do save snapshot, it's easy to reproduce. [rayzhang@ray Work]$ rhc app create myphp php-5.3 postgresql-9.2 Application Options ------------------- Namespace: chunchen Cartridges: php-5.3, postgresql-9.2 Gear Size: default Scaling: no Creating application 'myphp' ... done PostgreSQL 9.2 database added. Please make note of these credentials: Root User: admindbh5ay9 Root Password: 1axqLG4gRr9h Database Name: myphp Connection URL: postgresql://$OPENSHIFT_POSTGRESQL_DB_HOST:$OPENSHIFT_POSTGRESQL_DB_PORT Waiting for your DNS name to be available ... done Cloning into 'myphp'... The authenticity of host 'myphp-chunchen.dev.rhcloud.com (50.19.76.135)' can't be established. RSA key fingerprint is b4:85:aa:b6:1d:26:0d:3f:9c:39:54:8b:63:98:3e:09. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'myphp-chunchen.dev.rhcloud.com' (RSA) to the list of known hosts. rhc Your application 'myphp' is now available. URL: http://myphp-chunchen.dev.rhcloud.com/ SSH to: 5260cef5a51462851e0001d0.rhcloud.com Git remote: ssh://5260cef5a51462851e0001d0.rhcloud.com/~/git/myphp.git/ Cloned to: /home/rayzhang/Work/myphp Run 'rhc show-app myphp' for more details about your app. [rayzhang@ray Work]$ rhc snapshot save myphp Pulling down a snapshot to myphp.tar.gz... Creating and sending tar.gz Write failed: Broken pipe Error in trying to save snapshot. You can try to save manually by running: ssh 5260cef5a51462851e0001d0.rhcloud.com 'snapshot' > myphp.tar.gz
Created attachment 813632 [details] devenv_3912 broker development.log
Created attachment 813633 [details] devenv_3912 node platform.log
I can't get this problem to reproduce on any of the following networks from my laptop: 1. Locally on a devenv (loopback) 2. Mountain View Office, flaky network (>200ms latency, packet loss) 3. Mountain View Office, normal network 4. At home on cable-modem I'm chaining the commands together so that the snapshot executes immediately after the app create call returns. Also tried it entering the commands one after the other. devenv_3935, running rhc tools directly from the rhc repo. $ rhc app create myphp php-5.3 postgresql-9.2 ; rhc snapshot save myphp Application Options ------------------- Domain: rmillner0160 Cartridges: php-5.3, postgresql-9.2 Gear Size: default Scaling: no Creating application 'myphp' ... done PostgreSQL 9.2 database added. Please make note of these credentials: Root User: admin4jrssl6 Root Password: iTc4_SqJ1Z2Q Database Name: myphp Connection URL: postgresql://$OPENSHIFT_POSTGRESQL_DB_HOST:$OPENSHIFT_POSTGRESQL_DB_PORT Waiting for your DNS name to be available ... done Cloning into 'myphp'... Your application 'myphp' is now available. URL: http://myphp-rmillner0160.dev.rhcloud.com/ SSH to: 526816d766e140c78e000039.rhcloud.com Git remote: ssh://526816d766e140c78e000039.rhcloud.com/~/git/myphp.git/ Cloned to: /extra/tmp/myphp Run 'rhc show-app myphp' for more details about your app. Pulling down a snapshot to myphp.tar.gz... Creating and sending tar.gz RESULT: Success
Lets try this to debug the problem. 1. Create a script to run ssh in verbose mode. cat > ssh.sh <<EOF #!/bin/bash ssh -vvv "\$@" EOF chmod +x ssh.sh 2. Use the script above as the ssh command for the snapshot. rhc snapshot save myphp --ssh ./ssh.sh &> debug.txt 3. Send or attach debug.txt
After myphp53 created, run 'rhc snapshot save myphp --ssh ./ssh.sh &> debug.txt' [rayzhang@ray Work]$ cat debug.txt Pulling down a snapshot to myphp53.tar.gz... OpenSSH_5.3p1, OpenSSL 1.0.0-fips 29 Mar 2010 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug2: ssh_connect: needpriv 0 ssh: Could not resolve hostname : Name or service not known Error in trying to save snapshot. You can try to save manually by running: ./ssh.sh 5268d453fa7bab8715000183.rhcloud.com 'snapshot' > myphp53.tar.gz
This is a local machine or site issue. Is nscd running? If so, that's likely the root of the problem. The rhc tool does a direct DNS lookup on the application name and should not return until the name is available; but ssh is using the "gethostbyname" call which uses the local name service configuration. Nothing on the node or broker will fix this issue.
I encountered error while using 'rhc snapshot save' and forum searches did not help. Though this is old & Status is CLOSED, here is what I found and it worked after a 'rhc setup'! E:\winapps\XYZ>rhc snapshot save XYZ Password: ********* Pulling down a snapshot to XYZ.tar.gz... Error in trying to save snapshot. You can try to save manually by running: ssh 5243c3c25973caeb5a0000f7.com 'snapshot' > XYZ.tar.gz RUN THE SETUP AGAIN:- E:\winapps\XYZ>rhc setup OpenShift Client Tools (RHC) Setup Wizard : : Login to openshift.redhat.com: <account-email> Password: ********* : Your public ssh key must be uploaded to the OpenShift server. Would you like us to upload it for you? (yes/no) yes You can enter a name for your key, or leave it blank to use the default name. Using the same name as an existing key will overwrite the old key. Current Keys: : Checking for applications ... found * XYZ - http://XYZ.rhcloud.com/ * xyz - http://xyz.rhcloud.com/ Thank you for setting up your system. You can rerun this at any time by calling 'rhc setup'. E:\winapps\XYZ> E:\winapps\XYZ>rhc snapshot save XYZ Password: ********* Pulling down a snapshot to XYZ.tar.gz... RESULT: Success