Bug 975540

Summary: app create exits with code 504: D, [2013-06-18T13:32:10.694874 #96217] DEBUG -- : Response did not include a message from server: invalid character at "<html><bod"
Product: OpenShift Online Reporter: Dis <redhatbug>
Component: PodAssignee: Dan McPherson <dmcphers>
Status: CLOSED UPSTREAM QA Contact: libra bugs <libra-bugs>
Severity: low Docs Contact:
Priority: low    
Version: 2.xCC: hasari, redhatbug
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-08-27 18:46:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dis 2013-06-18 17:53:32 UTC
Description of problem:

When creating an app with --from-code, the instance is created but the command exits with a "504" error.

How reproducible:
100%

Steps to Reproduce:
1. rhc --debug app create ttrss2 php-5.3 postgresql-8.4 cron-1.4 --from-code=https://github.com/disconn3ct/tiny_tiny_rss-openshift-quickstart.git
2.
3.

Actual results:
...{cut}
Application Options
-------------------
DEBUG: Getting all domains
  Namespace:   foo
  Cartridges:  php-5.3, postgresql-8.4, cron-1.4
  Source Code: https://github.com/disconn3ct/tiny_tiny_rss-openshift-quickstart.git
  Gear Size:   default
  Scaling:     no

Creating application 'ttrss2' ... DEBUG: Creating application 'ttrss2' with these options - {:initial_git_url=>"https://github.com/disconn3ct/tiny_tiny_rss-openshift-quickstart.git", :cartridges=>["php-5.3", "postgresql-8.4", "cron-1.4"]}
DEBUG: Adding application ttrss2 to domain foo
DEBUG: Request POST https://openshift.redhat.com/broker/rest/domains/foo/applications
DEBUG:    code 504 301678 ms
D, [2013-06-18T13:32:10.694874 #96217] DEBUG -- : Response did not include a message from server: invalid character at "<html><bod"
Server returned an unexpected error code: 504

Expected results:
No errors, completed creation.

Additional info:

Might be unrelated, but post_start_postgresql-8.4 is not being called, per php/logs/deploy.log (from the action_hooks in the repo):
Entering build.
Configuring TTRSS: Configuration
Configuring Sphinx: Directories
Configuring Sphinx: Configuration
Leaving build.
Entering post-deploy
Starting Indexer
Stopping Searchd
Starting Searchd
Leaving post-deploy

Comment 1 Dis 2013-06-18 18:45:50 UTC
Just tried without from-code and got the same error:
[dis:~/Source] $ rhc app create ttrss2 php-5.3 postgresql-8.4 cron-1.4 
Application Options
-------------------
  Namespace:  foo
  Cartridges: php-5.3, postgresql-8.4, cron-1.4
  Gear Size:  default
  Scaling:    no

Creating application 'ttrss2' ... An error occurred while communicating with the server. This problem may only be temporary. Check that you have correctly specified your OpenShift
server 'https://openshift.redhat.com/broker/rest/domains/foo/applications'.
[dis:~/Source] 1 $

Comment 2 Dis 2013-06-18 19:29:02 UTC
Final post for today, I suspect there is a deeper problem somewhere. Now I am getting "application already exists"

Delete ttrss2, list apps (only ttrss, no ttrss2) and then create ttrss2 results in "Creating application 'ttrss2' ... The supplied application name 'ttrss2' already exists"

[dis:~/Source] $ rhc app delete ttrss2
This is a non-reversible action! Your application code and data will be permanently deleted if you continue!

Are you sure you want to delete the application 'ttrss2'? (yes|no): yes

Deleting application 'ttrss2' ... deleted

[dis:~/Source] $ rhc apps
ttrss @ http://ttrss-foo.rhcloud.com/ (uuid: 516c2e43e0b8cd35050001f5)
--------------------------------------------------------------------------
  Created: Apr 15 12:43 PM
  Gears:   1 (defaults to small)
  Git URL: ssh://12345.com/~/git/ttrss.git/
  SSH:     12345.com

  php-5.3 (PHP 5.3)
  -----------------
    Gears: Located with cron-1.4, postgresql-8.4

  cron-1.4 (Cron 1.4)
  -------------------
    Gears: Located with php-5.3, postgresql-8.4

  postgresql-8.4 (PostgreSQL Database 8.4)
  ----------------------------------------
    Gears:          Located with php-5.3, cron-1.4
    Connection URL: postgresql://$OPENSHIFT_POSTGRESQL_DB_HOST:$OPENSHIFT_POSTGRESQL_DB_PORT/
    Database Name:  ttrss
    Password:       foopassword
    Username:       admin

You have 1 applications
[dis:~/Source] $ rhc app create ttrss2 php-5.3 postgresql-8.4 cron-1.4 --from-code=https://github.com/disconn3ct/tiny_tiny_rss-openshift-quickstart.git --timeout=9999
Application Options
-------------------
  Namespace:   foo
  Cartridges:  php-5.3, postgresql-8.4, cron-1.4
  Source Code: https://github.com/disconn3ct/tiny_tiny_rss-openshift-quickstart.git
  Gear Size:   default
  Scaling:     no

Creating application 'ttrss2' ... The supplied application name 'ttrss2' already exists

Comment 3 Abhishek Gupta 2013-06-18 22:03:49 UTC
One thing that I noticed when trying to create an application using the git repo specified is that there is a syntax error in line 19 of this file --> https://github.com/disconn3ct/tiny_tiny_rss-openshift-quickstart/blob/master/.openshift/action_hooks/post_start_postgresql-8.4

There is a space missing before the closing square bracket of the if condition block. This is tripping up the post-configure step and causing a rollback to happen. The if condition needs to be like:

if [ ${OPENSHIFT_POSTGRESQL_DB_HOST}"x" == "x" ]


On the other issue of the application still being shown as existing even after deletion - I wasn't able to reproduce that issue.

Comment 4 Hiro Asari 2013-06-19 16:12:18 UTC
A similar syntax error exists on https://github.com/disconn3ct/tiny_tiny_rss-openshift-quickstart/blob/master/.openshift/action_hooks/post_deploy#L10.

Furthermore, https://github.com/disconn3ct/tiny_tiny_rss-openshift-quickstart/blob/master/.openshift/action_hooks/post_start_postgresql-8.4#L28 needs to be fixed. For example, there should be a space after '-f', '-h', etc., and there is a typo (${WOPENSHIFT_POSTGRESQL_DB_PASSWORD}).

Please consult http://www.postgresql.org/docs/8.4/static/app-psql.html to construct an appropriate command.

Comment 5 Dis 2013-06-19 16:35:45 UTC
For the 'already exists', there was a very long delay while creating before it returned the error. I haven't had time to do anything today but I suspect there was a (temporary?) problem upstream, since it occurred with or without the from-code.

I appreciate the syntax/typo checks, but fwiw post_deploy runs correctly despite that error:
Entering post-deploy
Starting Indexer
Stopping Searchd
Starting Searchd
Leaving post-deploy

The post_start_postgresql (with or without -8.4) is not automatically called at all. (As you see, the very very first thing it does is log it's presence.)

When called by hand, it completes correctly - the 'exit 1' is never triggered. 

I appreciate the look, but the bigger issue is "<html><bod"..

Comment 6 Abhishek Gupta 2013-06-19 20:32:48 UTC
I just tried with the latest version of the git repo and the application creation was successful and I got to the application's login screen. 

Are you still experiencing problems with the application creation?

Comment 8 Dis 2013-06-20 13:13:24 UTC
It is creating the app correctly, but returning 504:
[dis:~/Source] $ rhc app create ttrss2 php-5.3 postgresql-8.4 cron-1.4 --from-code=https://github.com/disconn3ct/tiny_tiny_rss-openshift-quickstart.git --timeout=9999
Application Options
-------------------
  Namespace:   foo
  Cartridges:  php-5.3, postgresql-8.4, cron-1.4
  Source Code: https://github.com/disconn3ct/tiny_tiny_rss-openshift-quickstart.git
  Gear Size:   default
  Scaling:     no

Creating application 'ttrss2' ... Server returned an unexpected error code: 504

At this point, I'm getting a 500 page from the webserver, It looks like the 504 came before the action hooks finished (or started)

[dis:~/Source/tiny_tiny_rss-openshift-quickstart] master ± rhc tail ttrss2
Could not find any files matching glob

When I ssh'd in, tail -f */logs/* had deploy.log just leaving build. So it is (eventually) building correctly, but the broker returns 504 before it even starts the hooks. (There is a fairly long delay before the 504. I can time it and/or run with debug if that helps but it is a couple minutes at least.)

Comment 9 Abhishek Gupta 2013-06-20 17:39:27 UTC
This is a limitation of the way things are implemented currently. When creating an application with multiple cartridges and using --from-code to specify the source code repository, the entire process can take a LONG time and trigger a timeout on the broker (multiple levels with different timeout values).

In the short-term, for quickstarts that take longer to configure/setup, the workaround is to create the application with the required components and then add the code later (add a remote git repo) and do a git push. Worst case, the application can be created with just the php cartridge and then the other cartridges can be added in the next step, followed by a third step to add the remote repo and do a git push.

In the mid-to-long term (refer trello card --> https://trello.com/c/33dEURdn) we would like to consider executing such tasks using a scheduler that is either not impacted by timeouts or handles timeout situations better instead of simply reporting back an error.

Please let us know if the workaround stated fixes your problem.