Bug 1128717 - Awaking of idled nodejs app return 503 for 2-3 minutes
Summary: Awaking of idled nodejs app return 503 for 2-3 minutes
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Image
Version: 2.x
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Maciej Szulik
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-08-11 12:02 UTC by Michal Fojtik
Modified: 2015-05-15 00:44 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-10-10 00:50:24 UTC


Attachments (Terms of Use)

Description Michal Fojtik 2014-08-11 12:02:20 UTC
Description of problem:

This is a clone of this GH issue[1] as more users started reporting this behavior on IRC.

[1] https://github.com/openshift-quickstart/openshift-ghost-quickstart/issues/18

The problem, as described by the users is that the 'ghost' quickstart, after un-idled reports 503 in browser and after few minutes it becomes available again.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. $ rhc app create ghost nodejs-0.10 --env NODE_ENV=production --from-code https://github.com/openshift-quickstart/openshift-ghost-quickstart.git
2. Idle the gear
3. Unindle the gear
4. Visit the website

Actual results:

Reports 503.

Expected results:

Should not report 503, but it should start the nodejs app immediately after the first request that un-idle the app is fired.

Additional info:

I'm currently trying to reproduce the bug myself, so I'm not 100% positive it is reproducable all the time. However, I after looking into the quickstart hooks and code, I don't think the issue is in the quickstart but in the nodejs cart itself.

Comment 1 Michal Fojtik 2014-08-11 12:59:14 UTC
fyi, I was not able to reproduce this issue :(

#!/bin/sh

for i in `seq 1 10`; do
  oo-admin-ctl-gears idlegear 53e8f198910c793d52000009
  sleep 2
  curl -sS http://ghost-mfojtik.dev.rhcloud.com
done 

All tries gave me normal 200, the app was un-idled successfully.
Also, the app was non-scalable (as reported by the users).

So until I'm missing something in idler configuration that is different in PROD vs devenv, I have no idea what could be wrong.

Comment 2 Ben Parees 2014-08-11 13:11:39 UTC
Michal, can you engage w/ ops on this?  Perhaps they know something or can see something in the logs.

Comment 3 Ben Parees 2014-08-11 13:12:14 UTC
(Assigning to Maciej since it's 2.x work)

Comment 4 Balazs Varga 2014-08-11 13:22:58 UTC
I'm able to reproduce it on current PROD.

1., create app
2., restart app

(beginning with restart)

nodejs.log
DEBUG: Sending SIGTERM to child...
DEBUG: Running node-supervisor with
DEBUG:   program 'index.js'
DEBUG:   --watch '/var/lib/openshift/53e8bf7ee0b8cda84b000080/app-root/data/.nodewatch'
DEBUG:   --ignore 'undefined'
DEBUG:   --extensions 'node|js|coffee'
DEBUG:   --exec 'node'
DEBUG: Starting child process with 'node index.js'
DEBUG: Watching directory '/var/lib/openshift/53e8bf7ee0b8cda84b000080/app-root/data/.nodewatch' for changes.
events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: listen EACCES
    at errnoException (net.js:901:11)
    at Server._listen2 (net.js:1020:19)
    at listen (net.js:1061:10)
    at net.js:1143:9
    at dns.js:72:18
    at process._tickDomainCallback (node.js:459:13)
    at process._tickFromSpinner (node.js:390:15)
DEBUG: Program node index.js exited with code 8
DEBUG: Starting child process with 'node index.js'
events.js:72
        throw er; // Unhandled 'error' event
              ^

rhc stop/start works

Comment 5 Maciej Szulik 2014-08-13 13:27:53 UTC
When creating app as described in #c0 do not add --env NODE_ENV=production than after restart the app fails with:

Migrations: Up to date at version 003
events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: listen EACCES
    at errnoException (net.js:901:11)
    at Server._listen2 (net.js:1020:19)
    at listen (net.js:1061:10)
    at net.js:1143:9
    at dns.js:72:18
    at process._tickCallback (node.js:415:13)
    at process._tickFromSpinner (node.js:390:15)
DEBUG: Program node index.js exited with code 8
DEBUG: Starting child process with 'node index.js'

Comment 6 Maciej Szulik 2014-08-13 20:13:03 UTC
The heart of the problem is the NODE_ENV=production variable, 
which restricts node server to use production configuration, 
which is the only one properly configured in the ghost-quickstart. 
So I've added pre_restart_nodejs action hook to the quickstart, 
to make it work immediately on openshift (see [1]),
before the proper solution will hit online, where I've added 
this var as a default see [2].

[1] https://github.com/openshift-quickstart/openshift-ghost-quickstart/pull/23
[2] https://github.com/openshift/origin-server/pull/5712

Comment 7 Yan Du 2014-08-14 03:14:55 UTC
Test on devenv_5074

1. $ rhc app create ghost nodejs-0.10 --env NODE_ENV=production --from-code https://github.com/openshift-quickstart/openshift-ghost-quickstart.git
2. Idle the gear
3. Unindle the gear
4. Visit the website, app could be accessed normally

Move bug to verified.

Comment 8 Maciej Szulik 2014-08-14 12:51:34 UTC
Please retest as I've made small adjustments according to comments on this PR.

Comment 9 openshift-github-bot 2014-08-14 14:03:09 UTC
Commits pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/97124748e41d01ace67df931908755f84ecfdbae
Bug 1128717 - set production as a default node environment

https://github.com/openshift/origin-server/commit/842d58b7de447f2ca689bc575a1dff3347969092
Merge pull request #5712 from soltysh/bug1128717

Merged by openshift-bot

Comment 10 Yan Du 2014-08-15 06:59:24 UTC
Retest on devenv_5079

Issue could not be reproduced. App could work normally in web after idle or unidle.


Note You need to log in before you can comment on or make changes to this bug.