Bug 964212 - openshift-node-web-proxy crashes while loading large number of apps [NEEDINFO]
openshift-node-web-proxy crashes while loading large number of apps
Status: CLOSED INSUFFICIENT_DATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers (Show other bugs)
1.2.0
All Linux
low Severity high
: ---
: ---
Assigned To: Jason DeTiberus
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-05-17 10:45 EDT by Jeremy Eder
Modified: 2017-03-08 12 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-02-04 13:44:57 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
jeder: needinfo? (jeder)


Attachments (Terms of Use)

  None (edit)
Description Jeremy Eder 2013-05-17 10:45:19 EDT
Using May 10 puddle.

While starting my server that has about 3000 php apps, eventually I get the error below, and the openshift-gears service stops starting gears.
If I restart the openshift-gears service, it begins starting gears again.

DEBUG: events.js:48
        throw arguments[1]; // Unhandled 'error' event
                       ^
Error: EACCES, open '/var/log/node-web-proxy/access.log'

DEBUG: Program node /usr/lib/node_modules/openshift-node-web-proxy/bin/web-proxy.js --config /etc/openshift/web-proxy-config.json exited with code 1

DEBUG: Starting child process with 'node /usr/lib/node_modules/openshift-node-web-proxy/bin/web-proxy.js --config /etc/openshift/web-proxy-config.json'
1368799446533:INFO:[17/May/2013:10:04:06 -0400] - ProxyServer using config '/etc/openshift/web-proxy-config.json'

Another problem with this is that when I started openshift-gears again, it starts from the beginning, goes through all the apps in serial...meaning it takes even more time to pick up where it left off.


I found this was triggered at least once here:  https://bugzilla.redhat.com/show_bug.cgi?id=797140

But it doesn't look like it is the same cause.  Any ideas how to fix/avoid this ?
Comment 2 Jason DeTiberus 2013-05-22 12:23:49 EDT
I am unable to reproduce the bugs here on a devenv or on the host in question.  Also, all relevant logs have either been rotated or overwritten, so I am not able to see the failure.

If you are able to reproduce, please attach the boot.log, audit.log and node-web-proxy logs.

I'll verify that the node-web-proxy is setting proper boot order and see if there are adjustments that can be made to the openshift-gears start process to only attempt to start gears that aren't already running.
Comment 3 openshift-github-bot 2013-05-23 17:39:26 EDT
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/b199ff58c613d97d432872973c6a687960528496
<node-proxy,port-proxy> Bug 964212 - Fix init script dependencies

https://bugzilla.redhat.com/show_bug.cgi?id=964212

Add INIT INFO to node-proxy and port-proxy to make sure required
services are started before init scripts are called during startup

Fix references to port-proxy in node-proxy init script
Comment 5 Luke Meyer 2013-06-28 14:36:45 EDT
Updates from comment 3 are rebased into OSE 1.2. Can you re-test to see if this made a difference or if you can supply the logs requested on failure?
Comment 6 Brenton Leanhardt 2013-08-15 10:44:15 EDT
Has upstream noticed this problem?
Comment 8 Mrunal Patel 2013-09-05 12:05:31 EDT
Brenton, Luke,
We haven't seen this particular issue in upstream. It might be easier to go look at a system that has this issue to debug it.

Thanks,
Mrunal
Comment 9 Luke Meyer 2013-09-05 14:07:51 EDT
Jeremy, is it possible for you to reproduce this in any of your work? No one else seems to be able to do so. Otherwise we might have to close it until someone comes up against it.

If reproduceable, attaching all the logs under /var/log/openshift/node would help.

Note You need to log in before you can comment on or make changes to this bug.