Bug 964212 - openshift-node-web-proxy crashes while loading large number of apps
Summary: openshift-node-web-proxy crashes while loading large number of apps
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 1.2.0
Hardware: All
OS: Linux
low
high
Target Milestone: ---
: ---
Assignee: Jason DeTiberus
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-05-17 14:45 UTC by Jeremy Eder
Modified: 2023-09-14 01:44 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-02-04 18:44:57 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Jeremy Eder 2013-05-17 14:45:19 UTC
Using May 10 puddle.

While starting my server that has about 3000 php apps, eventually I get the error below, and the openshift-gears service stops starting gears.
If I restart the openshift-gears service, it begins starting gears again.

DEBUG: events.js:48
        throw arguments[1]; // Unhandled 'error' event
                       ^
Error: EACCES, open '/var/log/node-web-proxy/access.log'

DEBUG: Program node /usr/lib/node_modules/openshift-node-web-proxy/bin/web-proxy.js --config /etc/openshift/web-proxy-config.json exited with code 1

DEBUG: Starting child process with 'node /usr/lib/node_modules/openshift-node-web-proxy/bin/web-proxy.js --config /etc/openshift/web-proxy-config.json'
1368799446533:INFO:[17/May/2013:10:04:06 -0400] - ProxyServer using config '/etc/openshift/web-proxy-config.json'

Another problem with this is that when I started openshift-gears again, it starts from the beginning, goes through all the apps in serial...meaning it takes even more time to pick up where it left off.


I found this was triggered at least once here:  https://bugzilla.redhat.com/show_bug.cgi?id=797140

But it doesn't look like it is the same cause.  Any ideas how to fix/avoid this ?

Comment 2 Jason DeTiberus 2013-05-22 16:23:49 UTC
I am unable to reproduce the bugs here on a devenv or on the host in question.  Also, all relevant logs have either been rotated or overwritten, so I am not able to see the failure.

If you are able to reproduce, please attach the boot.log, audit.log and node-web-proxy logs.

I'll verify that the node-web-proxy is setting proper boot order and see if there are adjustments that can be made to the openshift-gears start process to only attempt to start gears that aren't already running.

Comment 3 openshift-github-bot 2013-05-23 21:39:26 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/b199ff58c613d97d432872973c6a687960528496
<node-proxy,port-proxy> Bug 964212 - Fix init script dependencies

https://bugzilla.redhat.com/show_bug.cgi?id=964212

Add INIT INFO to node-proxy and port-proxy to make sure required
services are started before init scripts are called during startup

Fix references to port-proxy in node-proxy init script

Comment 5 Luke Meyer 2013-06-28 18:36:45 UTC
Updates from comment 3 are rebased into OSE 1.2. Can you re-test to see if this made a difference or if you can supply the logs requested on failure?

Comment 6 Brenton Leanhardt 2013-08-15 14:44:15 UTC
Has upstream noticed this problem?

Comment 8 Mrunal Patel 2013-09-05 16:05:31 UTC
Brenton, Luke,
We haven't seen this particular issue in upstream. It might be easier to go look at a system that has this issue to debug it.

Thanks,
Mrunal

Comment 9 Luke Meyer 2013-09-05 18:07:51 UTC
Jeremy, is it possible for you to reproduce this in any of your work? No one else seems to be able to do so. Otherwise we might have to close it until someone comes up against it.

If reproduceable, attaching all the logs under /var/log/openshift/node would help.

Comment 12 Red Hat Bugzilla 2023-09-14 01:44:16 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days


Note You need to log in before you can comment on or make changes to this bug.