Red Hat Bugzilla – Bug 964212
openshift-node-web-proxy crashes while loading large number of apps
Last modified: 2017-03-08 12:35 EST
Using May 10 puddle.
While starting my server that has about 3000 php apps, eventually I get the error below, and the openshift-gears service stops starting gears.
If I restart the openshift-gears service, it begins starting gears again.
throw arguments; // Unhandled 'error' event
Error: EACCES, open '/var/log/node-web-proxy/access.log'
DEBUG: Program node /usr/lib/node_modules/openshift-node-web-proxy/bin/web-proxy.js --config /etc/openshift/web-proxy-config.json exited with code 1
DEBUG: Starting child process with 'node /usr/lib/node_modules/openshift-node-web-proxy/bin/web-proxy.js --config /etc/openshift/web-proxy-config.json'
1368799446533:INFO:[17/May/2013:10:04:06 -0400] - ProxyServer using config '/etc/openshift/web-proxy-config.json'
Another problem with this is that when I started openshift-gears again, it starts from the beginning, goes through all the apps in serial...meaning it takes even more time to pick up where it left off.
I found this was triggered at least once here: https://bugzilla.redhat.com/show_bug.cgi?id=797140
But it doesn't look like it is the same cause. Any ideas how to fix/avoid this ?
I am unable to reproduce the bugs here on a devenv or on the host in question. Also, all relevant logs have either been rotated or overwritten, so I am not able to see the failure.
If you are able to reproduce, please attach the boot.log, audit.log and node-web-proxy logs.
I'll verify that the node-web-proxy is setting proper boot order and see if there are adjustments that can be made to the openshift-gears start process to only attempt to start gears that aren't already running.
Commit pushed to master at https://github.com/openshift/origin-server
<node-proxy,port-proxy> Bug 964212 - Fix init script dependencies
Add INIT INFO to node-proxy and port-proxy to make sure required
services are started before init scripts are called during startup
Fix references to port-proxy in node-proxy init script
Updates from comment 3 are rebased into OSE 1.2. Can you re-test to see if this made a difference or if you can supply the logs requested on failure?
Has upstream noticed this problem?
We haven't seen this particular issue in upstream. It might be easier to go look at a system that has this issue to debug it.
Jeremy, is it possible for you to reproduce this in any of your work? No one else seems to be able to do so. Otherwise we might have to close it until someone comes up against it.
If reproduceable, attaching all the logs under /var/log/openshift/node would help.