Bug 1030458

Description of problem:

Yesterday, Tim Kramer discovered that the CPU usage of node-web-proxy on an OpenShift Online node was much higher than normal.  Using strace, we found that the process was making thousands of accept() calls per second, all of which were failing because the process was out of file descriptors.

I was able to work around the problem by adjusting the process's limit:

echo -n "Max open files=4096:4096" > /proc/88047/limits

at which point it quickly handled existing connections at settled down to only having about 15 open files.

Version-Release number of selected component (if applicable):

openshift-origin-node-proxy-1.16.1-1.el6oso.noarch

How reproducible:

We have not yet attempted to reproduce this.  I suspect the if you set the ulimit to an artificially low number like 32, you could probably reproduce with a fairly small number of concurrent connections through the proxy.  

Actual result:

It seems that the system was in a state where it was not servicing existing  connections at all, yet still trying to accept new ones.

Expected result:

I would expect that when the process runs out of file descriptors, it should still be able service existing connections (or error out and close them) and simply reject incoming connections until enough file descriptors are closed to handle new connections.