Description of problem: /var/log/node-web-proxy/supervisor_log is growing out of control. It appears to have DEBUG enabled AND it's not being rotated. On one of our nodes, it's at 4.5GB and growing. Here is an excerpt. ==================================================== DEBUG: TypeError: Cannot call method 'split' of undefined at _get_routing_key (/usr/lib/node_modules/openshift-node-web-proxy/lib/proxy/ProxyRoutes.js:66:35) at ProxyRoutes.add (/usr/lib/node_modules/openshift-node-web-proxy/lib/proxy/ProxyRoutes.js:160:14) at ProxyRoutes.load (/usr/lib/node_modules/openshift-node-web-proxy/lib/proxy/ProxyRoutes.js:222:10) at /usr/lib/node_modules/openshift-node-web-proxy/lib/proxy/ProxyServer.js:721:44 at Array.forEach (native) at /usr/lib/node_modules/openshift-node-web-proxy/lib/proxy/ProxyServer.js:721:17 at /usr/lib/node_modules/openshift-node-web-proxy/lib/proxy/ProxyServer.js:452:14 at ChildProcess.exithandler (child_process.js:276:7) at ChildProcess.emit (events.js:70:17) at maybeExit (child_process.js:358:16) DEBUG: Program node /usr/lib/node_modules/openshift-node-web-proxy/bin/web-proxy.js --config /etc/openshift/web-proxy-config.json exited with code 1 DEBUG: Starting child process with 'node /usr/lib/node_modules/openshift-node-web-proxy/bin/web-proxy.js --config /etc/openshift/web-proxy-config.json' 1360684772452:INFO:[12/Feb/2013:10:59:32 -0500] - ProxyServer using config '/etc/openshift/web-proxy-config.json' 1360684772465:INFO:[12/Feb/2013:10:59:32 -0500] - Initializing ProxyServer ... 1360684772467:INFO:[12/Feb/2013:10:59:32 -0500] - Stopping protocol servers for: 1360684772478:INFO:[12/Feb/2013:10:59:32 -0500] - Creating protocol server for nonsecure_http_proxy 1360684772479:INFO:[12/Feb/2013:10:59:32 -0500] - Creating websocket server for nonsecure_http_proxy 1360684772481:INFO:[12/Feb/2013:10:59:32 -0500] - Creating protocol server for secure_http_proxy 1360684772522:INFO:[12/Feb/2013:10:59:32 -0500] - Creating websocket server for secure_http_proxy 1360684772523:INFO:[12/Feb/2013:10:59:32 -0500] - Initialized ProxyServer 1360684772523:INFO:[12/Feb/2013:10:59:32 -0500] - Starting protocol servers for: nonsecure_http_proxy,secure_http_proxy 1360684772523:INFO:[12/Feb/2013:10:59:32 -0500] - Starting protocol handler for nonsecure_http_proxy ... 1360684772524:INFO:[12/Feb/2013:10:59:32 -0500] - Starting protocol handler for secure_http_proxy ... 1360684772527:INFO:[12/Feb/2013:10:59:32 -0500] - secure_http_proxy listening on port 8000 1360684772528:INFO:[12/Feb/2013:10:59:32 -0500] - secure_http_proxy listening on port 8443 DEBUG: DEBUG: /usr/lib/node_modules/openshift-node-web-proxy/lib/proxy/ProxyRoutes.js:66 DEBUG: zparts = zuri.split('://')[1].split('/'); ============================================================================ Expected results: This file should not being growing out of control like it does. DEBUG should be turned off for production environments supervisor should setup a logrotate job to rotate this file
Passing to the resident NodeJS expert.
Matt, The basic issue is that there is a failure in parsing some URIs. We need to gather information about which URIs the proxy is failing to parse. Could you turn on debugs using this command and gather a few logs? kill -USR2 `cat /var/run/openshift-node-web-proxy.pid` The same command could be used to turn them back off. These debugs are different from the supervisor debugs that we see in the logs. Thanks, Mrunal
I let this one run for about 5 minutes with the kill -USR2 `cat /var/run/openshift-node-web-proxy.pid` command enabled. Can you tell me if this is what you are expecting to see? If not, I will attempt it again.
Created attachment 703019 [details] supervisor_log-2-26-am.gz here is log file
Created attachment 704292 [details] node supervisor log Here's more logs.
Created attachment 704296 [details] supervisor logs from stg node More supervisor logs.
Created attachment 704311 [details] routes.json from node Here's the routes.json files from the node.
Created attachment 704324 [details] routes.json from node Routes as separate files.
https://github.com/openshift/origin-server/pull/1515
Creating an app name starting with http/https should not cause proxy to fail as in the logs above.
Checked on devenv_2890, 1. tailf /var/log/node-web-proxy/supervisor_log 2. Create some apps 3. Control and access the apps 4. Only few logs generated. Mark bug as fixed. # tailf /var/log/node-web-proxy/supervisor_log 1362395131036:INFO:[04/Mar/2013:06:05:31 -0500] - Routing information was reloaded 1362395178189:INFO:[04/Mar/2013:06:06:18 -0500] - Routing information was reloaded