Bug 991824

Description of problem:
/var/log/openshift/node/cgroups-trace.log is completely filling up the /var partition in STG. This looks like a file used for debugging, and not something we want enabled for openshift production.

I noticed the problem last night and so I truncated the files to see how fast they grow. Within 8 hours, they've already grown to be over 3 gigs in size on some hosts.

I've tracked this down to watchman, here's a sample of what we see in that log file:

[...snip...]
August 04 00:59:48 INFO oo_spawn buffer(10/) 012d513770e04943b410a7a32d74b8fe/cpu.stat:nr_periods 0
012d513770e04943b410a7a32d74b8fe/cpu.stat:nr_throttled 0
012d513770e04943b410a7a32d74b8fe/cpu.stat:throttled_time 0
0148fcf65cb74864acf97b550e171aad/cpu.stat:nr_periods 0
0148fcf65cb74864acf97b550e171aad/cpu.stat:nr_throttled 0
0148fcf65cb74864acf97b550e171aad/cpu.stat:throttled_time 0
0188ce785dbe48f48b4a1e9c1443fd42/cpu.stat:nr_periods 0
0188ce785dbe48f48b4a1e9c1443fd42/cpu.stat:nr_throttled 0
[...snip...]


Version-Release number of selected component (if applicable):
rhc-node-1.12.6-1.el6oso.x86_64

How reproducible:
very

Steps to Reproduce:
1. create a lot of apps that are active (90 or so)
2. make sure watchman is running
3. watch the size of that log file grow quickly


Actual results:
This debug log is on by default, and there doesn't seem to be a way to turn it off.

As a result, it's filling up /var on our STG ex-nodes.

Expected results:
debug logs should either not be on by default, or there should at the very least be a way to shut them off.

The way to shut them off should be clearly documented in the release ticket.