Red Hat Bugzilla – Bug 1464125
logshifter indefinite hang
Last modified: 2017-08-11 18:24:14 EDT
Description of problem: When logshifter receives a significant amount of logging, it can hang indefinitely.
How reproducible: Always
Steps to Reproduce:
1) while true; do echo Testing; done | /usr/bin/logshifter -tag testing
2) Occasionally run `wc -l app-root/logs/testing.log` to see when logshifter stops writing logs.
Logshifter stops indefinitely. Some gears utilize a unix pipe between logshifter and the application (like Tomcat). This causes the pipe to "back-up" and block threads inside the application.
I would expect some slowness when logshifter is receiving a gratuitous amount of logging, however, not the indefinite hang.
Developers control the process within gears, and can choose where logs go / get directed!
By controlling what goes to standard out or does not go to standard out, a developer can choose to use or not use logshifter (for all cartridge the product ships).
So as a workaround to this, a customer can use tools with in cartridge frameworks to log to files directly and avoid logshifter.
If done, your on the hook to control / maintain your logs (and roll them), as this will likely impact gear storage.
With that said, most frameworks (like java) have good libraries (like log4j) that deal with this for you, and if you know your logging rate / storage limits, you can set well defined bounds for log retention.
As a result the severity of this can be set to "low"!
OpenShift 2.2 is EOL and given that this has a viable workarround, we should close this as WONTFIX - EOL.
Can you provide the logshifter configuration used in the test scenario?
Also, if you happen to be able to reproduce this somewhere again, could you try sending a SIGQUIT to the stuck logshifter process and attaching the resulting stack dump here?
I think I tracked down the problem and the fix. https://github.com/openshift/origin-server/pull/6449