Bug 1464125 - logshifter indefinite hang
logshifter indefinite hang
Status: CLOSED EOL
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging (Show other bugs)
2.2.0
Unspecified Unspecified
urgent Severity urgent
: ---
: ---
Assigned To: Abhishek Gupta
Xia Zhao
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-22 09:27 EDT by Robert Bost
Modified: 2017-08-11 18:24 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-06-29 15:33:41 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Robert Bost 2017-06-22 09:27:02 EDT
Description of problem: When logshifter receives a significant amount of logging, it can hang indefinitely.

How reproducible: Always


Steps to Reproduce:

1) while true; do echo Testing; done | /usr/bin/logshifter -tag testing
2) Occasionally run `wc -l app-root/logs/testing.log` to see when logshifter stops writing logs.

Actual results:

Logshifter stops indefinitely. Some gears utilize a unix pipe between logshifter and the application (like Tomcat). This causes the pipe to "back-up" and block threads inside the application.

Expected results:

I would expect some slowness when logshifter is receiving a gratuitous amount of logging, however, not the indefinite hang.
Comment 2 Eric Rich 2017-06-22 09:44:21 EDT
Developers control the process within gears, and can choose where logs go / get directed! 

By controlling what goes to standard out or does not go to standard out, a developer can choose to use or not use logshifter (for all cartridge the product ships). 

So as a workaround to this, a customer can use tools with in cartridge frameworks to log to files directly and avoid logshifter. 

If done, your on the hook to control / maintain your logs (and roll them), as this will likely impact gear storage. 

With that said, most frameworks (like java) have good libraries (like log4j) that deal with this for you, and if you know your logging rate / storage limits, you can set well defined bounds for log retention.

As a result the severity of this can be set to "low"!
Comment 4 Eric Rich 2017-06-29 15:33:41 EDT
OpenShift 2.2 is EOL and given  that this has a viable workarround, we should close this as WONTFIX - EOL.
Comment 5 Dan Mace 2017-07-20 14:07:45 EDT
Can you provide the logshifter configuration used in the test scenario?
Comment 6 Dan Mace 2017-07-20 14:18:58 EDT
Also, if you happen to be able to reproduce this somewhere again, could you try sending a SIGQUIT to the stuck logshifter process and attaching the resulting stack dump here?
Comment 7 Dan Mace 2017-07-21 11:17:50 EDT
I think I tracked down the problem and the fix. https://github.com/openshift/origin-server/pull/6449

Note You need to log in before you can comment on or make changes to this bug.