Bug 1464125

Summary:	logshifter indefinite hang
Product:	OpenShift Container Platform	Reporter:	Robert Bost <rbost>
Component:	Logging	Assignee:	Abhishek Gupta <abhgupta>
Status:	CLOSED EOL	QA Contact:	Xia Zhao <xiazhao>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	2.2.0	CC:	abhgupta, aos-bugs, byount, dmace, erich, gpei, pweil, rbost, rthrashe, xiazhao
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-06-29 19:33:41 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Robert Bost 2017-06-22 13:27:02 UTC

Description of problem: When logshifter receives a significant amount of logging, it can hang indefinitely.

How reproducible: Always


Steps to Reproduce:

1) while true; do echo Testing; done | /usr/bin/logshifter -tag testing
2) Occasionally run `wc -l app-root/logs/testing.log` to see when logshifter stops writing logs.

Actual results:

Logshifter stops indefinitely. Some gears utilize a unix pipe between logshifter and the application (like Tomcat). This causes the pipe to "back-up" and block threads inside the application.

Expected results:

I would expect some slowness when logshifter is receiving a gratuitous amount of logging, however, not the indefinite hang.

Comment 2 Eric Rich 2017-06-22 13:44:21 UTC

Developers control the process within gears, and can choose where logs go / get directed! 

By controlling what goes to standard out or does not go to standard out, a developer can choose to use or not use logshifter (for all cartridge the product ships). 

So as a workaround to this, a customer can use tools with in cartridge frameworks to log to files directly and avoid logshifter. 

If done, your on the hook to control / maintain your logs (and roll them), as this will likely impact gear storage. 

With that said, most frameworks (like java) have good libraries (like log4j) that deal with this for you, and if you know your logging rate / storage limits, you can set well defined bounds for log retention.

As a result the severity of this can be set to "low"!

Comment 4 Eric Rich 2017-06-29 19:33:41 UTC

OpenShift 2.2 is EOL and given  that this has a viable workarround, we should close this as WONTFIX - EOL.

Comment 5 Dan Mace 2017-07-20 18:07:45 UTC

Can you provide the logshifter configuration used in the test scenario?

Comment 6 Dan Mace 2017-07-20 18:18:58 UTC

Also, if you happen to be able to reproduce this somewhere again, could you try sending a SIGQUIT to the stuck logshifter process and attaching the resulting stack dump here?

Comment 7 Dan Mace 2017-07-21 15:17:50 UTC

I think I tracked down the problem and the fix. https://github.com/openshift/origin-server/pull/6449