Bug 1568361

Summary:	Rotate Elasticsearch log files on persistent volume, keeping a configured maximum
Product:	OpenShift Container Platform	Reporter:	Peter Portante <pportant>
Component:	Logging	Assignee:	Jeff Cantrill <jcantril>
Status:	CLOSED ERRATA	QA Contact:	Anping Li <anli>
Severity:	high	Docs Contact:
Priority:	high
Version:	3.9.0	CC:	aos-bugs, dmoessne, rmeggins
Target Milestone:	---	Keywords:	OpsBlocker
Target Release:	3.9.z
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:	undefined	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-06-06 15:46:20 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Peter Portante 2018-04-17 10:58:57 UTC

Currently Elasticsearch log files in a pod are logged to that pods ephemeral disk, but with out any log rotation.  Eventually the pod will fill up, and will crash.  When it crashes, there will be no evidence left of what occurred.

We need to consider logging to the persistent volume instead of to the ephemeral disk, and do with a configured maximum number of files to keep.

Comment 2 openshift-github-bot 2018-04-27 08:35:22 UTC

Commits pushed to master at https://github.com/openshift/openshift-ansible

https://github.com/openshift/openshift-ansible/commit/ea743beca4a272b326e2a5e9a5fc7c71eb7e4303
bug 1568361. Modify persistent directory for logs

https://github.com/openshift/openshift-ansible/commit/6f43f6eeb612d2720522b3ad34d4f554f85c5bb6
Merge pull request #8125 from jcantrill/1568361_persistent_logs

bug 1568361. Modify persistent directory for logs

Comment 3 Jeff Cantrill 2018-04-27 15:07:12 UTC

3.9 cherrypick https://github.com/openshift/openshift-ansible/pull/8180

Comment 4 Jeff Cantrill 2018-04-27 20:18:55 UTC

*** Bug 1506855 has been marked as a duplicate of this bug. ***

Comment 5 openshift-github-bot 2018-04-30 15:18:23 UTC

Commits pushed to master at https://github.com/openshift/origin-aggregated-logging

https://github.com/openshift/origin-aggregated-logging/commit/f909ab29d389a0657fd8e67cebd4565206385d56
bug 1568361. Move ES logs to persistent volume with max backup

https://github.com/openshift/origin-aggregated-logging/commit/6192416cfb346e4f732911088f1dd17049d9ba88
Merge pull request #1108 from jcantrill/1568361_rotate_logs

bug 1568361. Move ES logs to persistent volume with max backup

Comment 6 Jeff Cantrill 2018-04-30 15:21:17 UTC

3.9 cherrypick https://github.com/openshift/origin-aggregated-logging/pull/1127

Comment 8 Anping Li 2018-05-30 09:50:58 UTC

Verified in elasticsearch-3.9.30.

Comment 10 errata-xmlrpc 2018-06-06 15:46:20 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1796