Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1559219

Summary:	[RFE]-Add support for pre-creating indices for the next day to avoid scaling issues
Product:	OpenShift Container Platform	Reporter:	Peter Portante <pportant>
Component:	Logging	Assignee:	Josef Karasek <jkarasek>
Status:	CLOSED UPSTREAM	QA Contact:	Anping Li <anli>
Severity:	high	Docs Contact:
Priority:	high
Version:	3.9.0	CC:	aos-bugs, jcantril, rmeggins, tkatarki, wsun
Target Milestone:	---	Keywords:	OpsBlocker
Target Release:	3.9.z
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-05-11 20:07:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Peter Portante 2018-03-22 01:50:50 UTC

Add support for pre-creating indices for the next day to avoid scaling issues.

If a bulk request contains logs for an index that does not exist in Elasticsearch, those indices will be gratuitously created by ES as part of the bulk request processing.  When there are many indices present in Elasticsearch, this creation process can become very slow.

When a logging cluster has many namespaces (could be in the hundreds, and certainly in the thousands), pre-creating indices has been shown to alleviate the scaling issues by avoiding having bulk requests block for index creation.  The scaling is achieved by creating the indices directly the day before over the course of the day.

Without this work, large clusters with thousands of namespaces can see logging fail at some point.

Comment 3 Peter Portante 2018-04-09 01:33:52 UTC

The following gist is used on production on-line clusters successfully: https://gist.github.com/portante/1203459a9f5f98f4d08dd1958d078d2d

Comment 5 Peter Portante 2018-05-17 02:30:24 UTC

How do you propose to handle starter cluster scaling without this?