Bug 1493002

Summary:	[RFE] Set fluentd buffer parameters
Product:	[oVirt] ovirt-engine-metrics	Reporter:	Shirly Radco <sradco>
Component:	RFEs	Assignee:	Shirly Radco <sradco>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Lukas Svaty <lsvaty>
Severity:	medium	Docs Contact:
Priority:	high
Version:	1.0.5	CC:	bugs, lveyde, rmeggins, ylavi
Target Milestone:	ovirt-4.1.7	Keywords:	FutureFeature
Target Release:	1.0.7.1	Flags:	rule-engine: ovirt-4.1+ ylavi: planning_ack+ sradco: devel_ack+ lsvaty: testing_ack+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ovirt-engine-metrics-1.0.7.1	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-11-13 12:28:08 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Metrics	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1475135, 1493030

Description Shirly Radco 2017-09-19 07:07:29 UTC

Description of problem:
We need to set fluentd buffer parameter
Set buffer_type to file, buffer_path, buffer_chunk_limit and buffer_queue_limit
so that it can improve both the hosts and metrics store fluentd performance.

Comment 1 Shirly Radco 2017-09-28 09:17:11 UTC

Currently in the 0.12 version of fluentd there is a bug that prevents using file buffer.

This RFE handlse setting memory buffer parameters.
And making them as ansible variables.

This will allow easier performance tuning.

Rich, Do we need separate buffer configurations for metrics and logs?

Comment 2 Shirly Radco 2017-09-28 11:44:07 UTC

I separated the metrics and logs forwarding
and added dedicated buffer parameters  for each, due to different requirements.

Comment 3 Rich Megginson 2017-09-28 13:22:01 UTC

(In reply to Shirly Radco from comment #2)
> I separated the metrics and logs forwarding
> and added dedicated buffer parameters  for each, due to different
> requirements.

I don't think we have enough data yet to determine that.

But - note that with two different secure_forward configurations, that means each fluentd now has 2 connections to common logging instead of one, which means doubling the number of connections.

Comment 4 Lukas Svaty 2017-10-19 15:46:04 UTC

patch did not get through

Comment 5 Lukas Svaty 2017-10-25 10:20:23 UTC

sanity on fluentd, verified in ovirt-engine-metrics-1.0.7.1-1.el7ev.noarch