Bug 1493002 - [RFE] Set fluentd buffer parameters
Summary: [RFE] Set fluentd buffer parameters
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine-metrics
Classification: oVirt
Component: RFEs
Version: 1.0.5
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ovirt-4.1.7
: 1.0.7.1
Assignee: Shirly Radco
QA Contact: Lukas Svaty
URL:
Whiteboard:
Depends On:
Blocks: 1475135 1493030
TreeView+ depends on / blocked
 
Reported: 2017-09-19 07:07 UTC by Shirly Radco
Modified: 2017-11-14 11:43 UTC (History)
4 users (show)

Fixed In Version: ovirt-engine-metrics-1.0.7.1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-11-13 12:28:08 UTC
oVirt Team: Metrics
Embargoed:
rule-engine: ovirt-4.1+
ylavi: planning_ack+
sradco: devel_ack+
lsvaty: testing_ack+

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 81957 0 ovirt-engine-metrics-4.1 ABANDONED fluentd: set buffer configurations 2017-09-28 08:52:47 UTC
oVirt gerrit 82329 0 master MERGED fluentd: set buffer configurations 2017-09-28 19:46:36 UTC
oVirt gerrit 82330 0 ovirt-engine-metrics-4.1 MERGED fluentd: set buffer configurations 2017-09-28 19:56:48 UTC
oVirt gerrit 82422 0 master MERGED fluentd: set buffer configurations 2017-10-01 11:33:35 UTC
oVirt gerrit 82423 0 ovirt-engine-metrics-4.1 MERGED fluentd: fix bug in buffer configurations 2017-10-01 11:42:19 UTC

Description Shirly Radco 2017-09-19 07:07:29 UTC 
Description of problem:
We need to set fluentd buffer parameter
Set buffer_type to file, buffer_path, buffer_chunk_limit and buffer_queue_limit
so that it can improve both the hosts and metrics store fluentd performance.
Comment 1 Shirly Radco 2017-09-28 09:17:11 UTC 
Currently in the 0.12 version of fluentd there is a bug that prevents using file buffer.

This RFE handlse setting memory buffer parameters.
And making them as ansible variables.

This will allow easier performance tuning.

Rich, Do we need separate buffer configurations for metrics and logs?
Comment 2 Shirly Radco 2017-09-28 11:44:07 UTC 
I separated the metrics and logs forwarding
and added dedicated buffer parameters  for each, due to different requirements.
Comment 3 Rich Megginson 2017-09-28 13:22:01 UTC 
(In reply to Shirly Radco from comment #2)
> I separated the metrics and logs forwarding
> and added dedicated buffer parameters  for each, due to different
> requirements.

I don't think we have enough data yet to determine that.

But - note that with two different secure_forward configurations, that means each fluentd now has 2 connections to common logging instead of one, which means doubling the number of connections.
Comment 4 Lukas Svaty 2017-10-19 15:46:04 UTC 
patch did not get through
Comment 5 Lukas Svaty 2017-10-25 10:20:23 UTC 
sanity on fluentd, verified in ovirt-engine-metrics-1.0.7.1-1.el7ev.noarch
Note You need to log in before you can comment on or make changes to this bug.