Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1486936

Summary: BUFFER_SIZE_LIMIT & BUFFER_QUEUE_LIMIT variables are not available to logging-fluentd on OCP 3.4
Product: OpenShift Container Platform Reporter: Freddy E. Montero <fmontero>
Component: LoggingAssignee: Jeff Cantrill <jcantril>
Status: CLOSED DUPLICATE QA Contact: Xia Zhao <xiazhao>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.4.0CC: aos-bugs, fmontero, jokerman, mmccomas, rmeggins
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-31 16:11:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Freddy E. Montero 2017-08-30 20:41:38 UTC 
Description of problem:
All fluentd pods report the following issue:

2017-08-23 11:14:39 -0400 [warn]: emit transaction failed: error_class=Fluent::BufferQueueLimitError error="queue size exceeds limit" tag="kubernetes.var.log.containers.gio-integration-4-ge563_cipe-prod_gio-integration-0804333b230346551720fe1371be8f15c1a84a51e9b33a5279ed8236e5c07c2f.log"

2017-08-23 11:14:39 -0400 [warn]: suppressed same stacktrace

There are 2 BZ that shows similar behavior as the issue stated above:
https://bugzilla.redhat.com/show_bug.cgi?id=1413147
&
https://bugzilla.redhat.com/show_bug.cgi?id=1470862

as well as a KB article with a workaround to the issue:
https://access.redhat.com/solutions/3142351

Trying to implement this workaround on OCP 3.4 doesn't have any effect whatsoever.

Further investigation and reading latest repos for origin(https://github.com/openshift/origin-aggregated-logging) and ansible-openshift(https://github.com/openshift/openshift-ansible/tree/master/roles/openshift_logging), it seems that these variables can be set.

oc edit ds logging-fluentd
and setting the following
        - name: BUFFER_SIZE_LIMIT
          value: 2Mi
        - name: BUFFER_QUEUE_LIMIT
          value: "4096"
then redeploying the pods does not have any effect.
Digging more into this found out that the output plugin for elasticsearch (/etc/fluent/configs.d/openshift/output-es-config.conf) on all pods shows that it doesn't have the variables that can be tuned via ocp edit ds logging-fluentd  


Version-Release number of selected component (if applicable):
logging-fluentd:3.4.1-18

How reproducible:
Always
Comment 1 Rich Megginson 2017-08-30 23:16:47 UTC 
> There are 2 BZ that shows similar behavior as the issue stated above:
> https://bugzilla.redhat.com/show_bug.cgi?id=1413147

Different issue.  This is just a warning, not actually an error.

> https://bugzilla.redhat.com/show_bug.cgi?id=1470862

Different issue.  The problem here is Elasticsearch rejecting bulk index requests.

> as well as a KB article with a workaround to the issue:
> https://access.redhat.com/solutions/3142351

Different issue.  The problem here is Elasticsearch rejecting bulk index requests.
Comment 2 Rich Megginson 2017-08-30 23:18:16 UTC 
This looks like a dup of https://bugzilla.redhat.com/show_bug.cgi?id=1486473 - please confirm
Comment 4 Rich Megginson 2017-08-31 16:11:26 UTC 

*** This bug has been marked as a duplicate of bug 1486473 ***