Created attachment 1382428 [details] graphs generated after having set BUFFER_SIZE_LIMIT to 8MB
Francois could you please attache some additional information about the logging env: * ES DC/Pod yaml * Fluentd DS/pod yml * ES PV and disk size and info * Any info regarding the infra nodes hosting the ES and Fluent pods so we might better understand the disk and network IO * Provide data about the number of primary shards * The number of pending tasks over time A script that may be of interest: https://gist.githubusercontent.com/portante/c801acce8ae275fdcfc743b346981b13/raw/45cff354ec4a96882f6abc0725b2b6a646c5003c/health.sh
Commits pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/a24301bb9ae6bdfe3bca934a24c79ed68b1e8164 Bug 1534588 - Size of the emitted data exceeds buffer_chunk_limit If secure-forward outputs are additionally configured, the output count is also to be added to NUM_OUTPUTS, which is a divider to calculate the buffer size per output. Adding tests to cover the cases - having 1 then 2 intermediate forwarding fluentds. https://github.com/openshift/origin-aggregated-logging/commit/d5cd715df125f404af6f41a430b0a7fa42a9fb16 Merge pull request #926 from nhosoi/bz1534588 Automatic merge from submit-queue. Bug 1534588 - Size of the emitted data exceeds buffer_chunk_limit If secure-forward outputs are additionally configured, the output count is also to be added to NUM_OUTPUTS, which is a divider to calculate the buffer size per output. Adding tests to cover the cases - having 1 then 2 intermediate forwarding fluentds.
Commit pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/2d6b6043c2707d461d5a6eb8e38520d496c329f3 Removing an unnecessary line from the rebase mistake in pr/#926. Bug 1534588 - Size of the emitted data exceeds buffer_chunk_limit
Commits pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/6004e364013c333b445bda318cc599ac6a37ad00 Bug 1534588 - Size of the emitted data exceeds buffer_chunk_limit If secure-forward outputs are additionally configured, the output count is also to be added to NUM_OUTPUTS, which is a divider to calculate the buffer size per output. Adding tests to cover the cases - having 1 then 2 intermediate forwarding fluentds. https://github.com/openshift/origin-aggregated-logging/commit/9c429de93b8121f0854e3ee11079f86a3e145a30 Merge pull request #941 from openshift-cherrypick-robot/cherry-pick-926-to-es5.x [es5.x] Bug 1534588 - Size of the emitted data exceeds buffer_chunk_limit