Bug 1826861 - DiskPressure due to 80 GB /var/lib/fluentd
Summary: DiskPressure due to 80 GB /var/lib/fluentd
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.4
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.4.z
Assignee: Periklis Tsirakidis
QA Contact: Anping Li
Depends On: 1780698
Blocks: 1833226
TreeView+ depends on / blocked
Reported: 2020-04-22 16:27 UTC by Periklis Tsirakidis
Modified: 2020-07-13 07:48 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: On high incoming log rates Fluentd could possible flood the node's filesystem because the buffer queues were not limited. Consequence: A node under disk pressure could eventually crash the node and thus the applications would be rescheduled. Fix: The fluentd buffer queue per output is limited to a fixed amount of chunks (default 32). Result: Node disk pressure due to fluentd buffers should be omited by this fix.
Clone Of:
Last Closed: 2020-05-18 13:35:02 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift cluster-logging-operator pull 491 0 None closed [release-4.4] Bug 1826861: Fix disk pressure for fluentd buffer chunks 2021-02-11 14:47:47 UTC
Red Hat Product Errata RHBA-2020:2133 0 None None None 2020-05-18 13:35:23 UTC

Description Periklis Tsirakidis 2020-04-22 16:27:41 UTC
This bug was initially created as a copy of Bug #1780698

I am copying this bug because: 

Description of problem:
I have a cluster that was on 4.1.23 (upgraded continuously from about 4.1.4).

The upgrade to either 4.1.24 or 4.1.25 fails with a download error:

info: An upgrade is in progress. Unable to apply 4.1.25: could not download the update


4.1.25  quay.io/openshift-release-dev/ocp-release@sha256:5f824fa3b3c44c6a78a5fc6a77a82edc47cf2b495bb6b2b31e3e0a4d3d77684b
4.1.24  quay.io/openshift-release-dev/ocp-release@sha256:6f87fb66dfa907db03981e69474ea3069deab66358c18d965f6331bd727ff23f

oc get clusterversion
version   4.1.23    True        True          44h     Unable to apply 4.1.25: could not download the update

All Cluster Operators show on 4.1.23.

oc adm must-gather is at https://drive.google.com/open?id=18mqD6BpEwAQbApb1_5MD9j-cMBckBIA6

I can provide a kubeconfig as well to poke around there.

Comment 1 Kirsten Garrison 2020-04-22 21:55:59 UTC
For some reason the PR wasn't linked to this BZ, by the bot:


Comment 6 Anping Li 2020-05-08 11:01:55 UTC
Verified in clusterlogging.4.4.0-202005072005. 
Turn off the ES, the directory size didn't increase once it reached 257M. After the ES is turn back. the directory size decreased.

56M	/var/lib/fluentd/
56M	/var/lib/fluentd/clo_default_output_es
0	/var/lib/fluentd/retry_clo_default_output_es
121M	/var/lib/fluentd/
121M	/var/lib/fluentd/clo_default_output_es
0	/var/lib/fluentd/retry_clo_default_output_es

257M	/var/lib/fluentd/
257M	/var/lib/fluentd/clo_default_output_es
0	/var/lib/fluentd/retry_clo_default_output_es

Fri May  8 06:46:21 EDT 2020
257M	/var/lib/fluentd/
257M	/var/lib/fluentd/clo_default_output_es
0	/var/lib/fluentd/retry_clo_default_output_es
Fri May  8 06:51:23 EDT 2020
16M	/var/lib/fluentd/
16M	/var/lib/fluentd/clo_default_output_es
0	/var/lib/fluentd/retry_clo_default_output_es

Comment 8 errata-xmlrpc 2020-05-18 13:35:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.