Bug 1477513 - [3.4] Data loss of logs can occur if fluentd pod is terminated/restarted when Elasticsearch is unavailable
Summary: [3.4] Data loss of logs can occur if fluentd pod is terminated/restarted when...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.4.1
Hardware: All
OS: All
urgent
urgent
Target Milestone: ---
: 3.4.z
Assignee: Noriko Hosoi
QA Contact: Xia Zhao
URL:
Whiteboard:
Depends On: 1460749 1477515 1483114
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-08-02 09:55 UTC by Ruben Romero Montes
Modified: 2021-06-10 12:43 UTC (History)
7 users (show)

Fixed In Version: logging-fluentd-docker-3.4.1-27
Doc Type: Bug Fix
Doc Text:
Cause: Fluentd buffers the logs which are not immediately flushed to the storage such as Elaticsearch. The buffers are placed in memory, by default. Consequence: If there are some issue and the fluentd is forced to quit, the buffered logs would have been lost when the fluentd is restarted. Fix: Instead of placing the buffers in memory, they are configured on the permanent storage. Result: Even if the fluentd is accidentally restarted, the buffered logs would not be lost.
Clone Of: 1460749
Environment:
Last Closed: 2017-10-25 13:04:36 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:3049 0 normal SHIPPED_LIVE OpenShift Container Platform 3.6, 3.5, and 3.4 bug fix and enhancement update 2017-10-25 15:57:15 UTC

Comment 1 Noriko Hosoi 2017-09-19 21:20:11 UTC 
https://github.com/openshift/origin-aggregated-logging/pull/669
Comment 3 Noriko Hosoi 2017-09-25 20:39:16 UTC 
Pull request for template.
https://github.com/openshift/origin-aggregated-logging/pull/680
Comment 6 Xia Zhao 2017-09-28 09:49:13 UTC 

It's fixed, currently didn't meet data loss of logs any more when fluentd is restarted:

Verification steps:

1. Deploy logging stacks on openshift

2. Creat one project to populate logs.

3. Stop fluentd pods, and note down the last project logs in kibana

4. Wait for a while, and restart fluentd pods.

5. Check the subsequent logs after step 3, no logs is missing.

6. Repeat step 3 to 5, make sure no log is missing.

Image tested with:
logging-fluentd         3.4.1-27            5772acf0562d        7 days ago          235.7 MB

Test env:
# openshift version
openshift v3.4.1.44.22
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0
Comment 8 errata-xmlrpc 2017-10-25 13:04:36 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3049
Note You need to log in before you can comment on or make changes to this bug.