Bug 1477513 - [3.4] Data loss of logs can occur if fluentd pod is terminated/restarted when Elasticsearch is unavailable
[3.4] Data loss of logs can occur if fluentd pod is terminated/restarted when...
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging (Show other bugs)
3.4.1
All All
urgent Severity urgent
: ---
: 3.4.z
Assigned To: Noriko Hosoi
Xia Zhao
:
Depends On: 1460749 1477515 1483114
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-02 05:55 EDT by Ruben Romero Montes
Modified: 2017-10-25 09:04 EDT (History)
7 users (show)

See Also:
Fixed In Version: logging-fluentd-docker-3.4.1-27
Doc Type: Bug Fix
Doc Text:
Cause: Fluentd buffers the logs which are not immediately flushed to the storage such as Elaticsearch. The buffers are placed in memory, by default. Consequence: If there are some issue and the fluentd is forced to quit, the buffered logs would have been lost when the fluentd is restarted. Fix: Instead of placing the buffers in memory, they are configured on the permanent storage. Result: Even if the fluentd is accidentally restarted, the buffered logs would not be lost.
Story Points: ---
Clone Of: 1460749
Environment:
Last Closed: 2017-10-25 09:04:36 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:3049 normal SHIPPED_LIVE OpenShift Container Platform 3.6, 3.5, and 3.4 bug fix and enhancement update 2017-10-25 11:57:15 EDT

  None (edit)
Comment 3 Noriko Hosoi 2017-09-25 16:39:16 EDT
Pull request for template.
https://github.com/openshift/origin-aggregated-logging/pull/680
Comment 6 Xia Zhao 2017-09-28 05:49:13 EDT

It's fixed, currently didn't meet data loss of logs any more when fluentd is restarted:

Verification steps:

1. Deploy logging stacks on openshift

2. Creat one project to populate logs.

3. Stop fluentd pods, and note down the last project logs in kibana

4. Wait for a while, and restart fluentd pods.

5. Check the subsequent logs after step 3, no logs is missing.

6. Repeat step 3 to 5, make sure no log is missing.

Image tested with:
logging-fluentd         3.4.1-27            5772acf0562d        7 days ago          235.7 MB

Test env:
# openshift version
openshift v3.4.1.44.22
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0
Comment 8 errata-xmlrpc 2017-10-25 09:04:36 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3049

Note You need to log in before you can comment on or make changes to this bug.