Bug 1477515 - [3.5] Data loss of logs can occur if fluentd pod is terminated/restarted when Elasticsearch is unavailable
Summary: [3.5] Data loss of logs can occur if fluentd pod is terminated/restarted when...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.5.1
Hardware: All
OS: All
urgent
urgent
Target Milestone: ---
: 3.5.z
Assignee: Noriko Hosoi
QA Contact: Xia Zhao
URL:
Whiteboard:
Depends On: 1460749 1483114
Blocks: 1477513
TreeView+ depends on / blocked
 
Reported: 2017-08-02 09:55 UTC by Ruben Romero Montes
Modified: 2017-10-25 13:04 UTC (History)
8 users (show)

Fixed In Version: logging-fluentd-docker-3.5.0-31
Doc Type: Bug Fix
Doc Text:
Cause: Messages are read into fluentds memory buffer and are lost if the pod is restarted because fluentd considers them read but they have not been pushed to storage Consequence: Any message not stored but already read by fluentd is lost Fix: Replace the memory buffer with a file based buffer Result: File buffered messages are pushed to storage once fluentd restarts
Clone Of: 1460749
Environment:
Last Closed: 2017-10-25 13:04:36 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:3049 0 normal SHIPPED_LIVE OpenShift Container Platform 3.6, 3.5, and 3.4 bug fix and enhancement update 2017-10-25 15:57:15 UTC

Comment 1 Noriko Hosoi 2017-09-19 21:20:01 UTC 
https://github.com/openshift/origin-aggregated-logging/pull/670
Comment 3 Noriko Hosoi 2017-09-25 20:43:43 UTC 
Pull request for openshift-ansible.
https://github.com/openshift/openshift-ansible/pull/5528
Comment 5 Xia Zhao 2017-09-28 09:48:39 UTC 
It's fixed, currently didn't meet data loss of logs any more when fluentd is restarted:

Verification steps:

1. Deploy logging stacks on openshift

2. Creat one project to populate logs.

3. Stop fluentd pods, and note down the last project logs in kibana

4. Wait for a while, and restart fluentd pods.

5. Check the subsequent logs after step 3, no logs is missing.

6. Repeat step 3 to 5, make sure no log is missing.

Image tested with:
logging-fluentd         3.5.0-31            604adb699900        6 days ago          235.8 MB

Test env:
# openshift version
openshift v3.5.5.31.27
kubernetes v1.5.2+43a9be4
etcd 3.1.0
Comment 7 errata-xmlrpc 2017-10-25 13:04:36 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3049
Note You need to log in before you can comment on or make changes to this bug.