Bug 1477515 – [3.5] Data loss of logs can occur if fluentd pod is terminated/restarted when Elasticsearch is unavailable

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1477515 - [3.5] Data loss of logs can occur if fluentd pod is terminated/restarted when Elasticsearch is unavailable

Summary:	[3.5] Data loss of logs can occur if fluentd pod is terminated/restarted when...

Status:	CLOSED ERRATA

Aliases:	None

Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Logging (Show other bugs)
Sub Component:
Version:	3.5.1
Hardware:	All All

Priority	urgent Severity urgent
Target Milestone:	---
Target Release:	3.5.z
Assigned To:	Noriko Hosoi
QA Contact:	Xia Zhao
Docs Contact:

URL:
Whiteboard:
Keywords:

Depends On:	1460749 1483114
Blocks:	1477513
	Show dependency tree / graph

Reported:	2017-08-02 05:55 EDT by Ruben Romero Montes
Modified:	2017-10-25 09:04 EDT (History)
CC List:	8 users (show)

See Also:
Fixed In Version:	logging-fluentd-docker-3.5.0-31
Doc Type:	Bug Fix
Doc Text:	Cause: Messages are read into fluentds memory buffer and are lost if the pod is restarted because fluentd considers them read but they have not been pushed to storage Consequence: Any message not stored but already read by fluentd is lost Fix: Replace the memory buffer with a file based buffer Result: File buffered messages are pushed to storage once fluentd restarts
Story Points:	---
Clone Of:	1460749
Environment:
Last Closed:	2017-10-25 09:04:36 EDT
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2017:3049	normal	SHIPPED_LIVE	OpenShift Container Platform 3.6, 3.5, and 3.4 bug fix and enhancement update	2017-10-25 11:57:15 EDT

Groups: None (edit)

Comment 1 Noriko Hosoi 2017-09-19 17:20:01 EDT

https://github.com/openshift/origin-aggregated-logging/pull/670

Comment 3 Noriko Hosoi 2017-09-25 16:43:43 EDT

Pull request for openshift-ansible.
https://github.com/openshift/openshift-ansible/pull/5528

Comment 5 Xia Zhao 2017-09-28 05:48:39 EDT

It's fixed, currently didn't meet data loss of logs any more when fluentd is restarted:

Verification steps:

1. Deploy logging stacks on openshift

2. Creat one project to populate logs.

3. Stop fluentd pods, and note down the last project logs in kibana

4. Wait for a while, and restart fluentd pods.

5. Check the subsequent logs after step 3, no logs is missing.

6. Repeat step 3 to 5, make sure no log is missing.

Image tested with:
logging-fluentd         3.5.0-31            604adb699900        6 days ago          235.8 MB

Test env:
# openshift version
openshift v3.5.5.31.27
kubernetes v1.5.2+43a9be4
etcd 3.1.0

Comment 7 errata-xmlrpc 2017-10-25 09:04:36 EDT

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3049

Note You need to log in before you can comment on or make changes to this bug.