Bug 1496261

Summary:	log throttling reuses log position file
Product:	OpenShift Container Platform	Reporter:	Rich Megginson <rmeggins>
Component:	Logging	Assignee:	ewolinet
Status:	CLOSED ERRATA	QA Contact:	Anping Li <anli>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	3.7.0	CC:	aos-bugs, ewolinet, rmeggins
Target Milestone:	---
Target Release:	3.8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Enhancement
Doc Text:	Feature: When using log throttling for EFK, we create a new pos file for the different throttled inputs. Reason: Per fluentd recommendation, pos files should not be shared. Result: When using throttled logging, we expect to see multiple pos files per a throttle input. When throttling is disabled we copy the file positions back into the unified pos file to minimize duplication of records.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-03-28 14:07:14 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Rich Megginson 2017-09-26 20:07:32 UTC

Description of problem:
Using docker json-file, and there are several projects where logs have already been collected.  There is already a /var/log/es-containers.log.pos file which contains the position of the last log read from the json log files from the containers in the various projects.
Configure throttling for one or more of the projects
Throttling will reuse the same /var/log/es-containers.log.pos for the new in_tail configuration for the throttled projects.  This is a no-no:

https://docs.fluentd.org/v0.12/articles/in_tail#posfile-highly-recommended

"Don't share pos_file between in_tail configurations. It causes unexpected behavior, e.g. corrupt pos_file content."

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 ewolinet 2017-11-30 17:34:19 UTC

https://github.com/openshift/origin-aggregated-logging/pull/803

Comment 3 Anping Li 2018-01-10 07:02:50 UTC

Blocked by Bug 1532955

Comment 4 Anping Li 2018-01-16 08:37:34 UTC

Should there is a pos file for each container under /var/log/?   Should  es-containers.log.pos be deleted?

The following pos file are present during v3.9 testing. 
openshift3/logging-elasticsearch:v3.9.0-0.20.0.0
openshift-elasticsearch-plugin-2.4.4.17__redhat_1-3.el7.noarch
# ls /var/log/| grep pos
es-containers.log.pos
journal.pos

Comment 5 ewolinet 2018-01-16 18:35:33 UTC

No, there may still be a containers.log.pos file that exists, however if you are throttling a specific namespace you should not see any entries for that namespace in the containers.log.pos file.

To test this you will need to configure log throttling for Fluentd.
You should see a pos file for each throttled namespace you have configured.

Comment 6 Anping Li 2018-01-17 11:34:24 UTC

How could I throttle a specific namespace? could you give me a example?

Comment 8 Anping Li 2018-01-18 10:39:19 UTC

Verified in openshift3/logging-fluentd/images/v3.9.0-0.21.0.0

Once throttle a specific namespace, The pos file is created and the pos pointer are move to this file from es-containers.log.pos



-rw-r--r--. 1 root   root        1199 Jan 18 05:13 es-container-logging.log.pos
-rw-r--r--. 1 root   root         164 Jan 18 05:14 es-container-openshift-operations.log.pos
-rw-r--r--. 1 root   root        2430 Jan 18 05:13 es-containers.log.pos

Comment 11 errata-xmlrpc 2018-03-28 14:07:14 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489