Bug 1496261 - log throttling reuses log position file
Summary: log throttling reuses log position file
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 3.8.0
Assignee: ewolinet
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-09-26 20:07 UTC by Rich Megginson
Modified: 2018-04-25 02:59 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Feature: When using log throttling for EFK, we create a new pos file for the different throttled inputs. Reason: Per fluentd recommendation, pos files should not be shared. Result: When using throttled logging, we expect to see multiple pos files per a throttle input. When throttling is disabled we copy the file positions back into the unified pos file to minimize duplication of records.
Clone Of:
Environment:
Last Closed: 2018-03-28 14:07:14 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:0489 0 None None None 2018-03-28 14:07:53 UTC

Description Rich Megginson 2017-09-26 20:07:32 UTC
Description of problem:
Using docker json-file, and there are several projects where logs have already been collected.  There is already a /var/log/es-containers.log.pos file which contains the position of the last log read from the json log files from the containers in the various projects.
Configure throttling for one or more of the projects
Throttling will reuse the same /var/log/es-containers.log.pos for the new in_tail configuration for the throttled projects.  This is a no-no:

https://docs.fluentd.org/v0.12/articles/in_tail#posfile-highly-recommended

"Don't share pos_file between in_tail configurations. It causes unexpected behavior, e.g. corrupt pos_file content."

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 3 Anping Li 2018-01-10 07:02:50 UTC
Blocked by Bug 1532955

Comment 4 Anping Li 2018-01-16 08:37:34 UTC
Should there is a pos file for each container under /var/log/?   Should  es-containers.log.pos be deleted?

The following pos file are present during v3.9 testing. 
openshift3/logging-elasticsearch:v3.9.0-0.20.0.0
openshift-elasticsearch-plugin-2.4.4.17__redhat_1-3.el7.noarch
# ls /var/log/| grep pos
es-containers.log.pos
journal.pos

Comment 5 ewolinet 2018-01-16 18:35:33 UTC
No, there may still be a containers.log.pos file that exists, however if you are throttling a specific namespace you should not see any entries for that namespace in the containers.log.pos file.

To test this you will need to configure log throttling for Fluentd.
You should see a pos file for each throttled namespace you have configured.

Comment 6 Anping Li 2018-01-17 11:34:24 UTC
How could I throttle a specific namespace? could you give me a example?

Comment 8 Anping Li 2018-01-18 10:39:19 UTC
Verified in openshift3/logging-fluentd/images/v3.9.0-0.21.0.0

Once throttle a specific namespace, The pos file is created and the pos pointer are move to this file from es-containers.log.pos



-rw-r--r--. 1 root   root        1199 Jan 18 05:13 es-container-logging.log.pos
-rw-r--r--. 1 root   root         164 Jan 18 05:14 es-container-openshift-operations.log.pos
-rw-r--r--. 1 root   root        2430 Jan 18 05:13 es-containers.log.pos

Comment 11 errata-xmlrpc 2018-03-28 14:07:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489


Note You need to log in before you can comment on or make changes to this bug.