Bug 1410705 - Roll over log entries are visible when journal-read-from-head is false
Summary: Roll over log entries are visible when journal-read-from-head is false
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.4.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.4.z
Assignee: Rich Megginson
QA Contact: Xia Zhao
Depends On:
TreeView+ depends on / blocked
Reported: 2017-01-06 06:42 UTC by Xia Zhao
Modified: 2017-01-31 18:18 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2017-01-31 18:18:48 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0219 0 normal SHIPPED_LIVE Red Hat OpenShift Container Platform images bug fix update 2017-02-01 01:14:50 UTC

Description Xia Zhao 2017-01-06 06:42:36 UTC
Description of problem:
Deploy logging with use_journal = true, journal-read-from-head = false. Wait for a while until log entries for all necessary namespaces are ready, log in kibana UI, the roll over logs are displayed, and its index mapping was created in ES.

Version-Release number of selected component (if applicable):
ops registry
openshift3/logging-kibana    8a3df528c998
openshift3/logging-elasticsearch    583e04127ed6
openshift3/logging-auth-proxy    d9236074fecb
openshift3/logging-fluentd    43d549beb4c8
openshift3/logging-curator    5aadf9eb6168
openshift3/logging-deployer    7386facde449

How reproducible:

Steps to Reproduce:
1.Deploy logging with use_journal = true, journal-read-from-head = false
2.Double check and make sure read-from-head is diabled in fluentd:
$ oc rsh ${fluentd-pod}
sh-4.2# cat input-syslog-default-syslog.conf
  @type systemd
  @label @INGRESS
  path "/var/log/journal"
  pos_file /var/log/journal.pos
  tag journal
  read_from_head "false"

3.Tail ES log for namespaces that was created prior to EFK deployments
4.Log in kibana

Actual results:
3.Index mappings was created in ES for namespaces that was created prior to EFK deployments
4.Roll over logs are displayed

Expected results:
3.Index mappings should not be created in ES for namespaces that was created prior to EFK deployments
4.Roll over logs (and indices) should not exist

Additional info:

Comment 1 Xia Zhao 2017-01-06 08:59:15 UTC
Issue exist after removing the pos file on node, it recreates automatically after removing:

# cat /var/log/journal.pos 

# rm -rf journal.pos

# cat /var/log/journal.pos 

Comment 2 Rich Megginson 2017-01-06 17:25:57 UTC
There is definitely a bug here.  Investigating.

Comment 3 openshift-github-bot 2017-01-09 15:31:55 UTC
Commit pushed to master at https://github.com/openshift/origin-aggregated-logging

Bug 1410705 - Roll over log entries are visible when journal-read-from-head is false

The fluent-plugin-systemd plugin is parsing "false" as a string value, which
always evaluates to boolean true.  The fix is to not set read_from_head
when it is not true, this will use the default false internal value.
See also https://github.com/reevoo/fluent-plugin-systemd/issues/19
Also note that due to
there may be a few old entries read, even if read_from_head is false.

Comment 4 Rich Megginson 2017-01-09 17:02:47 UTC
Waiting for next release of 3.4 to release this fix - adding tdawson

To ssh://rmeggins@pkgs.devel.redhat.com/rpms/logging-fluentd-docker
   f8149e4..3a4733b  rhaos-3.4-rhel-7 -> rhaos-3.4-rhel-7

Just needs a rebuild of logging-fluentd-docker

Comment 5 Troy Dawson 2017-01-27 22:50:43 UTC
This should be in openshift3/logging-fluentd:3.4.1-2 or newer

Comment 7 Peter Ruan 2017-01-31 00:46:09 UTC
verified with
[root@host-8-175-197 ~]# oc version
oc v3.4.1.2
kubernetes v1.4.0+776c994
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://host-8-175-197.host.centralci.eng.rdu2.redhat.com:8443
openshift v3.4.1.2
kubernetes v1.4.0+776c994

Comment 9 errata-xmlrpc 2017-01-31 18:18:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.