Bug 1410705

Summary: Roll over log entries are visible when journal-read-from-head is false
Product: OpenShift Container Platform Reporter: Xia Zhao <xiazhao>
Component: LoggingAssignee: Rich Megginson <rmeggins>
Status: CLOSED ERRATA QA Contact: Xia Zhao <xiazhao>
Severity: low Docs Contact:
Priority: low    
Version: 3.4.0CC: aos-bugs, pruan, tdawson
Target Milestone: ---   
Target Release: 3.4.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-01-31 18:18:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Xia Zhao 2017-01-06 06:42:36 UTC
Description of problem:
Deploy logging with use_journal = true, journal-read-from-head = false. Wait for a while until log entries for all necessary namespaces are ready, log in kibana UI, the roll over logs are displayed, and its index mapping was created in ES.

Version-Release number of selected component (if applicable):
ops registry
openshift3/logging-kibana    8a3df528c998
openshift3/logging-elasticsearch    583e04127ed6
openshift3/logging-auth-proxy    d9236074fecb
openshift3/logging-fluentd    43d549beb4c8
openshift3/logging-curator    5aadf9eb6168
openshift3/logging-deployer    7386facde449

How reproducible:
Always

Steps to Reproduce:
1.Deploy logging with use_journal = true, journal-read-from-head = false
2.Double check and make sure read-from-head is diabled in fluentd:
$ oc rsh ${fluentd-pod}
sh-4.2# cat input-syslog-default-syslog.conf
<source>
  @type systemd
  @label @INGRESS
  path "/var/log/journal"
  pos_file /var/log/journal.pos
  tag journal
  read_from_head "false"
</source>

3.Tail ES log for namespaces that was created prior to EFK deployments
4.Log in kibana

Actual results:
3.Index mappings was created in ES for namespaces that was created prior to EFK deployments
4.Roll over logs are displayed

Expected results:
3.Index mappings should not be created in ES for namespaces that was created prior to EFK deployments
4.Roll over logs (and indices) should not exist

Additional info:

Comment 1 Xia Zhao 2017-01-06 08:59:15 UTC
Issue exist after removing the pos file on node, it recreates automatically after removing:

# cat /var/log/journal.pos 
s=e9708e5b637f4472942b54b9f7f660b6;i=4c3c2;b=8ff3def7672a47a09f1455574362d261;m=29dc7cafe;t=54565779e6744;x=40f8c1889c5f625e

# rm -rf journal.pos

# cat /var/log/journal.pos 
s=e9708e5b637f4472942b54b9f7f660b6;i=5e3fd;b=8ff3def7672a47a09f1455574362d261;m=305262080;t=54565defcbcc6;x=f093d8100b19e179

Comment 2 Rich Megginson 2017-01-06 17:25:57 UTC
There is definitely a bug here.  Investigating.

Comment 3 openshift-github-bot 2017-01-09 15:31:55 UTC
Commit pushed to master at https://github.com/openshift/origin-aggregated-logging

https://github.com/openshift/origin-aggregated-logging/commit/3f16fe806cf9c282d1916ce947fd5b5ab28a9e3c
Bug 1410705 - Roll over log entries are visible when journal-read-from-head is false

https://bugzilla.redhat.com/show_bug.cgi?id=1410705
The fluent-plugin-systemd plugin is parsing "false" as a string value, which
always evaluates to boolean true.  The fix is to not set read_from_head
when it is not true, this will use the default false internal value.
See also https://github.com/reevoo/fluent-plugin-systemd/issues/19
Also note that due to
https://github.com/reevoo/fluent-plugin-systemd/issues/20
and
https://github.com/ledbettj/systemd-journal/issues/64
there may be a few old entries read, even if read_from_head is false.

Comment 4 Rich Megginson 2017-01-09 17:02:47 UTC
Waiting for next release of 3.4 to release this fix - adding tdawson

To ssh://rmeggins.redhat.com/rpms/logging-fluentd-docker
   f8149e4..3a4733b  rhaos-3.4-rhel-7 -> rhaos-3.4-rhel-7

Just needs a rebuild of logging-fluentd-docker

Comment 5 Troy Dawson 2017-01-27 22:50:43 UTC
This should be in openshift3/logging-fluentd:3.4.1-2 or newer

Comment 7 Peter Ruan 2017-01-31 00:46:09 UTC
verified with
[root@host-8-175-197 ~]# oc version
oc v3.4.1.2
kubernetes v1.4.0+776c994
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://host-8-175-197.host.centralci.eng.rdu2.redhat.com:8443
openshift v3.4.1.2
kubernetes v1.4.0+776c994

Comment 9 errata-xmlrpc 2017-01-31 18:18:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0219