Bug 1410705 - Roll over log entries are visible when journal-read-from-head is false
Summary: Roll over log entries are visible when journal-read-from-head is false
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.4.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 3.4.z
Assignee: Rich Megginson
QA Contact: Xia Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-06 06:42 UTC by Xia Zhao
Modified: 2017-01-31 18:18 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-01-31 18:18:48 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0219 0 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.4.1.2 images bug fix update 2017-02-01 01:14:50 UTC

Description Xia Zhao 2017-01-06 06:42:36 UTC
Description of problem:
Deploy logging with use_journal = true, journal-read-from-head = false. Wait for a while until log entries for all necessary namespaces are ready, log in kibana UI, the roll over logs are displayed, and its index mapping was created in ES.

Version-Release number of selected component (if applicable):
ops registry
openshift3/logging-kibana    8a3df528c998
openshift3/logging-elasticsearch    583e04127ed6
openshift3/logging-auth-proxy    d9236074fecb
openshift3/logging-fluentd    43d549beb4c8
openshift3/logging-curator    5aadf9eb6168
openshift3/logging-deployer    7386facde449

How reproducible:
Always

Steps to Reproduce:
1.Deploy logging with use_journal = true, journal-read-from-head = false
2.Double check and make sure read-from-head is diabled in fluentd:
$ oc rsh ${fluentd-pod}
sh-4.2# cat input-syslog-default-syslog.conf
<source>
  @type systemd
  @label @INGRESS
  path "/var/log/journal"
  pos_file /var/log/journal.pos
  tag journal
  read_from_head "false"
</source>

3.Tail ES log for namespaces that was created prior to EFK deployments
4.Log in kibana

Actual results:
3.Index mappings was created in ES for namespaces that was created prior to EFK deployments
4.Roll over logs are displayed

Expected results:
3.Index mappings should not be created in ES for namespaces that was created prior to EFK deployments
4.Roll over logs (and indices) should not exist

Additional info:

Comment 1 Xia Zhao 2017-01-06 08:59:15 UTC
Issue exist after removing the pos file on node, it recreates automatically after removing:

# cat /var/log/journal.pos 
s=e9708e5b637f4472942b54b9f7f660b6;i=4c3c2;b=8ff3def7672a47a09f1455574362d261;m=29dc7cafe;t=54565779e6744;x=40f8c1889c5f625e

# rm -rf journal.pos

# cat /var/log/journal.pos 
s=e9708e5b637f4472942b54b9f7f660b6;i=5e3fd;b=8ff3def7672a47a09f1455574362d261;m=305262080;t=54565defcbcc6;x=f093d8100b19e179

Comment 2 Rich Megginson 2017-01-06 17:25:57 UTC
There is definitely a bug here.  Investigating.

Comment 3 openshift-github-bot 2017-01-09 15:31:55 UTC
Commit pushed to master at https://github.com/openshift/origin-aggregated-logging

https://github.com/openshift/origin-aggregated-logging/commit/3f16fe806cf9c282d1916ce947fd5b5ab28a9e3c
Bug 1410705 - Roll over log entries are visible when journal-read-from-head is false

https://bugzilla.redhat.com/show_bug.cgi?id=1410705
The fluent-plugin-systemd plugin is parsing "false" as a string value, which
always evaluates to boolean true.  The fix is to not set read_from_head
when it is not true, this will use the default false internal value.
See also https://github.com/reevoo/fluent-plugin-systemd/issues/19
Also note that due to
https://github.com/reevoo/fluent-plugin-systemd/issues/20
and
https://github.com/ledbettj/systemd-journal/issues/64
there may be a few old entries read, even if read_from_head is false.

Comment 4 Rich Megginson 2017-01-09 17:02:47 UTC
Waiting for next release of 3.4 to release this fix - adding tdawson

To ssh://rmeggins@pkgs.devel.redhat.com/rpms/logging-fluentd-docker
   f8149e4..3a4733b  rhaos-3.4-rhel-7 -> rhaos-3.4-rhel-7

Just needs a rebuild of logging-fluentd-docker

Comment 5 Troy Dawson 2017-01-27 22:50:43 UTC
This should be in openshift3/logging-fluentd:3.4.1-2 or newer

Comment 7 Peter Ruan 2017-01-31 00:46:09 UTC
verified with
[root@host-8-175-197 ~]# oc version
oc v3.4.1.2
kubernetes v1.4.0+776c994
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://host-8-175-197.host.centralci.eng.rdu2.redhat.com:8443
openshift v3.4.1.2
kubernetes v1.4.0+776c994

Comment 9 errata-xmlrpc 2017-01-31 18:18:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0219


Note You need to log in before you can comment on or make changes to this bug.