Description of problem: Deploy logging with use_journal = true, journal-read-from-head = false. Wait for a while until log entries for all necessary namespaces are ready, log in kibana UI, the roll over logs are displayed, and its index mapping was created in ES. Version-Release number of selected component (if applicable): ops registry openshift3/logging-kibana 8a3df528c998 openshift3/logging-elasticsearch 583e04127ed6 openshift3/logging-auth-proxy d9236074fecb openshift3/logging-fluentd 43d549beb4c8 openshift3/logging-curator 5aadf9eb6168 openshift3/logging-deployer 7386facde449 How reproducible: Always Steps to Reproduce: 1.Deploy logging with use_journal = true, journal-read-from-head = false 2.Double check and make sure read-from-head is diabled in fluentd: $ oc rsh ${fluentd-pod} sh-4.2# cat input-syslog-default-syslog.conf <source> @type systemd @label @INGRESS path "/var/log/journal" pos_file /var/log/journal.pos tag journal read_from_head "false" </source> 3.Tail ES log for namespaces that was created prior to EFK deployments 4.Log in kibana Actual results: 3.Index mappings was created in ES for namespaces that was created prior to EFK deployments 4.Roll over logs are displayed Expected results: 3.Index mappings should not be created in ES for namespaces that was created prior to EFK deployments 4.Roll over logs (and indices) should not exist Additional info:
Issue exist after removing the pos file on node, it recreates automatically after removing: # cat /var/log/journal.pos s=e9708e5b637f4472942b54b9f7f660b6;i=4c3c2;b=8ff3def7672a47a09f1455574362d261;m=29dc7cafe;t=54565779e6744;x=40f8c1889c5f625e # rm -rf journal.pos # cat /var/log/journal.pos s=e9708e5b637f4472942b54b9f7f660b6;i=5e3fd;b=8ff3def7672a47a09f1455574362d261;m=305262080;t=54565defcbcc6;x=f093d8100b19e179
There is definitely a bug here. Investigating.
Commit pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/3f16fe806cf9c282d1916ce947fd5b5ab28a9e3c Bug 1410705 - Roll over log entries are visible when journal-read-from-head is false https://bugzilla.redhat.com/show_bug.cgi?id=1410705 The fluent-plugin-systemd plugin is parsing "false" as a string value, which always evaluates to boolean true. The fix is to not set read_from_head when it is not true, this will use the default false internal value. See also https://github.com/reevoo/fluent-plugin-systemd/issues/19 Also note that due to https://github.com/reevoo/fluent-plugin-systemd/issues/20 and https://github.com/ledbettj/systemd-journal/issues/64 there may be a few old entries read, even if read_from_head is false.
Waiting for next release of 3.4 to release this fix - adding tdawson To ssh://rmeggins.redhat.com/rpms/logging-fluentd-docker f8149e4..3a4733b rhaos-3.4-rhel-7 -> rhaos-3.4-rhel-7 Just needs a rebuild of logging-fluentd-docker
This should be in openshift3/logging-fluentd:3.4.1-2 or newer
verified with [root@host-8-175-197 ~]# oc version oc v3.4.1.2 kubernetes v1.4.0+776c994 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://host-8-175-197.host.centralci.eng.rdu2.redhat.com:8443 openshift v3.4.1.2 kubernetes v1.4.0+776c994
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0219