Bug 1488941
| Summary: | JOURNAL_READ_FROM_HEAD="false" not honored - pre-existing pod logs indexed on logging-fluentd start | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Mike Fiedler <mifiedle> | ||||||
| Component: | Logging | Assignee: | Noriko Hosoi <nhosoi> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> | ||||||
| Severity: | medium | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 3.6.0 | CC: | anli, aos-bugs, jcantril, mifiedle, nhosoi, pportant, rmeggins | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | 3.7.0 | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: |
In the configuration for the fluentd systemd input plugin, read_from_head parameter was not set properly based on the environment variable JOURNAL_READ_FROM_HEAD or its corresponding ansible parameter openshift_logging_fluentd_journal_read_from_head.
Due to the problem, the full contents of pre-existing logs were indexed instead of the latest logs captured by "tail" when a pos_file does not exist, which happens when the logging system is initially deployed or a pos_file is deleted.
With this bug fix, the parameter is correctly set. And based on the setting, if JOURNAL_READ_FROM_HEAD=true, all the logs are indexed; if JOURNAL_READ_FROM_HEAD=false, logs read from "tail" are indexed when a pos_file does not exist.
|
Story Points: | --- | ||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2017-11-28 22:09:56 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
|
Description
Mike Fiedler
2017-09-06 13:39:05 UTC
Created attachment 1322668 [details]
syslog fluentd config
The default value is false: https://github.com/reevoo/fluent-plugin-systemd/blob/v0.0.8/lib/fluent/plugin/in_systemd.rb#L12 config_param :read_from_head, :bool, default: false due to an earlier bug in fluent-plugin-systemd, if you set JOURNAL_READ_FROM_HEAD=false, we just omit the setting `read_from_head` from the config, and let it use the default which is false: https://github.com/openshift/origin-aggregated-logging/blob/master/fluentd/generate_throttle_configs.rb#L136 # workaround for https://github.com/reevoo/fluent-plugin-systemd/issues/19 if ENV['JOURNAL_READ_FROM_HEAD'] == "true" file.write(<<-CONF2) read_from_head true CONF2 end which is why you don't see the setting. So, either this is a bug in fluent-plugin-systemd, or . . . Are you reconfiguring fluentd after the initial deployment? If so, then there may be an old /var/log/journal.pos file - if this file exists, fluentd will take this as the starting point to start reading from the journal, and will ignore the JOURNAL_READ_FROM_HEAD setting. Originally I was reconfiguring after deployment. I deployed a new cluster with openshift_logging_fluentd_read_from_head=false in the inventory and ended up in the same state. sh-4.2# env | grep HEAD JOURNAL_READ_FROM_HEAD= sh-4.2# pwd /etc/fluent/configs.d/dynamic sh-4.2# cat input-syslog-default-syslog.conf <source> @type systemd @label @INGRESS path "/var/log/journal" pos_file /var/log/journal.pos tag journal </source> To reiterate, this looks like it is behaving as described. If /var/log/journal.pos exists, then it just reads from that position in the journal. It seems since the version v3.6.173.0.27, the fluentd configuration files are modified including the JOURNAL_READ_FROM_HEAD handling.
+++ b/configs.d/openshift/input-pre-systemd.conf
@@ -0,0 +1,9 @@
+<source>
+ @type systemd
+ @label @INGRESS
+ path "#{ENV['JOURNAL_SOURCE'] || '/run/log/journal'}"
+ pos_file "#{ENV['JOURNAL_POS_FILE'] || '/var/log/journal.pos'}"
+ filters "#{ENV['JOURNAL_FILTERS_JSON'] || '[]'}"
+ tag journal
+ read_from_head "#{ENV['JOURNAL_READ_FROM_HEAD'] || 'false'}"
+</source>
Please note that the latest version in the brew is:
logging-fluentd-docker-v3.6.173.0.56-1
Mike, I guess you had a chance to upgrade your system since early September. Do you still have the problem? Thanks!
I do not see the problem on logging-fluentd 3.7.0-0.146.0. Not sure what that corresponds to for 3.6 as I have not tested there for a bit. Let me know if I should start up a 3.6 cluster and test there as well. Thanks for your quick response, Mike!! I think the QE team is verifying bugs as part of the next 3.6.x release. Since this is a unit test, can we ask someone in the QE team assigned to the tasks to verify with the latest 3.6.x? Do you happen to know to whom we could ask? @jcantril, in such a case, can we change the status of this bug to ON_QA directly or should we set to MODIFIED once and add this bz to some errata and let the errata process change the status to ON_QA? Thanks! I can take QE for this one on 3.6. I'll let Jeff answer how to get it into the errata flow. I think you either need to target this one for 3.6.z or create a clone targeted for 3.6.z, but better to let an expert answer. (In reply to Mike Fiedler from comment #8) > I can take QE for this one on 3.6. Once again, thanks a lot, Mike! @anli is this something you can verify is resolved? Jeff, will test it and add comment @richm, do you think this 'JOURNAL_READ_FROM_HEAD="false" not honored' issue had existed before your implementation to "Allow using both json-file and journald"? I'd rather think the problem appeared temporarily just with the image due to some mismatch, e.g., O_A and O_A_L. The reason I'm asking is the Doc... If this bug was not released, we should not doc it, I guess. (In reply to Noriko Hosoi from comment #12) > @richm, do you think this 'JOURNAL_READ_FROM_HEAD="false" not honored' issue > had existed before your implementation to "Allow using both json-file and > journald"? Yes, for quite some time. > I'd rather think the problem appeared temporarily just with the > image due to some mismatch, e.g., O_A and O_A_L. The reason I'm asking is > the Doc... If this bug was not released, we should not doc it, I guess. I'm not sure if customers have run into this. I think we should document it, just in case some customer has run into this. It also may have been a bug in fluent-plugin-systemd or systemd-journal ruby library, which has been fixed because we are now using a newer version. With logging-fluentd/v3.7.0-0.188.0.0. the test pass, so move bug to verified. When the JOURNAL_READ_FROM_HEAD is not set, The journal logs wasn't read from head. When set JOURNAL_READ_FROM_HEAD="false", The journal logs wasn't read from head Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3188 |