Bug 1347368 - fluentd should be run in the node timezone
Summary: fluentd should be run in the node timezone
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.2.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.2.1
Assignee: Rich Megginson
QA Contact: chunchen
Depends On:
Blocks: 1267746
TreeView+ depends on / blocked
Reported: 2016-06-16 15:49 UTC by François Cami
Modified: 2019-12-16 05:57 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, the logging deployer did not configure the local timezone for fluentd. The deployer has been updated to ensure new logging deployments properly configure fluentd for the local timezone. https://access.redhat.com/solutions/2377771 details how to re-configure existing deployments
Clone Of:
Last Closed: 2016-08-11 18:38:15 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2377771 0 None None None 2016-06-16 15:52:43 UTC
Red Hat Product Errata RHBA-2016:1608 0 normal SHIPPED_LIVE Red Hat OpenShift Enterprise bug fix and enhancement update 2016-08-11 22:37:53 UTC

Description François Cami 2016-06-16 15:49:50 UTC
Description of problem:
In OSE 3.2.0 + RHEL 7.2, the EFK templates do not set the timezone of fluentd to the timezone of the host, leading the year detection filter to fail and filling the logs with:
"no implicit conversion of nil into String "

Full logs:
2016-06-14 06:48:52 -0400 [info]: reading config file path="/etc/fluent/fluent.conf"
2016-06-14 06:49:00 -0400 [warn]: failed to expand `%Q[#{ Time.at(time) > Time.now ? (temp_time = Time.parse(Time.at(time).to_s.gsub(Time.at(time).year.to_s, (tag_parts[3].nil? ? Time.at(time).year.to_s : tag_parts[3][9,4]) )).to_datetime.to_s; Time.parse(temp_time) > Time.now ? Time.parse(temp_time.gsub(Time.parse(temp_time).year.to_s, (Time.parse(temp_time).year - 1).to_s )).to_datetime.to_s : Time.parse(temp_time).to_datetime.to_s ) : Time.at(time).to_datetime.to_s }]` error_class=TypeError error="no implicit conversion of nil into String"

The solution is to update the deployment config to use the same timezone in the fluentd container as in the node:
# oc edit dc logging-fluentd
## add under volumeMounts:
            - name: localtime
              mountPath: /etc/localtime
              readOnly: true
## add under volumes:
          - name: localtime
              path: /etc/localtime

Upstream fix: ff4526ad354a5d58ef6b12171e23e0b2c41b2c35
KCS: https://access.redhat.com/solutions/2377771

Comment 2 Luke Meyer 2016-06-21 15:00:45 UTC
There is probably a quick `oc volume` command to add this volume FWIW.

Comment 16 Rich Megginson 2016-08-03 21:28:53 UTC
Slight correction: part of the fix is to the fluentd image, so that it won't spew out errors when it encounters dates in the future.  So existing customers will get some benefit by updating the fluentd image. I'm not sure how that works - they'll have to redeploy fluentd with the latest image, something like oc deploy dc logging-fluentd --latest?

Comment 18 Xia Zhao 2016-08-04 07:41:11 UTC
To be clear, I'm doing a completely fresh install for the 3.2.1 level logging components on my machine.

Comment 23 errata-xmlrpc 2016-08-11 18:38:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.