Bug 1674170 - Master pods are not matched the timezone with worker nodes and its running host
Summary: Master pods are not matched the timezone with worker nodes and its running host
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.11.z
Assignee: Scott Dodson
QA Contact: Gaoyun Pei
Depends On:
Blocks: 1690557
TreeView+ depends on / blocked
Reported: 2019-02-09 14:57 UTC by Daein Park
Modified: 2019-06-06 10:11 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, timezone configuration was not mounted into the control plane static pods which led to logging timestamps not aligning with all other system logs. The host's timezone configuration is now mounted into the control plane pods.
Clone Of:
Last Closed: 2019-04-11 05:38:26 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0636 0 None None None 2019-04-11 05:38:35 UTC

Internal Links: 1717840

Description Daein Park 2019-02-09 14:57:43 UTC
Description of problem:

master services make same timezone with node kubelet.

Until v3.9 the master services(api, controller, etcd) are running as systemd unit(host processes) on the master nodes, it's same manner with "atomic-openshift-node.service", it's no problem.

But, as of v3.10 and v3.11 the mater services are changed as static pods which known as control plan pods, and the definitions of pod are not considered each timezone, so it's running as UTC(default), but "atomic-openshift-node.service" yet are running on the host configured other timezone.

e.g.> the master nodes and worker nodes are same timezone as JST, but master services are UTC.
master1 ~# date
Thu Feb  7 09:24:10 JST 2019

master1 ~# for pod in $(echo master-api-master1.example.com master-controllers-master1.example.com master-etcd-master1.example.com); do echo $pod; oc rsh $pod date; done

Thu Feb  7 00:24:41 UTC 2019

Thu Feb  7 00:24:42 UTC 2019

Thu Feb  7 00:24:43 UTC 2019

master1 ~# ssh node1 date
Thu Feb  7 09:24:55 JST 2019


The inconsistent timezone can cause not matching log timestamp and unexpected result on time dependent tasks, such as CronJob or Service Serving Certs and so on.

Version-Release number of the following components:
rpm -q openshift-ansible


rpm -q ansible


ansible --version

  ansible 2.6.11
    config file = /usr/share/ansible/openshift-ansible/ansible.cfg
    configured module search path = [u'/root/.ansible/plugins/modules',u'/usr/share/ansible/plugins/modules']
    ansible python module location = /usr/lib/python2.7/site-packages/ansible
    executable location = /usr/bin/ansible
    python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

How reproducible:

Always, the master services static pods are running on the master nodes with UTC as default.

Steps to Reproduce:

Actual results:

master services are UTC, but nodes hosts are other timezone.

Expected results:

The timezone is same with master nodes and worker nodes.

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Scott Dodson 2019-02-11 19:05:14 UTC
Can you try bind mounting /etc/localtime and /usr/share/zoneinfo into the static pod to see if this resolves the problem?

Comment 2 Daein Park 2019-02-11 23:45:51 UTC
Thank you for your workaround @Scott,

And the solution you provided is valid to resolve this issue. 
I've already opened the PR here after verifying same solution: https://github.com/openshift/openshift-ansible/pull/11160

I just mounted only a "/etc/localtime" to resolve it.
Do you think it had better to mount additional "/usr/share/zoneinfo" either ?

Comment 3 Scott Dodson 2019-02-12 15:38:01 UTC
I guess that means timezone data is installed inside the container image so it should be safe with only /etc/localtime as long as the base image for the control plane never changes which I can't see happening. I'd like to get some feedback from the pod teams on the change before we move forward.

Comment 4 Scott Dodson 2019-02-28 20:02:10 UTC
In openshift-ansible-3.11.89-1

Comment 6 Gaoyun Pei 2019-03-18 09:29:37 UTC
Verify this bug with openshift-ansible-3.11.96-1.git.0.c2a3f89.el7.noarch.rpm

[root@ip-172-18-8-68 ~]# for pod in $(echo master-api-ip-172-18-8-68.ec2.internal master-controllers-ip-172-18-8-68.ec2.internal master-etcd-ip-172-18-8-68.ec2.internal ); do echo $pod; oc rsh $pod date; done
Mon Mar 18 00:45:10 EDT 2019
Mon Mar 18 00:45:11 EDT 2019
Mon Mar 18 00:45:11 EDT 2019
[root@ip-172-18-8-68 ~]# date
Mon Mar 18 00:45:13 EDT 2019

Master pods: api, controller, etcd have the same timezone with the host.

Comment 8 errata-xmlrpc 2019-04-11 05:38:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.