Bug 1674170

Summary: Master pods are not matched the timezone with worker nodes and its running host
Product: OpenShift Container Platform Reporter: Daein Park <dapark>
Component: InstallerAssignee: Scott Dodson <sdodson>
Installer sub component: openshift-ansible QA Contact: Gaoyun Pei <gpei>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: unspecified CC: dapark, dcaldwel, dlbewley, gpei
Version: 3.11.0   
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, timezone configuration was not mounted into the control plane static pods which led to logging timestamps not aligning with all other system logs. The host's timezone configuration is now mounted into the control plane pods.
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-04-11 05:38:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1690557    

Description Daein Park 2019-02-09 14:57:43 UTC
Description of problem:

master services make same timezone with node kubelet.

Until v3.9 the master services(api, controller, etcd) are running as systemd unit(host processes) on the master nodes, it's same manner with "atomic-openshift-node.service", it's no problem.

But, as of v3.10 and v3.11 the mater services are changed as static pods which known as control plan pods, and the definitions of pod are not considered each timezone, so it's running as UTC(default), but "atomic-openshift-node.service" yet are running on the host configured other timezone.

e.g.> the master nodes and worker nodes are same timezone as JST, but master services are UTC.
~~~
master1 ~# date
Thu Feb  7 09:24:10 JST 2019

master1 ~# for pod in $(echo master-api-master1.example.com master-controllers-master1.example.com master-etcd-master1.example.com); do echo $pod; oc rsh $pod date; done

master-api-master1.example.com
Thu Feb  7 00:24:41 UTC 2019

master-controllers-master1.example.com
Thu Feb  7 00:24:42 UTC 2019

master-etcd-master1.example.com
Thu Feb  7 00:24:43 UTC 2019

master1 ~# ssh node1 date
Thu Feb  7 09:24:55 JST 2019

~~~

The inconsistent timezone can cause not matching log timestamp and unexpected result on time dependent tasks, such as CronJob or Service Serving Certs and so on.

Version-Release number of the following components:
rpm -q openshift-ansible

  openshift-ansible-3.11.59-1.git.0.ba8e948.el7.noarch

rpm -q ansible

  ansible-2.6.11-1.el7ae.noarch

ansible --version

  ansible 2.6.11
    config file = /usr/share/ansible/openshift-ansible/ansible.cfg
    configured module search path = [u'/root/.ansible/plugins/modules',u'/usr/share/ansible/plugins/modules']
    ansible python module location = /usr/lib/python2.7/site-packages/ansible
    executable location = /usr/bin/ansible
    python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

How reproducible:

Always, the master services static pods are running on the master nodes with UTC as default.

Steps to Reproduce:
1.
2.
3.

Actual results:

master services are UTC, but nodes hosts are other timezone.

Expected results:

The timezone is same with master nodes and worker nodes.

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Scott Dodson 2019-02-11 19:05:14 UTC
Can you try bind mounting /etc/localtime and /usr/share/zoneinfo into the static pod to see if this resolves the problem?

Comment 2 Daein Park 2019-02-11 23:45:51 UTC
Thank you for your workaround @Scott,

And the solution you provided is valid to resolve this issue. 
I've already opened the PR here after verifying same solution: https://github.com/openshift/openshift-ansible/pull/11160

I just mounted only a "/etc/localtime" to resolve it.
Do you think it had better to mount additional "/usr/share/zoneinfo" either ?

Comment 3 Scott Dodson 2019-02-12 15:38:01 UTC
I guess that means timezone data is installed inside the container image so it should be safe with only /etc/localtime as long as the base image for the control plane never changes which I can't see happening. I'd like to get some feedback from the pod teams on the change before we move forward.

Comment 4 Scott Dodson 2019-02-28 20:02:10 UTC
In openshift-ansible-3.11.89-1

Comment 6 Gaoyun Pei 2019-03-18 09:29:37 UTC
Verify this bug with openshift-ansible-3.11.96-1.git.0.c2a3f89.el7.noarch.rpm

[root@ip-172-18-8-68 ~]# for pod in $(echo master-api-ip-172-18-8-68.ec2.internal master-controllers-ip-172-18-8-68.ec2.internal master-etcd-ip-172-18-8-68.ec2.internal ); do echo $pod; oc rsh $pod date; done
master-api-ip-172-18-8-68.ec2.internal
Mon Mar 18 00:45:10 EDT 2019
master-controllers-ip-172-18-8-68.ec2.internal
Mon Mar 18 00:45:11 EDT 2019
master-etcd-ip-172-18-8-68.ec2.internal
Mon Mar 18 00:45:11 EDT 2019
[root@ip-172-18-8-68 ~]# date
Mon Mar 18 00:45:13 EDT 2019

Master pods: api, controller, etcd have the same timezone with the host.

Comment 8 errata-xmlrpc 2019-04-11 05:38:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0636