Description of problem:
master services make same timezone with node kubelet.
Until v3.9 the master services(api, controller, etcd) are running as systemd unit(host processes) on the master nodes, it's same manner with "atomic-openshift-node.service", it's no problem.
But, as of v3.10 and v3.11 the mater services are changed as static pods which known as control plan pods, and the definitions of pod are not considered each timezone, so it's running as UTC(default), but "atomic-openshift-node.service" yet are running on the host configured other timezone.
e.g.> the master nodes and worker nodes are same timezone as JST, but master services are UTC.
master1 ~# date
Thu Feb 7 09:24:10 JST 2019
master1 ~# for pod in $(echo master-api-master1.example.com master-controllers-master1.example.com master-etcd-master1.example.com); do echo $pod; oc rsh $pod date; done
Thu Feb 7 00:24:41 UTC 2019
Thu Feb 7 00:24:42 UTC 2019
Thu Feb 7 00:24:43 UTC 2019
master1 ~# ssh node1 date
Thu Feb 7 09:24:55 JST 2019
The inconsistent timezone can cause not matching log timestamp and unexpected result on time dependent tasks, such as CronJob or Service Serving Certs and so on.
Version-Release number of the following components:
rpm -q openshift-ansible
rpm -q ansible
config file = /usr/share/ansible/openshift-ansible/ansible.cfg
configured module search path = [u'/root/.ansible/plugins/modules',u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]
Always, the master services static pods are running on the master nodes with UTC as default.
Steps to Reproduce:
master services are UTC, but nodes hosts are other timezone.
The timezone is same with master nodes and worker nodes.
Please attach logs from ansible-playbook with the -vvv flag
Can you try bind mounting /etc/localtime and /usr/share/zoneinfo into the static pod to see if this resolves the problem?
Thank you for your workaround @Scott,
And the solution you provided is valid to resolve this issue.
I've already opened the PR here after verifying same solution: https://github.com/openshift/openshift-ansible/pull/11160
I just mounted only a "/etc/localtime" to resolve it.
Do you think it had better to mount additional "/usr/share/zoneinfo" either ?
I guess that means timezone data is installed inside the container image so it should be safe with only /etc/localtime as long as the base image for the control plane never changes which I can't see happening. I'd like to get some feedback from the pod teams on the change before we move forward.
Verify this bug with openshift-ansible-3.11.96-1.git.0.c2a3f89.el7.noarch.rpm
[root@ip-172-18-8-68 ~]# for pod in $(echo master-api-ip-172-18-8-68.ec2.internal master-controllers-ip-172-18-8-68.ec2.internal master-etcd-ip-172-18-8-68.ec2.internal ); do echo $pod; oc rsh $pod date; done
Mon Mar 18 00:45:10 EDT 2019
Mon Mar 18 00:45:11 EDT 2019
Mon Mar 18 00:45:11 EDT 2019
[root@ip-172-18-8-68 ~]# date
Mon Mar 18 00:45:13 EDT 2019
Master pods: api, controller, etcd have the same timezone with the host.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.