Description of problem: master services make same timezone with node kubelet. Until v3.9 the master services(api, controller, etcd) are running as systemd unit(host processes) on the master nodes, it's same manner with "atomic-openshift-node.service", it's no problem. But, as of v3.10 and v3.11 the mater services are changed as static pods which known as control plan pods, and the definitions of pod are not considered each timezone, so it's running as UTC(default), but "atomic-openshift-node.service" yet are running on the host configured other timezone. e.g.> the master nodes and worker nodes are same timezone as JST, but master services are UTC. ~~~ master1 ~# date Thu Feb 7 09:24:10 JST 2019 master1 ~# for pod in $(echo master-api-master1.example.com master-controllers-master1.example.com master-etcd-master1.example.com); do echo $pod; oc rsh $pod date; done master-api-master1.example.com Thu Feb 7 00:24:41 UTC 2019 master-controllers-master1.example.com Thu Feb 7 00:24:42 UTC 2019 master-etcd-master1.example.com Thu Feb 7 00:24:43 UTC 2019 master1 ~# ssh node1 date Thu Feb 7 09:24:55 JST 2019 ~~~ The inconsistent timezone can cause not matching log timestamp and unexpected result on time dependent tasks, such as CronJob or Service Serving Certs and so on. Version-Release number of the following components: rpm -q openshift-ansible openshift-ansible-3.11.59-1.git.0.ba8e948.el7.noarch rpm -q ansible ansible-2.6.11-1.el7ae.noarch ansible --version ansible 2.6.11 config file = /usr/share/ansible/openshift-ansible/ansible.cfg configured module search path = [u'/root/.ansible/plugins/modules',u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] How reproducible: Always, the master services static pods are running on the master nodes with UTC as default. Steps to Reproduce: 1. 2. 3. Actual results: master services are UTC, but nodes hosts are other timezone. Expected results: The timezone is same with master nodes and worker nodes. Additional info: Please attach logs from ansible-playbook with the -vvv flag
Can you try bind mounting /etc/localtime and /usr/share/zoneinfo into the static pod to see if this resolves the problem?
Thank you for your workaround @Scott, And the solution you provided is valid to resolve this issue. I've already opened the PR here after verifying same solution: https://github.com/openshift/openshift-ansible/pull/11160 I just mounted only a "/etc/localtime" to resolve it. Do you think it had better to mount additional "/usr/share/zoneinfo" either ?
I guess that means timezone data is installed inside the container image so it should be safe with only /etc/localtime as long as the base image for the control plane never changes which I can't see happening. I'd like to get some feedback from the pod teams on the change before we move forward.
In openshift-ansible-3.11.89-1
Verify this bug with openshift-ansible-3.11.96-1.git.0.c2a3f89.el7.noarch.rpm [root@ip-172-18-8-68 ~]# for pod in $(echo master-api-ip-172-18-8-68.ec2.internal master-controllers-ip-172-18-8-68.ec2.internal master-etcd-ip-172-18-8-68.ec2.internal ); do echo $pod; oc rsh $pod date; done master-api-ip-172-18-8-68.ec2.internal Mon Mar 18 00:45:10 EDT 2019 master-controllers-ip-172-18-8-68.ec2.internal Mon Mar 18 00:45:11 EDT 2019 master-etcd-ip-172-18-8-68.ec2.internal Mon Mar 18 00:45:11 EDT 2019 [root@ip-172-18-8-68 ~]# date Mon Mar 18 00:45:13 EDT 2019 Master pods: api, controller, etcd have the same timezone with the host.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0636