Created attachment 1724560 [details] Domain XML dump of the running VM Created attachment 1724560 [details] Domain XML dump of the running VM Description of problem: Configuration of timezone using <clock offset='timezone' timezone='America/New_York'/> is being ignored by the Guest OS. Instead, the local time of the guest OS is the same as of the host OS. By looking at the QEMU process it is seen that the TZ environment variable is indeed defined and configured. This issue happens in the CNV environment where the qemu and libvirt processes are running inside a container. The issue doesn't reproduce on the following versions (QEMU and libvirt also run in a container): libvirt version: 6.0.0, package: 16.fc31 (Unknown, 2020-04-07-15:55:55, ), qemu version: 4.2.0qemu-kvm-4.2.0-27.fc31, kernel: 4.18.0-227.el8.x86_64 Version-Release number of selected component (if applicable): libvirt version: 6.0.0, package: 25.4.module+el8.2.1+8060+c0c58169 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2020-09-11-18:58:56, ), qemu version: 4.2.0qemu-kvm-4.2.0-29.module+el8.2.1+7990+27f1e480.4, kernel: 4.18.0-193.24.1.el8_2.dt1.x86_64, hostname: testvmin6jnphzb9ssk4c2tft5tnhbg84r6dj5b8ppfj4bmsn6snrhs How reproducible: 100% Steps to Reproduce: 1. Create a VM with the same properties as in the attached domain XML dump. 2. Observe the Guest OS (Linux) local time using hwclock Actual results: Local time equal to the host OS time Expected results: local time adjusted to the timezone that was configured in the domain XML. Additional info: Please see the attached files.
Created attachment 1724561 [details] QEMU log of the running VM
Created attachment 1724563 [details] seaBIOS log of the running VM
Created attachment 1724564 [details] seaBIOS log of the running VM
Hello, can you please also share the following information, please? - QEMU command line that was created by libvirt - check whether there is a TZ=America/New_York environment variable of the QEMU process - output of `hwclock` from the host, container and guest Thank you.
TZ env variable is set as can be seen from attachment 1724561 [details]. Therefore I don't think this is a libvirt bug. Let's switch over to QEMU what they think.
Tried with qemu-kvm-5.2.0-0.module+el8.4.0+8855+a9e237a9.x86_64 and rhel8.4.0 guest: TZ=Asia/Shanghai \ /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -sandbox on \ -machine q35 \ ...... -rtc base=localtime \ ...... Guest time zone can't be set successfully. # timedatectl Local time: Mon 2020-12-14 04:10:30 EST Universal time: Mon 2020-12-14 09:10:30 UTC RTC time: Mon 2020-12-14 17:10:29 Time zone: America/New_York (EST, -0500)
From a qemu-kvm/general or API/CLI point of view, it looks to me like the Time zone is set - certainly the output from the timedatectl command displays the wrong thing. I'm not sure what Kubevirt/CNV uses in order to ascertain the time is wrong though. Amnon - seems to be some sort of guest/time based output thing - perhaps someone from your team could look a bit and provide some feedback. FWIW: My host has qemu-5.1.0-8.fc33 From the US/EST TZ, if I configure "<clock offset='timezone' timezone='America/New_York'/>" and boot I get the following in the guest: # timedatectl Local time: Mon 2020-12-21 10:59:06 EST Universal time: Mon 2020-12-21 15:59:06 UTC RTC time: Mon 2020-12-21 10:59:05 Time zone: America/New_York (EST, -0500) System clock synchronized: yes NTP service: active RTC in local TZ: no If I change that to "<clock offset='timezone' timezone='Asia/Shanghai'/>" and reboot/restart the guest I get the following in the guest: # timedatectl Local time: Mon 2020-12-21 11:02:47 EST Universal time: Mon 2020-12-21 16:02:47 UTC RTC time: Tue 2020-12-22 00:02:46 Time zone: America/New_York (EST, -0500) System clock synchronized: yes NTP service: active RTC in local TZ: no So, the guest does seem to be getting the RTC timezone change, but the "Time zone" output isn't right. For both the qemu-kvm command has '...-rtc base=localtime...' and I can see in my qemu guest output the 'TZ=America/New_York' or 'TZ=Asia/Shanghai' on the command line appropriately.
Igor - do you know (can you find out if not) how CNV/Kubevirt determines what the TZ is in the guest/container? What empirical data is used and can you add that in the bz? Using the same example from Comment 7, but adding hwclock output I get for America/New_York: # hwclock 2020-12-21 06:24:23.120958-05:00 # timedatectl Local time: Mon 2020-12-21 11:24:27 EST Universal time: Mon 2020-12-21 16:24:27 UTC RTC time: Mon 2020-12-21 11:24:26 Time zone: America/New_York (EST, -0500) System clock synchronized: yes NTP service: active RTC in local TZ: no and within minutes, for Asia/Shanghai: # hwclock 2020-12-21 19:27:22.259927-05:00 # timedatectl Local time: Mon 2020-12-21 11:27:24 EST Universal time: Mon 2020-12-21 16:27:24 UTC RTC time: Tue 2020-12-22 00:27:24 Time zone: America/New_York (EST, -0500) System clock synchronized: yes NTP service: active RTC in local TZ: no
Hi @john ferl we run "sudo hwclock --localtime" on the guest and parse the output. From the previous comments I see that another issue was found, not related to the one I've opened the bug about. I see that there is an issue with the output of timedatectl Time Zone string.
Hi: To isolate the issue, can you try with the same libvirt version to see if it's an issue of libvirt or not? Btw, for guest timezone, need to make sure it's not a configuration issue as what [1] said. [1] https://bugzilla.redhat.com/show_bug.cgi?id=859868#c10 Thanks
Created attachment 1743228 [details] latest reproduction
Hi, Sorry but I can't locate the specific libvirt version that you want me to use for isolation. It will be a time-heavy task since I will have to manually rebuild the containers in downstream. I hope that the latest reproduction with more info will help. Please see the attached "latest_reproduction.txt" file
(In reply to jason wang from comment #10) > Hi: > > To isolate the issue, can you try with the same libvirt version to see if > it's an issue of libvirt or not? > > Btw, for guest timezone, need to make sure it's not a configuration issue as > what [1] said. > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=859868#c10 > > Thanks I don't think this is related to libvirt. Libvirt should always put TZ= into the qemu's environment. Hence, I think this can be reproduced using solely qemu command line.
I have tried with following qemu versions, the issue always can be reproduced. qemu-kvm-5.1.0-2.module+el8.3.0+7652+b30e6901.x86_64 qemu-kvm-4.2.0-29.module+el8.2.1+8442+7a3eadf7.5.x86_64 qemu-kvm-4.2.0-27.module+el8.2.1+7092+9d345e72.x86_64 and like comment said, the guest RTC time is the same as TZ=xxx. Just the guest local time or UTC time and time zone are not correct. To avoid ntp, the guest is booted without network configure. The host and guest time zone are both America/New_York, and boot guest with TZ=Asia/Shanghai \ /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -sandbox on \ -machine q35 \ ...... -rtc base=localtime \ -net none \ ...... The results are attached. The left time is from host, the right time is from guest. We can see the guest RTC time is indeed Asia/Shanghai 2021-01-06 10:08:07 The guest UTC time will be set according to the RTC time and timezone America/New_York.
Created attachment 1744749 [details] time-results
Adding more info: If without TZ=xxx, only -rtc base=localtime other steps are the same with comment 14. The guest RTC time is the same with host local time. The results are attached with localtimenoTZ. If only -rtc base=utc, the guest RTC time is the same with host utc time, the results are attached with utcnoTZ. From all results, we can see the TZ=xxx only affects guest RTC time, will not change the guest time zone. The guest local time is set according to the RTC time and time zone in/etc/localtime and /etc/adjtime.
Created attachment 1744789 [details] localtimenoTZ
Created attachment 1744790 [details] utcnoTZ
So for some reason I thought that there might be some special ACPI/RTC/other API that can provide some default timezone to the OS from the HW, maybe only in some virtual RTC. I did not find anything like that however and, if I understand your request correctly, you want the guest OS timezone to be set from the host. I do not think this was ever possible and hence this is not really a bug. The XML snippet: <clock offset='timezone' timezone='America/New_York'/> Just tells QEMU that the RTC clock it provides to the guest is supposed to be set to whatever time is in that timezone. Guest does not see the timezone, it sees the time provided by QEMU and treats it as UTC. That might not work if the timezone is mistyped, in which case QEMU cannot find the timezone description and falls back to just providing UTC (my rough guess). So anyway, if you are in UTC-2, tell QEMU to provide RTC time of TZ UTC-4, then the guest OS will see that RTC has time of your time - 2 hours. But by default it treats it as UTC and on top of that the guest has some timezone configured, let's say UTC-1, so the local time in the guest will be your time - 5 hours. I believe that is what you are seeing and it all works correctly. Feel free to correct me if I am wrong, of course.
In my case, although TZ="Asia/Shanghai" && -rtc base=localtime , the RTC on the guest didn't been configured for that time. Following is the output from the Fedora 32 guest: --------------------------------------------------- [fedora@vmi-fedora ~]$ timedatectl Local time: Thu 2021-01-07 11:03:33 UTC Universal time: Thu 2021-01-07 11:03:33 UTC RTC time: Thu 2021-01-07 11:03:33 Time zone: Etc/UTC (UTC, +0000) System clock synchronized: no NTP service: active RTC in local TZ: no [fedora@vmi-fedora ~]$ TZ="Asia/Shanghai" date Thu 07 Jan 2021 07:06:52 PM CST [fedora@vmi-fedora ~]$ date Thu 07 Jan 2021 11:06:54 AM UTC Can you please try to reproduce the issue while running the qemu process from inside a container? TIA Igor
Setting the timezone field in libvirt XML does *not* do anything to configure the guest OS. The only effect is has is to initialize the RTC to the time according to that timezone, instead of initializing the RTC to UTC. The guest OS administrator still has the task of configuring their OS to use the RTC as if it was local time. Use of this timezone setting does not make sense for most OS, as any sane OS will want the RTC to be initialized to UTC, and exclusively control of timezone inside the guest. The only OS where it makes sense to set timezone in the libvirt XML is a Windows guest, as Windows will generally assume the RTC is in local time, not UTC. To see if libvirt is operating correctly you need to compare *only* the RTC time in host vs guest - ignore the guest OS timezone config. Essentially "hwclock" is the tool to use to check things In host: # hwclock 2021-01-13 15:47:03.997971+00:00 In guest running in UTC # virsh dumpxml f34x86_64 | grep '<clock' <clock offset='utc'> # ssh f34x86_64 $ hwclock 2021-01-13 15:49:57.589657+00:00 In guest running in a timezone that is 5 hours different # virsh dumpxml f34x86_64 | grep '<clock' <clock offset='timezone' timezone='America/New_York'> # ssh f34x86_64 $ hwclock 2021-01-13 10:51:36.287845+00:00 Overall, I don't see any bug here, rather a mis-understanding of effects of the libvirt timezone setting.
OK no problem, I will post the output of hwclock from guest and from the host (from the container I can't, permission denied). What I am trying to say is that in my case, when running the qemu from inside a container, RTC of the guest isn't being affected by the TZ passed to the qemu process. Why is it important to mention that the qemu is running in the container? because otherwise it's probably not reproduced.
Hi Daniel, Here is a reproduction of the issue I am talking about using the same steps you did host: sh-4.4# hwclock 2021-03-04 09:24:40.443602+00:00 In guest running UTC [root@vm-cirros /]# virsh dumpxml openshift-cnv_vm-cirros | grep '<clock' <clock offset='utc' adjustment='reset'/> # hwclock Thu Mar 4 09:36:08 2021 .157276 seconds In guest running in a timezone that is 5 hours different [root@vm-cirros /]# virsh dumpxml openshift-cnv_vm-cirros | grep '<clock' <clock offset='timezone' timezone='America/New_York'/> # hwclock Thu Mar 4 09:40:51 2021 .084068 seconds libvirt/qemu details: 2021-03-04 09:39:47.452+0000: starting up libvirt version: 6.6.0, package: 7.3.module+el8.3.0+9547+7d548490 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2021-01-21-04:46:26, ), qemu version: 5.1.0qemu-kvm-5.1.0-14.module+el8.3.0+8790+80f9c6d8.1, kernel: 4.18.0-240.10.1.el8_3.x86_64, hostname: vm-cirros Please advise.
Is your guest OS perhaps running NTP and synchronizing the HW clock back to UTC ? Also in the host OS container where QEMU is running, please also confirm that the simple 'date' tool sees effects of $TZ settings. $ date Thu 4 Mar 10:12:10 GMT 2021 $ TZ=America/New_York date Thu 4 Mar 05:12:19 EST 2021
(In reply to Daniel Berrangé from comment #24) > Is your guest OS perhaps running NTP and synchronizing the HW clock back to > UTC ? > Nope. the guest OS has no NTP installed on it > Also in the host OS container where QEMU is running, please also confirm > that the simple 'date' tool sees effects of $TZ settings. > > $ date > Thu 4 Mar 10:12:10 GMT 2021 > $ TZ=America/New_York date > Thu 4 Mar 05:12:19 EST 2021 That is strange, TZ has no effect on the host OS container [root@vm-cirros /]# date Thu Mar 4 10:54:06 UTC 2021 [root@vm-cirros /]# TZ=America/New_York date Thu Mar 4 10:54:14 America 2021 Currently I have no clue what mechanism could make TZ ineffective.
(In reply to Igor Bezukh from comment #25) > That is strange, TZ has no effect on the host OS container > > [root@vm-cirros /]# date > Thu Mar 4 10:54:06 UTC 2021 > [root@vm-cirros /]# TZ=America/New_York date > Thu Mar 4 10:54:14 America 2021 > > Currently I have no clue what mechanism could make TZ ineffective. Ok, so this is the root cause then. Do you have the 'tzdata' RPM present ? I'd hope so because IIRC it is a dependency of glibc-common
(In reply to Daniel Berrangé from comment #26) > (In reply to Igor Bezukh from comment #25) > > That is strange, TZ has no effect on the host OS container > > > > [root@vm-cirros /]# date > > Thu Mar 4 10:54:06 UTC 2021 > > [root@vm-cirros /]# TZ=America/New_York date > > Thu Mar 4 10:54:14 America 2021 > > > > Currently I have no clue what mechanism could make TZ ineffective. > > Ok, so this is the root cause then. > > Do you have the 'tzdata' RPM present ? I'd hope so because IIRC it is a > dependency of glibc-common Yep, I have it [root@vm-cirros /]# rpm -qa | grep tzdata tzdata-2020d-1.el8.noarch
Does /usr/share/zoneinfo/America/New_York exist, and does 'TZ=America/New_York strace date 2>&1 | grep open' show it being opened successfully ?
(In reply to Daniel Berrangé from comment #28) > Does /usr/share/zoneinfo/America/New_York exist, and does > 'TZ=America/New_York strace date 2>&1 | grep open' show it being opened > successfully ? Looks like the whole zoneinfo directory doesn't even exist: [root@vm-cirros /]# ls /usr/share/zoneinfo ls: cannot access '/usr/share/zoneinfo': No such file or directory
Ok, this definitely isn't a QEMU bug any more. Whatever is responsible for building the container image you're using appears to have purged all TZ data, and that needs to be fixed.
Daniel thank you for your help! Taking this bug back to CNV virt. It's an issue with our midstream build then.
Since this is a regression bug then no blocking impact expected on existing workloads. Target Release will be set to 4.8
Verify with: virt-operator-container-v4.9.0-59 [root@vm-cirros /]# rpm -qa | grep tzdata tzdata-2021a-1.el8.noarch [root@vm-cirros /]# ls /usr/share/zoneinfo/ Africa/ Australia/ Cuba Etc/ GMT-0 Indian/ Libya NZ-CHAT Portugal US/ iso3166.tab zone.tab America/ Brazil/ EET Europe/ GMT0 Iran MET Navajo ROC UTC leapseconds zone1970.tab Antarctica/ CET EST GB Greenwich Israel MST PRC ROK Universal posix/ Arctic/ CST6CDT EST5EDT GB-Eire HST Jamaica MST7MDT PST8PDT Singapore W-SU posixrules Asia/ Canada/ Egypt GMT Hongkong Japan Mexico/ Pacific/ Turkey WET right/ Atlantic/ Chile/ Eire GMT+0 Iceland Kwajalein NZ Poland UCT Zulu tzdata.zi
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Virtualization 4.9.0 Images security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:4104