When performing an interactive installation of RHV-H, the installed system is configured to interpret the hardware clock as the localtime, rather than the RHEL default of UTC. There is no option in the Anaconda UI to change this setting. As a result, if the configured timezone is in the western hemisphere (i.e. UTC-??), the system time will be in the future until the system has synced via NTP. This will cause the VDSM certificate in /etc/pki/vdsm/certs/vdsmcert.pem to be created with a "not before" time in the future, and libvirtd will fail to start.
Trying to install from the RHVH iso ? if so, which version ?
(In reply to Yuval Turgeman from comment #1) > Trying to install from the RHVH iso ? if so, which version ? I am using RHVH-4.1-20170706.1-RHVH-x86_64-dvd1.iso. Steps to reproduce: 1. Set system hardware clock to UTC. 2. Perform an interactive installation of RHV-H and specify an timezone in the western hemisphere (e.g. America/Los_Angeles) 3. After system reboots, note the following: - libvirtd fails to start - vdsmcert.pem not before date is in the future - timedatectl shows that the hardware clock is being interpreted as local time
I tried to follow the steps in comment #2 to reproduce the issue, but libvirtd was active, though the RTC was in local TZ. 1. After installation via RHVH-4.1-20170706.1-RHVH-x86_64-dvd1.iso finished, the time and date infos were: [root@dell-per730-34 ~]# timedatectl Local time: Mon 2017-08-14 07:22:05 PDT Universal time: Mon 2017-08-14 14:22:05 UTC RTC time: Mon 2017-08-14 07:22:05 Time zone: America/Los_Angeles (PDT, -0700) NTP enabled: no NTP synchronized: no RTC in local TZ: yes DST active: yes Last DST change: DST began at Sun 2017-03-12 01:59:59 PST Sun 2017-03-12 03:00:00 PDT Next DST change: DST ends (the clock jumps one hour backwards) at Sun 2017-11-05 01:59:59 PDT Sun 2017-11-05 01:00:00 PST Warning: The system is configured to read the RTC time in the local time zone. This mode can not be fully supported. It will create various problems with time zone changes and daylight saving time adjustments. The RTC time is never updated, it relies on external facilities to maintain it. If at all possible, use RTC in UTC by calling 'timedatectl set-local-rtc 0'. 2. the status of libvirtd was: [root@dell-per730-34 ~]# systemctl status libvirtd ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/libvirtd.service.d └─unlimited-core.conf Active: active (running) since Mon 2017-08-14 07:13:43 PDT; 8min ago Docs: man:libvirtd(8) http://libvirt.org Main PID: 1765 (libvirtd) CGroup: /system.slice/libvirtd.service └─1765 /usr/sbin/libvirtd --listen Aug 14 07:13:41 localhost.localdomain systemd[1]: Starting Virtualization daemon... Aug 14 07:13:43 localhost.localdomain systemd[1]: Started Virtualization daemon. 3. enable ntp [root@dell-per730-34 ~]# timedatectl set-ntp 1 [root@dell-per730-34 ~]# timedatectl Local time: Mon 2017-08-14 00:24:47 PDT Universal time: Mon 2017-08-14 07:24:47 UTC RTC time: Mon 2017-08-14 00:24:47 Time zone: America/Los_Angeles (PDT, -0700) NTP enabled: yes NTP synchronized: yes RTC in local TZ: yes DST active: yes Last DST change: DST began at Sun 2017-03-12 01:59:59 PST Sun 2017-03-12 03:00:00 PDT Next DST change: DST ends (the clock jumps one hour backwards) at Sun 2017-11-05 01:59:59 PDT Sun 2017-11-05 01:00:00 PST Warning: The system is configured to read the RTC time in the local time zone. This mode can not be fully supported. It will create various problems with time zone changes and daylight saving time adjustments. The RTC time is never updated, it relies on external facilities to maintain it. If at all possible, use RTC in UTC by calling 'timedatectl set-local-rtc 0'. The "RTC in local TZ" was set to "yes" initially, but the libvirtd service seems to be normal. Is there any thing that I missed to configure?
I tried to install RHEL ISO, after installation finished, the time and date infos were: [root@dell-per730-34 ~]# timedatectl Local time: Sun 2017-08-13 20:16:29 PDT Universal time: Mon 2017-08-14 03:16:29 UTC RTC time: Mon 2017-08-14 03:16:29 Time zone: America/Los_Angeles (PDT, -0700) NTP enabled: n/a NTP synchronized: no RTC in local TZ: no DST active: yes Last DST change: DST began at Sun 2017-03-12 01:59:59 PST Sun 2017-03-12 03:00:00 PDT Next DST change: DST ends (the clock jumps one hour backwards) at Sun 2017-11-05 01:59:59 PDT Sun 2017-11-05 01:00:00 PST The "RTC in local TZ" was set to "no" by default. So, the key point is why "RTC in local TZ" is set to "yes" by default in RHVH when it's installed via interactive Anaconda. If install RHVH via kickstart file with setting "timezone America/Los_Angeles --utc", then the hardware clock will be set to UTC time, "RTC in local TZ" will be set to "no" by default.
(In reply to Qin Yuan from comment #4) > So, the key point is why "RTC in local TZ" is set to "yes" by default in > RHVH when it's installed via interactive Anaconda. > If install RHVH via kickstart file with setting "timezone > America/Los_Angeles --utc", then the hardware clock will be set to UTC time, > "RTC in local TZ" will be set to "no" by default. Is there a checkbox in Anaconda to specify this? Was it used? If anaconda behaves differently between interactive installs and kickstarts, we may need to report a platform bug.
(In reply to Ryan Barry from comment #5) > Is there a checkbox in Anaconda to specify this? Was it used? AFAIK there is no way to modify this setting during an interactive install. > If anaconda behaves differently between interactive installs and kickstarts, > we may need to report a platform bug. I don't think that its so much a difference between interactive and kickstart. (When creating a kickstart file, one is expected to specify '--utc' on the timezone line.) Rather, I think the issue is that the interactive installation of RHV-H behaves differently than RHEL; RHV-H appears to be defaulting to localtime, where RHEL seems to default to UTC (which is what we probably want).
We are using platform Anaconda directly, so any difference in operation or configuration cannot be specific to RHVH, and must be in Anaconda itself. We do not do any configuration of time when the image is built. RHVH is delivered via live image, and this is not the first anaconda issue we've encountered where liveimg and "repo --url"... behave differently. I also can't reproduce this problem with a kickstarted install, and I do not have --utc specified.
I couldn't find a place on Anaconda GUI to set utc. There is an option '--utc' to control whether setting RTC to local time or UTC time when install RHVH via kickstart. But there is no such option when install RHVH via interactive Anaconda, and the RTC defaults to local time. I think we can add 'timezone --utc' to ks.cfg compressed in the ISO to make RTC default to UTC time. I wonder if it's valuable to add a checkbox on Anaconda GUI to let user to specify RTC to UTC time or local time.
Good call Qin, 'timezone --utc' to ks.cfg will set "RTC in local TZ" to no.
I tried many times without one reproducer of the libvirtd failure. I can only verify whether the RTC defaults to UTC time when there is a new ISO. Ian, could you please help to verify whether the libvirtd failure could disappear when the new ISO arrives?
Hello, I'm doing evaluation of RHEV and using RHVH-4.1-20170817.0-RHVH-x86_64-dvd1.iso with anaconda gui installation I have the same problem. No option given at install time about the clock. I chose Europe/Rome as timezone. I have to install a second host in a few days so let me now if I can be of any help debugging. I configured network and ntp during graphical installation and now I have this: [root@rhevora1 ~]# timedatectl Local time: Tue 2017-09-12 09:48:06 CEST Universal time: Tue 2017-09-12 07:48:06 UTC RTC time: Tue 2017-09-12 09:48:06 Time zone: Europe/Rome (CEST, +0200) NTP enabled: yes NTP synchronized: yes RTC in local TZ: yes DST active: yes Last DST change: DST began at Sun 2017-03-26 01:59:59 CET Sun 2017-03-26 03:00:00 CEST Next DST change: DST ends (the clock jumps one hour backwards) at Sun 2017-10-29 02:59:59 CEST Sun 2017-10-29 02:00:00 CET Warning: The system is configured to read the RTC time in the local time zone. This mode can not be fully supported. It will create various problems with time zone changes and daylight saving time adjustments. The RTC time is never updated, it relies on external facilities to maintain it. If at all possible, use RTC in UTC by calling 'timedatectl set-local-rtc 0'. [root@rhevora1 ~]# [root@rhevora1 ~]# nodectl check Status: OK Bootloader ... OK Layer boot entries ... OK Valid boot entries ... OK Mount points ... OK Separate /var ... OK Discard is used ... OK Basic storage ... OK Initialized VG ... OK Initialized Thin Pool ... OK Initialized LVs ... OK Thin storage ... OK Checking available space in thinpool ... OK Checking thinpool auto-extend ... OK vdsmd ... OK [root@rhevora1 ~]# [root@rhevora1 ~]# systemctl status libvirtd ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/libvirtd.service.d └─unlimited-core.conf Active: active (running) since Thu 2017-09-07 15:16:23 CEST; 4 days ago Docs: man:libvirtd(8) http://libvirt.org Main PID: 2305 (libvirtd) CGroup: /system.slice/libvirtd.service └─2305 /usr/sbin/libvirtd Sep 07 15:16:23 rhevora1.padana.locale systemd[1]: Starting Virtualization daemon... Sep 07 15:16:23 rhevora1.padana.locale systemd[1]: Started Virtualization daemon. [root@rhevora1 ~]# In the mean time can I use the proposed command to set to UTC in the hypervisor? timedatectl set-local-rtc 0 Thanks, Gianluca
Hi, the default was changed to UTC in the next version, for now I think you should be safe running `timedatectl set-local-rtc 0`. If something stops working please report here. Thanks !
Yes, but as the RTC was set to local time the effect has been to have date of the system ahead of 2 hours (in my case with Europe/Rome settings): [root@rhevora1 ~]# timedatectl Local time: Tue 2017-09-12 12:25:49 CEST Universal time: Tue 2017-09-12 10:25:49 UTC RTC time: Tue 2017-09-12 10:25:50 Time zone: Europe/Rome (CEST, +0200) NTP enabled: yes NTP synchronized: no RTC in local TZ: no DST active: yes Last DST change: DST began at Sun 2017-03-26 01:59:59 CET Sun 2017-03-26 03:00:00 CEST Next DST change: DST ends (the clock jumps one hour backwards) at Sun 2017-10-29 02:59:59 CEST Sun 2017-10-29 02:00:00 CET [root@rhevora1 ~]# So I also used: hwclock -w and then reboot to fix so that RTC is now in UTC and local time is correct with also ntp in sync [root@rhevora1 ~]# timedatectl Local time: Tue 2017-09-12 10:35:40 CEST Universal time: Tue 2017-09-12 08:35:40 UTC RTC time: Tue 2017-09-12 08:35:40 Time zone: Europe/Rome (CEST, +0200) NTP enabled: yes NTP synchronized: no RTC in local TZ: no DST active: yes Last DST change: DST began at Sun 2017-03-26 01:59:59 CET Sun 2017-03-26 03:00:00 CEST Next DST change: DST ends (the clock jumps one hour backwards) at Sun 2017-10-29 02:59:59 CEST Sun 2017-10-29 02:00:00 CET [root@rhevora1 ~]#
So is the system OK now (libvirt-wise etc) ?
Yes, but actually I'm not reproducing the particular problem. In my case I installed the iso on 7/9 so in my case I have openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text -noout ... Not Before: Sep 7 13:17:01 2017 GMT Not After : Sep 7 13:17:01 2018 GMT And now we are on 12/9. The problem I think should arise only if you are in UTC - (while I'm in UTC +) and ntp synchronyzes the time with the effect of going behind the "Not Before" of the certificate.
I did some tests on RHVH-4.1-20170706.1-RHVH-x86_64-dvd1.iso and RHVH-4.1-20170914.1-RHVH-x86_64-dvd1.iso: For RHVH-4.1-20170706.1-RHVH-x86_64-dvd1.iso, setting RTC to local time by default: 1. Set hardware clock to the real UTC time before installation(using `hwclock -w --utc`): 1) if you choose a timezone in UTC-? , such as America/Los_Angeles, during installation, then the vdsm certificate's 'not before' time will be in the future after time sync. 2) if you choose a timezone in UTC+?, such as Asia/Shanghai, during installation, then the vdsm certificate's 'not before' time will not be in the future after time sync. 2. Set hardware clock to the desired real local time before installation(using `hwclock -w --localtime`), such as the real local time of America/Los_Angeles, and choose the time zone America/Los_Angeles during installation, then the vdsm certificate's 'not before' time will not be in the future no matter time sync or not. For RHVH-4.1-20170914.1-RHVH-x86_64-dvd1.iso, setting RTC to UTC time by default: 1. Set hardware clock to the real UTC time before installation(using `hwclock -w --utc`), no matter what timezone you'd like to choose during installation, the vdsm certificate's 'not before' time will not be in the future no matter time sync or not. 2. Set hardware clock to the desired real local time before installation(using `hwclock -w --localtime`): 1) if set hardware clock to the real local time of a place in UTC-?, such as the real local time of America/Los_Angeles, before installation, then the vdsm certificate's 'not before' time will not be in the future after time sync. 2) if set hardware clock to the real local time of a place in UTC+?, such as the real local time of Asia/Shanghai, before installation, then the vdsm certificate's 'not before' time will be in the future after time sync. So, the key point is to set the hardware clock to the correct value no matter which RHVH ISO you choose. For the libvirtd.service failure，I tried to: 1. Set the hardware clock to UTC before installation 2. Install RHVH-4.1-20170706.1-RHVH-x86_64-dvd1.iso, choose timezone America/Los_Angeles, enable network time. 3. After the first boot, libvirtd.service was active, though the vdsm certification's 'not before' time was in the future. 4. Reboot again, this time the libvirtd.service was failed to start. The vdsm certificate is generated at the first boot. If the time sync is between generating vdsm certificate and starting libvirtd.service, then the libvirtd.service would be failed to start at the first boot. But this maybe happen by chance. Although the latest RHVH ISO defaults to setting RTC to UTC, it still depends on how the hardware clock is set before installation, and I think it's better to set hardware clock to the real UTC time before installation to avoid unpredictable problems. Since RHVH-4.1-20170914.1-RHVH-x86_64-dvd1.iso supports: 1. Set RTC to UTC by defaut 2. If set hardware clock to the real UTC time before installation, there is no libvirtd.service failure after the first boot and reboot. I will set the bug status to VERIFIED when it's set ON_QA.
checking the whole bug comments, when the issue happened, it failed vdsm registration we need to consider it a blocker.
Moving back to assigned for further investigation
I did a lot of testing on this today. It appears to be completely mitigated by enabling the network during Anaconda and selecting "network time". I realize that this is not an option for all customers (though almost all will install over PXE, so we can rely on time syncing). waitsync for chrony is not reliable, since we have no real guarantees that the network will be configured during an interactive install. So, there are two possible mitigations in lieu of this: Adding "hwclock --systohc" in %post And/or vdsm-tool should check both that the certificate exists AND that it's valid. This seems to be the better options...
Ryan, the better failsafe solution would be to create the VDSM certificate -24h to the past - just as we do with the engine CA cert, no? (Also note that a customer just complained about exactly this issue)
That would definitely be nicer. I'm checking to see what vdsm-tool does, mostly because I'm also not sure what happens if a host is deployed long enough for the certificate to expire (or whether this is a concern at all), Either way, it'll be a patch to vdsm-tool. I expect that creating it in the past will also be able to be broken by setting the date to some old default (arbitrarily, 2000/01/01, since many firmwares default to this) then installing, even if the certificate time is -24 hours. That's probably an acceptable failure case, though. In addition to everything else, we should definitely have a docs note about this
1. With enabling the network during Anaconda and selecting "network time": /var/log/messages for RHVH-4.1-20170706.1-RHVH-x86_64-dvd1.iso: Sep 21 01:12:34 hp-dl385pg8-14 journal: Runtime journal is using 8.0M (max allowed 1.5G, trying to leave 2.3G free of 15.5G available â<86><92> current limit 1.5G). ... Sep 21 01:12:34 hp-dl385pg8-14 kernel: rtc_cmos 00:04: setting system clock to 2017-09-21 08:12:34 UTC (1505981554) ... Sep 21 01:13:14 hp-dl385pg8-14 systemd: Switching root. Sep 21 01:13:14 hp-dl385pg8-14 journal: Journal stopped Sep 21 08:13:18 hp-dl385pg8-14 journal: Runtime journal is using 8.0M (max allowed 1.5G, trying to leave 2.3G free of 15.5G available â<86><92> current limit 1.5G). ... Sep 21 08:13:18 hp-dl385pg8-14 systemd[1]: RTC configured in localtime, applying delta of -420 minutes to system time. ... Sep 21 01:13:39 hp-dl385pg8-14 chronyd[1213]: Selected source 10.5.26.10 Sep 21 01:13:39 hp-dl385pg8-14 chronyd[1213]: System clock wrong by -25199.735189 seconds, adjustment started Sep 21 01:13:39 hp-dl385pg8-14 systemd: Time has been changed /var/log/messages for RHVH-4.1-20170914.1-RHVH-x86_64-dvd1.iso: Sep 21 07:39:27 localhost journal: Runtime journal is using 8.0M (max allowed 1.5G, trying to leave 2.3G free of 15.5G available → current limit 1.5G). ... Sep 21 07:39:27 localhost kernel: rtc_cmos 00:00: setting system clock to 2017-09-21 14:39:27 UTC (1506004767) ... Sep 21 07:39:33 localhost systemd: Switching root. Sep 21 07:39:33 localhost journal: Journal stopped Sep 21 07:39:34 localhost journal: Runtime journal is using 8.0M (max allowed 1.5G, trying to leave 2.3G free of 15.5G available → current limit 1.5G). ... Sep 21 07:39:54 localhost chronyd[1304]: Selected source 10.5.26.10 ... From the above messages, I think "rtc_cmos ... setting system clock to ... UTC" is the key point. If RTC defaults to localtime, then this will cause to generate error time. But if RTC defaults to UTC, this won't generate bad effect to time. 2. Without enabling the network during Anaconda: The time depends on the hardware clock, no matter using which RHVH iso. If the users want to use the RHVH iso with "timezone --utc", it's better to notify the them that they should enable network and select network time during Anaconda, otherwise they have to configure the hardware clock to UTC time before installation. (I think we don't have to use yesterday as the valid date for certificate, because for the latest iso with "timezone --utc", if there is network, and network time is on during Anaconda installation, then the time will be correct, but if there isn't network during installation, yesterday can't work if the hardware clock is, for example, 2000/01/01.)
Created attachment 1329120 [details] /var/log/messages with network for 0706 iso
Created attachment 1329121 [details] /var/log/messages with network for 0914 iso
(In reply to Qin Yuan from comment #22) > (I think we don't have to use yesterday as the valid date for certificate, > because for the latest iso with "timezone --utc", if there is network, and > network time is on during Anaconda installation, then the time will be > correct, but if there isn't network during installation, yesterday can't > work if the hardware clock is, for example, 2000/01/01.) Granted, as acknowledged in comment#21 However, as with the original patch, the current patch is "best effort", and matches engine behavior. We do recommend using ntp with RHV, and I would argue that a system with a clock which has drifted years/weeks/days is obviously not a supported configuration in any case. That said, if Dan will accept a larger patch, verifying and regenerating the SSL certificate is definitely possible. Imagine if RHVH is installed (or RHEL-H without ntp configured). A certificate is generated, and the host is added to engine. On a subsequent boot, time had been synchronized, and vdsm-tool sees the cert is invalid, then regenerates it. As far as I know, this certificate is self-signed and only used for libvirt, so this should be ok, but I could be wrong about that. Dan: how do you feel about cert validation in vdsm-tool?
vdsm-gencerts.sh creates silly self-signed certs, that are usable only until ovirt-host-deploy generates the proper certs for the host (but not only by libvirt; I believe that hosted-engine-setup is uses them as a client). Currently vdsm-gencerts.sh is called only if no (real) cert is found. I don't like the idea of allowing vdsm-gencerts.sh to override existing certs just because they have expired (or not yet valid). This seems dangerous. However, validating the cert in `vdsm-tool is-configured` seems reasonable.
The risk here seems to be that, if the cert isn't valid, vdsm startup will fail, as will libvirt, requiring manual intervention. This is no better than the current situation. I noticed that it generated self-signed certs which are only valid for a year. What I don't know is whether these are used forever, or only until it's registered to engine. In the current scenario, it seems that 366 days after deployment, they'd be bad, so they must only be used temporarily (at a guess). I can go source spelunking in vdsm to find out, but it's not as quick as asking ;) Given that they're self-signed and only used for host-deploy, and given that we know the risk is that the "Not-before" date is in the future, is it that dangerous for vdsm-tool to regenerate them in only this scenario?
The self-signed certs are used only until the the host is added to Engine. They are usable for in-host communication, such as between hosted-engine-setup and vdsm. Yes, they would expire after a year if the host is not added to Engine by then. I don't see that as a problem. The danger I am seeing is that if for some reason the *good* engine-supplied certs become invalid (say, a temporary glitch with the system clock), you would override it with the silly certs. This would render the host (and the VMs that it might be running) forever lost.
Clock set to UTC and activating certs starting "yesterday"
*** This bug has been marked as a duplicate of bug 1455441 ***