Bug 1474213 - RHV-H defaults to localtime hardware clock
RHV-H defaults to localtime hardware clock
Status: CLOSED DUPLICATE of bug 1455441
Product: ovirt-node
Classification: oVirt
Component: Installation & Update (Show other bugs)
4.1
Unspecified Unspecified
unspecified Severity high (vote)
: ovirt-4.1.7
: 4.1
Assigned To: Yuval Turgeman
Qin Yuan
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-24 03:23 EDT by Ian Pilcher
Modified: 2017-10-16 09:33 EDT (History)
20 users (show)

See Also:
Fixed In Version: redhat-virtualization-host-4.1-20170927.0
Doc Type: Bug Fix
Doc Text:
Cause: In cases where the hardware clock on a host is incorrect or set to local time, and time synchronization is not configured in Anaconda but is configured after installation, it is possible for the system time to jump backwards. Consequence: `vdsm-tool` is run on the first boot of Node, and generates a self-signed SSL certificate which is valid for one year from the time it was generated. If the time moves backwards, it is possible for this certificate to not be valid once the system time is corrected. Fix: oVirt Node now includes "--utc" as part of the default interactive Anaconda configuration to resolve differences between local and UTC hardware clocks. Additionally, vdsm-tool now generates certificates which are valid 24 hours prior to the time "vdsm-tool" is run (the first boot, in the case of Node). Result: System clock drift is mitigated, and Node now follows the same pattern for certificate validity as oVirt Engine, with a 24 hour window for validity.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-10-10 05:09:27 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Node
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
rule-engine: ovirt‑4.1+
rule-engine: blocker+
cshao: testing_plan_complete?
mgoldboi: planning_ack+
sbonazzo: devel_ack+
qiyuan: testing_ack+


Attachments (Terms of Use)
/var/log/messages with network for 0706 iso (168.35 KB, text/plain)
2017-09-21 13:59 EDT, Qin Yuan
no flags Details
/var/log/messages with network for 0914 iso (201.57 KB, text/plain)
2017-09-21 13:59 EDT, Qin Yuan
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 81861 master MERGED build: set clock to utc in install 2017-09-26 19:16 EDT
oVirt gerrit 82011 master MERGED gencerts: use yesterday as the valid date for the certificate 2017-09-27 06:18 EDT
oVirt gerrit 82259 ovirt-4.1 MERGED build: set clock to utc in install 2017-09-26 19:18 EDT
oVirt gerrit 82260 ovirt-4.1-pre MERGED build: set clock to utc in install 2017-09-26 19:18 EDT
oVirt gerrit 82261 ovirt-4.1-snapshot MERGED build: set clock to utc in install 2017-09-26 19:18 EDT
oVirt gerrit 82830 ovirt-4.1 MERGED gencerts: use yesterday as the valid date for the certificate 2017-10-17 12:03 EDT

  None (edit)
Description Ian Pilcher 2017-07-24 03:23:40 EDT
When performing an interactive installation of RHV-H, the installed system is configured to interpret the hardware clock as the localtime, rather than the RHEL default of UTC.  There is no option in the Anaconda UI to change this setting.

As a result, if the configured timezone is in the western hemisphere (i.e. UTC-??), the system time will be in the future until the system has synced via NTP.  This will cause the VDSM certificate in /etc/pki/vdsm/certs/vdsmcert.pem to be created with a "not before" time in the future, and libvirtd will fail to start.
Comment 1 Yuval Turgeman 2017-07-27 08:18:57 EDT
Trying to install from the RHVH iso ? if so, which version ?
Comment 2 Ian Pilcher 2017-08-07 10:18:10 EDT
(In reply to Yuval Turgeman from comment #1)
> Trying to install from the RHVH iso ? if so, which version ?

I am using RHVH-4.1-20170706.1-RHVH-x86_64-dvd1.iso.

Steps to reproduce:

1.  Set system hardware clock to UTC.
2.  Perform an interactive installation of RHV-H and specify an timezone
    in the western hemisphere (e.g. America/Los_Angeles)
3.  After system reboots, note the following:
    - libvirtd fails to start
    - vdsmcert.pem not before date is in the future
    - timedatectl shows that the hardware clock is being interpreted as
      local time
Comment 3 Qin Yuan 2017-08-14 03:47:21 EDT
I tried to follow the steps in comment #2 to reproduce the issue, but libvirtd was active, though the RTC was in local TZ.

1. After installation via RHVH-4.1-20170706.1-RHVH-x86_64-dvd1.iso finished, the  time and date infos were:

[root@dell-per730-34 ~]# timedatectl
      Local time: Mon 2017-08-14 07:22:05 PDT
  Universal time: Mon 2017-08-14 14:22:05 UTC
        RTC time: Mon 2017-08-14 07:22:05
       Time zone: America/Los_Angeles (PDT, -0700)
     NTP enabled: no
NTP synchronized: no
 RTC in local TZ: yes
      DST active: yes
 Last DST change: DST began at
                  Sun 2017-03-12 01:59:59 PST
                  Sun 2017-03-12 03:00:00 PDT
 Next DST change: DST ends (the clock jumps one hour backwards) at
                  Sun 2017-11-05 01:59:59 PDT
                  Sun 2017-11-05 01:00:00 PST

Warning: The system is configured to read the RTC time in the local time zone.
         This mode can not be fully supported. It will create various problems
         with time zone changes and daylight saving time adjustments. The RTC
         time is never updated, it relies on external facilities to maintain it.
         If at all possible, use RTC in UTC by calling
         'timedatectl set-local-rtc 0'.

2. the status of libvirtd was:

[root@dell-per730-34 ~]# systemctl status libvirtd
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/libvirtd.service.d
           └─unlimited-core.conf
   Active: active (running) since Mon 2017-08-14 07:13:43 PDT; 8min ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 1765 (libvirtd)
   CGroup: /system.slice/libvirtd.service
           └─1765 /usr/sbin/libvirtd --listen

Aug 14 07:13:41 localhost.localdomain systemd[1]: Starting Virtualization daemon...
Aug 14 07:13:43 localhost.localdomain systemd[1]: Started Virtualization daemon.

3. enable ntp
[root@dell-per730-34 ~]# timedatectl set-ntp 1
[root@dell-per730-34 ~]# timedatectl
      Local time: Mon 2017-08-14 00:24:47 PDT
  Universal time: Mon 2017-08-14 07:24:47 UTC
        RTC time: Mon 2017-08-14 00:24:47
       Time zone: America/Los_Angeles (PDT, -0700)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: yes
      DST active: yes
 Last DST change: DST began at
                  Sun 2017-03-12 01:59:59 PST
                  Sun 2017-03-12 03:00:00 PDT
 Next DST change: DST ends (the clock jumps one hour backwards) at
                  Sun 2017-11-05 01:59:59 PDT
                  Sun 2017-11-05 01:00:00 PST

Warning: The system is configured to read the RTC time in the local time zone.
         This mode can not be fully supported. It will create various problems
         with time zone changes and daylight saving time adjustments. The RTC
         time is never updated, it relies on external facilities to maintain it.
         If at all possible, use RTC in UTC by calling
         'timedatectl set-local-rtc 0'.


The "RTC in local TZ" was set to "yes" initially, but the libvirtd service seems  to be normal. Is there any thing that I missed to configure?
Comment 4 Qin Yuan 2017-08-14 06:29:23 EDT
I tried to install RHEL ISO, after installation finished, the time and date infos were:
[root@dell-per730-34 ~]# timedatectl
      Local time: Sun 2017-08-13 20:16:29 PDT
  Universal time: Mon 2017-08-14 03:16:29 UTC
        RTC time: Mon 2017-08-14 03:16:29
       Time zone: America/Los_Angeles (PDT, -0700)
     NTP enabled: n/a
NTP synchronized: no
 RTC in local TZ: no
      DST active: yes
 Last DST change: DST began at
                  Sun 2017-03-12 01:59:59 PST
                  Sun 2017-03-12 03:00:00 PDT
 Next DST change: DST ends (the clock jumps one hour backwards) at
                  Sun 2017-11-05 01:59:59 PDT
                  Sun 2017-11-05 01:00:00 PST

The "RTC in local TZ" was set to "no" by default.

So, the key point is why "RTC in local TZ" is set to "yes" by default in RHVH when it's installed via interactive Anaconda.
If install RHVH via kickstart file with setting "timezone America/Los_Angeles --utc", then the hardware clock will be set to UTC time, "RTC in local TZ" will be set to "no" by default.
Comment 5 Ryan Barry 2017-08-23 09:48:05 EDT
(In reply to Qin Yuan from comment #4)
> So, the key point is why "RTC in local TZ" is set to "yes" by default in
> RHVH when it's installed via interactive Anaconda.
> If install RHVH via kickstart file with setting "timezone
> America/Los_Angeles --utc", then the hardware clock will be set to UTC time,
> "RTC in local TZ" will be set to "no" by default.

Is there a checkbox in Anaconda to specify this? Was it used?

If anaconda behaves differently between interactive installs and kickstarts, we may need to report a platform bug.
Comment 6 Ian Pilcher 2017-08-23 11:02:49 EDT
(In reply to Ryan Barry from comment #5)
> Is there a checkbox in Anaconda to specify this? Was it used?

AFAIK there is no way to modify this setting during an interactive install.

> If anaconda behaves differently between interactive installs and kickstarts,
> we may need to report a platform bug.

I don't think that its so much a difference between interactive and kickstart.  (When creating a kickstart file, one is expected to specify '--utc' on the timezone line.)  Rather, I think the issue is that the interactive installation of RHV-H behaves differently than RHEL; RHV-H appears to be defaulting to localtime, where RHEL seems to default to UTC (which is what we probably want).
Comment 7 Ryan Barry 2017-08-23 12:14:07 EDT
We are using platform Anaconda directly, so any difference in operation or configuration cannot be specific to RHVH, and must be in Anaconda itself.

We do not do any configuration of time when the image is built.

RHVH is delivered via live image, and this is not the first anaconda issue we've encountered where liveimg and "repo --url"... behave differently.

I also can't reproduce this problem with a kickstarted install, and I do not have --utc specified.
Comment 8 Qin Yuan 2017-08-24 03:00:30 EDT
I couldn't find a place on Anaconda GUI to set utc.

There is an option '--utc' to control whether setting RTC to local time or UTC time when install RHVH via kickstart.

But there is no such option when install RHVH via interactive Anaconda, and the RTC defaults to local time.

I think we can add 'timezone --utc' to ks.cfg compressed in the ISO to make RTC default to UTC time.
I wonder if it's valuable to add a checkbox on Anaconda GUI to let user to specify RTC to UTC time or local time.
Comment 9 Yuval Turgeman 2017-09-04 08:32:50 EDT
Good call Qin, 'timezone --utc' to ks.cfg will set "RTC in local TZ" to no.
Comment 10 Qin Yuan 2017-09-07 03:47:10 EDT
I tried many times without one reproducer of the libvirtd failure. I can only verify whether the RTC defaults to UTC time when there is a new ISO.

Ian, could you please help to verify whether the libvirtd failure could disappear when the new ISO arrives?
Comment 11 Gianluca Cecchi 2017-09-12 03:58:10 EDT
Hello,
I'm doing evaluation of RHEV and using RHVH-4.1-20170817.0-RHVH-x86_64-dvd1.iso with anaconda gui installation I have the same problem.
No option given at install time about the clock.
I chose Europe/Rome as timezone. 
I have to install a second host in a few days so let me now if I can be of any help debugging.
I configured network and ntp during graphical installation and now I have this:

[root@rhevora1 ~]# timedatectl 
      Local time: Tue 2017-09-12 09:48:06 CEST
  Universal time: Tue 2017-09-12 07:48:06 UTC
        RTC time: Tue 2017-09-12 09:48:06
       Time zone: Europe/Rome (CEST, +0200)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: yes
      DST active: yes
 Last DST change: DST began at
                  Sun 2017-03-26 01:59:59 CET
                  Sun 2017-03-26 03:00:00 CEST
 Next DST change: DST ends (the clock jumps one hour backwards) at
                  Sun 2017-10-29 02:59:59 CEST
                  Sun 2017-10-29 02:00:00 CET

Warning: The system is configured to read the RTC time in the local time zone.
         This mode can not be fully supported. It will create various problems
         with time zone changes and daylight saving time adjustments. The RTC
         time is never updated, it relies on external facilities to maintain it.
         If at all possible, use RTC in UTC by calling
         'timedatectl set-local-rtc 0'.
[root@rhevora1 ~]# 

[root@rhevora1 ~]# nodectl check
Status: OK
Bootloader ... OK
  Layer boot entries ... OK
  Valid boot entries ... OK
Mount points ... OK
  Separate /var ... OK
  Discard is used ... OK
Basic storage ... OK
  Initialized VG ... OK
  Initialized Thin Pool ... OK
  Initialized LVs ... OK
Thin storage ... OK
  Checking available space in thinpool ... OK
  Checking thinpool auto-extend ... OK
vdsmd ... OK
[root@rhevora1 ~]# 

[root@rhevora1 ~]# systemctl status libvirtd
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/libvirtd.service.d
           └─unlimited-core.conf
   Active: active (running) since Thu 2017-09-07 15:16:23 CEST; 4 days ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 2305 (libvirtd)
   CGroup: /system.slice/libvirtd.service
           └─2305 /usr/sbin/libvirtd

Sep 07 15:16:23 rhevora1.padana.locale systemd[1]: Starting Virtualization daemon...
Sep 07 15:16:23 rhevora1.padana.locale systemd[1]: Started Virtualization daemon.
[root@rhevora1 ~]# 

In the mean time can I use the proposed command to set to UTC in the hypervisor?

timedatectl set-local-rtc 0

Thanks,
Gianluca
Comment 12 Yuval Turgeman 2017-09-12 04:09:01 EDT
Hi, the default was changed to UTC in the next version, for now I think you should be safe running `timedatectl set-local-rtc 0`.  If something stops working please report here.
Thanks !
Comment 13 Gianluca Cecchi 2017-09-12 04:37:26 EDT
Yes, but as the RTC was set to local time the effect has been to have date of the system ahead of 2 hours (in my case with Europe/Rome settings):

[root@rhevora1 ~]# timedatectl
      Local time: Tue 2017-09-12 12:25:49 CEST
  Universal time: Tue 2017-09-12 10:25:49 UTC
        RTC time: Tue 2017-09-12 10:25:50
       Time zone: Europe/Rome (CEST, +0200)
     NTP enabled: yes
NTP synchronized: no
 RTC in local TZ: no
      DST active: yes
 Last DST change: DST began at
                  Sun 2017-03-26 01:59:59 CET
                  Sun 2017-03-26 03:00:00 CEST
 Next DST change: DST ends (the clock jumps one hour backwards) at
                  Sun 2017-10-29 02:59:59 CEST
                  Sun 2017-10-29 02:00:00 CET
[root@rhevora1 ~]# 

So I also used:

hwclock -w

and then reboot to fix so that RTC is now in UTC and local time is correct with also ntp in sync

[root@rhevora1 ~]# timedatectl
      Local time: Tue 2017-09-12 10:35:40 CEST
  Universal time: Tue 2017-09-12 08:35:40 UTC
        RTC time: Tue 2017-09-12 08:35:40
       Time zone: Europe/Rome (CEST, +0200)
     NTP enabled: yes
NTP synchronized: no
 RTC in local TZ: no
      DST active: yes
 Last DST change: DST began at
                  Sun 2017-03-26 01:59:59 CET
                  Sun 2017-03-26 03:00:00 CEST
 Next DST change: DST ends (the clock jumps one hour backwards) at
                  Sun 2017-10-29 02:59:59 CEST
                  Sun 2017-10-29 02:00:00 CET
[root@rhevora1 ~]#
Comment 14 Yuval Turgeman 2017-09-12 04:41:41 EDT
So is the system OK now (libvirt-wise etc) ?
Comment 15 Gianluca Cecchi 2017-09-12 05:02:29 EDT
Yes, but actually I'm not reproducing the particular problem.
In my case I installed the iso on 7/9 so in my case I have

openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text -noout
...
            Not Before: Sep  7 13:17:01 2017 GMT
            Not After : Sep  7 13:17:01 2018 GMT

And now we are on 12/9.
The problem I think should arise only if you are in UTC - (while I'm in UTC +) and ntp synchronyzes the time with the effect of going behind the "Not Before" of the certificate.
Comment 16 Qin Yuan 2017-09-16 14:25:54 EDT
I did some tests on RHVH-4.1-20170706.1-RHVH-x86_64-dvd1.iso and RHVH-4.1-20170914.1-RHVH-x86_64-dvd1.iso:

For RHVH-4.1-20170706.1-RHVH-x86_64-dvd1.iso, setting RTC to local time by default:
1. Set hardware clock to the real UTC time before installation(using `hwclock -w --utc`):
1) if you choose a timezone in UTC-? , such as America/Los_Angeles, during installation, then the vdsm certificate's 'not before' time will be in the future after time sync. 

2) if you choose a timezone in UTC+?, such as  Asia/Shanghai, during installation, then the vdsm certificate's 'not before' time will not be in the future after time sync.

2. Set hardware clock to the desired real local time before installation(using `hwclock -w --localtime`), such as the real local time of America/Los_Angeles, and choose the time zone America/Los_Angeles during installation, then the vdsm certificate's 'not before' time will not be in the future no matter time sync or not.


For RHVH-4.1-20170914.1-RHVH-x86_64-dvd1.iso, setting RTC to UTC time by default:
1. Set hardware clock to the real UTC time before installation(using `hwclock -w --utc`), no matter what timezone you'd like to choose during installation, the vdsm certificate's 'not before' time will not be in the future no matter time sync or not.

2. Set hardware clock to the desired real local time before installation(using `hwclock -w --localtime`): 
1) if set hardware clock to the real local time of a place in UTC-?,   such as the real local time of America/Los_Angeles, before installation, then the vdsm certificate's 'not before' time will not be in the future after time sync.

2) if set hardware clock to the real local time of a place in UTC+?,   such as the real local time of Asia/Shanghai, before installation, then the vdsm certificate's 'not before' time will be in the future after time sync. 


So, the key point is to set the hardware clock to the correct value no matter which RHVH ISO you choose.


For the libvirtd.service failure,I tried to:
1. Set the hardware clock to UTC before installation
2. Install RHVH-4.1-20170706.1-RHVH-x86_64-dvd1.iso, choose timezone America/Los_Angeles, enable network time.
3. After the first boot, libvirtd.service was active, though the vdsm certification's 'not before' time was in the future.
4. Reboot again, this time the libvirtd.service was failed to start.

The vdsm certificate is generated at the first boot. If the time sync is between generating vdsm certificate and starting libvirtd.service, then the libvirtd.service would be failed to start at the first boot. But this maybe happen by chance.

Although the latest RHVH ISO defaults to setting RTC to UTC, it still depends on how the hardware clock is set before installation, and I think it's better to set hardware clock to the real UTC time before installation to avoid unpredictable problems.

Since RHVH-4.1-20170914.1-RHVH-x86_64-dvd1.iso supports:
1. Set RTC to UTC by defaut
2. If set hardware clock to the real UTC time before installation, there is no libvirtd.service failure after the first boot and reboot.

I will set the bug status to VERIFIED when it's set ON_QA.
Comment 17 Ying Cui 2017-09-18 07:33:19 EDT
checking the whole bug comments, when the issue happened, it failed vdsm registration we need to consider it a blocker.
Comment 18 Sandro Bonazzola 2017-09-18 08:41:24 EDT
Moving back to assigned for further investigation
Comment 19 Ryan Barry 2017-09-18 16:37:42 EDT
I did a lot of testing on this today.

It appears to be completely mitigated by enabling the network during Anaconda and selecting "network time". I realize that this is not an option for all customers (though almost all will install over PXE, so we can rely on time syncing).

waitsync for chrony is not reliable, since we have no real guarantees that the network will be configured during an interactive install.

So, there are two possible mitigations in lieu of this:

Adding "hwclock --systohc" in %post

And/or vdsm-tool should check both that the certificate exists AND that it's valid. This seems to be the better options...
Comment 20 Yaniv Kaul 2017-09-18 17:26:47 EDT
Ryan, the better failsafe solution would be to create the VDSM certificate -24h to the past - just as we do with the engine CA cert, no?

(Also note that a customer just complained about exactly this issue)
Comment 21 Ryan Barry 2017-09-18 17:31:27 EDT
That would definitely be nicer. I'm checking to see what vdsm-tool does, mostly because I'm also not sure what happens if a host is deployed long enough for the certificate to expire (or whether this is a concern at all),

Either way, it'll be a patch to vdsm-tool.

I expect that creating it in the past will also be able to be broken by setting the date to some old default (arbitrarily, 2000/01/01, since many firmwares default to this) then installing, even if the certificate time is -24 hours.

That's probably an acceptable failure case, though.

In addition to everything else, we should definitely have a docs note about this
Comment 22 Qin Yuan 2017-09-21 13:56:04 EDT
1. With enabling the network during Anaconda and selecting "network time":

/var/log/messages for RHVH-4.1-20170706.1-RHVH-x86_64-dvd1.iso:

Sep 21 01:12:34 hp-dl385pg8-14 journal: Runtime journal is using 8.0M (max allowed 1.5G, trying to leave 2.3G free of 15.5G available â<86><92> current limit 1.5G).
...
Sep 21 01:12:34 hp-dl385pg8-14 kernel: rtc_cmos 00:04: setting system clock to 2017-09-21 08:12:34 UTC (1505981554)
...
Sep 21 01:13:14 hp-dl385pg8-14 systemd: Switching root.
Sep 21 01:13:14 hp-dl385pg8-14 journal: Journal stopped
Sep 21 08:13:18 hp-dl385pg8-14 journal: Runtime journal is using 8.0M (max allowed 1.5G, trying to leave 2.3G free of 15.5G available â<86><92> current limit 1.5G).
...
Sep 21 08:13:18 hp-dl385pg8-14 systemd[1]: RTC configured in localtime, applying delta of -420 minutes to system time.
...
Sep 21 01:13:39 hp-dl385pg8-14 chronyd[1213]: Selected source 10.5.26.10
Sep 21 01:13:39 hp-dl385pg8-14 chronyd[1213]: System clock wrong by -25199.735189 seconds, adjustment started
Sep 21 01:13:39 hp-dl385pg8-14 systemd: Time has been changed


/var/log/messages for RHVH-4.1-20170914.1-RHVH-x86_64-dvd1.iso:

Sep 21 07:39:27 localhost journal: Runtime journal is using 8.0M (max allowed 1.5G, trying to leave 2.3G free of 15.5G available → current limit 1.5G).
...
Sep 21 07:39:27 localhost kernel: rtc_cmos 00:00: setting system clock to 2017-09-21 14:39:27 UTC (1506004767)
...
Sep 21 07:39:33 localhost systemd: Switching root.
Sep 21 07:39:33 localhost journal: Journal stopped
Sep 21 07:39:34 localhost journal: Runtime journal is using 8.0M (max allowed 1.5G, trying to leave 2.3G free of 15.5G available → current limit 1.5G).
...
Sep 21 07:39:54 localhost chronyd[1304]: Selected source 10.5.26.10
...


From the above messages, I think "rtc_cmos ... setting system clock to ... UTC" is the key point.
If RTC defaults to localtime, then this will cause to generate error time.
But if RTC defaults to UTC, this won't generate bad effect to time.


2. Without enabling the network during Anaconda:

The time depends on the hardware clock, no matter using which RHVH iso.

If the users want to use the RHVH iso with "timezone --utc", it's better to notify the them that they should enable network and select network time during Anaconda, otherwise they have to configure the hardware clock to UTC time before installation.

(I think we don't have to use yesterday as the valid date for certificate, because for the latest iso with "timezone --utc", if there is network, and network time is on during Anaconda installation, then the time will be correct, but if there isn't network during installation, yesterday can't work if the hardware clock is, for example, 2000/01/01.)
Comment 23 Qin Yuan 2017-09-21 13:59 EDT
Created attachment 1329120 [details]
/var/log/messages with network for 0706 iso
Comment 24 Qin Yuan 2017-09-21 13:59 EDT
Created attachment 1329121 [details]
/var/log/messages with network for 0914 iso
Comment 25 Ryan Barry 2017-09-21 15:56:27 EDT
(In reply to Qin Yuan from comment #22)
> (I think we don't have to use yesterday as the valid date for certificate,
> because for the latest iso with "timezone --utc", if there is network, and
> network time is on during Anaconda installation, then the time will be
> correct, but if there isn't network during installation, yesterday can't
> work if the hardware clock is, for example, 2000/01/01.)

Granted, as acknowledged in comment#21

However, as with the original patch, the current patch is "best effort", and matches engine behavior.

We do recommend using ntp with RHV, and I would argue that a system with a clock which has drifted years/weeks/days is obviously not a supported configuration in any case.

That said, if Dan will accept a larger patch, verifying and regenerating the SSL certificate is definitely possible. 

Imagine if RHVH is installed (or RHEL-H without ntp configured). A certificate is generated, and the host is added to engine. On a subsequent boot, time had been synchronized, and vdsm-tool sees the cert is invalid, then regenerates it.

As far as I know, this certificate is self-signed and only used for libvirt, so this should be ok, but I could be wrong about that.

Dan: how do you feel about cert validation in vdsm-tool?
Comment 26 Dan Kenigsberg 2017-09-21 17:58:57 EDT
vdsm-gencerts.sh creates silly self-signed certs, that are usable only until ovirt-host-deploy generates the proper certs for the host (but not only by libvirt; I believe that hosted-engine-setup is uses them as a client).

Currently vdsm-gencerts.sh is called only if no (real) cert is found. I don't like the idea of allowing vdsm-gencerts.sh to override existing certs just because they have expired (or not yet valid). This seems dangerous. However, validating the cert in `vdsm-tool is-configured` seems reasonable.
Comment 27 Ryan Barry 2017-09-21 18:05:47 EDT
The risk here seems to be that, if the cert isn't valid, vdsm startup will fail, as will libvirt, requiring manual intervention. This is no better than the current situation.

I noticed that it generated self-signed certs which are only valid for a year. What I don't know is whether these are used forever, or only until it's registered to engine. In the current scenario, it seems that 366 days after deployment, they'd be bad, so they must only be used temporarily (at a guess). I can go source spelunking in vdsm to find out, but it's not as quick as asking ;)

Given that they're self-signed and only used for host-deploy, and given that we know the risk is that the "Not-before" date is in the future, is it that dangerous for vdsm-tool to regenerate them in only this scenario?
Comment 28 Dan Kenigsberg 2017-09-27 06:05:53 EDT
The self-signed certs are used only until the the host is added to Engine. They are usable for in-host communication, such as between hosted-engine-setup and vdsm.

Yes, they would expire after a year if the host is not added to Engine by then. I don't see that as a problem.

The danger I am seeing is that if for some reason the *good* engine-supplied certs become invalid (say, a temporary glitch with the system clock), you would override it with the silly certs. This would render the host (and the VMs that it might be running) forever lost.
Comment 29 Yuval Turgeman 2017-10-02 03:36:55 EDT
Clock set to UTC and activating certs starting "yesterday"
Comment 30 Ryan Barry 2017-10-10 05:09:27 EDT

*** This bug has been marked as a duplicate of bug 1455441 ***

Note You need to log in before you can comment on or make changes to this bug.