+++ This bug is a downstream clone. The original bug is: +++ +++ bug 1454536 +++ ====================================================================== Description of problem: Fresh HostedEngine deployment fails because vdsmcert.pem NotBefore date not valid if host deployed with Timezone < UTC. If I select UTC-4 during Anaconda Install, certificate is not valid for the next 4 hours. UTC-3 during Anaconda Install, certificate is not valid for the next 3 hours. UTC during Anaconda Install, certificate is valid. UTC+10 during Anaconda Install, certificate is valid since 10 hours before install. Certificate being is generated during RHV-H Install, not during ovirt-hosted-engine-setup. Version-Release number of selected component (if applicable): RHVH-4.1-20170425.3 How reproducible: 100% Steps to Reproduce: 1. Boot Installation ISO 2. Select New York Timezone 3. Finish Install 4. Try to Deploy Hosted-Engine OR (short way) 1. Boot Installation ISO 2. Select New York Timezone 3. Finish Install 4. vdsm-tool configure --module libvirt --force Actual results: HE Deployment: [ INFO ] Stage: Environment setup [ ERROR ] Failed to execute stage 'Environment setup': Failed to reconfigure libvirt for VDSM [ INFO ] Stage: Clean up libvirt configuration fails, HostedEngine not deployed. VDSM-Tool: # vdsm-tool configure --module libvirt --force Checking configuration status... libvirt is already configured for vdsm SUCCESS: ssl configured to true. No conflicts Running configure... Reconfiguration of libvirt is done. Error: ServiceOperationError: _systemctlStart failed Job for libvirtd.service failed because the control process exited with error code. See "systemctl status libvirtd.service" and "journalctl -xe" for d etails. May 22 20:18:13 localhost journal: The server certificate /etc/pki/vdsm/certs/vdsmcert.pem is not yet active May 22 20:18:13 localhost systemd: libvirtd.service: main process exited, code=exited, status=6/NOTCONFIGURED May 22 20:18:13 localhost systemd: Failed to start Virtualization daemon. May 22 20:18:13 localhost systemd: Job vdsmd.service/start failed with result 'dependency'. Host was installed 3-4 minutes ago. # date Mon May 22 20:08:04 EDT 2017 # openssl verify -CAfile /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/certs/vdsmcert.pem: CN = VDSM Certificate Authority error 9 at 1 depth lookup:certificate is not yet valid # openssl x509 -noout -dates -in /etc/pki/vdsm/certs/vdsmcert.pem notBefore=May 23 04:03:59 2017 GMT notAfter=May 23 04:03:59 2018 GMT But 20:08 EDT is 00:08 GMT, not 04:08 GMT. So certificate is 4 hours ahead. Here the UTC-3 option during install. ------------------------------------- # openssl x509 -noout -dates -in /etc/pki/vdsm/certs/vdsmcert.pem notBefore=May 23 03:49:20 2017 GMT notAfter=May 23 03:49:20 2018 GMT # date Mon May 22 21:50:16 -03 2017 Again, now the certificate is 3 hours ahead. Here the UTC+10 option during install. -------------------------------------- # openssl x509 -noout -dates -in /etc/pki/vdsm/certs/vdsmcert.pem notBefore=May 22 15:05:08 2017 GMT notAfter=May 22 15:05:08 2018 GMT # date Tue May 23 11:06:11 AEST 2017 11:06 AEST is 01:06 GMT on the same day. Host was installed 3 minutes ago, but certificate is valid since 15:05 GMT, which is 10 hours before host install. Expected results: Certificate generated during installation should be valid regardless of selected Timezone. (Originally by Germano Veit Michel)
Same thing with /etc/pki/vdsm/certs/cacert.pem (Originally by Germano Veit Michel)
The certificate is not generated at install time, but rather on the first boot by vdsm itself. Reassigning... (Originally by Ryan Barry)
Can the customer also ensure that this isn't ntp/chrony catching the clock up after install? That is -- does the hardware clock change on the first boot (new in box server)? (Originally by Ryan Barry)
(In reply to Ryan Barry from comment #2) > The certificate is not generated at install time, but rather on the first > boot by vdsm itself. Reassigning... Hi Ryan, Right, thanks. I was actually expecting it to be generated during ovirt-hosted-engine-deploy using src/plugins/gr-he-setup/pki/vdsmpki.py. > Can the customer also ensure that this isn't ntp/chrony catching the clock > up after install? > > That is -- does the hardware clock change on the first boot (new in box > server)? I can reproduce it as many times as I want, without any NTP setting. (Originally by Germano Veit Michel)
Ryan, Are you saying this is generated during that vdsm_init sh script? So it would be through a vdsm-tool configure call? Any ideas on what debug to enable to get more info on this? (Originally by Germano Veit Michel)
(In reply to Germano Veit Michel from comment #5) > Ryan, > > Are you saying this is generated during that vdsm_init sh script? So it > would be through a vdsm-tool configure call? > > Any ideas on what debug to enable to get more info on this? Well, kind of. It's generated as part of the patches attached to https://bugzilla.redhat.com/show_bug.cgi?id=1429288 with 'vdsm-tool configure --force' You could disable this during RHVH install during %post, then start it at will, with debugging added to the systemd service. Ex: %post systemctl disable imgbase-config-vdsm imgbase init --layout %end But I'd expect this is also reproducible on RHEL. (Originally by Ryan Barry)
Simone can you have a look? (Originally by Sandro Bonazzola)
Reproduced just delpoying ovirt-node-ng-installer-ovirt-4.1-2017052309.iso without starting hosted-engine-setup. Just after node installation, vdsmcert.pem is already there but invalid due to not before. We need to understand what generates it, if it's already there hosted-engine-setup will simply keep that one. [root@node412 ~]# date -u mer 24 mag 2017, 18.26.47, UTC [root@node412 ~]# openssl x509 -noout -dates -in /etc/pki/vdsm/certs/vdsmcert.pem notBefore=May 25 00:10:05 2017 GMT notAfter=May 25 00:10:05 2018 GMT (Originally by Simone Tiraboschi)
This looks interesting, on my reproduction vdsmcert.pem has been generated at 17:10:05 [root@node412 ~]# ls --full-time /etc/pki/vdsm/certs/vdsmcert.pem -rw-------. 1 vdsm kvm 1224 2017-05-24 17:10:05.973100716 -0700 /etc/pki/vdsm/certs/vdsmcert.pem but at 17:13:16 chronyd moved back the time to 10:14:27; I assume that my cert is not valid due to that reason. mag 24 17:13:16 node412.localdomain NetworkManager[1063]: <info> [1495671196.9871] policy: set 'eth0' (eth0) as default for IPv6 routing and DNS mag 24 10:13:21 node412.localdomain systemd[1]: Time has been changed mag 24 17:13:21 node412.localdomain chronyd[997]: Selected source 212.45.144.206 mag 24 17:13:21 node412.localdomain chronyd[997]: System clock wrong by -25199.566962 seconds, adjustment started mag 24 10:13:21 node412.localdomain chronyd[997]: System clock was stepped by -25199.566962 seconds mag 24 10:14:27 node412.localdomain chronyd[997]: Selected source 85.199.214.99 (Originally by Simone Tiraboschi)
On my opinion this is node specific and the issue is that, at least on my system, imgbase-config-vdsm started configuring vdsm (and creating its certs) at 17:09:59 while only at 17:13:21 chronyd moved back the time to 10:14:27 and so the certs create at 17:10:05 was simply invalid because they was created in a relative future. Maybe we should make imgbase-config-vdsm waiting chronyd at systemd level [root@node412 ~]# journalctl -u imgbase-config-vdsm -- Logs begin at mer 2017-05-24 10:09:51 PDT, end at mer 2017-05-24 11:48:01 PDT. -- mag 24 17:09:59 node412.localdomain systemd[1]: Starting Reconfigure vdsm... mag 24 17:10:08 node412.localdomain usermod[1204]: add 'sanlock' to group 'kvm' mag 24 17:10:08 node412.localdomain usermod[1204]: add 'sanlock' to group 'qemu' mag 24 17:10:08 node412.localdomain usermod[1204]: add 'sanlock' to shadow group 'kvm' mag 24 17:10:08 node412.localdomain usermod[1204]: add 'sanlock' to shadow group 'qemu' mag 24 17:10:08 node412.localdomain vdsm-tool[951]: Checking configuration status... mag 24 17:10:08 node412.localdomain vdsm-tool[951]: multipath requires configuration mag 24 17:10:08 node412.localdomain vdsm-tool[951]: WARNING: LVM local configuration: /etc/lvm/lvmlocal.conf is not based on vdsm configuration mag 24 17:10:08 node412.localdomain vdsm-tool[951]: lvm requires configuration mag 24 17:10:08 node412.localdomain vdsm-tool[951]: libvirt is not configured for vdsm yet mag 24 17:10:08 node412.localdomain vdsm-tool[951]: FAILED: conflicting vdsm and libvirt-qemu tls configuration. mag 24 17:10:08 node412.localdomain vdsm-tool[951]: vdsm.conf with ssl=True requires the following changes: mag 24 17:10:08 node412.localdomain vdsm-tool[951]: libvirtd.conf: listen_tcp=0, auth_tcp="sasl", listen_tls=1 mag 24 17:10:08 node412.localdomain vdsm-tool[951]: qemu.conf: spice_tls=1. mag 24 17:10:08 node412.localdomain vdsm-tool[951]: Running configure... mag 24 17:10:08 node412.localdomain vdsm-tool[951]: Reconfiguration of certificates is done. mag 24 17:10:08 node412.localdomain vdsm-tool[951]: Reconfiguration of sebool is done. mag 24 17:10:08 node412.localdomain vdsm-tool[951]: Reconfiguration of multipath is done. mag 24 17:10:08 node412.localdomain vdsm-tool[951]: WARNING: LVM local configuration: /etc/lvm/lvmlocal.conf is not based on vdsm configuration mag 24 17:10:08 node412.localdomain vdsm-tool[951]: Backing up /etc/lvm/lvmlocal.conf to /etc/lvm/lvmlocal.conf.201705241710 mag 24 17:10:08 node412.localdomain vdsm-tool[951]: Installing /usr/share/vdsm/lvmlocal.conf at /etc/lvm/lvmlocal.conf mag 24 17:10:08 node412.localdomain vdsm-tool[951]: Reconfiguration of lvm is done. mag 24 17:10:08 node412.localdomain vdsm-tool[951]: Reconfiguration of passwd is done. mag 24 17:10:08 node412.localdomain vdsm-tool[951]: Reconfiguration of sanlock is done. mag 24 17:10:08 node412.localdomain vdsm-tool[951]: Reconfiguration of libvirt is done. mag 24 17:10:08 node412.localdomain vdsm-tool[951]: Done configuring modules to VDSM. mag 24 17:10:08 node412.localdomain systemd[1]: Started Reconfigure vdsm. (Originally by Simone Tiraboschi)
Checking the sos-reports from the original reporter, I think it happened exactly in the same way: May 22 18:30:59 ******** dracut: *** Including module: network *** May 22 14:30:59 ******** chronyd[1726]: Selected source ******* May 22 14:30:59 ******** systemd: Time has been changed May 22 14:30:59 ******** chronyd[1726]: System clock wrong by -14399.772337 seconds, adjustment started May 22 14:30:59 ******** chronyd[1726]: System clock was stepped by -14399.772337 seconds (Originally by Simone Tiraboschi)
QE can reproduce this issue. Test version: rhvh-4.1-0.20170421.0+1 RHVH-4.1-20170425.3-RHVH-x86_64-dvd1.iso imgbased-0.9.23-0.1.el7ev.noarch libvirt-2.0.0-10.el7_3.5.x86_64 vdsm-4.19.10.1-1.el7ev.x86_64 ovirt-hosted-engine-ha-2.1.0.5-2.el7ev.noarch ovirt-hosted-engine-setup-2.1.0.5-1.el7ev.noarch rhvm-appliance-20170511.0-1.x86_64.rhevm.ova Test steps: 1. Boot Installation ISO 2. Select New York Timezone 3. Finish Install 4. vdsm-tool configure --module libvirt --force 5. Try to Deploy Hosted-Engine via cockpit Test result: Hosted Engine deployment failed. =================================================== # imgbase w [INFO] You are on rhvh-4.1-0.20170421.0+1 # date Wed May 24 22:58:47 EDT 2017 # ls --full-time /etc/pki/vdsm/certs/vdsmcert.pem -rw-------. 1 vdsm kvm 1224 2017-05-25 02:55:49.477178065 -0400 /etc/pki/vdsm/certs/vdsmcert.pem [root@dhcp-9-70 ~]# date Wed May 24 23:02:06 EDT 2017 # date Wed May 24 23:03:26 EDT 2017 [root@dhcp-9-70 ~]# date -u Thu May 25 03:03:29 UTC 2017 # vdsm-tool configure --module libvirt --force Checking configuration status... libvirt is already configured for vdsm SUCCESS: ssl configured to true. No conflicts Running configure... Reconfiguration of libvirt is done. Error: ServiceOperationError: _systemctlStart failed Job for libvirtd.service failed because the control process exited with error code. See "systemctl status libvirtd.service" and "journalctl -xe" for details. May 24 23:04:14 dhcp-9-70.nay.redhat.com libvirtd[21246]: The server certificate /etc/pki/vdsm/certs/vdsmcert.pem is not yet active May 24 23:04:14 dhcp-9-70.nay.redhat.com systemd[1]: libvirtd.service: main process exited, code=exited, status=6/NOTCONFIGURED May 24 23:04:14 dhcp-9-70.nay.redhat.com systemd[1]: Failed to start Virtualization daemon. May 24 23:04:14 dhcp-9-70.nay.redhat.com systemd[1]: Dependency failed for Virtual Desktop Server Manager network restoration. May 24 23:04:14 dhcp-9-70.nay.redhat.com systemd[1]: Dependency failed for Virtual Desktop Server Manager. May 24 23:04:14 dhcp-9-70.nay.redhat.com systemd[1]: Dependency failed for MOM instance configured for VDSM purposes. May 24 23:04:14 dhcp-9-70.nay.redhat.com systemd[1]: Job mom-vdsm.service/start failed with result 'dependency'. May 24 23:04:14 dhcp-9-70.nay.redhat.com systemd[1]: Job vdsmd.service/start failed with result 'dependency'. May 24 23:04:14 dhcp-9-70.nay.redhat.com systemd[1]: Job vdsm-network.service/start failed with result 'dependency'. Host was installed 20 minutes ago. # date Wed May 24 23:18:54 EDT 2017 # openssl verify -CAfile /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/certs/vdsmcert.pem: CN = VDSM Certificate Authority error 9 at 1 depth lookup:certificate is not yet valid # openssl x509 -noout -dates -in /etc/pki/vdsm/certs/vdsmcert.pem notBefore=May 25 06:55:49 2017 GMT notAfter=May 25 06:55:49 2018 GMT (Originally by Chen Shao)
(In reply to shaochen from comment #14) > QE can reproduce this issue. shaochen, could you please check if also in your reproduction, chronyd changed the system clock after cert creation by imgbase-config-vdsm? You should look for 'systemd: Time has been changed' in systemd journal or on /var/log/messages. (Originally by Simone Tiraboschi)
(In reply to Simone Tiraboschi from comment #15) > (In reply to shaochen from comment #14) > > QE can reproduce this issue. > > shaochen, could you please check if also in your reproduction, chronyd > changed the system clock after cert creation by imgbase-config-vdsm? Yes. > You should look for 'systemd: Time has been changed' in systemd journal or > on /var/log/messages. May 24 22:55:57 dhcp-9-70 chronyd[892]: System clock wrong by -14400.420517 seconds, adjustment started May 24 22:55:57 dhcp-9-70 systemd: Time has been changed May 24 22:55:57 dhcp-9-70 chronyd[892]: System clock was stepped by -14400.420517 seconds Attach all logs for you reference. (Originally by Chen Shao)
Created attachment 1282136 [details] all log info (Originally by Chen Shao)
Can not install rhvm-appliance on rhvh-4.1-0.20170531.0+1(imgbased-0.9.30-0.1.el7ev) since bug 1457684, once it's been fixed, will verify this bug.
Just verified on rhvh-4.1-0.20170609.0+1 Test version: RHVH-4.1-20170609.0 imgbased-0.9.31-0.1.el7ev.noarch Test Steps: 1. Boot Installation ISO 2. Select New York Timezone 3. Finish Install 4. vdsm-tool configure --module libvirt --force 5. Try to deploy Hosted engine Test result: 1. [root@hp-dl385g8-03 ~]# date Tue Jun 13 05:21:34 EDT 2017 2. There is no error in the output of step 4: [root@hp-dl385g8-03 ~]# vdsm-tool configure --module libvirt --force Checking configuration status... libvirt is already configured for vdsm SUCCESS: ssl configured to true. No conflicts Running configure... Reconfiguration of libvirt is done. Done configuring modules to VDSM. 3. The Hosted engine was successfully deployed
For this is related with bug 1474213 and got more info, I just did a re-test with build 0609 iso according to comment 21, but got a failure result. There is a significant step which was missed from comment 21 that did not select "network connect automatically on" while configuring the public ip during anaconda installation, this can cause no ip address after first reboot. Thus, the time was not synced. But, when network is on, the issue can be caught 100%. So considering this should be re-assign.
As noted in comment#23, this is "best effort". For a complete fix, we need https://bugzilla.redhat.com/show_bug.cgi?id=1474213 At this point, I'd suggest that they're duplicates
*** Bug 1474213 has been marked as a duplicate of this bug. ***
Hi yzhao, We need do more testing on this bug, so move back the bug status to ON_QA firstly.
1. I did additional test scenarios to verify this bug completely according to the duplicated Bug 1474213 - RHV-H defaults to localtime hardware clock. 2. The patch https://gerrit.ovirt.org/#/c/82011/ in bug 1474213 does not take effect in latest RHVH 4.1.7 build(rhvh-4.1-0.20171012.0+1). 3. check the file "/usr/libexec/vdsm/vdsm-gencerts.sh", the file don't contain the code "activation_date..." <snip> ... organization = "VDSM Certificate" cn = "$VDSM_FQDN" email = "root@$VDSM_FQDN" signing_key encryption_key tls_www_server ... </snip> Need to devel re-check on this bug. And attach my test steps as well for clear explanation. 4. Test version: rhvh-4.1-0.20171012.0+1 vdsm-4.19.34-1.el7ev.x86_64 imgbased-0.9.48-0.1.el7ev.noarch Test steps: 1. Set the hwclock is UTC+ 2. Install RHVH via ISO 3. Do not enable nic and network time 4. Continue to finish other installation steps, then reboot 5. Check whether the RTC is default to UTC using "timedatectl" 6. check vdsm certificate using: #openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text -noout #openssl verify -CAfile /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/certs/vdsmcert.pem 7. Check libvirtd.service status 8. Enable network and sync time 9. Repeat step6,7 10. Try to deploy HostedEngine via cockpit Test Results: 1. After step9, the vdsm certificate is failed: [root@dhcp-10-244 ~]# openssl verify -CAfile /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/certs/vdsmcert.pem: CN = VDSM Certificate Authority error 9 at 1 depth lookup:certificate is not yet valid So , the bug's status need to be changed to assigned.
(In reply to Yihui Zhao from comment #32) > 1. I did additional test scenarios to verify this bug completely according > to the duplicated Bug 1474213 - RHV-H defaults to localtime hardware clock. > > 2. The patch https://gerrit.ovirt.org/#/c/82011/ in bug 1474213 does not > take effect in latest RHVH 4.1.7 build(rhvh-4.1-0.20171012.0+1). > > > 3. check the file "/usr/libexec/vdsm/vdsm-gencerts.sh", the file don't > contain the code "activation_date..." > > <snip> > ... > organization = "VDSM Certificate" > cn = "$VDSM_FQDN" > email = "root@$VDSM_FQDN" > signing_key > encryption_key > tls_www_server > ... > </snip> > > Need to devel re-check on this bug. And attach my test steps as well for > clear explanation. > > > 4. > Test version: > rhvh-4.1-0.20171012.0+1 > vdsm-4.19.34-1.el7ev.x86_64 > imgbased-0.9.48-0.1.el7ev.noarch > > Test steps: > 1. Set the hwclock is UTC+ > 2. Install RHVH via ISO > 3. Do not enable nic and network time > 4. Continue to finish other installation steps, then reboot > 5. Check whether the RTC is default to UTC using "timedatectl" > 6. check vdsm certificate using: > #openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text -noout > #openssl verify -CAfile /etc/pki/vdsm/certs/cacert.pem > /etc/pki/vdsm/certs/vdsmcert.pem > 7. Check libvirtd.service status > 8. Enable network and sync time > 9. Repeat step6,7 > 10. Try to deploy HostedEngine via cockpit > > Test Results: > 1. After step9, the vdsm certificate is failed: > [root@dhcp-10-244 ~]# openssl verify -CAfile /etc/pki/vdsm/certs/cacert.pem > /etc/pki/vdsm/certs/vdsmcert.pem > /etc/pki/vdsm/certs/vdsmcert.pem: CN = VDSM Certificate Authority > error 9 at 1 depth lookup:certificate is not yet valid > > So , the bug's status need to be changed to assigned. Add: the step3: Do not enable nic and network time , and select the Asia/Shanghai timezone
Dan - I don't see a cherry-pick of this patch to ovirt-4.1, but I don't know the normal vdsm development flow. Can we have https://gerrit.ovirt.org/#/c/82011/ picked, please?
The author of the patch (in this case, you) is expected to suggest the backport to ovirt-4.1 branch. Once posted and verified, I'd be glad to approve and merge it.
1. I did additional test scenarios to verify this bug completely according to the duplicated Bug 1474213 - RHV-H defaults to localtime hardware clock. Need to devel re-check on this bug. And attach my test steps as well for clear explanation. Test version: vdsm-4.19.35-1.el7ev.x86_64 rhvh-4.1-0.20171025.0+1 imgbased-0.9.48-0.1.el7ev.noarch RHVH-4.1-20171025.0-RHVH-x86_64-dvd1.iso rhvm-appliance-4.1.20171019.0-1.el7.noarch Test steps: 1. Set the hwclock is UTC+ 2. Install RHVH via ISO 3. Do not enable nic and network time 4. Continue to finish other installation steps, then reboot 5. Check whether the RTC is default to UTC using "timedatectl" 6. check vdsm certificate using: #openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text -noout |grep Not #openssl verify -CAfile /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/certs/vdsmcert.pem 7. Check libvirtd.service status 8. Enable network and sync time 9. Repeat step6,7 10. Try to deploy HostedEngine via cockpit Test Results: 1. After step 5: [root@dhcp-65-102 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text -noout |grep Not Not Before: Nov 1 15:24:26 2017 GMT Not After : Nov 2 15:24:26 2018 GMT [root@dhcp-65-102 ~]# openssl verify -CAfile /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/certs/vdsmcert.pem: OK 2. After step 9: [root@dhcp-65-102 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text -noout |grep Not Not Before: Nov 1 15:24:26 2017 GMT Not After : Nov 2 15:24:26 2018 GMT [root@dhcp-65-102 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text -noout |grep Not Not Before: Nov 1 15:24:26 2017 GMT Not After : Nov 2 15:24:26 2018 GMT [root@dhcp-65-102 ~]# openssl verify -CAfile /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/certs/vdsmcert.pem: CN = VDSM Certificate Authority error 9 at 1 depth lookup:certificate is not yet valid 3. After step10, cockpit shows: """ Failed to execute stage 'Environment setup': Failed to reconfigure libvirt for VDSM Hosted Engine deployment failed """ Additional info: [root@dhcp-65-102 ~]# openssl x509 -in /etc/pki/vdsm/certs/cacert.pem -text -noout |grep Not Not Before: Nov 2 15:24:26 2017 GMT Not After : Nov 2 15:24:26 2018 GMT So , the bug's status need to be changed to assigned.
I've been trying most of the day for a reproducer here without success. I do not understand how your system possibly got a date before Nov 1 15:24:26 2017 GMT Can you also show the output of 'date'? Steps taken: timedatectl set-timezone Asia/Shanghai timedatectl set-local-rtc true #reboot to ensure the system doesn't know what UTC is Install RHVH #perform the listed steps The certificate is always valid. In truth, syncing the time hardly has any drift. Anaconda seems to do the "right thing" here, even when the RTC is off, and I'm not sure how. Screenshots are attached. Can you please provide a test system? Ideally after step 7, but please enable the network (and don't sync time)
Created attachment 1347119 [details] Times before installing (with a terminal showing correct values on top)
Created attachment 1347120 [details] Times after installation
[root@dhcp-65-102 ~]# ntpdate -u clock.redhat.com 3 Nov 23:26:33 ntpdate[27630]: step time server 10.5.27.10 offset -28798.741205 sec [root@dhcp-65-102 ~]# timedatectl Local time: Fri 2017-11-03 23:26:38 CST Universal time: Fri 2017-11-03 15:26:38 UTC RTC time: Fri 2017-11-03 23:26:38 Time zone: Asia/Shanghai (CST, +0800) NTP enabled: no NTP synchronized: no RTC in local TZ: no DST active: n/a [root@dhcp-65-102 ~]# openssl verify -CAfile /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/certs/vdsmcert.pem: CN = VDSM Certificate Authority error 9 at 1 depth lookup:certificate is not yet valid [root@dhcp-65-102 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text -noout | grep Not Not Before: Nov 2 23:03:59 2017 GMT Not After : Nov 3 23:03:59 2018 GMT [root@dhcp-65-102 ~]# date Fri Nov 3 23:27:02 CST 2017 [root@dhcp-65-102 ~]# openssl x509 -in /etc/pki/vdsm/certs/cacert.pem -text -noout | grep Not Not Before: Nov 3 23:03:59 2017 GMT Not After : Nov 3 23:03:59 2018 GMT The problem seems to be that cacert.pem also needs an activation date in the past.
According to https://gerrit.ovirt.org/#/c/83599/, I did some experiments. The results indicated that setting 'activation_date=$(date -u +"%Y-%m-%d %H:%M:%S-0000")' has the same effect with not setting activation_date. My testing steps are as below: 1. Install RHVH-4.1-20171025.0-RHVH-x86_64-dvd1.iso, do not enable network during installation. 2. Run `timedatectl`: [root@dell-per730-34 certs]# timedatectl Local time: Mon 2017-11-06 05:32:08 CST Universal time: Sun 2017-11-05 21:32:08 UTC RTC time: Sun 2017-11-05 21:32:07 Time zone: Asia/Shanghai (CST, +0800) NTP enabled: no NTP synchronized: no RTC in local TZ: no DST active: n/a The time was wrong, the correct one should be(checked on my computer): [root@localhost ~]# timedatectl Local time: Sun 2017-11-05 21:32:38 CST Universal time: Sun 2017-11-05 13:32:38 UTC RTC time: Sun 2017-11-05 13:32:39 Time zone: Asia/Shanghai (CST, +0800) Network time on: yes NTP synchronized: yes RTC in local TZ: no 3. Remove cacert.pem and vdsmcert.pem under /etc/pki/vdsm/certs 4. vi /usr/libexec/vdsm/vdsm-gencerts.sh, delete 'activation_date = $(date -d "yesterday" +"%Y-%m-%d %H:%M:%S")' 5. Run `vdsm-tool configure --module certificates` 6. Run `openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text -noout |grep Not` : [root@dell-per730-34 certs]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text -noout |grep Not Not Before: Nov 5 21:33:10 2017 GMT Not After : Nov 5 21:33:10 2018 GMT 7. Repeat step3 to step6, except adding 'activation_date = $(date -u +"%Y-%m-%d %H:%M:%S-0000")': [root@dell-per730-34 certs]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text -noout |grep Not Not Before: Nov 5 21:35:07 2017 GMT Not After : Nov 5 21:35:07 2018 GMT As you can see, the results of step6 and step7 are similar. And, if enable NTP: [root@dell-per730-34 certs]# timedatectl set-ntp 1 [root@dell-per730-34 certs]# timedatectl Local time: Sun 2017-11-05 21:38:57 CST Universal time: Sun 2017-11-05 13:38:57 UTC RTC time: Sun 2017-11-05 13:38:57 Time zone: Asia/Shanghai (CST, +0800) NTP enabled: yes NTP synchronized: yes RTC in local TZ: no DST active: n/a The time was correct, but the vdsmcert would be invalid then: [root@dell-per730-34 certs]# openssl verify -CAfile /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/certs/vdsmcert.pem: CN = VDSM Certificate Authority error 9 at 1 depth lookup:certificate is not yet valid So, if my experiment steps are correct, I think we cannot use 'activation_date=$(date -u +"%Y-%m-%d %H:%M:%S-0000")', but using yesterday would be better. If I'm wrong, please correct me :)
Now, OE have not the ovirt-4.1.8 version, so move the bug's status to MODIFIED.
Qin Yuan, thanks for reporting the problem with my suggested patch. It should not delay this bug.
1. Cannot reproduce this issue. Test version: RHVH-4.1-20171127.0-RHVH-x86_64-dvd1.iso vdsm-4.19.40-1.el7ev.x86_64 imgbased-0.9.51-0.1.el7ev.noarch Test steps: 1. Set the hwclock is UTC+ 2. Install RHVH via ISO 3. Do not enable nic and network time 4. Continue to finish other installation steps, then reboot 5. Check whether the RTC is default to UTC using "timedatectl" 6. check vdsm certificate using: #openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text -noout |grep Not #openssl verify -CAfile /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/certs/vdsmcert.pem 7. Check libvirtd.service status 8. Enable network and sync time 9. Repeat step6,7 10. Try to deploy HostedEngine via cockpit Test Results: 1. After step 5: [root@dhcp-8-151 certs]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text -noout |grep Not Not Before: Nov 30 12:13:40 2017 GMT Not After : Dec 1 12:13:40 2018 GMT [root@dhcp-8-151 certs]# openssl x509 -in /etc/pki/vdsm/certs/cacert.pem -text -noout |grep Not Not Before: Nov 30 12:13:40 2017 GMT Not After : Dec 1 12:13:40 2018 GMT [root@dhcp-8-151 certs]# openssl verify -CAfile /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/certs/vdsmcert.pem: OK 2. After step 9: [root@dhcp-8-151 certs]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -text -noout |grep Not Not Before: Nov 30 12:13:40 2017 GMT Not After : Dec 1 12:13:40 2018 GMT [root@dhcp-8-151 certs]# openssl x509 -in /etc/pki/vdsm/certs/cacert.pem -text -noout |grep Not Not Before: Nov 30 12:13:40 2017 GMT Not After : Dec 1 12:13:40 2018 GMT [root@dhcp-8-151 certs]# openssl verify -CAfile /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/certs/vdsmcert.pem: OK 3. After step10, Deploy HE successfully. Due to the result of the step5,9,10, so this issue cannot reproduce, and change the bug's status to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:3431