Bug 1291161

Summary: Engine grants invalid certificate if host clock is restarted prior installation
Product: [oVirt] ovirt-node Reporter: Artyom <alukiano>
Component: legacy-ovirt-node-plugin-vdsmAssignee: Fabian Deutsch <fdeutsch>
Status: CLOSED WONTFIX QA Contact: Ying Cui <ycui>
Severity: high Docs Contact:
Priority: unspecified    
Version: ---CC: alonbl, alukiano, bugs, cshao, dguo, fdeutsch, gklein, huzhao, mburman, oourfali, ycui
Target Milestone: ---Flags: oourfali: ovirt-3.6.z?
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: node
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-12-22 11:46:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
sos report from rhev-h none

Description Artyom 2015-12-14 07:55:33 UTC
Created attachment 1105510 [details]
sos report from rhev-h

Description of problem:
libvirt failed to start, because certificate problem
Dec 14 07:26:08 localhost libvirtd: 20180: info : libvirt version: 1.2.17, package: 13.el7_2.2 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2015-11-23-07:46:04, x86-019.build.eng.bos.redhat.com)
Dec 14 07:26:08 localhost libvirtd: 20180: error : virNetTLSContextCheckCertTimes:165 : The server certificate /etc/pki/vdsm/certs/vdsmcert.pem is not yet active

# openssl verify -CAfile /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/certs/vdsmcert.pem
/etc/pki/vdsm/certs/vdsmcert.pem: CN = VDSM Certificate Authority
error 9 at 1 depth lookup:certificate is not yet valid

Looks like we generate vdsm certificates, before we have configured network and receive correct date from NTP server, so we have
openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -text 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            56:6e:7f:88:10:58:86:fa:38:42:26:3d
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=VDSM Certificate Authority
        Validity
            Not Before: Dec 14 08:36:24 2015 GMT
            Not After : Dec 13 08:36:24 2016 GMT

and
# date
Mon Dec 14 07:38:05 UTC 2015

Version-Release number of selected component (if applicable):
Red Hat Enterprise Virtualization Hypervisor (Beta) release 7.2 (20151210.1.el7ev)
I also encounter this one in rhev-hypervisor7-7.1-20151015.0

How reproducible:
always

Steps to Reproduce:
1. Deploy RHEV-H from usb-key
2. 
3.

Actual results:
After deployment libvirtd failed to start because certificate problem

Expected results:
After deployment libvirtd succeed to start without any errors

Additional info:

Comment 1 Fabian Deutsch 2015-12-14 08:48:06 UTC
This appears on Node, but doesn't seem to be a node specific problem, as this can also happen on RHEL hosts.

Comment 2 Dan Kenigsberg 2015-12-16 16:21:35 UTC
We cannot wait for NTP to sync the host time.

Prior to ovirt-host-deploy we attempted copying time from from Engine to the host.

Also, we made our certificate is valid few hours into the past, so reasonably out-of-sync hosts can be added.

Would reintroducing one of these hacks make sense?

Comment 3 Alon Bar-Lev 2015-12-17 07:45:26 UTC
(In reply to Dan Kenigsberg from comment #2)
> We cannot wait for NTP to sync the host time.
> 
> Prior to ovirt-host-deploy we attempted copying time from from Engine to the
> host.
> 
> Also, we made our certificate is valid few hours into the past, so
> reasonably out-of-sync hosts can be added.
> 
> Would reintroducing one of these hacks make sense?

Engine grant a valid certificate based on engine clock minus one day.
Host-deploy still sync clock if it is out of sync.
This one time sync is useless, as after reboot you will probably be out of sync once again... but still we do this.

Instead of attaching irrelevant sos reports, please attach host-deploy log, so we can probably see the above.

Comment 4 Artyom 2015-12-17 11:08:41 UTC
I opened bug on RHEV-H and problem that I have libvirtd service failed to start before I add host to any engine, so not host-deploy.log

Comment 5 Alon Bar-Lev 2015-12-17 11:18:37 UTC
there is no sense in starting libvirt pre-deploy, nor am I sure how /etc/pki/vdsm/certs/vdsmcert.pem was created.

back to ovirt-node, I suggest to close it as NOTABUG.

Comment 6 Fabian Deutsch 2015-12-22 11:46:43 UTC
libvirtd is started by default on Node, which can cause this problem.

because it is an issue, but appearing not to often I'm closing this bug. Please reopen if you encounter this problem often.

Comment 7 Artyom 2015-12-22 14:10:31 UTC
For me it happen pretty often, for example from five RHEV-H installation it can happen twice. It also can create additional problem if you want to run automation on RHEV-H. Can we just add some additional hook on network setup, that will regenerate libvirt certificate?

Comment 8 Gil Klein 2015-12-22 17:14:29 UTC
Can we consider setting the certificate validity, 1 day before the current date?

This can easily workaround this kind of problem.

Comment 9 Fabian Deutsch 2015-12-22 17:18:26 UTC
According to comment 3 this is already the case.

Comment 10 Fabian Deutsch 2016-01-04 14:40:22 UTC
Artyom, have you got an idea why the host is initially so out of sync?
Does the DHCP server also provide NTP servers?

Comment 11 Artyom 2016-01-05 14:04:55 UTC
Do not really know, maybe RHEV-H use some default time zone?
Yes DHCP server also provide NTP.