Bug 1291161 - Engine grants invalid certificate if host clock is restarted prior installation
Engine grants invalid certificate if host clock is restarted prior installation
Status: CLOSED WONTFIX
Product: ovirt-node
Classification: oVirt
Component: legacy-ovirt-node-plugin-vdsm (Show other bugs)
---
x86_64 Linux
unspecified Severity high (vote)
: ---
: ---
Assigned To: Fabian Deutsch
Ying Cui
node
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-14 02:55 EST by Artyom
Modified: 2016-08-03 01:14 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-22 06:46:43 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Node
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
oourfali: ovirt‑3.6.z?
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?


Attachments (Terms of Use)
sos report from rhev-h (5.70 MB, application/x-xz)
2015-12-14 02:55 EST, Artyom
no flags Details

  None (edit)
Description Artyom 2015-12-14 02:55:33 EST
Created attachment 1105510 [details]
sos report from rhev-h

Description of problem:
libvirt failed to start, because certificate problem
Dec 14 07:26:08 localhost libvirtd: 20180: info : libvirt version: 1.2.17, package: 13.el7_2.2 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2015-11-23-07:46:04, x86-019.build.eng.bos.redhat.com)
Dec 14 07:26:08 localhost libvirtd: 20180: error : virNetTLSContextCheckCertTimes:165 : The server certificate /etc/pki/vdsm/certs/vdsmcert.pem is not yet active

# openssl verify -CAfile /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/certs/vdsmcert.pem
/etc/pki/vdsm/certs/vdsmcert.pem: CN = VDSM Certificate Authority
error 9 at 1 depth lookup:certificate is not yet valid

Looks like we generate vdsm certificates, before we have configured network and receive correct date from NTP server, so we have
openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -text 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            56:6e:7f:88:10:58:86:fa:38:42:26:3d
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=VDSM Certificate Authority
        Validity
            Not Before: Dec 14 08:36:24 2015 GMT
            Not After : Dec 13 08:36:24 2016 GMT

and
# date
Mon Dec 14 07:38:05 UTC 2015

Version-Release number of selected component (if applicable):
Red Hat Enterprise Virtualization Hypervisor (Beta) release 7.2 (20151210.1.el7ev)
I also encounter this one in rhev-hypervisor7-7.1-20151015.0

How reproducible:
always

Steps to Reproduce:
1. Deploy RHEV-H from usb-key
2. 
3.

Actual results:
After deployment libvirtd failed to start because certificate problem

Expected results:
After deployment libvirtd succeed to start without any errors

Additional info:
Comment 1 Fabian Deutsch 2015-12-14 03:48:06 EST
This appears on Node, but doesn't seem to be a node specific problem, as this can also happen on RHEL hosts.
Comment 2 Dan Kenigsberg 2015-12-16 11:21:35 EST
We cannot wait for NTP to sync the host time.

Prior to ovirt-host-deploy we attempted copying time from from Engine to the host.

Also, we made our certificate is valid few hours into the past, so reasonably out-of-sync hosts can be added.

Would reintroducing one of these hacks make sense?
Comment 3 Alon Bar-Lev 2015-12-17 02:45:26 EST
(In reply to Dan Kenigsberg from comment #2)
> We cannot wait for NTP to sync the host time.
> 
> Prior to ovirt-host-deploy we attempted copying time from from Engine to the
> host.
> 
> Also, we made our certificate is valid few hours into the past, so
> reasonably out-of-sync hosts can be added.
> 
> Would reintroducing one of these hacks make sense?

Engine grant a valid certificate based on engine clock minus one day.
Host-deploy still sync clock if it is out of sync.
This one time sync is useless, as after reboot you will probably be out of sync once again... but still we do this.

Instead of attaching irrelevant sos reports, please attach host-deploy log, so we can probably see the above.
Comment 4 Artyom 2015-12-17 06:08:41 EST
I opened bug on RHEV-H and problem that I have libvirtd service failed to start before I add host to any engine, so not host-deploy.log
Comment 5 Alon Bar-Lev 2015-12-17 06:18:37 EST
there is no sense in starting libvirt pre-deploy, nor am I sure how /etc/pki/vdsm/certs/vdsmcert.pem was created.

back to ovirt-node, I suggest to close it as NOTABUG.
Comment 6 Fabian Deutsch 2015-12-22 06:46:43 EST
libvirtd is started by default on Node, which can cause this problem.

because it is an issue, but appearing not to often I'm closing this bug. Please reopen if you encounter this problem often.
Comment 7 Artyom 2015-12-22 09:10:31 EST
For me it happen pretty often, for example from five RHEV-H installation it can happen twice. It also can create additional problem if you want to run automation on RHEV-H. Can we just add some additional hook on network setup, that will regenerate libvirt certificate?
Comment 8 Gil Klein 2015-12-22 12:14:29 EST
Can we consider setting the certificate validity, 1 day before the current date?

This can easily workaround this kind of problem.
Comment 9 Fabian Deutsch 2015-12-22 12:18:26 EST
According to comment 3 this is already the case.
Comment 10 Fabian Deutsch 2016-01-04 09:40:22 EST
Artyom, have you got an idea why the host is initially so out of sync?
Does the DHCP server also provide NTP servers?
Comment 11 Artyom 2016-01-05 09:04:55 EST
Do not really know, maybe RHEV-H use some default time zone?
Yes DHCP server also provide NTP.

Note You need to log in before you can comment on or make changes to this bug.