Bug 1259634

Summary: [RFE][PKI] renew important certificate when about to expire during engine-setup
Product: Red Hat Enterprise Virtualization Manager Reporter: rhev-integ
Component: ovirt-engine-setupAssignee: Yedidyah Bar David <didi>
Status: CLOSED CURRENTRELEASE QA Contact: Pavel Stehlik <pstehlik>
Severity: high Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: adahms, alonbl, bazulay, didi, ecohen, gklein, iheim, jbelka, lsurette, mkalinin, rbalakri, Rhev-m-bugs, rhodain, sbonazzo, yeylon, ylavi
Target Milestone: ---Keywords: FutureFeature, Improvement, ZStream
Target Release: 3.5.4   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: integration
Fixed In Version: Doc Type: Enhancement
Doc Text:
With this release, the engine-setup script now checks if the relevant certificates - including the internal CA certificate and those it signs - are set to expire soon or have already expired. If so, engine-setup now prompts users whether to renew the certificates. If users reply 'yes', the certificates are renewed. If users reply 'no', the certificates are not renewed, and users are prompted with the same question the next time they run engine-setup. This feature addresses the situation where some older setups were at risk of the certificate expiring without the user knowing. Now, users are notified of impending expiry, and can renew the certificate in advance. This functionality is not restricted to updates; users can run engine-setup at any time to check and renew the relevant certificates. Renewing the certificate does not renew the certificates of hosts; you must manually reinstall all hosts and update browsers to accept the new certificate. For more details, see the following - https://access.redhat.com/solutions/1572983
Story Points: ---
Clone Of: 1259601 Environment:
Last Closed: 2015-09-04 12:50:50 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1259601    
Bug Blocks:    

Description rhev-integ 2015-09-03 08:48:49 UTC
+++ This bug is a RHEV-M zstream clone. The original bug is: +++
+++   https://bugzilla.redhat.com/show_bug.cgi?id=1259601. +++
+++ Requested by "sbonazzo" +++
======================================================================



----------------------------------------------------------------------
Following comment by didi on September 03 at 07:11:12, 2015

+++ This bug was initially created as a clone of Bug #1214860 +++

During engine setup we can renew selected certificates that are important for system operation:

1. ca certificate - without modifying key nor serial.

2. engine certificate - without modifying key as it incorrectly being used for database field encryption.

3. apache certificate - if issued by engine ca.

All other certificates will not be renewed automatically, including these that are at remote servers.

VDSM certificates should be renewed using host redeploy.

--- Additional comment from Itamar Heim on 2015-04-27 12:03:02 IDT ---

don't we still need to distribute the new CA certificate to hosts for them to accept the engine certificate coming from a new CA certificate?
(my concern is post engine start with new certificate, all hosts would become non-responsive)

--- Additional comment from Alon Bar-Lev on 2015-04-27 16:39:40 IDT ---

(In reply to Itamar Heim from comment #1)
> don't we still need to distribute the new CA certificate to hosts for them
> to accept the engine certificate coming from a new CA certificate?
> (my concern is post engine start with new certificate, all hosts would
> become non-responsive)

we will need to redeploy all hosts in any case to renew the CA certificate and the host certificate. but at least this will work as CA certificate and engine certificate at engine machine will be valid.

--- Additional comment from Sandro Bonazzola on 2015-04-29 10:41:02 IDT ---

Yaniv, looks like this bug should be targeted to 3.5.3 not 3.5.4 or the PKI change will be incomplete.

--- Additional comment from Yaniv Dary on 2015-04-30 00:58:53 IDT ---

(In reply to Sandro Bonazzola from comment #3)
> Yaniv, looks like this bug should be targeted to 3.5.3 not 3.5.4 or the PKI
> change will be incomplete.

Should this bug be on POST? what is the tests needed to verify the fix?

--- Additional comment from Sandro Bonazzola on 2015-05-05 10:23:43 IDT ---

(In reply to Yaniv Dary from comment #4)
> (In reply to Sandro Bonazzola from comment #3)
> > Yaniv, looks like this bug should be targeted to 3.5.3 not 3.5.4 or the PKI
> > change will be incomplete.
> 
> Should this bug be on POST?

Yes, it's still on post having patches still to be merged on 3.5 and 3.5.3 branches.

> what is the tests needed to verify the fix?

Alon, can you answer this?

--- Additional comment from Alon Bar-Lev on 2015-05-05 10:39:06 IDT ---

If certificate (CA, engine, websocket) is invalid (bug#1210486) or about to expire (less than a year), a certificate will be renewed.

Special notice should be taken regarding the CA certificate, as it should be distributed to all locations. Browser for example will ask user to confirm the new CA certificate.

engine->vdsm communication should not be effected by this renew, all hosts (which are not expired) should be up.

The process is performed only locally on engine machine, distributed configuration should manually re-issue certificates. for example, host-deploy should be performed for all hosts to renew their certificate and install the new certificate authority.

QA note:
Probably best to adjust clock 2 years back install engine and move clock forward.
If a 3.0/2.2 configuration is available, then try to upgrade and notice all certificates are renewed.

Handy command:
$ openssl x509 -in <certificate> -text

--- Additional comment from Alon Bar-Lev on 2015-06-02 11:55:21 IDT ---

An issue with setup rollback was found and fixed, moved to modified.

--- Additional comment from Alon Bar-Lev on 2015-06-02 12:26:17 IDT ---

(In reply to Alon Bar-Lev from comment #7)
> An issue with setup rollback was found and fixed, moved to modified.

sorry, my bad, should be post.

--- Additional comment from Yedidyah Bar David on 2015-06-10 08:47:32 IDT ---

Yaniv, please be more specific about what we want to do with this bug for 3.5.4.

If just to enable the existing patches, we need to:

1. Revert [1] - Disable renew - was pushed to 3.5 and 3.5.3

2. Backport to 3.5 [2] - Prompt before renew - was pushed to master

We discussed also other things, such as somehow automatically update certs on the hosts, perhaps somehow help/notify the admin to update certs on clients (browsers), etc.

[1] https://gerrit.ovirt.org/#/q/Ib0de9b1f1c3aa3aa1f3d81d47a91d44f458e7192
[2] https://gerrit.ovirt.org/41876

--- Additional comment from Oved Ourfali on 2015-06-10 08:59:06 IDT ---

(In reply to Yedidyah Bar David from comment #9)
> Yaniv, please be more specific about what we want to do with this bug for
> 3.5.4.
> 
> If just to enable the existing patches, we need to:
> 
> 1. Revert [1] - Disable renew - was pushed to 3.5 and 3.5.3
> 
> 2. Backport to 3.5 [2] - Prompt before renew - was pushed to master
> 
> We discussed also other things, such as somehow automatically update certs
> on the hosts, perhaps somehow help/notify the admin to update certs on
> clients (browsers), etc.
> 
> [1] https://gerrit.ovirt.org/#/q/Ib0de9b1f1c3aa3aa1f3d81d47a91d44f458e7192
> [2] https://gerrit.ovirt.org/41876

This one is only for renewal of the engine certificate in the setup process, as originally implemented by Alon.

--- Additional comment from Gil Klein on 2015-06-15 15:35:41 IDT ---

Note for QE verification:

Please verify based on Alon's comment in the upstream BZ:
https://bugzilla.redhat.com/show_bug.cgi?id=1210486#c3

1. 3.5.0 -> 3.5.3 upgrade - existing hosts, websocket proxy, engine https should continue to work without an issue.

2. 3.5.0 -> 3.5.3 post upgrade - adding new host should succeed, migration between hosts with new and old certificates should work.

3. 3.0 -> 3.5.3 upgrade - should work, ca certificate should be renewed, the .truststore should contain only the new ca certificate, the AIA in cacert.conf should be valid.

--- Additional comment from Yaniv Dary on 2015-06-28 15:04:45 IDT ---

(In reply to Yedidyah Bar David from comment #9)
> Yaniv, please be more specific about what we want to do with this bug for
> 3.5.4.
> 
> If just to enable the existing patches, we need to:
> 
> 1. Revert [1] - Disable renew - was pushed to 3.5 and 3.5.3
> 
> 2. Backport to 3.5 [2] - Prompt before renew - was pushed to master

Yes this is the scope.

> 
> We discussed also other things, such as somehow automatically update certs
> on the hosts, perhaps somehow help/notify the admin to update certs on
> clients (browsers), etc.

This is tracked in another RFE.

> 
> [1] https://gerrit.ovirt.org/#/q/Ib0de9b1f1c3aa3aa1f3d81d47a91d44f458e7192
> [2] https://gerrit.ovirt.org/41876

--- Additional comment from Jiri Belka on 2015-07-29 15:12:10 IDT ---

Please write down/clarify current verification steps. There were discussions if and what should be done with the issue and this can cause confusion for other readers on this BZ. Not sure if info in #11 is still valid. Thank you.

--- Additional comment from Yedidyah Bar David on 2015-07-29 15:49:14 IDT ---

(In reply to Jiri Belka from comment #13)
> Please write down/clarify current verification steps. There were discussions
> if and what should be done with the issue and this can cause confusion for
> other readers on this BZ. Not sure if info in #11 is still valid. Thank you.

In principle, comment 11 is correct. Fixes/additions:

1. s/3.5.3/3.5.4/

2. 3.5.4 does not renew automatically, but asks. If you reply 'no', nothing is changed, and so on a next attempt you are asked again (please verify this too).

3. If you reply 'yes', comment 11 flows apply.

Note that you still might need to renew hosts' certs (simplest is by reinstalling them) and remove the ca cert from your browser (or you'll get an error, depending on browser). Please verify these too, including what happens with hosts that already expired their cert - make sure you can reinstall and that then they work, and detail what you had to do in your browser - iirc Pavel tried this with different browsers and documented results somewhere, not sure.

--- Additional comment from Jiri Belka on 2015-07-31 11:25:01 IDT ---

ok, test based on #14 / rhevm-setup-3.5.4-1.2.el6ev.noarch

my FF 38.0.1 doesn't have problem with new rhevm cert (after renewal).

little comments:

- websocket-proxy has to be restarted to load new certs
- hosts won't get /etc/pki/vdsm/libvirt-spice/ca-cert.pem updated,
  i had to put the host into maintenance and do reinstall to get
  the file updated

not sure if we will have a 'known issue' documentation about certs renewal,
if so then above comments should be probably taken into account.

--- Additional comment from Jiri Belka on 2015-07-31 11:26:10 IDT ---

Please check my #15 comments.

--- Additional comment from Alon Bar-Lev on 2015-08-12 12:28:42 IDT ---

This change broke template upgrade.
This what happens when I do not review.

--- Additional comment from Sandro Bonazzola on 2015-08-12 12:47:30 IDT ---

(In reply to Alon Bar-Lev from comment #17)
> This change broke template upgrade.
> This what happens when I do not review.

I'll be sure to wait for your +1 before merging anything PKI related.

--- Additional comment from Alon Bar-Lev on 2015-08-12 12:50:38 IDT ---

(In reply to Sandro Bonazzola from comment #18)
> (In reply to Alon Bar-Lev from comment #17)
> > This change broke template upgrade.
> > This what happens when I do not review.
> 
> I'll be sure to wait for your +1 before merging anything PKI related.

PKI is integration now... please take full ownership.
You introduce complexity that I rejected... you are on your own as we discussed.
I cannot maintain knowledge anymore in the complexity you introduce to the solutions.
Just do not break anything.

--- Additional comment from Sandro Bonazzola on 2015-08-12 13:02:05 IDT ---

(In reply to Alon Bar-Lev from comment #17)
> This change broke template upgrade.

Can you please attach logs or describe how it's currently broken without your fix?
I'm not the PKI maintainer and he won't be available for the rest of the month.
I've not fully understood the impact and the issue you found here.

> This what happens when I do not review.

--- Additional comment from Alon Bar-Lev on 2015-08-12 13:33:05 IDT ---

(In reply to Sandro Bonazzola from comment #20)
> (In reply to Alon Bar-Lev from comment #17)
> > This change broke template upgrade.
> 
> Can you please attach logs or describe how it's currently broken without
> your fix?
> I'm not the PKI maintainer and he won't be available for the rest of the
> month.
> I've not fully understood the impact and the issue you found here.

the templates were not upgraded, they should be upgraded always.

--- Additional comment from Gil Klein on 2015-08-17 16:11:22 IDT ---

Didi, could you please reply on comment #15 ?

--- Additional comment from Jiri Belka on 2015-08-21 18:57:19 IDT ---

ok, rhevm-setup-3.5.4.2-1.3.el6ev.noarch

- update from older 3.5.0 which had UTCTime issue but not close expiration
  of certificates, renewal for CA (IE ok, FF as based on NSS warns about serial)
- clean 3.5.4.2-1.3, UTCTime in valid encoding
- testing for close expiration (should expire in 2 montgs, switched time to 2010
  and changed DAYS in pki-create-ca.sh to have all certs close to expire on
  3.5.3.1-1.4); all certs on engine host got renewed correctly

testing also included migrations between hosts added in the beginning and later, opening console, spice-html consoles...

----------------------------------------------------------------------
Following comment by didi on September 03 at 07:16:26, 2015

Cloned from upstream bug 1214860 to get it in the release notes.

Comment 1 Yedidyah Bar David 2015-09-03 09:09:40 UTC
See also https://access.redhat.com/solutions/1572983