Bug 1259601 - [RFE][PKI] renew important certificate when about to expire during engine-setup
[RFE][PKI] renew important certificate when about to expire during engine-setup
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine (Show other bugs)
unspecified
Unspecified Unspecified
unspecified Severity high
: ovirt-3.6.0-rc
: 3.6.0
Assigned To: Yedidyah Bar David
Pavel Stehlik
: FutureFeature, Improvement, ZStream
Depends On: 1210486 1214860
Blocks: 1188759 1259634
  Show dependency treegraph
 
Reported: 2015-09-03 03:11 EDT by Yedidyah Bar David
Modified: 2016-03-09 16:12 EST (History)
16 users (show)

See Also:
Fixed In Version: org.ovirt.engine-root-3.5.3-4
Doc Type: Enhancement
Doc Text:
Running engine-setup now checks the expiration date of relevant certificates (including the internal CA certificate and those signed by it) and asks the user whether to renew any certificate that has expired, or is close to expiring. If the user selects 'Yes', those certificates are renewed; 'No' will not alter anything.
Story Points: ---
Clone Of: 1214860
: 1259634 (view as bug list)
Environment:
Last Closed: 2016-03-09 16:12:50 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Integration
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 40330 None None None Never
oVirt gerrit 40508 None None None Never
oVirt gerrit 40509 None None None Never
oVirt gerrit 41820 None None None Never
oVirt gerrit 41863 None None None Never
oVirt gerrit 41864 None None None Never
oVirt gerrit 41876 None None None Never
oVirt gerrit 42649 None None None Never
oVirt gerrit 42650 None None None Never
oVirt gerrit 42726 None None None Never
oVirt gerrit 42729 None None None Never
oVirt gerrit 44743 None None None Never
oVirt gerrit 44744 None None None Never
oVirt gerrit 44745 None None None Never
oVirt gerrit 45002 None None None Never

  None (edit)
Description Yedidyah Bar David 2015-09-03 03:11:12 EDT
+++ This bug was initially created as a clone of Bug #1214860 +++

During engine setup we can renew selected certificates that are important for system operation:

1. ca certificate - without modifying key nor serial.

2. engine certificate - without modifying key as it incorrectly being used for database field encryption.

3. apache certificate - if issued by engine ca.

All other certificates will not be renewed automatically, including these that are at remote servers.

VDSM certificates should be renewed using host redeploy.

--- Additional comment from Itamar Heim on 2015-04-27 12:03:02 IDT ---

don't we still need to distribute the new CA certificate to hosts for them to accept the engine certificate coming from a new CA certificate?
(my concern is post engine start with new certificate, all hosts would become non-responsive)

--- Additional comment from Alon Bar-Lev on 2015-04-27 16:39:40 IDT ---

(In reply to Itamar Heim from comment #1)
> don't we still need to distribute the new CA certificate to hosts for them
> to accept the engine certificate coming from a new CA certificate?
> (my concern is post engine start with new certificate, all hosts would
> become non-responsive)

we will need to redeploy all hosts in any case to renew the CA certificate and the host certificate. but at least this will work as CA certificate and engine certificate at engine machine will be valid.

--- Additional comment from Sandro Bonazzola on 2015-04-29 10:41:02 IDT ---

Yaniv, looks like this bug should be targeted to 3.5.3 not 3.5.4 or the PKI change will be incomplete.

--- Additional comment from Yaniv Dary on 2015-04-30 00:58:53 IDT ---

(In reply to Sandro Bonazzola from comment #3)
> Yaniv, looks like this bug should be targeted to 3.5.3 not 3.5.4 or the PKI
> change will be incomplete.

Should this bug be on POST? what is the tests needed to verify the fix?

--- Additional comment from Sandro Bonazzola on 2015-05-05 10:23:43 IDT ---

(In reply to Yaniv Dary from comment #4)
> (In reply to Sandro Bonazzola from comment #3)
> > Yaniv, looks like this bug should be targeted to 3.5.3 not 3.5.4 or the PKI
> > change will be incomplete.
> 
> Should this bug be on POST?

Yes, it's still on post having patches still to be merged on 3.5 and 3.5.3 branches.

> what is the tests needed to verify the fix?

Alon, can you answer this?

--- Additional comment from Alon Bar-Lev on 2015-05-05 10:39:06 IDT ---

If certificate (CA, engine, websocket) is invalid (bug#1210486) or about to expire (less than a year), a certificate will be renewed.

Special notice should be taken regarding the CA certificate, as it should be distributed to all locations. Browser for example will ask user to confirm the new CA certificate.

engine->vdsm communication should not be effected by this renew, all hosts (which are not expired) should be up.

The process is performed only locally on engine machine, distributed configuration should manually re-issue certificates. for example, host-deploy should be performed for all hosts to renew their certificate and install the new certificate authority.

QA note:
Probably best to adjust clock 2 years back install engine and move clock forward.
If a 3.0/2.2 configuration is available, then try to upgrade and notice all certificates are renewed.

Handy command:
$ openssl x509 -in <certificate> -text

--- Additional comment from Alon Bar-Lev on 2015-06-02 11:55:21 IDT ---

An issue with setup rollback was found and fixed, moved to modified.

--- Additional comment from Alon Bar-Lev on 2015-06-02 12:26:17 IDT ---

(In reply to Alon Bar-Lev from comment #7)
> An issue with setup rollback was found and fixed, moved to modified.

sorry, my bad, should be post.

--- Additional comment from Yedidyah Bar David on 2015-06-10 08:47:32 IDT ---

Yaniv, please be more specific about what we want to do with this bug for 3.5.4.

If just to enable the existing patches, we need to:

1. Revert [1] - Disable renew - was pushed to 3.5 and 3.5.3

2. Backport to 3.5 [2] - Prompt before renew - was pushed to master

We discussed also other things, such as somehow automatically update certs on the hosts, perhaps somehow help/notify the admin to update certs on clients (browsers), etc.

[1] https://gerrit.ovirt.org/#/q/Ib0de9b1f1c3aa3aa1f3d81d47a91d44f458e7192
[2] https://gerrit.ovirt.org/41876

--- Additional comment from Oved Ourfali on 2015-06-10 08:59:06 IDT ---

(In reply to Yedidyah Bar David from comment #9)
> Yaniv, please be more specific about what we want to do with this bug for
> 3.5.4.
> 
> If just to enable the existing patches, we need to:
> 
> 1. Revert [1] - Disable renew - was pushed to 3.5 and 3.5.3
> 
> 2. Backport to 3.5 [2] - Prompt before renew - was pushed to master
> 
> We discussed also other things, such as somehow automatically update certs
> on the hosts, perhaps somehow help/notify the admin to update certs on
> clients (browsers), etc.
> 
> [1] https://gerrit.ovirt.org/#/q/Ib0de9b1f1c3aa3aa1f3d81d47a91d44f458e7192
> [2] https://gerrit.ovirt.org/41876

This one is only for renewal of the engine certificate in the setup process, as originally implemented by Alon.

--- Additional comment from Gil Klein on 2015-06-15 15:35:41 IDT ---

Note for QE verification:

Please verify based on Alon's comment in the upstream BZ:
https://bugzilla.redhat.com/show_bug.cgi?id=1210486#c3

1. 3.5.0 -> 3.5.3 upgrade - existing hosts, websocket proxy, engine https should continue to work without an issue.

2. 3.5.0 -> 3.5.3 post upgrade - adding new host should succeed, migration between hosts with new and old certificates should work.

3. 3.0 -> 3.5.3 upgrade - should work, ca certificate should be renewed, the .truststore should contain only the new ca certificate, the AIA in cacert.conf should be valid.

--- Additional comment from Yaniv Dary on 2015-06-28 15:04:45 IDT ---

(In reply to Yedidyah Bar David from comment #9)
> Yaniv, please be more specific about what we want to do with this bug for
> 3.5.4.
> 
> If just to enable the existing patches, we need to:
> 
> 1. Revert [1] - Disable renew - was pushed to 3.5 and 3.5.3
> 
> 2. Backport to 3.5 [2] - Prompt before renew - was pushed to master

Yes this is the scope.

> 
> We discussed also other things, such as somehow automatically update certs
> on the hosts, perhaps somehow help/notify the admin to update certs on
> clients (browsers), etc.

This is tracked in another RFE.

> 
> [1] https://gerrit.ovirt.org/#/q/Ib0de9b1f1c3aa3aa1f3d81d47a91d44f458e7192
> [2] https://gerrit.ovirt.org/41876

--- Additional comment from Jiri Belka on 2015-07-29 15:12:10 IDT ---

Please write down/clarify current verification steps. There were discussions if and what should be done with the issue and this can cause confusion for other readers on this BZ. Not sure if info in #11 is still valid. Thank you.

--- Additional comment from Yedidyah Bar David on 2015-07-29 15:49:14 IDT ---

(In reply to Jiri Belka from comment #13)
> Please write down/clarify current verification steps. There were discussions
> if and what should be done with the issue and this can cause confusion for
> other readers on this BZ. Not sure if info in #11 is still valid. Thank you.

In principle, comment 11 is correct. Fixes/additions:

1. s/3.5.3/3.5.4/

2. 3.5.4 does not renew automatically, but asks. If you reply 'no', nothing is changed, and so on a next attempt you are asked again (please verify this too).

3. If you reply 'yes', comment 11 flows apply.

Note that you still might need to renew hosts' certs (simplest is by reinstalling them) and remove the ca cert from your browser (or you'll get an error, depending on browser). Please verify these too, including what happens with hosts that already expired their cert - make sure you can reinstall and that then they work, and detail what you had to do in your browser - iirc Pavel tried this with different browsers and documented results somewhere, not sure.

--- Additional comment from Jiri Belka on 2015-07-31 11:25:01 IDT ---

ok, test based on #14 / rhevm-setup-3.5.4-1.2.el6ev.noarch

my FF 38.0.1 doesn't have problem with new rhevm cert (after renewal).

little comments:

- websocket-proxy has to be restarted to load new certs
- hosts won't get /etc/pki/vdsm/libvirt-spice/ca-cert.pem updated,
  i had to put the host into maintenance and do reinstall to get
  the file updated

not sure if we will have a 'known issue' documentation about certs renewal,
if so then above comments should be probably taken into account.

--- Additional comment from Jiri Belka on 2015-07-31 11:26:10 IDT ---

Please check my #15 comments.

--- Additional comment from Alon Bar-Lev on 2015-08-12 12:28:42 IDT ---

This change broke template upgrade.
This what happens when I do not review.

--- Additional comment from Sandro Bonazzola on 2015-08-12 12:47:30 IDT ---

(In reply to Alon Bar-Lev from comment #17)
> This change broke template upgrade.
> This what happens when I do not review.

I'll be sure to wait for your +1 before merging anything PKI related.

--- Additional comment from Alon Bar-Lev on 2015-08-12 12:50:38 IDT ---

(In reply to Sandro Bonazzola from comment #18)
> (In reply to Alon Bar-Lev from comment #17)
> > This change broke template upgrade.
> > This what happens when I do not review.
> 
> I'll be sure to wait for your +1 before merging anything PKI related.

PKI is integration now... please take full ownership.
You introduce complexity that I rejected... you are on your own as we discussed.
I cannot maintain knowledge anymore in the complexity you introduce to the solutions.
Just do not break anything.

--- Additional comment from Sandro Bonazzola on 2015-08-12 13:02:05 IDT ---

(In reply to Alon Bar-Lev from comment #17)
> This change broke template upgrade.

Can you please attach logs or describe how it's currently broken without your fix?
I'm not the PKI maintainer and he won't be available for the rest of the month.
I've not fully understood the impact and the issue you found here.

> This what happens when I do not review.

--- Additional comment from Alon Bar-Lev on 2015-08-12 13:33:05 IDT ---

(In reply to Sandro Bonazzola from comment #20)
> (In reply to Alon Bar-Lev from comment #17)
> > This change broke template upgrade.
> 
> Can you please attach logs or describe how it's currently broken without
> your fix?
> I'm not the PKI maintainer and he won't be available for the rest of the
> month.
> I've not fully understood the impact and the issue you found here.

the templates were not upgraded, they should be upgraded always.

--- Additional comment from Gil Klein on 2015-08-17 16:11:22 IDT ---

Didi, could you please reply on comment #15 ?

--- Additional comment from Jiri Belka on 2015-08-21 18:57:19 IDT ---

ok, rhevm-setup-3.5.4.2-1.3.el6ev.noarch

- update from older 3.5.0 which had UTCTime issue but not close expiration
  of certificates, renewal for CA (IE ok, FF as based on NSS warns about serial)
- clean 3.5.4.2-1.3, UTCTime in valid encoding
- testing for close expiration (should expire in 2 montgs, switched time to 2010
  and changed DAYS in pki-create-ca.sh to have all certs close to expire on
  3.5.3.1-1.4); all certs on engine host got renewed correctly

testing also included migrations between hosts added in the beginning and later, opening console, spice-html consoles...
Comment 1 Yedidyah Bar David 2015-09-03 03:16:26 EDT
Cloned from upstream bug 1214860 to get it in the release notes.
Comment 5 errata-xmlrpc 2016-03-09 16:12:50 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0376.html

Note You need to log in before you can comment on or make changes to this bug.