Bug 1214860 - [RFE][PKI] renew important certificate when about to expire during engine-setup
Summary: [RFE][PKI] renew important certificate when about to expire during engine-setup
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: oVirt
Classification: Retired
Component: ovirt-engine-core
Version: unspecified
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.5.4
Assignee: Yedidyah Bar David
QA Contact: Jiri Belka
URL:
Whiteboard: integration
Depends On: 1210486
Blocks: 1188759 1259601
TreeView+ depends on / blocked
 
Reported: 2015-04-23 17:16 UTC by Alon Bar-Lev
Modified: 2015-09-21 09:33 UTC (History)
14 users (show)

Fixed In Version: org.ovirt.engine-root-3.5.3-4
Clone Of:
: 1259601 (view as bug list)
Environment:
Last Closed: 2015-09-03 13:54:11 UTC
oVirt Team: ---
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1210486 0 unspecified CLOSED [PKI] CA certificate notBefore should confirm to rfc2459 2021-02-22 00:41:40 UTC
oVirt gerrit 40330 0 master MERGED packaging: setup: pki: renew about to expire certificates 2020-12-07 08:05:23 UTC
oVirt gerrit 40508 0 ovirt-engine-3.5 MERGED packaging: setup: pki: renew about to expire certificates 2020-12-07 08:05:50 UTC
oVirt gerrit 40509 0 ovirt-engine-3.5.3 MERGED packaging: setup: pki: renew about to expire certificates 2020-12-07 08:05:23 UTC
oVirt gerrit 41820 0 master MERGED packaging: setup: pki: use local transaction when extracting PKCS#12 2020-12-07 08:05:22 UTC
oVirt gerrit 41863 0 ovirt-engine-3.5 MERGED packaging: setup: pki: use local transaction when extracting PKCS#12 2020-12-07 08:05:24 UTC
oVirt gerrit 41864 0 ovirt-engine-3.5.3 MERGED packaging: setup: pki: use local transaction when extracting PKCS#12 2020-12-07 08:05:22 UTC
oVirt gerrit 41876 0 master MERGED packaging: setup: pki: Prompt before renewal 2020-12-07 08:05:24 UTC
oVirt gerrit 42649 0 ovirt-engine-3.5 MERGED Revert "packaging: setup: Disable pki renewal for 3.5" 2020-12-07 08:05:24 UTC
oVirt gerrit 42650 0 ovirt-engine-3.5 MERGED packaging: setup: pki: Prompt before renewal 2020-12-07 08:05:25 UTC
oVirt gerrit 42726 0 master MERGED packaging: setup: Clarify prompt text on certs renew 2020-12-07 08:05:23 UTC
oVirt gerrit 42729 0 ovirt-engine-3.5 MERGED packaging: setup: Clarify prompt text on certs renew 2020-12-07 08:05:23 UTC
oVirt gerrit 44743 0 master MERGED packaging: setup: pki fix breakage introduced by Idca016 2020-12-07 08:05:25 UTC
oVirt gerrit 44744 0 ovirt-engine-3.6 MERGED packaging: setup: pki fix breakage introduced by Idca016 2020-12-07 08:05:25 UTC
oVirt gerrit 44745 0 ovirt-engine-3.5 MERGED packaging: setup: pki fix breakage introduced by Idca016 2020-12-07 08:05:25 UTC
oVirt gerrit 45002 0 ovirt-engine-3.5.4 MERGED packaging: setup: pki fix breakage introduced by Idca016 2020-12-07 08:05:53 UTC

Internal Links: 1210486

Description Alon Bar-Lev 2015-04-23 17:16:07 UTC
During engine setup we can renew selected certificates that are important for system operation:

1. ca certificate - without modifying key nor serial.

2. engine certificate - without modifying key as it incorrectly being used for database field encryption.

3. apache certificate - if issued by engine ca.

All other certificates will not be renewed automatically, including these that are at remote servers.

VDSM certificates should be renewed using host redeploy.

Comment 1 Itamar Heim 2015-04-27 09:03:02 UTC
don't we still need to distribute the new CA certificate to hosts for them to accept the engine certificate coming from a new CA certificate?
(my concern is post engine start with new certificate, all hosts would become non-responsive)

Comment 2 Alon Bar-Lev 2015-04-27 13:39:40 UTC
(In reply to Itamar Heim from comment #1)
> don't we still need to distribute the new CA certificate to hosts for them
> to accept the engine certificate coming from a new CA certificate?
> (my concern is post engine start with new certificate, all hosts would
> become non-responsive)

we will need to redeploy all hosts in any case to renew the CA certificate and the host certificate. but at least this will work as CA certificate and engine certificate at engine machine will be valid.

Comment 3 Sandro Bonazzola 2015-04-29 07:41:02 UTC
Yaniv, looks like this bug should be targeted to 3.5.3 not 3.5.4 or the PKI change will be incomplete.

Comment 4 Yaniv Lavi 2015-04-29 21:58:53 UTC
(In reply to Sandro Bonazzola from comment #3)
> Yaniv, looks like this bug should be targeted to 3.5.3 not 3.5.4 or the PKI
> change will be incomplete.

Should this bug be on POST? what is the tests needed to verify the fix?

Comment 5 Sandro Bonazzola 2015-05-05 07:23:43 UTC
(In reply to Yaniv Dary from comment #4)
> (In reply to Sandro Bonazzola from comment #3)
> > Yaniv, looks like this bug should be targeted to 3.5.3 not 3.5.4 or the PKI
> > change will be incomplete.
> 
> Should this bug be on POST?

Yes, it's still on post having patches still to be merged on 3.5 and 3.5.3 branches.

> what is the tests needed to verify the fix?

Alon, can you answer this?

Comment 6 Alon Bar-Lev 2015-05-05 07:39:06 UTC
If certificate (CA, engine, websocket) is invalid (bug#1210486) or about to expire (less than a year), a certificate will be renewed.

Special notice should be taken regarding the CA certificate, as it should be distributed to all locations. Browser for example will ask user to confirm the new CA certificate.

engine->vdsm communication should not be effected by this renew, all hosts (which are not expired) should be up.

The process is performed only locally on engine machine, distributed configuration should manually re-issue certificates. for example, host-deploy should be performed for all hosts to renew their certificate and install the new certificate authority.

QA note:
Probably best to adjust clock 2 years back install engine and move clock forward.
If a 3.0/2.2 configuration is available, then try to upgrade and notice all certificates are renewed.

Handy command:
$ openssl x509 -in <certificate> -text

Comment 7 Alon Bar-Lev 2015-06-02 08:55:21 UTC
An issue with setup rollback was found and fixed, moved to modified.

Comment 8 Alon Bar-Lev 2015-06-02 09:26:17 UTC
(In reply to Alon Bar-Lev from comment #7)
> An issue with setup rollback was found and fixed, moved to modified.

sorry, my bad, should be post.

Comment 9 Yedidyah Bar David 2015-06-10 05:47:32 UTC
Yaniv, please be more specific about what we want to do with this bug for 3.5.4.

If just to enable the existing patches, we need to:

1. Revert [1] - Disable renew - was pushed to 3.5 and 3.5.3

2. Backport to 3.5 [2] - Prompt before renew - was pushed to master

We discussed also other things, such as somehow automatically update certs on the hosts, perhaps somehow help/notify the admin to update certs on clients (browsers), etc.

[1] https://gerrit.ovirt.org/#/q/Ib0de9b1f1c3aa3aa1f3d81d47a91d44f458e7192
[2] https://gerrit.ovirt.org/41876

Comment 10 Oved Ourfali 2015-06-10 05:59:06 UTC
(In reply to Yedidyah Bar David from comment #9)
> Yaniv, please be more specific about what we want to do with this bug for
> 3.5.4.
> 
> If just to enable the existing patches, we need to:
> 
> 1. Revert [1] - Disable renew - was pushed to 3.5 and 3.5.3
> 
> 2. Backport to 3.5 [2] - Prompt before renew - was pushed to master
> 
> We discussed also other things, such as somehow automatically update certs
> on the hosts, perhaps somehow help/notify the admin to update certs on
> clients (browsers), etc.
> 
> [1] https://gerrit.ovirt.org/#/q/Ib0de9b1f1c3aa3aa1f3d81d47a91d44f458e7192
> [2] https://gerrit.ovirt.org/41876

This one is only for renewal of the engine certificate in the setup process, as originally implemented by Alon.

Comment 11 Gil Klein 2015-06-15 12:35:41 UTC
Note for QE verification:

Please verify based on Alon's comment in the upstream BZ:
https://bugzilla.redhat.com/show_bug.cgi?id=1210486#c3

1. 3.5.0 -> 3.5.3 upgrade - existing hosts, websocket proxy, engine https should continue to work without an issue.

2. 3.5.0 -> 3.5.3 post upgrade - adding new host should succeed, migration between hosts with new and old certificates should work.

3. 3.0 -> 3.5.3 upgrade - should work, ca certificate should be renewed, the .truststore should contain only the new ca certificate, the AIA in cacert.conf should be valid.

Comment 12 Yaniv Lavi 2015-06-28 12:04:45 UTC
(In reply to Yedidyah Bar David from comment #9)
> Yaniv, please be more specific about what we want to do with this bug for
> 3.5.4.
> 
> If just to enable the existing patches, we need to:
> 
> 1. Revert [1] - Disable renew - was pushed to 3.5 and 3.5.3
> 
> 2. Backport to 3.5 [2] - Prompt before renew - was pushed to master

Yes this is the scope.

> 
> We discussed also other things, such as somehow automatically update certs
> on the hosts, perhaps somehow help/notify the admin to update certs on
> clients (browsers), etc.

This is tracked in another RFE.

> 
> [1] https://gerrit.ovirt.org/#/q/Ib0de9b1f1c3aa3aa1f3d81d47a91d44f458e7192
> [2] https://gerrit.ovirt.org/41876

Comment 13 Jiri Belka 2015-07-29 12:12:10 UTC
Please write down/clarify current verification steps. There were discussions if and what should be done with the issue and this can cause confusion for other readers on this BZ. Not sure if info in #11 is still valid. Thank you.

Comment 14 Yedidyah Bar David 2015-07-29 12:49:14 UTC
(In reply to Jiri Belka from comment #13)
> Please write down/clarify current verification steps. There were discussions
> if and what should be done with the issue and this can cause confusion for
> other readers on this BZ. Not sure if info in #11 is still valid. Thank you.

In principle, comment 11 is correct. Fixes/additions:

1. s/3.5.3/3.5.4/

2. 3.5.4 does not renew automatically, but asks. If you reply 'no', nothing is changed, and so on a next attempt you are asked again (please verify this too).

3. If you reply 'yes', comment 11 flows apply.

Note that you still might need to renew hosts' certs (simplest is by reinstalling them) and remove the ca cert from your browser (or you'll get an error, depending on browser). Please verify these too, including what happens with hosts that already expired their cert - make sure you can reinstall and that then they work, and detail what you had to do in your browser - iirc Pavel tried this with different browsers and documented results somewhere, not sure.

Comment 15 Jiri Belka 2015-07-31 08:25:01 UTC
ok, test based on #14 / rhevm-setup-3.5.4-1.2.el6ev.noarch

my FF 38.0.1 doesn't have problem with new rhevm cert (after renewal).

little comments:

- websocket-proxy has to be restarted to load new certs
- hosts won't get /etc/pki/vdsm/libvirt-spice/ca-cert.pem updated,
  i had to put the host into maintenance and do reinstall to get
  the file updated

not sure if we will have a 'known issue' documentation about certs renewal,
if so then above comments should be probably taken into account.

Comment 16 Jiri Belka 2015-07-31 08:26:10 UTC
Please check my #15 comments.

Comment 17 Alon Bar-Lev 2015-08-12 09:28:42 UTC
This change broke template upgrade.
This what happens when I do not review.

Comment 18 Sandro Bonazzola 2015-08-12 09:47:30 UTC
(In reply to Alon Bar-Lev from comment #17)
> This change broke template upgrade.
> This what happens when I do not review.

I'll be sure to wait for your +1 before merging anything PKI related.

Comment 19 Alon Bar-Lev 2015-08-12 09:50:38 UTC
(In reply to Sandro Bonazzola from comment #18)
> (In reply to Alon Bar-Lev from comment #17)
> > This change broke template upgrade.
> > This what happens when I do not review.
> 
> I'll be sure to wait for your +1 before merging anything PKI related.

PKI is integration now... please take full ownership.
You introduce complexity that I rejected... you are on your own as we discussed.
I cannot maintain knowledge anymore in the complexity you introduce to the solutions.
Just do not break anything.

Comment 20 Sandro Bonazzola 2015-08-12 10:02:05 UTC
(In reply to Alon Bar-Lev from comment #17)
> This change broke template upgrade.

Can you please attach logs or describe how it's currently broken without your fix?
I'm not the PKI maintainer and he won't be available for the rest of the month.
I've not fully understood the impact and the issue you found here.

> This what happens when I do not review.

Comment 21 Alon Bar-Lev 2015-08-12 10:33:05 UTC
(In reply to Sandro Bonazzola from comment #20)
> (In reply to Alon Bar-Lev from comment #17)
> > This change broke template upgrade.
> 
> Can you please attach logs or describe how it's currently broken without
> your fix?
> I'm not the PKI maintainer and he won't be available for the rest of the
> month.
> I've not fully understood the impact and the issue you found here.

the templates were not upgraded, they should be upgraded always.

Comment 22 Gil Klein 2015-08-17 13:11:22 UTC
Didi, could you please reply on comment #15 ?

Comment 23 Jiri Belka 2015-08-21 15:57:19 UTC
ok, rhevm-setup-3.5.4.2-1.3.el6ev.noarch

- update from older 3.5.0 which had UTCTime issue but not close expiration
  of certificates, renewal for CA (IE ok, FF as based on NSS warns about serial)
- clean 3.5.4.2-1.3, UTCTime in valid encoding
- testing for close expiration (should expire in 2 montgs, switched time to 2010
  and changed DAYS in pki-create-ca.sh to have all certs close to expire on
  3.5.3.1-1.4); all certs on engine host got renewed correctly

testing also included migrations between hosts added in the beginning and later, opening console, spice-html consoles...

Comment 24 Sandro Bonazzola 2015-09-03 13:54:11 UTC
This is an automated message.
oVirt 3.5.4 has been released on September 3rd 2015 and should include the fix for this BZ. Moving to closed current release.


Note You need to log in before you can comment on or make changes to this bug.