Bug 2075352 - upgrading RHV-H does not renew certificate
Summary: upgrading RHV-H does not renew certificate
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm
Version: 4.4.10
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ovirt-4.5.0-1
: ---
Assignee: Michal Skrivanek
QA Contact: cshao
URL:
Whiteboard:
Depends On: 2079890
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-04-14 00:00 UTC by Marcus West
Modified: 2022-05-26 16:24 UTC (History)
9 users (show)

Fixed In Version: 4.5.0-10
Doc Type: Rebase: Bug Fixes and Enhancements
Doc Text:
The following changes have been made to the way certificates are generated: Internal CA is issued for 20 years. Internal certificates are valid for 5 years. Internal HTTPS certificates (apache, websocket proxy) are valid for 398 days. CA is renewed 60 days before expiration. Certificates are renewed 365 days before expiration (CertExpirationWarnPeriodInDays configurable via engine-config). CertExpirationAlertPeriodInDays (defaulting to 30) is now also configurable by engine-config. Note that engine certificates and CA are checked/renewed only during engine-setup. Certificates on hosts are renewed/checked during host upgrade or a manual Enroll certificates action.
Clone Of:
Environment:
Last Closed: 2022-05-26 16:24:18 UTC
oVirt Team: Infra
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHV-45664 0 None None None 2022-04-14 00:05:57 UTC
Red Hat Knowledge Base (Solution) 6865861 0 None None None 2022-04-14 04:24:34 UTC
Red Hat Product Errata RHSA-2022:4711 0 None None None 2022-05-26 16:24:28 UTC

Description Marcus West 2022-04-14 00:00:06 UTC
Description of problem:

Upgrading RHV-H does not renew certificate


Version-Release number of selected component (if applicable):

ovirt-engine-4.4.10.7-0.4.el8ev.noarch
rhvh-4.4.6.1-0.20210527.0, upgrade to
rhvh-4.4.10.3-0.20220321.0


How reproducible:

Just once so far


Steps to Reproduce:
1. Note host cert expiry date (openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -dates)
2. Click 'Cluster -> Upgrade'
3. Observe host upgrades successfully
4. check certificate again


Actual results:

[root@mwest-rhvh1 ~]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -dates
notBefore=Jun 20 09:52:22 2021 GMT
notAfter=Jul 24 09:52:22 2022 GMT


Expected results:

Host certificate should be renewed for another 13 months


Additional info:

Now that the expiry time for host certificates has been shortened, users need to pay more attention to regularly renewing them.  If the opportunity to renew when the host is down (for upgrading) is not taken, then the user will have to come back at a later date and shut down the host again (migrating VM's etc), just to renew the host certificate.

AFAIK the process of just renewing the certificates cannot be easily automated.

Comment 5 Martin Perina 2022-04-20 14:35:10 UTC
cluster_upgrade executes 'Host Upgrade' flow for each host and as a part of cluster upgrade certificates are renewed only 7 days before expiring (this period has been extended to 30 days as a part of BZ2056126).

If you want to force renewing certificate, you need to run 'Enroll Certificate' for each affected host.

Comment 6 Christoph Dwertmann 2022-04-22 07:54:19 UTC
@mperina Consider the following scenario:

On day 1 I install RHV. Between days 2 and 368 I may or may not upgrade the RHV cluster from time to time, but the certificates do not get renewed during this period. Between days 369 and 398 I don't upgrade the cluster, maybe because there is no new update available. On day 399 my production cluster falls apart as the certificates have expired.

I manage all RHV clusters via ansible only and never use the UI, so I didn't see any of the cert expiry warning popups.

I know that the "re-enroll" function renews the certificate regardless of how many days are left in the validity period. What is the logic behind the "renew certs during upgrade only when there is less than 30 days validity remaining" rule?

Comment 7 Michal Skrivanek 2022-04-28 15:14:56 UTC
(In reply to Christoph Dwertmann from comment #6)
> @mperina Consider the following scenario:
> 
> On day 1 I install RHV. Between days 2 and 368 I may or may not upgrade the
> RHV cluster from time to time, but the certificates do not get renewed
> during this period. Between days 369 and 398 I don't upgrade the cluster,
> maybe because there is no new update available. On day 399 my production
> cluster falls apart as the certificates have expired.

correct

> I manage all RHV clusters via ansible only and never use the UI, so I didn't
> see any of the cert expiry warning popups.

you should route the warnings (e.g. using ovirt-engine-notifier) to the system that you use, SNMP traps, emails...
Not watching for any alert/error coming from RHV is not a good idea in general.

> I know that the "re-enroll" function renews the certificate regardless of
> how many days are left in the validity period. What is the logic behind the
> "renew certs during upgrade only when there is less than 30 days validity
> remaining" rule?

it's not the best logic, it's a result of various changes and adjustment in the past, and it's indeed problematic. We're fixing that in bug 2079890

Comment 10 Michal Skrivanek 2022-05-09 13:41:13 UTC
upstream bug is moving to 4.5.0-1, moving this accordingly

Comment 14 cshao 2022-05-11 11:05:12 UTC
Test version:
rhvh-4.4.6.1-0.20210527.0
rhvh-4.4.10.3-0.20220321.0
engine 4.5.0-10


Test steps:
1. Install rhvh-4.4.6.1-0.20210527.0
2. register to engine 4.5.0-10
3. upgrade to rhvh-4.4.10.3-0.20220321.0
4. manual Enroll certificates

Test result:
1. After step 3, Certificates on hosts are not renewed
2. After step 4, Certificates on hosts are renewed.


# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -dates
notBefore=May 10 17:29:07 2022 GMT
notAfter=May 12 17:29:07 2027 GMT
# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -dates
notBefore=May 10 18:38:07 2022 GMT
notAfter=May 12 18:38:07 2027 GMT

So the bug is fixed, change bug status to VERIFIED.

Comment 19 errata-xmlrpc 2022-05-26 16:24:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: RHV Manager (ovirt-engine) [ovirt-4.5.0] security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:4711


Note You need to log in before you can comment on or make changes to this bug.