Bug 1975505

Summary: Auto-generated certificates doesn't work properly on undercloud
Product: Red Hat OpenStack Reporter: Cristian Muresanu <cmuresan>
Component: puppet-certmongerAssignee: Ade Lee <alee>
Status: CLOSED DUPLICATE QA Contact: Jeremy Agee <jagee>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 16.1 (Train)CC: alee, cmuresan, dmendiza, jjoyce, jschluet, mircea.vutcovici, slinaber, tvignaud
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-19 14:06:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Cristian Muresanu 2021-06-23 20:03:47 UTC
Description of problem:

When using Auto-generated certificates on undercloud according with documentation [1], certmonger doesn't update the CA trust when local root CA is renewed. Also certmonger doesn't run the "update-ca-trust extract"

Workaround:
# cp /etc/pki/ca-trust/source/anchors/cm-local-ca.pem{,-$(date +%F_%H%M%S)}
# openssl pkcs12 -in /var/lib/certmonger/local/creds -out /etc/pki/ca-trust/source/anchors/cm-local-ca.pem -nokeys -nodes -passin pass:""
# sudo update-ca-trust extract

[1] "7.13. Creating an environment file for undercloud CA trust"
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/director_installation_and_usage/index#creating-an-environment-file-for-undercloud-ca-trust

Version-Release number of selected component (if applicable):


How reproducible:
All the time

Steps to Reproduce:
1.
2.
3.

Actual results:
(undercloud) [stack@director ~]$ openstack baremetal node list
Failed to discover available identity versions when contacting https://10.226.36.11:13000. Attempting to parse version from URL.
Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. SSL exception connecting to https://10.226.36.11:13000: HTTPSConnectionPool(host='10.226.36.11', port=13000): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))

Comment 1 Ade Lee 2021-06-28 16:31:29 UTC
Just a few notes on this.

First, this issue only becomes an issue when an undercloud update is performed.  I tested this by manually removing
the /var/lib/certmonger/local/creds file and restarting certmonger.  This resulted in a new creds file befing generated - 
which is essentially the same as the certmonger local ca cert being recreated on expiry.

When I did this, I was able to access the openstack API just fine because the /etc/pki/ca-trust/source/anchors/cm-local-ca.pem
file had not been modified and the certificate contained therein had already been trusted.

There was an issue with updating and running the trust command which was fixed by the following upstream commit:

https://github.com/openstack/puppet-tripleo/commit/53007c0b30fd48ca8ee2bfdca41be15de6ac57de

The issue here was that the /etc/pki/ca-trust/source/anchors/cm-local-ca.pem file was not updated and trusted because of a
conditional whereby the file was only updated when the cert in /etc/pki/ca-trust/source/anchors/cm-local-ca.pem had expired,
even if the underlying creds file had changed.

In the meantime, though, as part of the update, a new HA proxy cert was issued through certmonger - using the new creds - and so
a mismatch existed between the ca cert the signed the ha proxy cert  (the new creds) and the cert that was trusted in  /etc/pki/ca-trust/source/anchors/cm-local-ca.pem

The solution to the above problem was to simplify have  /etc/pki/ca-trust/source/anchors/cm-local-ca.pem always be updated when
an update occurs.

Now we need to look to see when the above fix was delivered in 16.1/ 16.2

Note that this fix was delivered in OSP 13z16: https://bugzilla.redhat.com/show_bug.cgi?id=1888898 in puppet-tripleo-8.5.1-22.el7ost

Comment 2 Ade Lee 2021-06-28 16:47:52 UTC
It looks like the fix to this issue was delivered as part of 16.1.6 in https://bugzilla.redhat.com/show_bug.cgi?id=1921691
This is puppet-tripleo-11.5.0-1.20210406223722.f716ef5.el8ost.noarch.rpm

Please confirm that this version of puppet-tripleo is greater than or equal to this version,
and/or check for the fix in :

https://github.com/openstack/puppet-tripleo/commit/53007c0b30fd48ca8ee2bfdca41be15de6ac57de
/usr/share/openstack-puppet/modules/tripleo/manifests/certmonger/ca/local.pp

If this fix is present, then something else is going on.

Comment 4 Douglas Mendizábal 2021-07-19 14:06:25 UTC

*** This bug has been marked as a duplicate of bug 1921691 ***

Comment 5 Red Hat Bugzilla 2023-09-18 00:27:51 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days