Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1661621

Summary: When using novajoin and IPA with an external CA chain, haproxy is not configured with the proper intermediate cert chain
Product: Red Hat OpenStack Reporter: Brian J. Atkisson <batkisso>
Component: openstack-tripleo-heat-templatesAssignee: Emilien Macchi <emacchi>
Status: CLOSED DUPLICATE QA Contact: Gurenko Alex <agurenko>
Severity: medium Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: bperkins, dhill, hrybacki, mburns, nkinder
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-04-05 16:04:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1595876    
Bug Blocks:    

Description Brian J. Atkisson 2018-12-21 18:49:58 UTC
Description of problem:

We are using novajoin internally for our IT OpenStack deployment. We
have been having TLS problems with horizon's VM console. We just noticed
that the overcloud services are not sending a cert chain with
all the necessary intermediate CA certs.

In our environment, we have the following chain:

1. Root CA (CN=Red Hat IT Root CA)
2. Intermediate CA (CN=Intermediate Certificate Authority)
3. IPA CA (O=IPA.REDHAT.COM/CN=Certificate Authority)

We used novajoin and director to setup everything. It appears that none
of the haproxy front-ended services are configured to send the cert
chain, only sending the actual server cert.

To fix this, we pretty much just did the following on the controllers:
cat /etc/ipa/ca.crt >> /etc/pki/tls/private/overcloud_endpoint.pem
cat /etc/ipa/ca.crt >>
/etc/pki/tls/certs/haproxy/overcloud-haproxy-internal_api.pem

It seems like this needs to be configured as part of certmonger's post command. 

Note: I have seen problems with certmonger renewing certs when CA certs are included the cert pem file that certmonger manages.


Version-Release number of selected component (if applicable):

OSP 13

On Director:

puppet-openstacklib-12.4.0-1.el7ost.noarch
openstack-mistral-api-6.0.5-1.el7ost.noarch
openstack-neutron-12.0.4-2.el7ost.noarch
openstack-ironic-conductor-10.1.6-2.el7ost.noarch
openstack-nova-placement-api-17.0.7-2.el7ost.noarch
openstack-heat-api-10.0.2-2.el7ost.noarch
openstack-swift-proxy-2.17.1-0.20180911215500.b38a303.el7ost.noarch
python2-openstackclient-3.14.2-2.el7ost.noarch
openstack-tripleo-heat-templates-8.0.7-4.el7ost.noarch
openstack-ironic-common-10.1.6-2.el7ost.noarch
openstack-tripleo-image-elements-8.0.1-1.el7ost.noarch
openstack-mistral-engine-6.0.5-1.el7ost.noarch
openstack-nova-compute-17.0.7-2.el7ost.noarch
openstack-neutron-openvswitch-12.0.4-2.el7ost.noarch
openstack-ironic-api-10.1.6-2.el7ost.noarch
openstack-ironic-inspector-7.2.1-4.el7ost.noarch
openstack-swift-object-2.17.1-0.20180911215500.b38a303.el7ost.noarch
openstack-tripleo-validations-8.4.4-1.el7ost.noarch
python-openstackclient-lang-3.14.2-2.el7ost.noarch
openstack-tripleo-common-8.6.6-2.el7ost.noarch
openstack-heat-common-10.0.2-2.el7ost.noarch
openstack-neutron-common-12.0.4-2.el7ost.noarch
openstack-mistral-common-6.0.5-1.el7ost.noarch
openstack-keystone-13.0.1-2.el7ost.noarch
openstack-nova-conductor-17.0.7-2.el7ost.noarch
openstack-nova-api-17.0.7-2.el7ost.noarch
openstack-ironic-staging-drivers-0.9.1-1.el7ost.noarch
openstack-heat-engine-10.0.2-2.el7ost.noarch
openstack-tempest-18.0.0-4.el7ost.noarch
openstack-swift-account-2.17.1-0.20180911215500.b38a303.el7ost.noarch
openstack-tripleo-ui-8.3.2-2.el7ost.noarch
openstack-zaqar-6.0.1-1.el7ost.noarch
puppet-openstack_extras-12.4.1-0.20180831193234.7fed86a.el7ost.noarch
openstack-tripleo-common-containers-8.6.6-2.el7ost.noarch
openstack-tripleo-puppet-elements-8.0.1-1.el7ost.noarch
openstack-glance-16.0.1-5.el7ost.noarch
openstack-nova-scheduler-17.0.7-2.el7ost.noarch
openstack-heat-api-cfn-10.0.2-2.el7ost.noarch
openstack-swift-container-2.17.1-0.20180911215500.b38a303.el7ost.noarch
openstack-selinux-0.8.15-1.el7ost.noarch
python2-openstacksdk-0.11.3-1.el7ost.noarch
openstack-nova-common-17.0.7-2.el7ost.noarch
openstack-mistral-executor-6.0.5-1.el7ost.noarch
openstack-neutron-ml2-12.0.4-2.el7ost.noarch


How reproducible:
Always

Steps to Reproduce:
1. Install environment with IPA integration. IPA's CA cert needs to be signed by an external CA
2. openstack deploy
3. openssl s_client -showcert -connect overcloud:443

Actual results:
Only the server cert is returned

Expected results:
Intermediate certs to be delivered as part of TLS initialization, providing the cert chain + server cert in the following order:

1. server cert
2. IPA CA cert
3. Intermediate CA cert

Additional info:

Most browsers cache intermediate CA certs these days, so the problem is not immediately noticeable. However, it appears when using non-browser tooling (CLI / API integration / ansible / etc).

Comment 2 Nathan Kinder 2018-12-21 19:28:20 UTC
This is likely a bug.  We use certmonger's "post-save" command to update
the certs/keys for HAProxy.  You can see the details of this post save
command on your controllers like this:

----------------------------------------------------------------------
[heat-admin@overcloud-controller-0 ~]$ sudo getcert list -i
haproxy-external-cert
Number of certificates and requests being tracked: 17.
Request ID 'haproxy-external-cert':
	status: MONITORING
	stuck: no
	key pair storage:
type=FILE,location='/etc/pki/tls/private/haproxy/overcloud-haproxy-external.key'
	certificate:
type=FILE,location='/etc/pki/tls/certs/haproxy/overcloud-haproxy-external.crt'
	CA: IPA
	issuer: CN=Certificate Authority,O=OOO.TEST
	subject: CN=overcloud.ooo.test,O=OOO.TEST
	expires: 2020-12-15 01:19:52 UTC
	dns: overcloud.ooo.test
	principal name: haproxy/overcloud.ooo.test
	key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
	eku: id-kp-serverAuth,id-kp-clientAuth
	pre-save command:
	post-save command: "cat
/etc/pki/tls/certs/haproxy/overcloud-haproxy-external.crt
/etc/pki/tls/private/haproxy/overcloud-haproxy-external.key >
/etc/pki/tls/private/overcloud_endpoint.pem && if systemctl -q is-active
haproxy; then systemctl reload haproxy; else true; fi"
	track: yes
	auto-renew: yes
----------------------------------------------------------------------

The post-save command used for HAProxy appears to be configurable at the
puppet-tripleo level:

----------------------------------------------------------------------
https://github.com/openstack/puppet-tripleo/blob/392d629d9f83bd2b2445d0e473dde4dde3510294/manifests/certmonger/haproxy.pp#L47-L49

https://github.com/openstack/puppet-tripleo/blob/392d629d9f83bd2b2445d0e473dde4dde3510294/manifests/profile/base/certmonger_user.pp#L55-L58
----------------------------------------------------------------------

I don't see anywhere we can specify this via tripleo-heat-templates
though, which is what you would need to be able to pass a custom command
in.  I believe we need to add some flexibility here so you can have
other certificates injected into the HAProxy bundle.  This would be
needed for cases where some outside CA is used for the external certs.

The other thing we should do here is to change the default behavior when
"TLS everywhere" is used and IPA is used for the external certs.  In
this case, we should just automatically add /etc/ipa/ca.crt in the
post-save command instead of requiring the user to override it.

Comment 5 Red Hat Bugzilla 2023-09-14 04:44:11 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days