Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1827578

Summary:	User provided certificate OctaviaClientCert is missing after deployment
Product:	Red Hat OpenStack	Reporter:	Gregory Thiemonge <gthiemon>
Component:	openstack-tripleo-heat-templates	Assignee:	Gregory Thiemonge <gthiemon>
Status:	CLOSED ERRATA	QA Contact:	Omer Schwartz <oschwart>
Severity:	high	Docs Contact:
Priority:	high
Version:	16.0 (Train)	CC:	cgoncalves, mburns, mvalsecc, oschwart, pkundal
Target Milestone:	z2	Keywords:	Triaged
Target Release:	16.1 (Train on RHEL 8.2)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-11.3.2-0.20200724133402.e4d56f1.el8ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-10-28 15:37:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Gregory Thiemonge 2020-04-24 09:02:20 UTC

Description of problem:

I tried to deploy Octavia with my own certificates and keys (https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html/networking_guide/sec-octavia#config-octavia-certs-keys).
I used the following parameter file (octavia_parameters.yaml):

parameter_defaults:
    OctaviaCaCert: |
        -----BEGIN CERTIFICATE-----
        <EDITED>
        -----END CERTIFICATE-----

    OctaviaCaKey: |
        -----BEGIN RSA PRIVATE KEY-----
        <EDITED>
        -----END RSA PRIVATE KEY-----

    OctaviaClientCert: |
        -----BEGIN CERTIFICATE-----
        <EDITED>
        -----END CERTIFICATE-----
        -----BEGIN PRIVATE KEY-----
        <EDITED>
        -----END PRIVATE KEY-----

    OctaviaCaKeyPassphrase: <EDITED>

    OctaviaGenerateCerts: false


Included in my overcloud_deploy.sh script (penultimate line):

openstack overcloud deploy \
--timeout 100 \
--templates /usr/share/openstack-tripleo-heat-templates \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \
  --environment-file /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
--stack overcloud \
--libvirt-type kvm \
--ntp-server clock1.rdu2.redhat.com \
-e /home/stack/virt/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/enable-tls.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/public_vip.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
-e /home/stack/virt/hostnames.yml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-ha.yaml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e ~/containers-prepare-parameter.yaml \
-e /home/stack/virt/docker-images.yaml \
-e /home/stack/octavia_parameters.yaml \
--log-file overcloud_deployment_90.log


After deployment, OctaviaClientCert (/var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/client.pem) is missing on the controllers.

[root@controller-0 ~]# find /var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/
/var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/
/var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/private
/var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/private/cakey.pem
/var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/ca_01.pem


And Octavia services throws exceptions when trying to communicate with an amphora:


2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/octavia/controller/worker/v1/tasks/amphora_driver_tasks.py", line 329, in execute          
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server     amp_info = self.amphora_driver.get_info(amphora)                                                                                
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 368, in get_info                
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server     self._populate_amphora_api_version(amphora)                                                                                     
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 105, in _populate_amphora_api_version                                                                                                                                                                                          
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server     amphora)['api_version']                                                                                                         
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 702, in get_api_version         
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server     r = self.get(amp, retry_404=False)                                                                 
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 662, in request                 
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server     r = _request(**reqargs)                                                                                                         
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 546, in get                                                    
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server     return self.request('GET', url, **kwargs)                                                                                       
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request                                                
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server     resp = self.send(prep, **send_kwargs)                                                                                           
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send                                                   
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server     r = adapter.send(request, **kwargs)                                                                                             
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 416, in send                                                   
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server     self.cert_verify(conn, request.url, verify, cert)                                     
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 586, in cert_verify             
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server     self).cert_verify(conn, url, verify, cert)                                                                                      
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 250, in cert_verify                                            
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server     "invalid path: {}".format(conn.cert_file))                                             
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server OSError: Could not find the TLS certificate file, invalid path: /etc/octavia/certs/client.pem                
2020-04-24 08:58:21.616 24 ERROR oslo_messaging.rpc.server     


Version-Release number of selected component (if applicable):
16.1


How reproducible:
100%


Steps to Reproduce:
1. Deploy Octavia with user-provided certificates (using the parameter file in the description)


Actual results:
client.pem is missing on controllers, Octavia cannot configure amphorae.


Expected results:
client.pem should be present on controllers, and Octavia services should be able to communicate with amphorae


Additional info:

Comment 3 Gregory Thiemonge 2020-07-22 14:26:24 UTC

*** Bug 1858609 has been marked as a duplicate of this bug. ***

Comment 9 Omer Schwartz 2020-10-06 07:15:48 UTC

After verification process that involved these steps:

I deployed Octavia with my own certificates and keys (By the steps which are provided in this link: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html/networking_guide/sec-octavia#config-octavia-certs-keys).

I used the following parameter file (octavia_parameters.yaml):

parameter_defaults:
    OctaviaCaCert: |
        -----BEGIN CERTIFICATE-----
        <EDITED>
        -----END CERTIFICATE-----

    OctaviaCaKey: |
        -----BEGIN RSA PRIVATE KEY-----
        <EDITED>
        -----END RSA PRIVATE KEY-----

    OctaviaClientCert: |
        -----BEGIN CERTIFICATE-----
        <EDITED>
        -----END CERTIFICATE-----
        -----BEGIN PRIVATE KEY-----
        <EDITED>
        -----END PRIVATE KEY-----

    OctaviaCaKeyPassphrase: <EDITED>

    OctaviaGenerateCerts: false



Included octavia_parameters.yaml in my overcloud_deploy.sh script.



After deployment, OctaviaClientCert (/var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/client.pem) appears on the 
controllers:

[root@controller-0 ~]# find /var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/
/var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/
/var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/private
/var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/private/cakey.pem
/var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/ca_01.pem
/var/lib/config-data/puppet-generated/octavia/etc/octavia/certs/client.pem            <~~ This is the one



And Octavia service works - when trying to communicate with an amphora (by sending traffic):

(overcloud) [stack@undercloud-0 ~]$ req="curl $LB_FIP"; for i in {1..10}; do $req;echo; done
octaviaclientcerttest-server1-7dd736b4f2a6
octaviaclientcerttest-server2-ddh3angkkpjk
octaviaclientcerttest-server1-7dd736b4f2a6
octaviaclientcerttest-server2-ddh3angkkpjk
octaviaclientcerttest-server1-7dd736b4f2a6
octaviaclientcerttest-server2-ddh3angkkpjk
octaviaclientcerttest-server1-7dd736b4f2a6
octaviaclientcerttest-server2-ddh3angkkpjk
octaviaclientcerttest-server1-7dd736b4f2a6
octaviaclientcerttest-server2-ddh3angkkpjk


(overcloud) [stack@undercloud-0 ~]$ cat /var/lib/rhos-release/latest-installed
16.1  -p RHOS-16.1-RHEL-8-20200917.n.3


client.pem appears on controllers, and Octavia services are able to communicate with amphorae.

Looks good to me, moving this BZ to MODIFIED.

Comment 14 errata-xmlrpc 2020-10-28 15:37:32 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:4284