Bug 1553520

Summary: Cannot create listener with TLS termination
Product: Red Hat OpenStack Reporter: Carlos Goncalves <cgoncalves>
Component: openstack-octaviaAssignee: Carlos Goncalves <cgoncalves>
Status: CLOSED ERRATA QA Contact: Alexander Stafeyev <astafeye>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 13.0 (Queens)CC: amuller, bcafarel, ihrachys, jschluet, lpeer, majopela, nmanos, nyechiel, rhbz, samccann
Target Milestone: rcKeywords: Triaged
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-octavia-2.0.1-0.20180327200337.e06b95f.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-27 13:35:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1553517, 1553521, 1554336, 1554409, 1556933, 1576436    
Bug Blocks: 1433523    

Description Carlos Goncalves 2018-03-09 01:29:07 UTC
Description of problem:

Octavia requires python2-cryptography!=2.0,>=1.9 [1] and is synced with global-requirement.txt [2]. RHEL/CentOS7 provides python2-cryptography-1.7.2-1.el7 which is not good enough and throws exceptions on load balancer create in Octavia:

2018-03-08 23:45:46.453 24634 ERROR octavia.controller.worker.controller_worker   File "/usr/lib/python2.7/site-packages/octavia/certificates/common/pkcs12.py", line 35, in get_certificate
2018-03-08 23:45:46.453 24634 ERROR octavia.controller.worker.controller_worker     return self.certificate.to_cryptography().public_bytes(
2018-03-08 23:45:46.453 24634 ERROR octavia.controller.worker.controller_worker AttributeError: 'X509' object has no attribute 'to_cryptography'

Version-Release number of selected component (if applicable):

openstack-octavia-api-2.0.0-1.el7.noarch
openstack-octavia-common-2.0.0-1.el7.noarch
openstack-octavia-health-manager-2.0.0-1.el7.noarch
openstack-octavia-housekeeping-2.0.0-1.el7.noarch
openstack-octavia-worker-2.0.0-1.el7.noarch
python2-octaviaclient-1.4.0-1.el7.noarch
python-octavia-2.0.0-1.el7.noarch

How reproducible: 100%


Steps to Reproduce:
1. openstack loadbalancer create lb2
2. openstack loadbalancer listener create --protocol-port 443 --protocol TERMINATED_HTTPS --name listener2 --default-tls-container=http://<ommitted>:9311/v1/secrets/50a1b6e0-b53c-4b33-a06d-0544eaaf02f0 lb2


Actual results:

2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/controller/queue/endpoint.py", line 68, in create_listener
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server     self.worker.create_listener(listener_id)
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/controller/worker/controller_worker.py", line 206, in create_listener
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server     create_listener_tf.run()
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server     for _state in self.run_iter(timeout=timeout):
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server     failure.Failure.reraise_if_any(er_failures)
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server     failures[0].reraise()
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server     six.reraise(*self._exc_info)
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server     result = task.execute(**arguments)
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py", line 56, in execute
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server     self.amphora_driver.update(listener, loadbalancer.vip)
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 67, in update
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server     certs = self._process_tls_certificates(listener)
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 175, in _process_tls_certificates
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server     self.cert_manager, listener)
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/common/tls_utils/cert_parser.py", line 350, in load_certificates_data
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server     check_only=True))
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/common/tls_utils/cert_parser.py", line 363, in _map_cert_tls_container
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server     primary_cn=get_primary_cn(cert),
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/common/tls_utils/cert_parser.py", line 373, in get_primary_cn
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server     return get_host_names(tls_cert.get_certificate())['cn']
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/certificates/common/pkcs12.py", line 35, in get_certificate
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server     return self.certificate.to_cryptography().public_bytes(
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server AttributeError: 'X509' object has no attribute 'to_cryptography'
2018-03-08 23:45:46.482 24634 ERROR oslo_messaging.rpc.server 



Thanks to Iain MacDonnell for finding and help reporting the bug!

Comment 2 Carlos Goncalves 2018-03-12 13:48:39 UTC
To clarify, the root cause of this bug is two-fold:

1) Wrong minimum python2-cryptography set in octavia.spec (set to >= 1.7.2 [1]) while upstream it's set correctly [2].

2) Minimum required version of python-pyOpenSSL was not reflected in octavia/requirements.txt. A bump of minimum version should had been proposed to requirements/global-requirements.txt by octavia folks but that slipped.


So, to fix this bug we need:
- python-cryptography >= 1.9
- python-pyOpenSSL >= 17.1.0


[1] https://github.com/rdo-packages/octavia-distgit/blob/queens-rdo/openstack-octavia.spec#L115-L116
[2] https://github.com/openstack/octavia/blob/stable/queens/requirements.txt#L47

Comment 3 Noam Manos 2018-04-03 09:27:49 UTC
On puddle 2018-03-20.2 - There's no "python-cryptography" nor "python2-cryptography" packages at all, only "cryptography" version 1.7.2.

----

(overcloud) [stack@undercloud-0 ~]$ cat /etc/yum.repos.d/latest-installed 
13   -p 2018-03-20.2
(overcloud) [stack@undercloud-0 ~]$ pip list | egrep *cryptography
cryptography                     1.7.2            
(overcloud) [stack@undercloud-0 ~]$ pip list | egrep *OpenSSL*
pyOpenSSL                        17.3.0           

----

Comment 4 Carlos Goncalves 2018-04-03 09:50:41 UTC
Use yum, not pip: rpm -qa | grep *cryptography*

Comment 5 Noam Manos 2018-04-03 11:29:15 UTC
(In reply to Carlos Goncalves from comment #4)
> Use yum, not pip: rpm -qa | grep *cryptography*

Nothing either.

(overcloud) [stack@undercloud-0 ~]$ rpm -qa | grep *cryptography*
(overcloud) [stack@undercloud-0 ~]$

Comment 6 Carlos Goncalves 2018-04-03 11:33:40 UTC
You're running it on the undercloud, sorry I missed that from before. You have to run it in the octavia-worker docker container which runs on the controller nodes.

Comment 7 Bernard Cafarelli 2018-04-03 13:12:49 UTC
Indeed, for example I got:
[heat-admin@controller-0 ~]$ sudo docker exec -ti octavia_worker bash
()[octavia@controller-0 /]$ rpm -qi python2-cryptography
Name        : python2-cryptography
Version     : 2.1.4
Release     : 1.el7ost
[...]

(latest puddle)

Comment 8 Carlos Goncalves 2018-04-03 13:32:34 UTC
Right, but octavia's .spec file is not yet enforcing latest minimum required dependency versions. It's pending import from RDO.

Comment 20 errata-xmlrpc 2018-06-27 13:35:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086